Skip to main content

Is it possible to upload a file as Multipart Upload without first writing it to the filesystem?

I have this working:

But this requires to first write the attribute to a file and then using the filepath to upload.

I tried base64 encoding the binary attribute and uploading. But I dont know what to set the mime-type to.

_response_body (string: UTF-8): {
"args": {},
"data": {},
"files": {
"temp.zip": "data:application/octet-stream;base64,AAAAAAAAAAAAAAAA="
},
"form": {
"token": "tokentokentokentokentoken",
"f": "json",
"archive": "true"
},
"headers": {
"host": "postman-echo.com",
"x-request-start": "t=1722253286.223",
"content-length": "201671",
"x-forwarded-proto": "https",
"x-forwarded-port": "443",
"x-amzn-trace-id": "Root=1-66a77fe5-2101668c0d99d0fd01e61cc4",
"user-agent": "FME/2021.7.42.21821 libcurl/7.79.1 (OpenSSL/1.1.1n) Schannel zlib/1.2.11 WinIDN libssh2/1.9.0 nghttp2/1.44.0",
"accept": "*/*",
"accept-encoding": "deflate, gzip",
"content-type": "multipart/form-data; boundary=baecc94b-9aee-474a-8581-94dc6b6bacd6",
"content-transfer-encoding": "binary"
},
"json": null,
"url": "https://postman-echo.com/post"
}

Above the response from postman-echo if I redirect the working response there.

Should I add data:application/octet-stream;base64, in front of the base64 string?

 

 

 

Whats the reason for not wanting to write it out?

You can use the tempPathnameCreator which will create a temporary folder that you can write out to and then will delete it once the process has finished


Because it's unnecessary overhead. File is already in memory, has to be writen to filesystem, read from filesystem into memory again and then uploaded. Could be creating base64 is actually slower then reading from file.

But feels strange to need to create a tempPath just because HTTPCaller misses a function.


An HTTP transaction are basically just text files being exchanged, and multipart is a syntax for incorporating different data chunks into this file. Binary content is encoded to text.

So an uploaded file is not sent as a file path, but as a heavily encoded (UTF-8, Base64, ...) data chunk in the multipart framework.

It’s doable with raw HTTP, but I don’t think HTTPCaller will give you that kind of access. Instead it needs to encode the file by reading it.

I think you should accept the overhead of storing the file temporarily, it’s a much nicer and high-level approach.

Cheers


Uploading an import file to Relatics (SE software) can be done via a SOAP call where the file is Base64 encoded, using a HTTPCaller.

Method = POST

Content-Type = application/soap+xml (via a header as it is not an option in the drop down list)

Body = 

<?xml version="1.0" encoding="UTF-8"?>
<soap12:Envelope xmlns:soap12="http://www.w3.org/2003/05/soap-envelope">
    <soap12:Body>
        <Import xmlns="http://www.relatics.com/">
            <Operation>$(OperationNameImport)</Operation>
            <Identification>
                <Identification>
                    <Workspace>@Value(WorkspaceId)</Workspace>
                </Identification>
            </Identification>
            <Authentication>
                <Authentication>
                    <Entrycode>$(EntryCodeImport)</Entrycode>
                </Authentication>
            </Authentication>
            <Filename>import.csv</Filename> 
                <Data><! CDATA@Value(text_line_data)]]></Data>
        </Import>
    </soap12:Body>
</soap12:Envelope>

The text_line_data attribute contains the the Base64 encoded string of the file.


Because it's unnecessary overhead. File is already in memory, has to be writen to filesystem, read from filesystem into memory again and then uploaded. Could be creating base64 is actually slower then reading from file.

But feels strange to need to create a tempPath just because HTTPCaller misses a function.

Curious how your getting the file already in memory, is it from a previous request?

I just want to promote my own idea here about making it easier to create temp files.

 

This doesn’t solve your problem, however, if that file you have in memory is from a previous HTTPCaller then being able to output directly to a temp location would make your workspace cleaner. 


@virtualcitymatt Yes, it's basicly doing this.

HTTPCaller download zip-file → HTTPCaller upload zip-file from one rest endpoint to another.

Because file upload is basicly string upload where the file is string encoded I just need to know what the string should look like and my problem is solved.

I know it should be base64 encoded. But I think it needs more info.


According to this post:

https://stackoverflow.com/questions/3085990/post-a-file-string-using-curl-in-php

 

The result looks like this:

-----------------------------7da16b2e4026c
Content-Disposition: form-data; name="NewFile"; filename="test.jpg"
Content-Type: image/jpeg

(...raw JPEG data here...)
-----------------------------7da16b2e4026c
Content-Disposition: form-data; name="otherformfield"

content of otherformfield is this text
-----------------------------7da16b2e4026c--

Most of the work is done by FME, the separator line, the content disposition, i could set the Content-Type and the name. The (..raw JPEG data here..) is the attribute value. Looks like the only thing I can't influence is the filename="data.zip”

 


Reply