Skip to main content

I want to upload some files for a specific workbench using HTTP.

I'm using Python for this with requests library, but I'm having difficulty getting it to work.

My python code is as follows:

    with open(file_path, 'rb') as file:
        url = f"{fme_url}/fmedataupload/{script_location}"
        data = {
            'custom_file': {
                'value': file.read(),
                'options': {
                    'contentType': 'application/octet-stream',
                    'filename': file_name,
                }
            },
            'opt_namespace': directory,
            'opt_fullpath': True
        }
        token = get_token()
        headers = {
            'content-type': 'application/x-www-form-urlencoded',
            'Authorization': f"fmetoken token={token}",
        }
        return requests.post(url, data=data, headers=headers, json=True)

I do get a 200 OK response, so authorization is fine, but the file isn't uploaded.

It even creates a directory at the right spot.

The response is as follows

<Response u200]>
{"serviceResponse": {
   "statusInfo": {"status": "success"},
   "session": "foo",
   "files": {
      "path": "",
      "folder": l{
         "path": "$(FME_DATA_REPOSITORY)/main_repository/script_name.fmw/foo",
         "name": ""
      }]
   }
}}

 

The data upload documentation (https://docs.safe.com/fme/html/FME_Server_Documentation/ReferenceManual/service_dataupload.htm) simply lists 'Upload a single file or multiple files using a simple form submission that uses HTTP POST.', which is not helpful.

 

I'm using Python 3.8 and FME server 2019.2

 

My goal is to upload the required files, then start a workbench, and finally retrieve the result, all using Python. Preferably, the FME server account that Python uses has as little permissions as possible.

@arjanboogaart Have you tried FMEServerResourceConnector or HTTPCaller?


@arjanboogaart Have you tried FMEServerResourceConnector or HTTPCaller?

My question is about using the FME server rest API in python. This is not from inside an FME workbench, but as a standalone python application.


Hi @arjanboogaart,

So you want to upload some files for a particular workspace... are they to persist?

I'm not sure if this is going to help you but you could try the FME Server API for uploading a resource for a workspace and then delete the resource at some time later when no longer needed.

I can only share this as guidance and unfortunately, I've not attempted to port it to python... but you could likely have some success.

From Postman

POST the resource or file:

DELETE the resource or file:

 


@arjanboogaart, Can you confirm the version of FME Server you are using here? Python 3?

It could be an issue we need to look at for sure. Thanks for posting!


@arjanboogaart,

I got curious about your mention that the file was NOT being uploaded so I did go test this in Postman and used the POST. In my testing, using FME Server 2019.2, I can confirm a file is uploaded and a folder is created.

Are you looking in the /system/temp/upload location? Does the token you are using have enough permissions (I would expect an error if so - so this is likely not your problem)

Because the files are being uploaded by the Data Upload Service, they are volatile (non-persistent) you will find them in the FME Server temp file location.

That location is determined in the service properties file.

By default:

# UPLOAD_DIR

UPLOAD_DIR=<System share>/resources/system/temp/upload

 


@arjanboogaart, Can you confirm the version of FME Server you are using here? Python 3?

It could be an issue we need to look at for sure. Thanks for posting!

Python 3.8 and FME server 2019.2. I have a suspicion it has to do with how python requests sends the form data, as we have a similar setup in Node.js which is working.

It's strange how it sends a 200 OK response, I would expect a 400 if there's something wrong with my request.


@arjanboogaart,

I got curious about your mention that the file was NOT being uploaded so I did go test this in Postman and used the POST. In my testing, using FME Server 2019.2, I can confirm a file is uploaded and a folder is created.

Are you looking in the /system/temp/upload location? Does the token you are using have enough permissions (I would expect an error if so - so this is likely not your problem)

Because the files are being uploaded by the Data Upload Service, they are volatile (non-persistent) you will find them in the FME Server temp file location.

That location is determined in the service properties file.

By default:

# UPLOAD_DIR

UPLOAD_DIR=<System share>/resources/system/temp/upload

 

Do you mind sharing the postman request? I believe there is an 'export' option. That might help me recreate it in Python


Python 3.8 and FME server 2019.2. I have a suspicion it has to do with how python requests sends the form data, as we have a similar setup in Node.js which is working.

It's strange how it sends a 200 OK response, I would expect a 400 if there's something wrong with my request.

@arjanboogaart, I suspect you are not using python3.8 with FME... it won't work as we've not added support for that yet.

Here's the export.

UploadExamplePostman.json.zip

Let us know if this helps.


@arjanboogaart,

I got curious about your mention that the file was NOT being uploaded so I did go test this in Postman and used the POST. In my testing, using FME Server 2019.2, I can confirm a file is uploaded and a folder is created.

Are you looking in the /system/temp/upload location? Does the token you are using have enough permissions (I would expect an error if so - so this is likely not your problem)

Because the files are being uploaded by the Data Upload Service, they are volatile (non-persistent) you will find them in the FME Server temp file location.

That location is determined in the service properties file.

By default:

# UPLOAD_DIR

UPLOAD_DIR=<System share>/resources/system/temp/upload

 

This example uses HTTP PUT instead of POST. PUT doesn't support custom directory names.


@arjanboogaart​ @steveatsafe​ did this issue ever get resolved? I am running into the same issue where I am trying to upload a file through Python and am getting a 200 response without a file name or path, and no file is uploaded. Using Python 3.7 and FME Server 2021.2.2.


In case anyone else is having trouble with this, I have found success with simply passing the binary file to the “data” parameter of requests.post(). Taking OP’s code as a base, this works for me when uploading binary files:

with open(file_path, 'rb') as file:
url = f"{fme_url}/fmedataupload/{script_location}"
token = get_token()
headers = {
'Content-Type': 'application/octet-stream',
'Authorization': f"fmetoken token={token}",
'Accept': 'application/json',
}
return requests.post(url, data=file, headers=headers)

I’ve successfully done this with AutoCAD .dwg files, passing them to a RealDWG reader. With text files, I left ‘Content-Type’ as ‘text/plain’.

Also, a couple things I ran into that were confusing at first:

  • Trying to step through the code in VS Code debug mode seems to trigger the workspace twice (at least for me), once with the file I’ve uploaded, and a second time with the default parameters, in my case this made it appear that the file wasn’t successfully being uploaded because the download link was giving me the file of the second run, but upon inspection of the completed jobs I was able to see that it was working. Running outside of debug mode hasn’t caused the double workspace run issue for me.
  • If you are writing output data to a Resources folder as in this tutorial https://support.safe.com/hc/en-us/articles/25407584866445-Send-Source-Data-to-FME-Flow-with-HTTP-POST , be sure you delete any files that are there after a run with the Job Submitter service. The tutorial covers this, but I didn’t fully realize why it’s important. It seems if files are already present at workspace completion, those files are what the download link will return rather than the files generated by your subsequent workspace runs with the File Download service. I’m honestly not sure if the one is the direct cause of the other, but that is what I’ve observed so far.