Question

Uploading files using python via data upload service


Badge

I want to upload some files for a specific workbench using HTTP.

I'm using Python for this with requests library, but I'm having difficulty getting it to work.

My python code is as follows:

    with open(file_path, 'rb') as file:
        url = f"{fme_url}/fmedataupload/{script_location}"
        data = {
            'custom_file': {
                'value': file.read(),
                'options': {
                    'contentType': 'application/octet-stream',
                    'filename': file_name,
                }
            },
            'opt_namespace': directory,
            'opt_fullpath': True
        }
        token = get_token()
        headers = {
            'content-type': 'application/x-www-form-urlencoded',
            'Authorization': f"fmetoken token={token}",
        }
        return requests.post(url, data=data, headers=headers, json=True)

I do get a 200 OK response, so authorization is fine, but the file isn't uploaded.

It even creates a directory at the right spot.

The response is as follows

<Response [200]>
{"serviceResponse": {
   "statusInfo": {"status": "success"},
   "session": "foo",
   "files": {
      "path": "",
      "folder": [{
         "path": "$(FME_DATA_REPOSITORY)/main_repository/script_name.fmw/foo",
         "name": ""
      }]
   }
}}

 

The data upload documentation (https://docs.safe.com/fme/html/FME_Server_Documentation/ReferenceManual/service_dataupload.htm) simply lists 'Upload a single file or multiple files using a simple form submission that uses HTTP POST.', which is not helpful.

 

I'm using Python 3.8 and FME server 2019.2

 

My goal is to upload the required files, then start a workbench, and finally retrieve the result, all using Python. Preferably, the FME server account that Python uses has as little permissions as possible.


10 replies

Badge +2

@arjanboogaart Have you tried FMEServerResourceConnector or HTTPCaller?

Badge

@arjanboogaart Have you tried FMEServerResourceConnector or HTTPCaller?

My question is about using the FME server rest API in python. This is not from inside an FME workbench, but as a standalone python application.

Badge +11

Hi @arjanboogaart,

So you want to upload some files for a particular workspace... are they to persist?

I'm not sure if this is going to help you but you could try the FME Server API for uploading a resource for a workspace and then delete the resource at some time later when no longer needed.

I can only share this as guidance and unfortunately, I've not attempted to port it to python... but you could likely have some success.

From Postman

POST the resource or file:

DELETE the resource or file:

 

Badge +11

@arjanboogaart, Can you confirm the version of FME Server you are using here? Python 3?

It could be an issue we need to look at for sure. Thanks for posting!

Badge +11

@arjanboogaart,

I got curious about your mention that the file was NOT being uploaded so I did go test this in Postman and used the POST. In my testing, using FME Server 2019.2, I can confirm a file is uploaded and a folder is created.

Are you looking in the /system/temp/upload location? Does the token you are using have enough permissions (I would expect an error if so - so this is likely not your problem)

Because the files are being uploaded by the Data Upload Service, they are volatile (non-persistent) you will find them in the FME Server temp file location.

That location is determined in the service properties file.

By default:

# UPLOAD_DIR

UPLOAD_DIR=<System share>/resources/system/temp/upload

 

Badge

@arjanboogaart, Can you confirm the version of FME Server you are using here? Python 3?

It could be an issue we need to look at for sure. Thanks for posting!

Python 3.8 and FME server 2019.2. I have a suspicion it has to do with how python requests sends the form data, as we have a similar setup in Node.js which is working.

It's strange how it sends a 200 OK response, I would expect a 400 if there's something wrong with my request.

Badge

@arjanboogaart,

I got curious about your mention that the file was NOT being uploaded so I did go test this in Postman and used the POST. In my testing, using FME Server 2019.2, I can confirm a file is uploaded and a folder is created.

Are you looking in the /system/temp/upload location? Does the token you are using have enough permissions (I would expect an error if so - so this is likely not your problem)

Because the files are being uploaded by the Data Upload Service, they are volatile (non-persistent) you will find them in the FME Server temp file location.

That location is determined in the service properties file.

By default:

# UPLOAD_DIR

UPLOAD_DIR=<System share>/resources/system/temp/upload

 

Do you mind sharing the postman request? I believe there is an 'export' option. That might help me recreate it in Python

Badge +11

Python 3.8 and FME server 2019.2. I have a suspicion it has to do with how python requests sends the form data, as we have a similar setup in Node.js which is working.

It's strange how it sends a 200 OK response, I would expect a 400 if there's something wrong with my request.

@arjanboogaart, I suspect you are not using python3.8 with FME... it won't work as we've not added support for that yet.

Here's the export.

UploadExamplePostman.json.zip

Let us know if this helps.

Badge

@arjanboogaart,

I got curious about your mention that the file was NOT being uploaded so I did go test this in Postman and used the POST. In my testing, using FME Server 2019.2, I can confirm a file is uploaded and a folder is created.

Are you looking in the /system/temp/upload location? Does the token you are using have enough permissions (I would expect an error if so - so this is likely not your problem)

Because the files are being uploaded by the Data Upload Service, they are volatile (non-persistent) you will find them in the FME Server temp file location.

That location is determined in the service properties file.

By default:

# UPLOAD_DIR

UPLOAD_DIR=<System share>/resources/system/temp/upload

 

This example uses HTTP PUT instead of POST. PUT doesn't support custom directory names.

Badge +2

@arjanboogaart​ @steveatsafe​ did this issue ever get resolved? I am running into the same issue where I am trying to upload a file through Python and am getting a 200 response without a file name or path, and no file is uploaded. Using Python 3.7 and FME Server 2021.2.2.

Reply