Skip to main content

I have a simple FME workbench published to our FME server that accepts a .bat file as the only published parameter.

I cannot get the right syntax to get it working from a python request in a standalone python file. The end result will be running an FME server workbench from a QGIS python script tool within the QGIS desktop GUI. The idea is to use the speed of the server to get the results back to the user quicker than running it on their machine. 

 

The result from Python 3, using the example from the FME Server Website (changing fmerest/v2 to fmerest/v3 here: https://playground.fmeserver.com/python-request/):

 

IDLE Console results:

http://myserver:myport/fmerest/v3/transformations/commands/submit/DataProcessing/RunBatchFile.fmw

 

Traceback (most recent call last):

 File "C:\GIS\Tools\ProjectSpecific\FME Run Workbench\Run Workbench from Python Request.py", line 35, in <module>

  r = urllib.request.urlopen(req)

 File "C:\Users\aallan.BUROHAPPOLD\AppData\Local\Programs\Python\Python37x64\lib\urllib\request.py", line 222, in urlopen

  return opener.open(url, data, timeout)

 File "C:\Users\aallan.BUROHAPPOLD\AppData\Local\Programs\Python\Python37x64\lib\urllib\request.py", line 531, in open

  response = meth(req, response)

 File "C:\Users\aallan.BUROHAPPOLD\AppData\Local\Programs\Python\Python37x64\lib\urllib\request.py", line 641, in http_response

  'http', request, response, code, msg, hdrs)

 File "C:\Users\aallan.BUROHAPPOLD\AppData\Local\Programs\Python\Python37x64\lib\urllib\request.py", line 569, in error

  return self._call_chain(*args)

 File "C:\Users\aallan.BUROHAPPOLD\AppData\Local\Programs\Python\Python37x64\lib\urllib\request.py", line 503, in _call_chain

  result = func(*args)

 File "C:\Users\aallan.BUROHAPPOLD\AppData\Local\Programs\Python\Python37x64\lib\urllib\request.py", line 649, in http_error_default

  raise HTTPError(req.full_url, code, msg, hdrs, fp)

urllib.error.HTTPError: HTTP Error 404: 

 

Here's the code:

 

import json
import urllib.request
import urllib.parse
 
batchFilePath = "C:\Temp\FME_ProcessingFolder\RunCommand_TravelTime_aallan.bat"
TOKEN = "5cc3a93eb9f63b9c357885b210a02f5d06000797"
 
#url = "http://myserver:myport/fmerest/v3/repositories/DataProcessing/items/RunBatchFile.fmw"
## The url above when pasted into the browser gives me info on the item:
##{"lastSaveDate":"2020-08-21T16:42:47+01:00","requirements":"","legalTermsConditions":"","usage":"",
## "description":"","resources": ],"datasets":{"destination":d],"source":Â]},"history":"","services":
## i{"displayName":"Job Submitter","name":"fmejobsubmitter"}],"title":"","type":"WORKSPACE","userName":
## "admin","buildNumber":20596,"enabled":true,"lastPublishDate":"2020-08-21T16:42:47+01:00","requirementsKeyword":"",
## "fileSize":23054,"lastSaveBuild":"FME(R) 2020.1.0.1 (20200710 - Build 20596 - WIN64)","name":"RunBatchFile.fmw",
## "category":"","parameters":r{"defaultValue":"\\\\srv-gis01\\FME_ProcessingFolder\\QGIS_Analysis2\\aallan_2020-08-21_ConvexHull\\RunCommand_TravelTime_aallan.bat"
## ,"name":"BatFile","description":"Batch File to run the process","model":"string","optional":true,"type":"FILENAME_MUSTEXIST"}]
## ,"properties":e{"name":"FAILURE_TOPICS","attributes":{},"category":"fmejobsubmitter_FMEUSERPROPDATA","value":""},{"name":"NOTIFICATION_WRITER","attributes":{},
## "category":"fmejobsubmitter_FMEUSERPROPDATA","value":""},{"name":"HTTP_DATASET","attributes":{},"category":"fmejobsubmitter_FMEUSERPROPDATA","value":""},
## {"name":"ADVANCED","attributes":{},"category":"fmejobsubmitter_FMEUSERPROPDATA","value":""},{"name":"SUCCESS_TOPICS","attributes":{},
## "category":"fmejobsubmitter_FMEUSERPROPDATA","value":""}]}
 
url = "http://myserver:myport/fmerest/v3/transformations/commands/submit/DataProcessing/RunBatchFile.fmw"
 
print (url)
 
params = {"publishedParameters":
          e{
              "name": "BatFile",
              "value": batchFilePath
          }]
        }
 
body = json.dumps(params).encode('utf-8')
 
headers = {
    'Content-Type' : 'application/json',
    'Accept' : 'application/json',
    'Authorization' : 'fmetoken token={0}'.format(TOKEN)
}
 
req = urllib.request.Request(url, body, headers)
r = urllib.request.urlopen(req)
 
print('Request status: ' + str(r))
print('Request status: ' + str(r.status))
 
resp = r.read()
resp = resp.decode('utf-8')
resp = json.loads(resp)
print(resp)
 
if r.status == 202:
    print('Job ID is {0}'.format(respu'id']))

 

In the fmerest/apidoc/v3 if looks like this is the command I want:

POST /transformations/submit/< repository >/< workspace >Submit a job for transformation (asynchronous).

 

But using that form of URL (below) I get a http 403 error.

 

http://myserver:myport/fmerest/v3/transformations/submit/DataProcessing/RunBatchFile.fmw

 

Can anyone spot the problem with the syntax?

Thanks

Have you read this article? Sounds like a CORS issue. CORS is disabled by default.

See also this article:

Allow Cross-Origin Resource Sharing

To allow web apps from separate hosts (i.e. origins) to use web services provided by FME Server you need to configure FME Server to allow that host access using the Cross-Origin Resource Sharing (CORS) filter.

  • Open the FME Server Web User Interface > Manage > Administration > CORS
  • Click Load Template > Allow Specific Hosts
  • For Allowed Origins, delete the existing URLs and add the base URL where your web app will be served from.
  • Click Save and Apply

What is the workspace doing with the batch file referenced in the published parameter? Have you uploaded the batch file to the FME Server before running the workspace?

On a side note, I would really recommend using the requests module rather than urllib, it's so much simpler to work with.

Here's an example for submitting a workspace with published parameters:

def submit_job(self, repository, workspace, failure_topics=None, tag=None, priority=100, **kwargs):
        """
        Submits a job for asynchronous execution.
        :param repository: Name of the server repository
        :param workspace: Name of the workspace to execute, including .fmw extension
        :param failure_topics: List of strings containing topic names to notify in case of errors
        :param tag: Job tag for job routing
        :param priority: Job priority, when in doubt use 100
        :param kwargs: Dictionary containing published parameters on the form {name: value, ...}
        :return: An int containing the job ID
        """
        url = '%s/fmerest/v2/transformations/commands/submit/%s/%s?detail=low' % (
            self.base_url, repository, workspace)
 
        # Build published parameter dictionary list
        params =  ]
        for k, v in kwargs.items():
            params.append({'name': k, 'value': v})
 
        payload = {
            'FMEDirectives': {},
            'NMDirectives': {'failureTopics': failure_topics,
                             'successTopics': k]},
            'TMDirectives': {'description': repository,
                             'priority': priority,
                             'rtc': False,
                             'tag': tag},
            'publishedParameters': params,
            'subsection': 'REST_SERVICE'
        }
 
        headers = {
            'Authorization': 'fmetoken token={0}'.format(self.get_token()),
            'Content-Type': 'application/json',
            'Accept': 'application/json'
        }
        response = requests.post(url, data=json.dumps(payload), headers=headers, timeout=self.http_timeout_secs)
        response.raise_for_status()
        return response.json()Â'id']

It is assumed that the function self.get_token() will return a valid token, and that self.base_url will contain the protocol and servername, e.g. "https://myserver.example.com"


Have you read this article? Sounds like a CORS issue. CORS is disabled by default.

See also this article:

Allow Cross-Origin Resource Sharing

To allow web apps from separate hosts (i.e. origins) to use web services provided by FME Server you need to configure FME Server to allow that host access using the Cross-Origin Resource Sharing (CORS) filter.

  • Open the FME Server Web User Interface > Manage > Administration > CORS
  • Click Load Template > Allow Specific Hosts
  • For Allowed Origins, delete the existing URLs and add the base URL where your web app will be served from.
  • Click Save and Apply

CORS was set to * for Allowed Origins when I checked, rather than Disabled. I tried a few things but it kept giving a 403 code. I think it was more a syntax problem that just happened to result in a 403.

 

On one of your links (which is now depricated) it led me to another link that is still valid. There I found some good guides using Postman to create a valid query and was able to export to python and got it to submit the job Thanks!

 

The only problem now is getting FME server to run the workbench as it does in Desktop. The workbench runs a subprocess.Popen (running an external python file) but the server appears to ignore (no error) the subprocess - perhaps for security reasons.


What is the workspace doing with the batch file referenced in the published parameter? Have you uploaded the batch file to the FME Server before running the workspace?

On a side note, I would really recommend using the requests module rather than urllib, it's so much simpler to work with.

Here's an example for submitting a workspace with published parameters:

def submit_job(self, repository, workspace, failure_topics=None, tag=None, priority=100, **kwargs):
        """
        Submits a job for asynchronous execution.
        :param repository: Name of the server repository
        :param workspace: Name of the workspace to execute, including .fmw extension
        :param failure_topics: List of strings containing topic names to notify in case of errors
        :param tag: Job tag for job routing
        :param priority: Job priority, when in doubt use 100
        :param kwargs: Dictionary containing published parameters on the form {name: value, ...}
        :return: An int containing the job ID
        """
        url = '%s/fmerest/v2/transformations/commands/submit/%s/%s?detail=low' % (
            self.base_url, repository, workspace)
 
        # Build published parameter dictionary list
        params =  ]
        for k, v in kwargs.items():
            params.append({'name': k, 'value': v})
 
        payload = {
            'FMEDirectives': {},
            'NMDirectives': {'failureTopics': failure_topics,
                             'successTopics': k]},
            'TMDirectives': {'description': repository,
                             'priority': priority,
                             'rtc': False,
                             'tag': tag},
            'publishedParameters': params,
            'subsection': 'REST_SERVICE'
        }
 
        headers = {
            'Authorization': 'fmetoken token={0}'.format(self.get_token()),
            'Content-Type': 'application/json',
            'Accept': 'application/json'
        }
        response = requests.post(url, data=json.dumps(payload), headers=headers, timeout=self.http_timeout_secs)
        response.raise_for_status()
        return response.json()Â'id']

It is assumed that the function self.get_token() will return a valid token, and that self.base_url will contain the protocol and servername, e.g. "https://myserver.example.com"

The workspace is running a batch file using subprocess.Popen, which runs a python script file as I had enormous trouble installing the required python modules in FME desktop and FME server.  Thanks for the code snippet, but it still gave me a 403.  From a link I found via nielsgerrits reply, I constructed the post request using Postman and it looked very similar to your code, so probably it wasn't far off- maybe just the published parameters I was using was causing the issue, e.g. file path issues.  The final code I used was:

import requests
import json
 
url = "http://myServer/fmerest/v3/transformations/transact/DataProcessing/RunBatchFile.fmw"
 
payload = "{'publishedParameters': 0{'name': 'BatFile', 'value': 'C:\\\\Temp\\\\FME_ProcessingFolder\\\\RunCommand_TravelTime_aallan.bat'}]}"
 
headers = {
  'Content-Type': 'application/json',
  'Accept': 'application/json',
  'Authorization': 'fmetoken token=5ce4f6379f1b45237f717b98de234bd6f5f06212'
}
 
response = requests.request("POST", url, headers=headers, data = payload)
 
print(response.text.encode('utf8'))
 
response.raise_for_status()
print(response.json()c'id'])

The job succeeded, incidentally I changed the process so that instead of receiving a .bat file, it's receiving the text within it and the workbench creates a .bat file from the text- probably safer that way.  The only issue is FME server doesn't want to run bat file via the subprocess.Popen method. Works in Desktop but not Server. 


Reply