Skip to main content

If this is an answered question, please excuse me and point me in the right direction.

I'm working on a web-based system that leverages FME server to run serveral tasks and provides the result to the user.

By programatically building parameters and calling the FME Server data download endpoint at

$"{repository.Url}/fmedatadownload/{repository.RepositoryName}/{analysis.WorkspaceId}";

and setting the parameter opt_servicemode to sync I get a response on the form:

{  
   "serviceResponse":{  
      "jobID":10261,
      "statusInfo":{  
         "mode":"sync",
         "status":"success"
      },
      "fmeTransformationResult":{  
         "fmeEngineResponse":{  
            "outputLocation":"C:\\\\ProgramData\\\\Safe Software\\\\FME Server\\\\resources\\\\system\\\\temp\\\\engineresults\\\\FME_176D3A2A_1504696983768_3368.zip",
            "resultRootDir":"/fmedatadownload/results",
            "statusNumber":"0",
            "logFileName":"job_10261.log",
            "downloadUrl":"http://server/fmedatadownload/results/FME_176D3A2A_1504696983768_3368.zip",
            "numFeaturesOutput":"49",
            "statusMessage":"Translation Successful",
            "notificationLocation":"C:\\\\ProgramData\\\\Safe Software\\\\FME Server\\\\resources\\\\system\\\\temp\\\\engineresults\\\\FME_176D3A2A_1504696983768_3368_nw"
         },
         "fmeServerResponse":{  
            "request":"\\"            Samples/austinApartments/austinApartments.fmw\\" -XML_PARAMETER <?xml version=\\"1.0\\" encoding=\\"UTF-8\\"?>\<TRANSFORMATION_REQUEST><PUBLISHED_PARAMETER name=\\"FME_SERVER_REQUEST_HEADERS\\">            {  
               \\"remote-addr\\":\\"192.168.46.110\\",
               \\"host\\":\\"192.168.4.148\\"
            }            <\\/PUBLISHED_PARAMETER><PUBLISHED_PARAMETER name=\\"FME_SERVER_REQUEST_URI\\">http://server//fmedatadownload/Samples/austinApartments.fmw<\\/PUBLISHED_PARAMETER><PUBLISHED_PARAMETER name=\\"FME_SERVER_REQUEST_PARAMETERS\\">            {  
               \\"opt_responseformat\\":\\"json\\",
               \\"opt_servicemode\\":\\"sync\\"
            }            <\\/PUBLISHED_PARAMETER><\\/TRANSFORMATION_REQUEST> --FME_SECURITY_USER \\"admin\\" --FME_SECURITY_ROLES \\"fmeadmin fmesuperuser\\" --FME_SERVER_HOST \\"localhost\\" --FME_SERVER_PORT \\"7071\\" --FME_SERVER_WEB_URL \\"http:            //192.168.4.148:80            \\" -OGCKML_1_DATASET \\"!FME_AUTO_DIR_NAME!.zip\\\\austinApartments.kml\\"",
            "jobStatus":"SUCCESS",
            "timeRequested":"Wed-06-Sep-2017 01:23:03 PM",
            "requestKeyword":"FILE_DOWNLOAD_SERVICE",
            "description":"",
            "timeStarted":"2017-09-06 01:23:03",
            "requesterHost":"192.168.4.148",
            "priority":"100",
            "result":"0:Translation Successful|NotificationLocation=C:\\\\ProgramData\\\\Safe Software\\\\FME Server\\\\resources\\\\system\\\\temp\\\\engineresults\\\\FME_176D3A2A_1504696983768_3368_nw|OutputLocation=C:\\\\ProgramData\\\\Safe Software\\\\FME Server\\\\resources\\\\system\\\\temp\\\\engineresults\\\\FME_176D3A2A_1504696983768_3368.zip|ResultRootDir=/fmedatadownload/results|NumFeaturesOutput=49|LogFileName=job_10261.log|downloadUrl=http://server/fmedatadownload/results/FME_176D3A2A_1504696983768_3368.zip",
            "requesterResultPort":"55697",
            "resultSuccess":"true",
            "serviceSuccess":"true",
            "id":"10261",
            "timeFinished":"2017-09-06 01:23:04",
            "serviceMsg":""
         }
      },
      "url":"http://server/fmedatadownload/results/FME_176D3A2A_1504696983768_3368.zip"
   }
}

Downloading the result at url works as expected.

However. As far as my understanding of FME server goes, under heavy loads things are queued up. This means that a job submitted to the DataDownload endpoint might "run" for more than 10 minutes. This would make the HTTP request time out. 

Because of this I'm thinking that opt_servicemode = async is a better solution. If I set this, the response I get is:

{     "serviceResponse":{        "jobID":10266,      "statusInfo":{           "mode":"async",         "status":"success"      }   }}

That is, almost no info. But I do get a jobID, so I'm guessing that this serves as a reference to the job. 

And, indeed, if i navigate my browser to http://server/fmeserver/#/job/10266/summary i see a "Data DownloadURL" on top of the page, as well as in the "Result Data" section at the bottom of the page. 

However, i cannot find this url in either the response to the DataDownload request, neither as a response to any of the endpoints at http://server/fmerest/v2/transformations/jobs/id/10266

So, my questions are basically:

1) How is the data download endpoint supposed to work in async mode? Am I supposed to get a link somehow?

2) Where is the data-download-link provided?

NOTE: The sync download works as expected, but this is not the problem I'm trying to solve. I am also aware that FME server can send the result by email if I specify opt_requesteremail, but this is not what I want either. 

Any answers or pointers welcome. 

I just tested the following. Submit a job using the fmedatadownload service in async mode:

http://server/fmedatadownload/Samples/austinApartments.fmw?opt_servicemode=async

I got the following response:

{
    "serviceResponse": {
        "jobID": 4,
        "statusInfo": {
            "mode": "async",
            "status": "success"
        }
    }
}

Then I retrieve the job status based on the returned "jobID" value:

http://server/fmerest/v3/transformations/jobs/id/4/result

I get the following result JSON, checking that "status" = "SUCCESS", this indicates that the job has successfully terminated (i.e. not in the queue anymore):

{
  "resultDatasetDownloadUrl": "http://server/fmedatadownload/results/FME_58184A14_1504700044541_3240.zip",
  "timeRequested": "2017-09-06T14:12:18+02:00",
  "requesterResultPort": -1,
  "numFeaturesOutput": 49,
  "requesterHost": "x.x.x.x",
  "timeStarted": "2017-09-06T14:14:04+02:00",
  "id": 4,
  "timeFinished": "2017-09-06T14:14:06+02:00",
  "priority": 100,
  "statusMessage": "Translation Successful",
  "status": "SUCCESS"
}

I can then download the result dataset using the link given in "resultDatasetDownloadUrl":

http://server/fmedatadownload/results/FME_58184A14_1504700044541_3240.zip

This is using the API V3 in FME Server 2017, but I think it should be identical in FME 2016.

Remember that you have full access to the API documentation on your local server:


Ah, thanks!

Seems like the resultDatasetDownloadUrl parameter was introduced in the api v3:

 

http://server/fmerest/v2/transformations/jobs/id/10269/result?detail=low

 

returns:

{
  "timeRequested": "2017-09-06T14:21:27",
  "requesterResultPort": -1,
  "numFeaturesOutput": 49,
  "requesterHost": "192.168.4.148",
  "timeStarted": "2017-09-06T14:21:27",
  "id": 10269,
  "timeFinished": "2017-09-06T14:21:28",
  "priority": 100,
  "statusMessage": "Translation Successful",
  "status": "SUCCESS"
}

whereas, 

http://server/fmerest/v3/transformations/jobs/id/10269/result

returns 

{
  "resultDatasetDownloadUrl": "http://server/fmedatadownload/results/FME_176D3A2A_1504700487510_6860.zip",
  "timeRequested": "2017-09-06T14:21:27+02:00",
  "requesterResultPort": -1,
  "numFeaturesOutput": 49,
  "requesterHost": "192.168.4.148",
  "timeStarted": "2017-09-06T14:21:27+02:00",
  "id": 10269,
  "timeFinished": "2017-09-06T14:21:28+02:00",
  "priority": 100,
  "statusMessage": "Translation Successful",
  "status": "SUCCESS"
}

So, thank you for pointing this out!


Ah, thanks!

Seems like the resultDatasetDownloadUrl parameter was introduced in the api v3:

 

http://server/fmerest/v2/transformations/jobs/id/10269/result?detail=low

 

returns:

{
  "timeRequested": "2017-09-06T14:21:27",
  "requesterResultPort": -1,
  "numFeaturesOutput": 49,
  "requesterHost": "192.168.4.148",
  "timeStarted": "2017-09-06T14:21:27",
  "id": 10269,
  "timeFinished": "2017-09-06T14:21:28",
  "priority": 100,
  "statusMessage": "Translation Successful",
  "status": "SUCCESS"
}

whereas, 

http://server/fmerest/v3/transformations/jobs/id/10269/result

returns 

{
  "resultDatasetDownloadUrl": "http://server/fmedatadownload/results/FME_176D3A2A_1504700487510_6860.zip",
  "timeRequested": "2017-09-06T14:21:27+02:00",
  "requesterResultPort": -1,
  "numFeaturesOutput": 49,
  "requesterHost": "192.168.4.148",
  "timeStarted": "2017-09-06T14:21:27+02:00",
  "id": 10269,
  "timeFinished": "2017-09-06T14:21:28+02:00",
  "priority": 100,
  "statusMessage": "Translation Successful",
  "status": "SUCCESS"
}

So, thank you for pointing this out!

Interesting, didn't know that (rarely ever use the fmedatadownload service). Thanks for sharing!

If you are forced to use FME Server 2016, you can subscribe to the DATADOWNLOAD_ASYNC_JOB_SUCCESS topic and the notification json message contains the download link (downloadUrl).

{
"workspace":"austinDownload.fmw",
"timeRequested":"Wed-06-Sep-2017 09:07:13 AM",
"urlPrefix":"http://fme2016",
"ws_topic":"DATADOWNLOAD_ASYNC_JOB_SUCCESS",
"NumFeaturesOutput":"929",
"downloadUrl":"http://fme2016/fmedatadownload/results/FME_737C7632_1504703233427_2944.zip",
"requestKeyword":"FILE_DOWNLOAD_SERVICE",
"timeStarted":"Wed-06-Sep-2017 09:07:13 AM",
"OutputLocation":"C:\\ProgramData\\Safe Software\\FME Server\\resources\\system\\temp\\engineresults\\FME_737C7632_1504703233427_2944.zip",
"repository":"Samples",
"jobsuccess_topic":"DATADOWNLOAD_ASYNC_JOB_SUCCESS",
"LogFileName":"job_185781.log",
"jobfailure_topic":"DATADOWNLOAD_ASYNC_JOB_FAILURE",
"StatusMessage":"Translation Successful",
"ResultRootDir":"/fmedatadownload/results",
"StatusNumber":"0",
"timeFinished":"Wed-06-Sep-2017 09:07:16 AM",
"subscriber_folder":"C:\\ProgramData\\Safe Software\\FME Server\\resources\\system\\temp\\engineresults\\FME_737C7632_1504703233427_2944_nw",
"id":"185781",
"logHome":"C:/ProgramData/Safe Software/FME Server///resources/logs",
"logUrl":"http://FME2016:80/fmerest/v3/transformations/jobs/id/185,781/log"
}


If you are forced to use FME Server 2016, you can subscribe to the DATADOWNLOAD_ASYNC_JOB_SUCCESS topic and the notification json message contains the download link (downloadUrl).

{
"workspace":"austinDownload.fmw",
"timeRequested":"Wed-06-Sep-2017 09:07:13 AM",
"urlPrefix":"http://fme2016",
"ws_topic":"DATADOWNLOAD_ASYNC_JOB_SUCCESS",
"NumFeaturesOutput":"929",
"downloadUrl":"http://fme2016/fmedatadownload/results/FME_737C7632_1504703233427_2944.zip",
"requestKeyword":"FILE_DOWNLOAD_SERVICE",
"timeStarted":"Wed-06-Sep-2017 09:07:13 AM",
"OutputLocation":"C:\\ProgramData\\Safe Software\\FME Server\\resources\\system\\temp\\engineresults\\FME_737C7632_1504703233427_2944.zip",
"repository":"Samples",
"jobsuccess_topic":"DATADOWNLOAD_ASYNC_JOB_SUCCESS",
"LogFileName":"job_185781.log",
"jobfailure_topic":"DATADOWNLOAD_ASYNC_JOB_FAILURE",
"StatusMessage":"Translation Successful",
"ResultRootDir":"/fmedatadownload/results",
"StatusNumber":"0",
"timeFinished":"Wed-06-Sep-2017 09:07:16 AM",
"subscriber_folder":"C:\\ProgramData\\Safe Software\\FME Server\\resources\\system\\temp\\engineresults\\FME_737C7632_1504703233427_2944_nw",
"id":"185781",
"logHome":"C:/ProgramData/Safe Software/FME Server///resources/logs",
"logUrl":"http://FME2016:80/fmerest/v3/transformations/jobs/id/185,781/log"
}

Fortunately I'm able to use v3, but thanks for sharing. May be useful for others!

 

 


If you are forced to use FME Server 2016, you can subscribe to the DATADOWNLOAD_ASYNC_JOB_SUCCESS topic and the notification json message contains the download link (downloadUrl).

{
"workspace":"austinDownload.fmw",
"timeRequested":"Wed-06-Sep-2017 09:07:13 AM",
"urlPrefix":"http://fme2016",
"ws_topic":"DATADOWNLOAD_ASYNC_JOB_SUCCESS",
"NumFeaturesOutput":"929",
"downloadUrl":"http://fme2016/fmedatadownload/results/FME_737C7632_1504703233427_2944.zip",
"requestKeyword":"FILE_DOWNLOAD_SERVICE",
"timeStarted":"Wed-06-Sep-2017 09:07:13 AM",
"OutputLocation":"C:\\ProgramData\\Safe Software\\FME Server\\resources\\system\\temp\\engineresults\\FME_737C7632_1504703233427_2944.zip",
"repository":"Samples",
"jobsuccess_topic":"DATADOWNLOAD_ASYNC_JOB_SUCCESS",
"LogFileName":"job_185781.log",
"jobfailure_topic":"DATADOWNLOAD_ASYNC_JOB_FAILURE",
"StatusMessage":"Translation Successful",
"ResultRootDir":"/fmedatadownload/results",
"StatusNumber":"0",
"timeFinished":"Wed-06-Sep-2017 09:07:16 AM",
"subscriber_folder":"C:\\ProgramData\\Safe Software\\FME Server\\resources\\system\\temp\\engineresults\\FME_737C7632_1504703233427_2944_nw",
"id":"185781",
"logHome":"C:/ProgramData/Safe Software/FME Server///resources/logs",
"logUrl":"http://FME2016:80/fmerest/v3/transformations/jobs/id/185,781/log"
}

Hi, sorry for the question, first time working with FME here. But how can you suscribe to DATADOWNLOAD_ASYNC_JOB_SUCCESS, is it some kind of webhook? I'm browsing through the docs. but can't find anything about it.

 

I was able to run the job with fmedatadownload service async. But I need a way to query the status of the job without using the Rest API. Any idea how?

 

 


Hi, sorry for the question, first time working with FME here. But how can you suscribe to DATADOWNLOAD_ASYNC_JOB_SUCCESS, is it some kind of webhook? I'm browsing through the docs. but can't find anything about it.

 

I was able to run the job with fmedatadownload service async. But I need a way to query the status of the job without using the Rest API. Any idea how?

 

 

You have to create a subscription on FME Server and subscribe to the DATADOWNLOAD_ASYNC_JOB_SUCCESS topic.

 

Depending on the type of subscription created (I guess email or push should be used), you'll be able to send an email or trigger another workspace as an example to deal with the notification information.

Reply