Question

Dynamic Server Job submitter?


Hi,

I have an FME server job submitter workbench. A creator submits the jobs, then goes to a job waiter and logfile retriever.

In order to not tie up an engine with the submitting bench due to the job waiter, in 2016 I write job metadata to a spread sheet using a feature writer and then output to another job submitter that has the job waiter inside that set to run last.

If that job submitter detects any failures, it will attempt to rerun that bench once more (and only once more).

I want to be able to write more metadata to the same spreadsheet, and have the bench that runs a second attempt at failures, dynamically run what the spreadsheet says to be run again. What we do now is copy the original Submitter bench, and then use a manually defined attribute filter to match workbench names from the spreadsheet to do the second run.

Anyone have any ideas on how I can achieve this? Thinking HTTP caller with some of the system server parameters exposed from the original Submitter on server. But some of the server parameters are blank or will fail the bench if I expose them (FME_SERVER_DEST_DIR for example).

What would be ideal is having the Job Submitter transformer allow for "repository name", "selected item name" (workbench_name.fmw), "wait for job", "job priority", "job tag" to be set by attributes

Then by default use "As Specified in Workspace and Parameters" and by default use the parameters set when published.

Thanks,

Aaron

Edit 1:

I've been successful using the HTTP caller with manually grabbing the DIRECT URL EXAMPLE from the workspace on the server.

However I would need to be able to dynamically retrieve the direct url example for each bench run.

But even this method is not ideal because a Job ID is not retrieved in the end (to check if success or fail and email accordingly) and the bench running the HTTP caller (using get method) still needs to wait for the a response before finishing the workspace (thus still tying up 2 engines).


2 replies

Badge +2
Hi, did you try to use the REST interface? I'm facing a similar problem publishing an FME workspace that schedules multiple sub-workspaces. I use an HTTPCaller to post a REST API call. The REST API (never thought I would use it, however it is pretty nice and easy) is accessible through http://myfmeserver:8080/fmerest, then select API button. I put all the DIRECT URL EXAMPLES in a JSON text file and determine which one to use. Parameter values are substituted into the uploadbody by a JSONTemplater. After all it comes down to constructing the URL (having generated a token from FME Server) and sending an appropriate uploadbody containing all parameter values for the job. The URL that I post is for example: 'http://ourfmeserver:8080/fmerest/v2/transformations/commands/submit/LDR/Stap_4_BerekenSectiesMetStationing.fmw?token=4566443334daef40e8bc340c3a63b3ff434&detail;=low' The Upload body is like { "FMEDirectives" : {}, "NMDirectives" : { "failureTopics" : [ ], "successTopics" : [ ] }, "TMDirectives" : { "rtc" : false, "description" : "LDR autorun LDR_Trigger_Handle_Notification", "priority" : 5, "tag" : "linux" }, "publishedParameters" : [ { "name" : "SourceDataset_FFS", "value" : "D:\\\\Data\\\\FME\\\\FME%20Server\\\\Data\\\\LDR\\\\1_Secties\\\\*.ffs" }, { "name" : "FANOUT_DIRECTORY", "value" : "D:\\\\Data\\\\FME\\\\FME%20Server\\\\Data\\\\LDR\\\\4_SectiesMetStationing" }, { "name" : "DestDataset_MSSQL_ADO", "value" : "GPL" }, { "name" : "MSSQL_ADO_OUT_SERVER_MSSQL_ADO", "value" : "GUHQTSQL1282\\\\TDBMSA" } ], "subsection" : "REST_SERVICE" }'
Badge +16

I agree with @helmoet the rest API provides easy access to multiple functions on fme server and can replace the jobsubmitter.

Reply