Question

FMEServerJobSubmitter - Job Submitter only?

  • 8 March 2021
  • 9 replies
  • 46 views
dbaldacchino1
Rank
Badge +3

Hi,

 

So with this transformer you can pick from workspaces that are on FME Server, regardless of what services have been enabled for those workspace. It is not clear to me whether this works only for those configured as "Job Submitter" or any of them. The main questions I have are:

  1. If there are multiple services enabled for a workspace, is there a default it will use?
  2. Since you cannot specify the service when configuring the settings of this transformer, does it mean it will just run the workspace regardless, as if it was configured for "Job Submitter" only?
  3. If the workspace is configured for "Data Streaming" only, I suppose it would just run and you won't actually see the results as FME Server would get the data but the user won't see it, correct? I suppose I can test this myelf...

 

Thanks for any insight you might share. if you're interested in further commentary as to why I'm asking about this, read on 😃

 

I was trying to see if there was a way I could create a workspace App on FME Server that could decide (through built-in logic) whether to stream data back to the user's browser (ex: stream back a file) or to write it to a specific location on a file server. Currently I think you have to have two separate Apps: one that streams and another as Job Submitter. But I was hoping to combine that into one App. For example if the user fills out the project number and we find the network path where to store the file, we save it there but if not, we stream it back to the user. I think it would be valuable to be able to perform these types of things without having to create multiple apps. Thanks.

9 replies

drc43
Rank
Badge +10

Hi again @dbaldacchino1​ ,

 

You bring up some interesting questions, and unfortunately I don't have many answers for you. But I think I have an example of what you're talking about that might help. I have an app that allows the user to select which output they want, web map or database, in a parameter. The workspace then routes this as follows.

outputIf they choose web map, it reprojects it, creates the HTML document and writes it out, streaming back the result. But if they choose database, it first writes it to the database, and then creates a simple HTML page that it streams back to the user letting them know that it is done and where they can find the resulting data. This way the app is only set up for Data Streaming and it can handle both methods. My process also includes an optional parameter for the user to fill in if they choose database to allow them to name part of the resulting feature class. (eg. parameter = Bayside, feature class = TEMP_GEOCODE_Bayside) This is also streamed back to the user so they know what to look for. Is that the kind of thing you're looking to do?

dbaldacchino1
Rank
Badge +3

Hi again @dbaldacchino1​ ,

 

You bring up some interesting questions, and unfortunately I don't have many answers for you. But I think I have an example of what you're talking about that might help. I have an app that allows the user to select which output they want, web map or database, in a parameter. The workspace then routes this as follows.

outputIf they choose web map, it reprojects it, creates the HTML document and writes it out, streaming back the result. But if they choose database, it first writes it to the database, and then creates a simple HTML page that it streams back to the user letting them know that it is done and where they can find the resulting data. This way the app is only set up for Data Streaming and it can handle both methods. My process also includes an optional parameter for the user to fill in if they choose database to allow them to name part of the resulting feature class. (eg. parameter = Bayside, feature class = TEMP_GEOCODE_Bayside) This is also streamed back to the user so they know what to look for. Is that the kind of thing you're looking to do?

Thanks a lot @drc43​ for taking the time to post this example.

This is a very good technique and example, even though it is not exactly what I had in mind. However to be honest, when it comes to Server apps, I do prefer to set them as Data Streaming almost by default these days so I can deliver a meaningful status to the end user, as crude as the HTML page might be! But at least I can communicate with whoever runs it; explain the handled failure in detail, or customize the result so they can try again, etc. If the app ran a Job Submitter workspace, the end result is not the best for the end user; it returns a success or failure and some other data, but it's not terribly informative.

I might try something along the lines of your example, but I want to try streaming two things in my case: an HTML "results" page and if applicable, also stream back a file (I have never tried streaming two outputs at the same time). Thanks again and I'll definitely report back on this one.

david_r
Rank
Userlevel 4

The FMEServerJobSubmitter is, as the name implies, a job submitter (although I suspect that it uses the REST API rather than the jobsubmitter service). You cannot get the results from e.g. the Data Streaming service using the FMEServerJobSubmitter, in that case it would be better to use the HTTPCaller and the FME Server REST API.

drc43
Rank
Badge +10

Thanks a lot @drc43​ for taking the time to post this example.

This is a very good technique and example, even though it is not exactly what I had in mind. However to be honest, when it comes to Server apps, I do prefer to set them as Data Streaming almost by default these days so I can deliver a meaningful status to the end user, as crude as the HTML page might be! But at least I can communicate with whoever runs it; explain the handled failure in detail, or customize the result so they can try again, etc. If the app ran a Job Submitter workspace, the end result is not the best for the end user; it returns a success or failure and some other data, but it's not terribly informative.

I might try something along the lines of your example, but I want to try streaming two things in my case: an HTML "results" page and if applicable, also stream back a file (I have never tried streaming two outputs at the same time). Thanks again and I'll definitely report back on this one.

@dbaldacchino1​ ,

I'm pretty sure I've read somewhere that if you try to stream back multiple files, it will convert it to a Data Download and not stream back anything. Yeah, here it is under requirements: http://docs.safe.com/fme/html/FME_Server_Documentation/ReferenceManual/service_datastreaming.htm

And I totally agree with using the Data Streaming to deliver a meaningful message to the user, that's the direction I'm taking with most of my apps now too.

dbaldacchino1
Rank
Badge +3

The FMEServerJobSubmitter is, as the name implies, a job submitter (although I suspect that it uses the REST API rather than the jobsubmitter service). You cannot get the results from e.g. the Data Streaming service using the FMEServerJobSubmitter, in that case it would be better to use the HTTPCaller and the FME Server REST API.

Thanks @david_r​ , that's what I thought but was thinking that if that were truly the case, that it would only allow me to pick workspaces that had the "Job Submitter" service enabled. However it also allows you to pick workspaces that only have the "Data Streaming" service enabled...is that really by design? It sounded like a bug to me.

dbaldacchino1
Rank
Badge +3

Thanks a lot @drc43​ for taking the time to post this example.

This is a very good technique and example, even though it is not exactly what I had in mind. However to be honest, when it comes to Server apps, I do prefer to set them as Data Streaming almost by default these days so I can deliver a meaningful status to the end user, as crude as the HTML page might be! But at least I can communicate with whoever runs it; explain the handled failure in detail, or customize the result so they can try again, etc. If the app ran a Job Submitter workspace, the end result is not the best for the end user; it returns a success or failure and some other data, but it's not terribly informative.

I might try something along the lines of your example, but I want to try streaming two things in my case: an HTML "results" page and if applicable, also stream back a file (I have never tried streaming two outputs at the same time). Thanks again and I'll definitely report back on this one.

Oh bummer. Yeah it says

The resulting dataset is one file only. After the translation has finished, this dataset is streamed with the appropriate content-type (MIME type) over HTTP back to the client making the request. For example, a web browser or Google Earth are clients that could make the request.

So my suggestion above would need to be tweaked. Might have to figure out how to provide a download link and include it in the HTML page instead. Will post my findings, thanks for the heads up.

david_r
Rank
Userlevel 4

Thanks @david_r​ , that's what I thought but was thinking that if that were truly the case, that it would only allow me to pick workspaces that had the "Job Submitter" service enabled. However it also allows you to pick workspaces that only have the "Data Streaming" service enabled...is that really by design? It sounded like a bug to me.

I don't think it's a bug (the REST API and the jobsubmitter/datastreaming service are entirely separate), but you should signal it to Safe if you think it is.

dbaldacchino1
Rank
Badge +3

The FMEServerJobSubmitter is, as the name implies, a job submitter (although I suspect that it uses the REST API rather than the jobsubmitter service). You cannot get the results from e.g. the Data Streaming service using the FMEServerJobSubmitter, in that case it would be better to use the HTTPCaller and the FME Server REST API.

@david_r​ support confirmed that workspaces triggered by this transformer will be run by REST service regardless of what service it's actually registered to, and should act like the job submitter. They will update the documentation.

david_r
Rank
Userlevel 4

@david_r​ support confirmed that workspaces triggered by this transformer will be run by REST service regardless of what service it's actually registered to, and should act like the job submitter. They will update the documentation.

Thanks for confirming my suspicions!

Reply


Community Stats

28,729
Posts
109,527
Replies
36,565
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.