Hello,
I have a workbench that called children in sequence using FMEServerJobSubmitter transformer.
I have a strange behaviour. A job in one branch should finish before another job in another branch starts.
My understanding is that the run is sequence should affect only the job in the same branch not in other branches.
Why is that the children called from the three different branches of the ETL can not be run in parallel (i.e. at the same time)?
Thanks
Would it be possible to set the FMEServerJobsubmitter in sequence, like this:
Make sure to set the Wait to Yes!
Then the transformers will not be run in parallel.
Would it be possible to set the FMEServerJobsubmitter in sequence, like this:
Make sure to set the Wait to Yes!
Then the transformers will not be run in parallel.
Hi @erik_jan
What I want is to have the jobs for each branch being submitted on the server in parallel so that the three jobs could run at the same time.
Have you set "Wait for jobs to complete" = Yes? That might explain it.
Try something like this:
Take a look at this KB article:
https://knowledge.safe.com/articles/55594/workflow-management-example-multiple-engines.html
The combination of settings for "Waits for Jobs to Complete" and "Submit Jobs " will influence the parallel Engine behaviour.
I assume that you also have multiple Engines available in your license...?
Have you set "Wait for jobs to complete" = Yes? That might explain it.
Try something like this:
Hello @david_r
Your solution is not exactly what I want.
Let say, I want to run the 3 branches simulateneously and that each branch with receive 10 features.
Is it possible to have such behavoiur with some branches being set to run in parallel while other run in sequence?
I read this article posted by @nic_ran without being able to find the solution https://knowledge.safe.com/articles/55594/workflow-management-example-multiple-engines.html
Take a look at this KB article:
https://knowledge.safe.com/articles/55594/workflow-management-example-multiple-engines.html
The combination of settings for "Waits for Jobs to Complete" and "Submit Jobs " will influence the parallel Engine behaviour.
I assume that you also have multiple Engines available in your license...?
Yes @nic_ran I do have multiple engines.
Is it possible to set some branck to run run in parallel while other run in sequence in a way that the ETL from the three branches run at the same time?
Hello @david_r
Your solution is not exactly what I want.
Let say, I want to run the 3 branches simulateneously and that each branch with receive 10 features.
Is it possible to have such behavoiur with some branches being set to run in parallel while other run in sequence?
I read this article posted by @nic_ran without being able to find the solution https://knowledge.safe.com/articles/55594/workflow-management-example-multiple-engines.html
If so you should use two different groups of FMEServerJobSubmitters, one configure for parallel and one configured for sequential runs. You can then use e.g. a TestFilter to route the jobs to either group according to your criteria.
If so you should use two different groups of FMEServerJobSubmitters, one configure for parallel and one configured for sequential runs. You can then use e.g. a TestFilter to route the jobs to either group according to your criteria.
This is exactly what I did.
My issue issue is that, the group configured for sequential runs end first before the group configured for paralle runs starts.
Is there a way to make it more efficient so that the both groups configures for sequential runs and parallel runs are executed simultaneous?
Take a look at this KB article:
https://knowledge.safe.com/articles/55594/workflow-management-example-multiple-engines.html
The combination of settings for "Waits for Jobs to Complete" and "Submit Jobs " will influence the parallel Engine behaviour.
I assume that you also have multiple Engines available in your license...?
@arthy, I have played around with this with a test workspace and would agree with you that the behaviour does seem to contradict the documentation. I was not able to get parallel processes to run unless I set the "Wait for jobs to complete" parameter to "No". The "Submit Jobs" "In Sequence / In Parallel" option did not have any effect.
I would submit a support request to Safe to ask them to look into this. I have attached my test workspaces for your convenience if you would like to use them when submitting the support request.
Parallel job run.fmw
@arthy, I have played around with this with a test workspace and would agree with you that the behaviour does seem to contradict the documentation. I was not able to get parallel processes to run unless I set the "Wait for jobs to complete" parameter to "No". The "Submit Jobs" "In Sequence / In Parallel" option did not have any effect.
I would submit a support request to Safe to ask them to look into this. I have attached my test workspaces for your convenience if you would like to use them when submitting the support request.
Parallel job run.fmw
Thanks @nic_ran
Hi @arthy,
The problem here is how FME works opposed to the transformer. FME is parallel. So no matter how many transformers you have only one process will happen at a time. Think about writers where no matter how many writers are in the workspace they do not write at the same time. This is the same for the FMEServerJobSubmitter so the delay you are seeing is not infarct anything to do with the job submitter but rather how the features move in the workspace. Since you have all your FMEServerJobSubmitters set to wait to yes this process is limiting the next transformer.
To get around this you could put a featureHolder before each FMEServerJobSubmitter set to Sequential so the Parallel transformers kick off first, but more or less that would be the same as putting the FMEServerJobSubmitters in a line, one after another.