Question

FME Server Automations and Multi-Engine Task Completion

moberdries
Rank
Badge +3

I have an FME Server Automation, triggered as a schedule, that has these basic steps:

 

  1. Reads binary files from a SFTPS site and writes to staging folder (A)
  2. Reads binary files from staging folder (A), decode files, and writes out to staging folder (B)
  3. Reads decoded files from the staging folder (B) and writes to a Vertica database
  4. Cleans up on staging folders (A) and (B), ready for next schedule

 

The decoding bit in step (2) takes its file input from a PATH Reader and can take some time to finish, so I run the workspace in parallel against a FME Server queue across 4 engines.

 

My issue is that the Automation moves to step (3) as soon as 1 of the 4 engines becomes available. Step (3) and step (4) are fast and complete before the remaining 3 engines running step (2) complete. This results is 3 files arriving in the staging folder (B) after the Automation has finished.

 

QUESTION: Is there a ”simple” way to make the Automation wait until step (2) has fully completed on all 4 engines, before moving to step (3)?

 

The Automation Merger looked promising, but I would have to split the output from the PATH Reader in Step (2) across 4 workspaces and use something like a Sampler to split the file feeds.

 

I looked at using a Decelerator and/or a FMEServerJobWaiter, but neither seemed very elegant – any ideas?

 

Regards

Mike

10 replies

itay
Rank
Badge +16

Hi @moberdries,

This is something you can solve by adding job queues and priorities see doc here.

This way you can set up step 2 with a higher priority than the rest of the steps and make sure the step 2 workspaces run with specific engines.

The other steps should be assigned different engines with a lower priority.

Hope this helps.

Itay

H
Badge +2

Hi @moberdries,

Are all of these steps currently triggered from a single schedule? If you don't want Step 3 to run until Step 2 is complete why not look at chaining these jobs in the Automation so the workspace in step 3 will only be sent to the queue to run on successful completion of step 2? This way the job isn't waiting in the queue yet so even if the engines free up it won't matter.

moberdries
Rank
Badge +3

Hi @moberdries,

Are all of these steps currently triggered from a single schedule? If you don't want Step 3 to run until Step 2 is complete why not look at chaining these jobs in the Automation so the workspace in step 3 will only be sent to the queue to run on successful completion of step 2? This way the job isn't waiting in the queue yet so even if the engines free up it won't matter.

Yes, the automation is triggered from a single schedule and the workspaces in play are in the same repository, which is assigned to a single queue.

You say "why not look at chaining these jobs in the Automation so the workspace in step 3 will only be sent to the queue to run on successful completion of step 2? " - can you expand on how I would do this without sacrificing the ability to process on 4 engines in parallel at step 2.

Effectively what I am looking for here is a FeatureHolder for Automations.

I think Italy's solution below using priorities might work - I will investigate.

Mike

H
Badge +2

Yes, the automation is triggered from a single schedule and the workspaces in play are in the same repository, which is assigned to a single queue.

You say "why not look at chaining these jobs in the Automation so the workspace in step 3 will only be sent to the queue to run on successful completion of step 2? " - can you expand on how I would do this without sacrificing the ability to process on 4 engines in parallel at step 2.

Effectively what I am looking for here is a FeatureHolder for Automations.

I think Italy's solution below using priorities might work - I will investigate.

Mike

Hi @moberdries,

How are you submitting Step 2 in parallel? Do you just have multiple Run Workspace action in the Automation that make up step two or a single parent workspace that feeds the binary file paths into an FMEServerJobSubmitter transformer to run them in parallel? If it is the latter this was my thought process.

Automation:

 

Job History:

 

If it is the former then you're right, the merge action is the closest we have to the FeatureHolder idea, but it sounds like that doesn't meet your needs as you can't pass information into the path reader as you'd like? Are you able to share a screenshot of your Automation set up so far and I'll see if I can think of any other ideas for you?

 

Given that you do not want the Workspace in Step 3 to run until Step 2 is completed, I'm not sure setting Job priority will give you the desired result. Job priority is set on a per repository basis and allows you to have some control over what order jobs are run in once they are in the queue. However in order to prevent Step 3 from running early you'd have to assign it to a queue with an engine that you can guarantee would be the last one with jobs from step 2 running - otherwise you'll still encounter the same issue that as soon as the engine is free - the job will run. Is there always one job in Step 2 that takes longest to run?

moberdries
Rank
Badge +3

Hi @moberdries,

Are all of these steps currently triggered from a single schedule? If you don't want Step 3 to run until Step 2 is complete why not look at chaining these jobs in the Automation so the workspace in step 3 will only be sent to the queue to run on successful completion of step 2? This way the job isn't waiting in the queue yet so even if the engines free up it won't matter.

Automation currently looks like this

H
Badge +2

Automation currently looks like this

Hi @moberdries,

Ok, so you do have it set up in the way I was first thinking. In the Master workspace are you using an FMEServerJobSubmitter, try setting the Wait for Jobs to Complete = Yes and that way the Master translation won't finish until all the child jobs are complete and so step 3 won't be triggered early.

moberdries
Rank
Badge +3

Hi @moberdries,

Are all of these steps currently triggered from a single schedule? If you don't want Step 3 to run until Step 2 is complete why not look at chaining these jobs in the Automation so the workspace in step 3 will only be sent to the queue to run on successful completion of step 2? This way the job isn't waiting in the queue yet so even if the engines free up it won't matter.

Nope - step 2 has a workspace that begins with a PATH Reader - each feature being passed is a file "to be decoded". The files are read from the "staging folder" and the decoding process in step 2 is complex and time consuming, so we spread it across 4 engines for performance reasons, i.e. we process 4 files at a time as 4 jobs running in parallel, on 4 separate engines. No file is the same size so jobs finish a different times on different engines. When we get to the last 4 files in the staging folder, one file is going to be decoded quicker than the others, meaning an engine becomes available. That engine, rather than waiting for the other engines to complete, moves to step 3, while step 2 is still processing. This is unwanted behaviour because it means that step 2 output is still being generated after the Automation has completed. Make sense?

H
Badge +2

Nope - step 2 has a workspace that begins with a PATH Reader - each feature being passed is a file "to be decoded". The files are read from the "staging folder" and the decoding process in step 2 is complex and time consuming, so we spread it across 4 engines for performance reasons, i.e. we process 4 files at a time as 4 jobs running in parallel, on 4 separate engines. No file is the same size so jobs finish a different times on different engines. When we get to the last 4 files in the staging folder, one file is going to be decoded quicker than the others, meaning an engine becomes available. That engine, rather than waiting for the other engines to complete, moves to step 3, while step 2 is still processing. This is unwanted behaviour because it means that step 2 output is still being generated after the Automation has completed. Make sense?

Hi @moberdries,

Yes, I understand the issue you are trying to overcome but I'm having a hard time understanding how you are submitting jobs in parallel if you are not using a JobSubmitter or Rest API. Using the Path Reader will allow you to run multiple files through a single workflow but if your master workspace isn't launching additional jobs then this will remain running on a single engine through one job - so how are you making the jobs run in parallel across multiple engines from this single Automation?

Perhaps it may be easier to jump on a call so I can view your Master workspace. If you would find this helpful then please submit a case through Safe Software Support and include 'Attn Holly' in the subject.

moberdries
Rank
Badge +3

Hi @moberdries,

Are all of these steps currently triggered from a single schedule? If you don't want Step 3 to run until Step 2 is complete why not look at chaining these jobs in the Automation so the workspace in step 3 will only be sent to the queue to run on successful completion of step 2? This way the job isn't waiting in the queue yet so even if the engines free up it won't matter.

My bad Holly - you are right, step 2 is a Master workspace calling a child, via an FMEServerJobSubmitter. The FMEServerJobSubmitter is set to "run in parallel", which is what allows us to run 4 instances of the child workspace on 4 separate engines. Under the hood the Master workspace (step 2) looks like this:

The config of the FMEJServerJob Submitter looks like this:

 

moberdries
Rank
Badge +3

Hi @moberdries,

Are all of these steps currently triggered from a single schedule? If you don't want Step 3 to run until Step 2 is complete why not look at chaining these jobs in the Automation so the workspace in step 3 will only be sent to the queue to run on successful completion of step 2? This way the job isn't waiting in the queue yet so even if the engines free up it won't matter.

Hi Holly - in the end I decided to modify my Master workspace and republish. If I use a ModuloCounter transformer I can split the files coming from the PATH Reader into 4 separate streams. Each stream can then call a FMEServerJobSubmitter (running in series this time). I can then place a FeatureHolder transformer downstream of the 4x FMEServerJobSubmitter transformers which will ensure all jobs complete before moving to the next step in the FME Server automation. This will still result in 4 instances of the child workspace running on FME Server across 4 engines, giving me the parallel processing I require.

Reply


Community Stats

28,662
Posts
109,327
Replies
36,529
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.