Skip to main content
  • EN

Hello FME users,

I hope somebody can help me with this situation:

I have a workspace that reads a list of feature ID's.

For each feature I want to run the F020 workspace (connecting to a database and creating a file).

To create 4 parallel processes, I split up the list and pass an ID to each of the 4 WorkspaceRunners (each using their own database connection).

The WorkspaceRunners are set to wait, but that results in all processes being run in sequence.

If I change the setting to "Wait = No", the file is not created in time for the next step in the process.

Any idea how I can run the 4 F020 WorkspaceRunners in parallel, but run the FileExistenceChecker for each process after the F020 has created its output file?

All suggestions are appreciated.

 

For similar issues in the past, i've checked for a certain outcome, e.g. existence of a file, and if it doesn't exist wait a minute and loop and check again until it does exist. Although have to be careful with this approach as sometimes the file can exist without all the data being complete.

 

The lag between the parent workspace saying it's complete and the actual completion can be pretty big.

 

Trust you have voted for this idea?

https://knowledge.safe.com/idea/22570/workspacerunner-identify-when-all-child-processes.html

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.

For similar issues in the past, i've checked for a certain outcome, e.g. existence of a file, and if it doesn't exist wait a minute and loop and check again until it does exist. Although have to be careful with this approach as sometimes the file can exist without all the data being complete.

 

The lag between the parent workspace saying it's complete and the actual completion can be pretty big.

 

Trust you have voted for this idea?

https://knowledge.safe.com/idea/22570/workspacerunner-identify-when-all-child-processes.html

Thanks @egomm I did vote for David's idea.

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.

Thanks @jdh I think I will give that a try.

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.

I implemented this suggestion and it works great!

Thanks @jdh

I implemented this suggestion and it works great!

Thanks @jdh

Your welcome.

 

Just make sure to keep the modulo under the number of concurrent licenses -1 (usually 7).

Reply


Community Stats

30,405
Posts
115,520
Replies
38,209
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.