Skip to main content
Question

Load balancing using WorkspaceRunner transformers


erik_jan
Contributor
Forum|alt.badge.img+18

Hello FME users,

I hope somebody can help me with this situation:

I have a workspace that reads a list of feature ID's.

For each feature I want to run the F020 workspace (connecting to a database and creating a file).

To create 4 parallel processes, I split up the list and pass an ID to each of the 4 WorkspaceRunners (each using their own database connection).

The WorkspaceRunners are set to wait, but that results in all processes being run in sequence.

If I change the setting to "Wait = No", the file is not created in time for the next step in the process.

Any idea how I can run the 4 F020 WorkspaceRunners in parallel, but run the FileExistenceChecker for each process after the F020 has created its output file?

All suggestions are appreciated.

 

6 replies

ebygomm
Influencer
Forum|alt.badge.img+39
  • Influencer
  • June 6, 2019

For similar issues in the past, i've checked for a certain outcome, e.g. existence of a file, and if it doesn't exist wait a minute and loop and check again until it does exist. Although have to be careful with this approach as sometimes the file can exist without all the data being complete.

 

The lag between the parent workspace saying it's complete and the actual completion can be pretty big.

 

Trust you have voted for this idea?

https://knowledge.safe.com/idea/22570/workspacerunner-identify-when-all-child-processes.html


jdh
Contributor
Forum|alt.badge.img+28
  • Contributor
  • June 6, 2019

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.


erik_jan
Contributor
Forum|alt.badge.img+18
  • Author
  • Contributor
  • June 6, 2019
ebygomm wrote:

For similar issues in the past, i've checked for a certain outcome, e.g. existence of a file, and if it doesn't exist wait a minute and loop and check again until it does exist. Although have to be careful with this approach as sometimes the file can exist without all the data being complete.

 

The lag between the parent workspace saying it's complete and the actual completion can be pretty big.

 

Trust you have voted for this idea?

https://knowledge.safe.com/idea/22570/workspacerunner-identify-when-all-child-processes.html

Thanks @egomm I did vote for David's idea.


erik_jan
Contributor
Forum|alt.badge.img+18
  • Author
  • Contributor
  • June 6, 2019
jdh wrote:

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.

Thanks @jdh I think I will give that a try.


erik_jan
Contributor
Forum|alt.badge.img+18
  • Author
  • Contributor
  • June 7, 2019
jdh wrote:

In the past I have put a moduloCounter on the feature, wrapped the WorkspaceRunner (Wait for jobs=Yes) in a custom transformer and set the parallel process group-by to the modulo.

I implemented this suggestion and it works great!

Thanks @jdh


jdh
Contributor
Forum|alt.badge.img+28
  • Contributor
  • June 7, 2019
erik_jan wrote:

I implemented this suggestion and it works great!

Thanks @jdh

Your welcome.

 

Just make sure to keep the modulo under the number of concurrent licenses -1 (usually 7).

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings