FME Automation and Multiple Instances

It sounds like someone else asked a similar question only using Python instead of C#. Maybe it would be a bit similar? https://knowledge.safe.com/questions/25014/get-number-of-concurrent-fmeprocesses-and-fmeworke.html but does not give me quite enough info.

 

 

 

Hi @kirk, just to make sure I'm understanding your requirement. Are you looking to find out the number of FME processes currently running to determine how many allowed concurrent processes you have left in the same FME instance? Are you looking to use this information to set the Maximum Number of Concurrent Workspaces parameter in the WorkspaceRunner?

 

 

xiaomengatsafe wrote:

Hi @kirk, just to make sure I'm understanding your requirement. Are you looking to find out the number of FME processes currently running to determine how many allowed concurrent processes you have left in the same FME instance? Are you looking to use this information to set the Maximum Number of Concurrent Workspaces parameter in the WorkspaceRunner?

 

 

Hi @Xiaomeng, my goal is to use the FMEObject classes to run concurrent workspaces (I am currently using C#). Perhaps there is a different way to initialize the FMESession so that it instantiates a new FME instance for each FMESession, but I do not know how to do that. It appears that all the FMESessions are using a single instance of FME. Issuing commandline calls may be one alternative method, but I would prefer to use the FMEObject classes if possible. For now I am limiting the number of instances to 4 so should not run into the limit of 8. Maybe I need to set some additional properties so the workspaces will run concurrently?

 

 

xiaomengatsafe wrote:

Hi @kirk Is there a reason that you prefer using FMEobject? If your goal is to enhance performance, have you considered using a parent workspace calling multiple child workspaces using WorkspaceRunner? If you set "Wait for Job to Complete" to No, it can run multiple workspaces in parallel. Using fmeObject to run multiple processes shouldn't be significant performance advantages over using a simple parent workspace to call Multiple WorkspaceRunners.

If you need higher scalability and performance, FME Server with multiple Engines would be the most recommended approach.

Here are some examples of using workspaceRunner: https://knowledge.safe.com/articles/1469/batch-processing-using-the-workspacerunner-1.html

Here are some more information on FME Server Scalability: https://knowledge.safe.com/articles/1544/fme-server-scalability-1.html

Some more background on concurrent FME processes: https://knowledge.safe.com/articles/139/how-many-concurrent-fme-processes-can-i-run-at-my.html

I was just made aware that you've chatted with some of my colleagues and have discussed the option of using WorkspaceRunner transformer in workbench. It won't work for you, as you want to embed FME in your own workflow. I'm also no expert on the topic of fme object, but I think I know who might be able to provide some help, and I'll try to get some update for you here within the next couple of days.

 

In the mean time, this doesn't rule out FME Server option, as you can deploy the multiple workspaces to FME Server, and use one parent workspace with FMEServerJobSubmitters to call them. And from your own workflow, you can call the parent workspace using FME Server REST API. (There might even be more elegant method with FME Server)

 

kirk wrote:

Hi @Xiaomeng, my goal is to use the FMEObject classes to run concurrent workspaces (I am currently using C#). Perhaps there is a different way to initialize the FMESession so that it instantiates a new FME instance for each FMESession, but I do not know how to do that. It appears that all the FMESessions are using a single instance of FME. Issuing commandline calls may be one alternative method, but I would prefer to use the FMEObject classes if possible. For now I am limiting the number of instances to 4 so should not run into the limit of 8. Maybe I need to set some additional properties so the workspaces will run concurrently?

 

 

Hi @kirk, Would it be ok if we reached out to you via email privately to discuss the details of your specific need, and help figure out what is the best approach going forward? Thanks!