Hi Everyone,
We are trying to execute FME workspaces in batches of eight in parallel to avoid license limit issue. The requests are received dynamically above 8 by a java/ python program but the processing should be done in batches of 8 always. Ideally the requests above 8 should go in queue and wait for the last batch of 8 to finish.
You may want to look into using a master workspace that uses WorkspaceCaller(s) to dispatch the jobs in groups of 8.
Otherwise this sounds like a perfect task for FME Server.
You may want to look into using a master workspace that uses WorkspaceCaller(s) to dispatch the jobs in groups of 8.
Otherwise this sounds like a perfect task for FME Server.
Hi David,
Thanks for your answer. We actually are getting calls dynamically one by one from the user and not in control, so even if we get one job, the workbench has to execute immediately. If another job is requested when the first job is in progress, then the second job should also start in parallel However, the total no. of jobs should not go above 8. Hope I am clear here.
Hi David,
Thanks for your answer. We actually are getting calls dynamically one by one from the user and not in control, so even if we get one job, the workbench has to execute immediately. If another job is requested when the first job is in progress, then the second job should also start in parallel However, the total no. of jobs should not go above 8. Hope I am clear here.
I would still highly recommend FME Server (or FME Cloud, depending on where your data lives). It can be pricy but it has the queuing system built right into it.
On regular desktop I think you can simply run multiple FME instances from the command line and it should cap out at 8, but then you'd have to implement a queuing system yourself.
I would still highly recommend FME Server (or FME Cloud, depending on where your data lives). It can be pricy but it has the queuing system built right into it.
On regular desktop I think you can simply run multiple FME instances from the command line and it should cap out at 8, but then you'd have to implement a queuing system yourself.
Thanks. We are currently doing the same in a program using java but only one instance of FME is seen in taskbar and the next job goes in queue. We do not want to use shell as we need to process the workbench output status in java for each request.
Thanks. We are currently doing the same in a program using java but only one instance of FME is seen in taskbar and the next job goes in queue. We do not want to use shell as we need to process the workbench output status in java for each request.
If you want to use FME Desktop to emulate the queuing and job routing functionality in FME Server, you'll basically have to implement it yourself as there are no such mechanisms in FME Desktop.
Hi @ashitosh
You should really consider FME Server or FME Cloud. You have to consider where the data is, how much data and how often the process is running. You can get an FMECloud trial with some free credits (but remember to PAUSE the instance after you are done testing to reduce the costs).
https://www.safe.com/fme/fme-cloud/
Thanks Sigtill and David! Unfortunately, due to budget constraints at client's end, we are not able to upgrade to server as of now. The task is handled programmatically in batch on FME desktop. Whenever a file request is available, we have to execute but not more than 8 threads at any time. Not sure if workspacecallers will help, but will definitely try out.