Skip to main content
  • EN

Hi,

 

I'm looking for some additional explanation on parallel processing using the workspacerunner transformer.

 

There's 20 batches of GDB-data (all separate gdb's to prevent locking).

When I use the workspace runner to use 1 workspace to run 4 parallel instances using different input and output geodatabases (no locking is occuring) for all 20 batches something goes wrong. I cant tell what is causing issues because the logs are incomprehensible. Does every instance keep overwriting the log file?

I see that it executes the command to run in a rapid order while each feature is expected to run around 5 minutes. After waiting and seeing all FME processes finish in my Task Manager i only see around 4 output geodatabases made. I need to see: 20.

 

However: I tried to fix it. I use Text-reader and Text-writer to create 1 .fmw for each batch: so 20 fme models. I use each model 1 time in each parallel process. Meaning: unique input gdb, unique output gdb, using unique fme workspace. This way: i have no issues.

 

What is going on? What is going Wrong?

Do I misunderstand how WorkspaceRunner is supposed to work? It is not expected behaviour that each process needs a unique .fmw right?

 

Thanks in advance,

takanaka

Your understanding is correct. You need 1 parent and 1 child workspace. The parent has the workspacerunner, which reads the sources (directory and file reader?) and devides the work to the child workspace, passing the source paths using published parameters to the child workspace. I suspect this is what is going wrong, not using the published parameters in the child and the workspacerunner. If not, you need to share more about what you have. Screenshots or the parent workspace.

Your understanding is correct. You need 1 parent and 1 child workspace. The parent has the workspacerunner, which reads the sources (directory and file reader?) and devides the work to the child workspace, passing the source paths using published parameters to the child workspace. I suspect this is what is going wrong, not using the published parameters in the child and the workspacerunner. If not, you need to share more about what you have. Screenshots or the parent workspace.

Thanks for your reply. I now fixed the problem. I discovered i can create log files for each parallel instance as the log file of a workspace can be linked to a user parameter. So i now create 20 log files (i did not know it was possible). Finally i could read why it was an issue. There was FME_MF_DIR linked inside the user parameters that was fed to the model. This was not working because the FME MF DIR of the 20 batched models was in a different folder than what i otherwise use as the not-batched model.

 

Thank you for your help,

takanaka

I think another of the issues is the limit of licenses. I often see issues that the license requested is not available message in the logs.

I think another of the issues is the limit of licenses. I often see issues that the license requested is not available message in the logs.

FME? Or external license?

FME? Or external license?

External i think.

Reply


Community Stats

30,162
Posts
114,483
Replies
37,809
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.