Skip to main content
Solved

Workspacerunner parallel processing same worspace

takanaka
Contributor
Forum|alt.badge.img+4

Hi,

 

I'm looking for some additional explanation on parallel processing using the workspacerunner transformer.

 

There's 20 batches of GDB-data (all separate gdb's to prevent locking).

When I use the workspace runner to use 1 workspace to run 4 parallel instances using different input and output geodatabases (no locking is occuring) for all 20 batches something goes wrong. I cant tell what is causing issues because the logs are incomprehensible. Does every instance keep overwriting the log file?

I see that it executes the command to run in a rapid order while each feature is expected to run around 5 minutes. After waiting and seeing all FME processes finish in my Task Manager i only see around 4 output geodatabases made. I need to see: 20.

 

However: I tried to fix it. I use Text-reader and Text-writer to create 1 .fmw for each batch: so 20 fme models. I use each model 1 time in each parallel process. Meaning: unique input gdb, unique output gdb, using unique fme workspace. This way: i have no issues.

 

What is going on? What is going Wrong?

Do I misunderstand how WorkspaceRunner is supposed to work? It is not expected behaviour that each process needs a unique .fmw right?

 

Thanks in advance,

takanaka

Best answer by nielsgerrits

Your understanding is correct. You need 1 parent and 1 child workspace. The parent has the workspacerunner, which reads the sources (directory and file reader?) and devides the work to the child workspace, passing the source paths using published parameters to the child workspace. I suspect this is what is going wrong, not using the published parameters in the child and the workspacerunner. If not, you need to share more about what you have. Screenshots or the parent workspace.

View original
Did this help you find an answer to your question?

5 replies

nielsgerrits
VIP

Your understanding is correct. You need 1 parent and 1 child workspace. The parent has the workspacerunner, which reads the sources (directory and file reader?) and devides the work to the child workspace, passing the source paths using published parameters to the child workspace. I suspect this is what is going wrong, not using the published parameters in the child and the workspacerunner. If not, you need to share more about what you have. Screenshots or the parent workspace.

takanaka
Contributor
Forum|alt.badge.img+4
  • takanakaContributor
  • Author
  • Contributor
  • July 7, 2023
nielsgerrits wrote:

Your understanding is correct. You need 1 parent and 1 child workspace. The parent has the workspacerunner, which reads the sources (directory and file reader?) and devides the work to the child workspace, passing the source paths using published parameters to the child workspace. I suspect this is what is going wrong, not using the published parameters in the child and the workspacerunner. If not, you need to share more about what you have. Screenshots or the parent workspace.

Thanks for your reply. I now fixed the problem. I discovered i can create log files for each parallel instance as the log file of a workspace can be linked to a user parameter. So i now create 20 log files (i did not know it was possible). Finally i could read why it was an issue. There was FME_MF_DIR linked inside the user parameters that was fed to the model. This was not working because the FME MF DIR of the 20 batched models was in a different folder than what i otherwise use as the not-batched model.

 

Thank you for your help,

takanaka

takanaka
Contributor
Forum|alt.badge.img+4
  • takanakaContributor
  • Author
  • Contributor
  • July 10, 2023

I think another of the issues is the limit of licenses. I often see issues that the license requested is not available message in the logs.

nielsgerrits
VIP
takanaka wrote:

I think another of the issues is the limit of licenses. I often see issues that the license requested is not available message in the logs.

FME? Or external license?

takanaka
Contributor
Forum|alt.badge.img+4
  • takanakaContributor
  • Author
  • Contributor
  • July 10, 2023
nielsgerrits wrote:

FME? Or external license?

External i think.

Reply


Helpful Members This Week

Community Stats

30,921
Posts
117,182
Replies
38,725
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.