I have a workbench that runs a lot of geometry intersection tasks with many source datasets and writes the results in a database. I grouped the different tasks into Custom Transformers, and most of the source datasets go into only one Custom Transformer and are not needed anywhere else. My problem is that as soon as a group based transformer (Feature Mergers and Aggregators mainly) is reached in one of the custom transformers, more source data is loaded and the Workbench runs out of memory once the intersection tasks start.
Is it possible to configure FME Workbench in a way to read only the data relevant to one custom transformer and force it to finish before loading more source data for another custom transformer?
Page 1 / 1
Not particularly, no. You can order the readers, but as soon as a group-based transformer is reached, it's difficult to say what will happen. I tried it and, like you, a second reader starts to read.
So... what you could do is replace the reader with a FeatureReader transformer. Then you could get the output from your group transformers, select/sample a single feature, and use it to read the other data at that point with a FeatureReader.
I haven't looked at the situation in depth, but that's the first thought that comes to mind.
This could be an easy way.
- In the custom transformer definition, create published parameters from the "Parallel Process By" (FME_PROCESS_GROUP_BY) parameter and the "Parallel Process Groups are Ordered" (FME_PROCESS_GROUPS_ORDERED) parameter, as shown in the screenshot below.
- In the Main canvas, set an attribute which identifies each dataset - e.g. "fme_dataset" to the former parameter and set "Yes" to the latter parameter.
Sounds like a job for the workspacerunner. It allows you to set a ‘wait for job to complete’ parameter. Combine this with feature readers and writers in the called workspace. Depending on your workflow you might need to write some temporary data to process after all the custom transformormers have completed.