I have a workbench that runs a lot of geometry intersection tasks with many source datasets and writes the results in a database. I grouped the different tasks into Custom Transformers, and most of the source datasets go into only one Custom Transformer and are not needed anywhere else. My problem is that as soon as a group based transformer (Feature Mergers and Aggregators mainly) is reached in one of the custom transformers, more source data is loaded and the Workbench runs out of memory once the intersection tasks start.
Is it possible to configure FME Workbench in a way to read only the data relevant to one custom transformer and force it to finish before loading more source data for another custom transformer?
Solved
Is it possible to force finish execution of custom transformer before reading more source data?
Best answer by mark2atsafe
Not particularly, no. You can order the readers, but as soon as a group-based transformer is reached, it's difficult to say what will happen. I tried it and, like you, a second reader starts to read.
So... what you could do is replace the reader with a FeatureReader transformer. Then you could get the output from your group transformers, select/sample a single feature, and use it to read the other data at that point with a FeatureReader.
I haven't looked at the situation in depth, but that's the first thought that comes to mind.
+43- Safer
- Best Answer
Not particularly, no. You can order the readers, but as soon as a group-based transformer is reached, it's difficult to say what will happen. I tried it and, like you, a second reader starts to read.
So... what you could do is replace the reader with a FeatureReader transformer. Then you could get the output from your group transformers, select/sample a single feature, and use it to read the other data at that point with a FeatureReader.
I haven't looked at the situation in depth, but that's the first thought that comes to mind.
+21- Contributor
This could be an easy way.
- In the custom transformer definition, create published parameters from the "Parallel Process By" (FME_PROCESS_GROUP_BY) parameter and the "Parallel Process Groups are Ordered" (FME_PROCESS_GROUPS_ORDERED) parameter, as shown in the screenshot below.
- In the Main canvas, set an attribute which identifies each dataset - e.g. "fme_dataset" to the former parameter and set "Yes" to the latter parameter.
Sounds like a job for the workspacerunner. It allows you to set a ‘wait for job to complete’ parameter. Combine this with feature readers and writers in the called workspace. Depending on your workflow you might need to write some temporary data to process after all the custom transformormers have completed.
Reply
Rich Text Editor, editor1
Editor toolbars
Press ALT 0 for help
Helpful Members This Week
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.