Skip to main content
  • EN

Dear all,

 

I use FME to perform a complex comparison analysis between two large datasets. I am looking for a way to analyze various subpopulations in succession, in order to reduce memory usage and increase performance. I have defined the desired subpopulations in a Reader file, so they can feed the SQLExecutor transformers to obtain the subsets. After performing the analysis on the first subset (i.e. 'Apple') and writing the results to a report file, the next subset (i.e. 'Banana') should be analyzed and so on.

 

fme 

How can I implement such a 'for each loop' within FME?

 

Thanks in advance.

 

Remy

You're going to want to look at group based custom transformers

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Transformers/Group_Based_Transformers.htm

https://community.safe.com/s/question/0D54Q0000849jVa/how-use-group-by-on-custom-transformer

 

And looping in custom transformers

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Workbench/Workbench/transformers_custom_looping.htm

I'd personally use a parent workspace with a workspace runner to run the workspace for each line in your file, you should be able to use send Source_1 and Source_2 as parameters in the child workspace

Workspace runner or a batch file will be your best bet.

 

If you are running something in a specific order and one at a time the runner works well.

 

If you running a comparison between the first two, comparing the result of that run to the next, etc. A batch file with a series of calls/start would let you spawn all the instances at once as independent processes. You can run up to 8 processes at a time.

 

I'd personally use a parent workspace with a workspace runner to run the workspace for each line in your file, you should be able to use send Source_1 and Source_2 as parameters in the child workspace

Exactly what I was looking for!

 

Firstly, I added two Published User Parameters to the initial (child) workspace:

01 

After which I changed both SQLExecutors to SQLCreators that use the published parameters as SQL Statement.02At last, I created a parent workspace that affects the published parameters.

03 

Thanks a lot!

Reply


Community Stats

30,185
Posts
114,572
Replies
37,848
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.