Question

Processing Features in Groups based on Attribute Value, e.g. using AttributeFilter

  • 29 September 2021
  • 4 replies
  • 92 views
A
Badge

So I just got my toes wet with customer transformers and loops. Fascinating. For the first time the worlds of programming and FME seem to really come together here in my mind. Limitless potential.

I put together a custom transformers that takes a bunch of point features, then identifies a certain point, pops it out into feature holder and feeds the rest of the points back to the start of the loop. Essentially, this allows me to put these points in any order I would like for further processing, using my own business logic.

Now, this works great for one set of points. What if I have 10K points and would like to use my custom transformers on batches of these. I can split them out in to groups/stream using an AttributeFilter but how do I connect that to the custom transformer? I could certainly connect each stream to a copy of the transformer but that would be crude and crazy clutter. Is there something like a FeatureHolder or some other trick - some form of GroupBy - before we get to the custom transformer that will process all features with , say, NUM = 1, before starting on those with NUM = 2?

I might not be seeing the forest for the trees here. Appreciate any suggestions.

4 replies

virtualcitymatt
Rank
Userlevel 4
Badge +26

Perhaps I don't fully unterstand the question but custom transformers can do group-by. https://community.safe.com/s/article/parallel-processing-using-custom-transformers

Does this help?

A
Badge

Thanks, @virtualcitymatt​ 

Honestly, I find the custom transformer creation a bit convoluted. Totally understand that the inner workings may be non-trivial. But after several attempts, it seems that I can either create a looping custom transformer that is of the linked type, or I can create one - I guess 'embedded' that allows group-by. I haven't been able to do both. Whichever way I go, likely be design, I don't have all the options or parameters to do both. I'll keep playing with this. The only way I can envision this working right now is by upping the complexity and using a custom transformer inside a custom transformer... not sure that is possible. But FME always surprises with great features. 😀

virtualcitymatt
Rank
Userlevel 4
Badge +26

Thanks, @virtualcitymatt​ 

Honestly, I find the custom transformer creation a bit convoluted. Totally understand that the inner workings may be non-trivial. But after several attempts, it seems that I can either create a looping custom transformer that is of the linked type, or I can create one - I guess 'embedded' that allows group-by. I haven't been able to do both. Whichever way I go, likely be design, I don't have all the options or parameters to do both. I'll keep playing with this. The only way I can envision this working right now is by upping the complexity and using a custom transformer inside a custom transformer... not sure that is possible. But FME always surprises with great features. 😀

Hmm, linked custom transformers should also allow group by - but the process is indeed convoluted. I can honestly say that I've almost never had to use looping in FME.

Mostly I've been able to get away with a combination of Cloners, Samples, Testers, Sorters and StatisticsCalculators. This usually can give me a result I want.

A
Badge

Got it working now... you first do your Group By, as explained here. Then you export your custom transformers as linked, as described here. Sweet. There is a lot or processing that goes on here. In my case there is a GroupBy happening going into the Custom Transformer, and then I'm using a GroupBy with the re-ordered data coming out the Transformer. But it does exactly what I wanted.

@virtualcitymatt​ - thanks again for chiming in. Not sure how often I will use this. Part of the drill here was to merely wrap my brain around it.

Reply


Community Stats

28,662
Posts
109,327
Replies
36,529
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.