Solved

Tester vs FeatureJoiner Performance

  • 25 November 2019
  • 2 replies
  • 5 views

Badge +3

Relatively new FME 2018 user and just trying to see if there some improvements to my use of Tester in the workspace. Now, I've been using Tester a lot to do simple things like using it like a WHERE condition, but in 2018 at least it seems to do a naive, feature-by-feature test of the Tester conditions. For example if I use Tester on a 2M feature input with a condition AttributeValue = 'A', then this takes around 2 minutes to complete.

However, I've later discovered FeatureJoiner as a go-to transformer for significantly better performance as this seems to first index the Attribute(s) in question first before testing them, as well as passing out the Unjoined features without any post-processing overhead as essentially "untouched" features.

If I instead use an alternative of Creator(1 feature)->AttributeCreator(Set AttributeValue='A') to make a single Test Feature, and then pass this into the RHS of a FeatureJoiner transformer to test against the 2M feature set on the LHS, then the same filter/test then only takes 11 seconds to complete, because it appears FeatureJoiner first creates an index on the feature set to be tested/joined against vs Tester which doesn't.

So, am I using Tester wrong or applying it in the wrong context? I know it can do a lot of other test types but "=" type queries I would have thought would be its bread-and-butter? Is there any other way to wring out attribute indices in other Transformers like Tester, FeatureMerger etc. that similarly get outperformed by transformers that make use of indices like FeatureJoiner?

icon

Best answer by mark2atsafe 27 November 2019, 18:23

View original

2 replies

Userlevel 4
Badge +25

Good question. The Tester got an update in 2019.2, so that it should perform as well as the FeatureJoiner (there's a new background technique we call Bulk Mode that has only just been added to the Tester). 

From the Change Log file:

-------------------------- FME 2019.2 b19704 20190612 -------------------------- 
Transformers: Added Bulk Mode support to several transformers that use the TestFactory, 
including the Tester and TestFilter [FMEENGINE-56180]

So that should help greatly. I do like the idea of using a Creator/FeatureJoiner combination, but it should no longer be necessary. 

You'll probably want to keep an eye on what other transformers and formats you use. Not all of them use bulk mode yet either. A non-bulk transformer before a bulk-transformer, can cause the bulk-transformer to lose performance. We don't have an official list, but in general 2019 (especially 2019.2) has a lot better support and should be way faster overall.

Badge +3

Good question. The Tester got an update in 2019.2, so that it should perform as well as the FeatureJoiner (there's a new background technique we call Bulk Mode that has only just been added to the Tester). 

From the Change Log file:

-------------------------- FME 2019.2 b19704 20190612 -------------------------- 
Transformers: Added Bulk Mode support to several transformers that use the TestFactory, 
including the Tester and TestFilter [FMEENGINE-56180]

So that should help greatly. I do like the idea of using a Creator/FeatureJoiner combination, but it should no longer be necessary. 

You'll probably want to keep an eye on what other transformers and formats you use. Not all of them use bulk mode yet either. A non-bulk transformer before a bulk-transformer, can cause the bulk-transformer to lose performance. We don't have an official list, but in general 2019 (especially 2019.2) has a lot better support and should be way faster overall.

Thanks @mark2atsafe, good to know has been addressed in 2019.2!

Reply