"Feature Tables" - the buzz word heard around the 2017 FME UC, is now referred to by a more intuitive term "Bulk Mode"... The CSV format has been updated for amazing performance increases, but for what transformers and formats can I expect to see benefits from Bulk Mode?
The list is currently...
FME VersionTransformersFormats2017.0
2019.0
There are also FME functions supporting this technology...
@Concatenate, @CopyAttributes, @FeatureType, @Log, @LogCount, @RemoveAttributes, @RenameAttributes, @SupplyAttributes, and @Value
The list is currently...
FME VersionTransformersFormats2017.0
2019.0
There are also FME functions supporting this technology...
@Concatenate, @CopyAttributes, @FeatureType, @Log, @LogCount, @RemoveAttributes, @RenameAttributes, @SupplyAttributes, and @Value
FME 2017.0
Additionally, some blocking transformers have learned to utilize bulk mode inputs when input features are in bulk mode and group-by is not used.
These transformers include:
Note: these transformers do not output features in bulk mode
What about which reader / writers and featurereader/featurewriters?
What about which reader / writers and featurereader/featurewriters?
Edit: An amazing editor cleaned this up to display as a table (way easier to read) versus the list structure I had originally posted!
Would be nice if InlineQuerier, SQLCreator and SQLExtractor would be added to the list.
Would be nice if InlineQuerier, SQLCreator and SQLExtractor would be added to the list.
Absolutely, I use the SQLExecutor a lot.
Would be nice if InlineQuerier, SQLCreator and SQLExtractor would be added to the list.
Hi @takashi and @david_r!
Good news—The SQLCreator and SQLExecutor in 2019.0 support Bulk Mode (formerly known as Feature Tables), *if* you are already reading from a format that supports Bulk Mode (see above for formats). The InLineQuerier is not there *yet* but keep your eyes peeled for updates on this.
Please note two key things:
1) If you are reading a format that doesn't support bulk mode, then the SQLCreator and SQLExecutor will not be utilizing bulk mode.
2) If you are opening/editing a workspace authored in 2018 or earlier with a SQLCreator/Executor and would like to leverage bulk mode (assuming you are reading in a bulk mode enabled format) in 2019, you will need to replace your existing SQLCreator/Executor transformer in the FME 2019 workspace.
If you have any questions or feedback about this, please don't hesitate to reach out!
Best,
Annabelle
Hi @takashi and @david_r!
Good news—The SQLCreator and SQLExecutor in 2019.0 support Bulk Mode (formerly known as Feature Tables), *if* you are already reading from a format that supports Bulk Mode (see above for formats). The InLineQuerier is not there *yet* but keep your eyes peeled for updates on this.
Please note two key things:
1) If you are reading a format that doesn't support bulk mode, then the SQLCreator and SQLExecutor will not be utilizing bulk mode.
2) If you are opening/editing a workspace authored in 2018 or earlier with a SQLCreator/Executor and would like to leverage bulk mode (assuming you are reading in a bulk mode enabled format) in 2019, you will need to replace your existing SQLCreator/Executor transformer in the FME 2019 workspace.
If you have any questions or feedback about this, please don't hesitate to reach out!
Best,
Annabelle
H @annabelleatsafe, that's a great news. Thank you!
Will it at some point be possible to use Bulk Mode / Feature Tables in custom Scripts via the PythonCaller?
Will it at some point be possible to use Bulk Mode / Feature Tables in custom Scripts via the PythonCaller?
Hi, @denniswilhelm1. Thank you for raising this question. There is currently not yet a plan to add Bulk Mode support to the PythonCaller. However, we value your feedback, and would love to understand how important this is to users like yourself. I added this as an idea here: https://knowledge.safe.com/idea/90648/support-bulk-mode-in-pythoncaller.html?
Would you mind adding your vote and comment on the importance of this enhancement on that idea? Thanks!
Hi Safers, is this list still up to date, or has anything changed in FME 2020? Perhaps it would make sense to have this information in an article rather than a forum post?
@annabelleatsafe @rylanatsafe @andreaatsafe
Hi Safers, is this list still up to date, or has anything changed in FME 2020? Perhaps it would make sense to have this information in an article rather than a forum post?
Would be nice for an update.
There is at least one Transformer not appearing in the above list for 2018.1+: Junction is also Bulk Mode Enabled, as it is just a reskinned Cloner. That is sort of Implicit but it probably should be explicitly listed in case anyone is trying to find out what behavior a Junction has.
Would be nice for an update.
There is at least one Transformer not appearing in the above list for 2018.1+: Junction is also Bulk Mode Enabled, as it is just a reskinned Cloner. That is sort of Implicit but it probably should be explicitly listed in case anyone is trying to find out what behavior a Junction has.
Good catch about the Junction, definitely agree that it needs to be explicitely stated.
Lots has changed and continues to change. I understand the desire to have a complete list of what transformers/readers/writers understand bulk mode natively, but unfortunately it turns out that it is a bit harder than you'd think for us to keep and maintain such a list. (Sometimes we do infrastructure changes that suddenly cause many many transformers to preserve bulk mode -- and we haven't felt it worthwhile to specifically enumerate them).
There is also a difference between preserving bulk mode and being optimized for it. The Tester in FME 2020 preserves bulk mode and is quite a bit faster than in 2019, BUT in FME 2021 the Tester is optimized for Bulk Mode and, well, just rips.
I can advise that the first 20 transformers listed here https://www.safe.com/transformers/ are all on our hit list, and we're currently closing in on 75% done there. (FeatureMerger is about to get the treatment -- we hoped everyone would forget about it and just use the FeatureJoiner, but it seems that isn't going to happen. When we did a similar treatment to the PointOnAreaOverlayer (that work is ongoing now), we saw memory use for large dataset scenarios drop to 1/7th the original need. Taking the "Ferocious" out of "Ferocious Memory Eater" (some customer's write-in suggestion for what FME stands for...).
One more thing -- it is the case that if you choose some options in the transformers that have the "bulk mode treatment", you will lose bulk mode. Lists are incompatible -- so asking to produce a list is a ticket out of bulk mode. (We're looking to provide relational options in these cases -- so you instead of a list get the input rows duplicated for each value that would have otherwise gone in the list...).
All that said, we'll see if we can put something a bit more pleasing together for FME 2021.
AttributeKeeper now supports Bulk Mode in FME 2020. Previously AttributeKeeper used to split feature tables into features. AttributeKeeper now also supports the ability to create a feature table from the input.
AttributeKeeper now supports Bulk Mode in FME 2020. Previously AttributeKeeper used to split feature tables into features. AttributeKeeper now also supports the ability to create a feature table from the input.
Well that is great tip @Mark Stoakes because AttributeKeeper in 2018.1 has been my goto Transformer when I needed to break out of Bulk Mode and force the workflow back into Feature-by-Feature transformations! Means going to have to carefully go through these in FME 2020 workflows.
Why use AttributeKeeper like this might you ask? Well, at least in 2018.1, there are unfortunately some very weird and wonderful things that happen in Bulk Mode, particularly when you have a Feature Table where the Features within the Table had a different Schema from each other Eg. "Empty" values for some of the Features and "Non-Empty" values for others.
Some of the Attribute Merging/Conflict detection goes haywire in say, FeatureJoiner if it sees that the first Features in a Feature Table on the Left Hand Side have a non-empty Attribute, and decides that for conflict resolution, no Attribute of the same name on the Right Hand Side will then override or vice-versa. This produces the bizarre result where you can have an Empty Attribute entering on the Left Hand Side and a Non-Empty Attribute of the same name entering on the Right Hand Side, but the Join Output is still an Empty Attribute, despite there being a non-empty Attribute from either the Left or Right hand sides.
Breaking the workflow into Feature-by-Feature mode with AttributeKeeper forces the Transformers to do Feature-by-Feature comparison/attribute conflict-detection rather than Bulk Mode which tries to "guess" a rule to apply over the entire Feature Table Feature dataset which seemed to be based on whether one, rather than all, Features in the Schema have a non-empty value or not. For the opposite reason, FeatureJoiner is my goto Transformer when I need to put the workflow back into Bulk mode.
Similarly some other Bulk Mode workflows sometimes hold onto not-easily-exposable Attributes/List data that is no longer present in the WorkBench Schema/should have been removed and AttributeKeeper was a good way to "clean" the Schema by switching the workflow temporarily back into feature-by-feature Schema setting to put all the Features back into the desired Schema.
Lots has changed and continues to change. I understand the desire to have a complete list of what transformers/readers/writers understand bulk mode natively, but unfortunately it turns out that it is a bit harder than you'd think for us to keep and maintain such a list. (Sometimes we do infrastructure changes that suddenly cause many many transformers to preserve bulk mode -- and we haven't felt it worthwhile to specifically enumerate them).
There is also a difference between preserving bulk mode and being optimized for it. The Tester in FME 2020 preserves bulk mode and is quite a bit faster than in 2019, BUT in FME 2021 the Tester is optimized for Bulk Mode and, well, just rips.
I can advise that the first 20 transformers listed here https://www.safe.com/transformers/ are all on our hit list, and we're currently closing in on 75% done there. (FeatureMerger is about to get the treatment -- we hoped everyone would forget about it and just use the FeatureJoiner, but it seems that isn't going to happen. When we did a similar treatment to the PointOnAreaOverlayer (that work is ongoing now), we saw memory use for large dataset scenarios drop to 1/7th the original need. Taking the "Ferocious" out of "Ferocious Memory Eater" (some customer's write-in suggestion for what FME stands for...).
One more thing -- it is the case that if you choose some options in the transformers that have the "bulk mode treatment", you will lose bulk mode. Lists are incompatible -- so asking to produce a list is a ticket out of bulk mode. (We're looking to provide relational options in these cases -- so you instead of a list get the input rows duplicated for each value that would have otherwise gone in the list...).
All that said, we'll see if we can put something a bit more pleasing together for FME 2021.
Hi Dale, thanks a lot for your detailed answer, that actually makes a lot of sense.
Well that is great tip @Mark Stoakes because AttributeKeeper in 2018.1 has been my goto Transformer when I needed to break out of Bulk Mode and force the workflow back into Feature-by-Feature transformations! Means going to have to carefully go through these in FME 2020 workflows.
Why use AttributeKeeper like this might you ask? Well, at least in 2018.1, there are unfortunately some very weird and wonderful things that happen in Bulk Mode, particularly when you have a Feature Table where the Features within the Table had a different Schema from each other Eg. "Empty" values for some of the Features and "Non-Empty" values for others.
Some of the Attribute Merging/Conflict detection goes haywire in say, FeatureJoiner if it sees that the first Features in a Feature Table on the Left Hand Side have a non-empty Attribute, and decides that for conflict resolution, no Attribute of the same name on the Right Hand Side will then override or vice-versa. This produces the bizarre result where you can have an Empty Attribute entering on the Left Hand Side and a Non-Empty Attribute of the same name entering on the Right Hand Side, but the Join Output is still an Empty Attribute, despite there being a non-empty Attribute from either the Left or Right hand sides.
Breaking the workflow into Feature-by-Feature mode with AttributeKeeper forces the Transformers to do Feature-by-Feature comparison/attribute conflict-detection rather than Bulk Mode which tries to "guess" a rule to apply over the entire Feature Table Feature dataset which seemed to be based on whether one, rather than all, Features in the Schema have a non-empty value or not. For the opposite reason, FeatureJoiner is my goto Transformer when I need to put the workflow back into Bulk mode.
Similarly some other Bulk Mode workflows sometimes hold onto not-easily-exposable Attributes/List data that is no longer present in the WorkBench Schema/should have been removed and AttributeKeeper was a good way to "clean" the Schema by switching the workflow temporarily back into feature-by-feature Schema setting to put all the Features back into the desired Schema.
I'm kind of hoping that the various issues with bulk mode (and sadly there are a few*) may also have been addressed in 2020 so an AttributeKeeper may not be needed in quite the same way (maybe I'm too optimistic).
Otherwise there's always the pythoncaller!
*I've not encountered the FeatureJoiner issue, but i know at least one of the issues i've had to break bulk mode for has been fixed in 2020
Well that is great tip @Mark Stoakes because AttributeKeeper in 2018.1 has been my goto Transformer when I needed to break out of Bulk Mode and force the workflow back into Feature-by-Feature transformations! Means going to have to carefully go through these in FME 2020 workflows.
Why use AttributeKeeper like this might you ask? Well, at least in 2018.1, there are unfortunately some very weird and wonderful things that happen in Bulk Mode, particularly when you have a Feature Table where the Features within the Table had a different Schema from each other Eg. "Empty" values for some of the Features and "Non-Empty" values for others.
Some of the Attribute Merging/Conflict detection goes haywire in say, FeatureJoiner if it sees that the first Features in a Feature Table on the Left Hand Side have a non-empty Attribute, and decides that for conflict resolution, no Attribute of the same name on the Right Hand Side will then override or vice-versa. This produces the bizarre result where you can have an Empty Attribute entering on the Left Hand Side and a Non-Empty Attribute of the same name entering on the Right Hand Side, but the Join Output is still an Empty Attribute, despite there being a non-empty Attribute from either the Left or Right hand sides.
Breaking the workflow into Feature-by-Feature mode with AttributeKeeper forces the Transformers to do Feature-by-Feature comparison/attribute conflict-detection rather than Bulk Mode which tries to "guess" a rule to apply over the entire Feature Table Feature dataset which seemed to be based on whether one, rather than all, Features in the Schema have a non-empty value or not. For the opposite reason, FeatureJoiner is my goto Transformer when I need to put the workflow back into Bulk mode.
Similarly some other Bulk Mode workflows sometimes hold onto not-easily-exposable Attributes/List data that is no longer present in the WorkBench Schema/should have been removed and AttributeKeeper was a good way to "clean" the Schema by switching the workflow temporarily back into feature-by-feature Schema setting to put all the Features back into the desired Schema.
@bwn You'll have to check your workspaces. AttributeKeeperwas not versioned so will have to check those workspaces.
We hope we have addressed most of the issues related to the join conflicts in FeatureJoiner, the causes for this are in fact why AttributeKeeper was last transformer to get the Bulk Mode treatment.
We definitely want to know about any other bulk mode issues so please don't hesitate to enumerate them (and I'll check with the team to be sure we're not sitting on any).
Note that top people are examining how we could get PythonCaller working in Bulk Mode at least in some fashion, so that trick may fly out the window. It feels like (and it would be a bit of a fail if we had to do this) we should just make a DeBulker or something like that that exists only to split bulk mode, then you won't get surprised when <insert random obscure transformer here> suddenly stops splitting and operates in bulk. And we'd promise to never "upgrade" the DeBulker.
Your alternate names coveted.
Actually after thinking about it, how about using Decelerator with a per feature delay of 0 -- that works now to split a feature table and we could guarantee it always always will forever more. Aside: it has a wee bit of overhead that we could tighten up, but if one is splitting the feature table in the first place, perhaps loosing a few seconds in the overall run isn't a deal breaker anyway.
Actually after thinking about it, how about using Decelerator with a per feature delay of 0 -- that works now to split a feature table and we could guarantee it always always will forever more. Aside: it has a wee bit of overhead that we could tighten up, but if one is splitting the feature table in the first place, perhaps loosing a few seconds in the overall run isn't a deal breaker anyway.
It's a great idea, as long as it's easy to find (alias?) for those who aren't intimately familiar with bulk mode.
SequentialModeForcer?
Update -- FME 2021 is getting the "optimized" per feature delay of 0 breaking apart of Bulk Mode "Feature Tables" -- will be in betas by next week for sure. Aliases added for SequentialModeForcer and BulkModeSplitter.
The list is currently...
FME VersionTransformersFormats2017.0
2019.0
There are also FME functions supporting this technology...
@Concatenate, @CopyAttributes, @FeatureType, @Log, @LogCount, @RemoveAttributes, @RenameAttributes, @SupplyAttributes, and @Value
Is it possible to get an updated list as of FME 2022.1 ?