Skip to main content
  • EN

Hi.

I'm trying to build a simple workspace that extracts logged features from a log file.

Since a logged feature in the log file is wrapped by a starter line and an end line, I'm using a variable to keep track of whether I'm inside a logged section or not.

I'm reading the log file as a text file, and forking the read line feature 3-ways with connection order: (1) check for start line, (2) output if started, and (3) check for end line. In (1) and (3) I'm changing the variable to 1 or 0 resp., while in (2) I'm checking whether the retrieved variable value is 1 before doing an output.

Unfortunately it seems that my check in (2) is always true ??

It looks like the FeatureReader is sending all line features into each connection, one by one, so first (1) receives all features, then (2) receives all features, etc.

I inserted a FeatureHolder immidiatedly after the FeatureReader, in case it was a problem with the reader, but it changed nothing.

As far as I understand, a single feature should be processed into all three connections before the next feature is processed. Is my understanding of this incorrect ?

Using 2020.0

Cheers.

 

 

 

 

I'm not sure whether this is the issue here, but there is a change so that feature order is not preserved across output pots by default

I think in your case, I'd use a testfilter instead of separate testers if i understand what you are trying to do

 

I'm not sure whether this is the issue here, but there is a change so that feature order is not preserved across output pots by default

I think in your case, I'd use a testfilter instead of separate testers if i understand what you are trying to do

 

Thanks, but where is that setting located ?

Thanks, but where is that setting located ?

This is how i'd arrange the workflow

Because feature order is preserved across output ports in a single transformer, but I don't think that feature order would be preserved when you start splitting into separate testers

I just tried to do a super-simple workspace, with a single creator making 10 features connected to 3 loggers. The 3 loggers are called for each feature in sequence, so something works as expected.

Hi @lifalin2016, since a few years ago (maybe 2018) FME supports so called "bulk mode" or "feature table", some readers and transformers process and pass every feature at once to subsequent object. The Text File reader supports the bulk mode.

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_QuickTranslator/Workbench/Improving-Performance-Bulk-Mode.htm

The bulk mode has improved performance drastically in some scenarios, but as a side effect, the connection order may not work as expected, as you observed.

A possible workaround is, insert a PythonCaller immediately after the reader. The PythonCaller decomposes the "feature table" into individual features, then send them to subsequent connection one by one.

As @nic_ran suggested in this thread, I also hope an option will be added to bulk writers/transformers to disable bulk mode.

https://knowledge.safe.com/questions/102034/fme-20192-a-new-parameter-for-feature-order.html

 

I'm not sure whether this is the issue here, but there is a change so that feature order is not preserved across output pots by default

I think in your case, I'd use a testfilter instead of separate testers if i understand what you are trying to do

 

Thanks, but it still doesn't explain why the connection order isn't respected as it should be. This is a work-around.

Thanks, but it still doesn't explain why the connection order isn't respected as it should be. This is a work-around.

See @takashi's answer above. Bulk mode is present for some readers and transformers and not others so it's not always safe to rely on the feature order unless you work around it

Attached is a workspace that demonstrates how feature order at the inspector is different depending on input source and needs to be accounted for if feature order is critical

feature_order_demo.fmwt

Hi @lifalin2016, since a few years ago (maybe 2018) FME supports so called "bulk mode" or "feature table", some readers and transformers process and pass every feature at once to subsequent object. The Text File reader supports the bulk mode.

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_QuickTranslator/Workbench/Improving-Performance-Bulk-Mode.htm

The bulk mode has improved performance drastically in some scenarios, but as a side effect, the connection order may not work as expected, as you observed.

A possible workaround is, insert a PythonCaller immediately after the reader. The PythonCaller decomposes the "feature table" into individual features, then send them to subsequent connection one by one.

As @nic_ran suggested in this thread, I also hope an option will be added to bulk writers/transformers to disable bulk mode.

https://knowledge.safe.com/questions/102034/fme-20192-a-new-parameter-for-feature-order.html

 

Thanks. But wouldn't the added FeatureHolder after the FeatureReader remedy that ? It doesn't.

Thanks. But wouldn't the added FeatureHolder after the FeatureReader remedy that ? It doesn't.

No, a featureholder doesn't break the bulk mode

Hi @lifalin2016, since a few years ago (maybe 2018) FME supports so called "bulk mode" or "feature table", some readers and transformers process and pass every feature at once to subsequent object. The Text File reader supports the bulk mode.

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_QuickTranslator/Workbench/Improving-Performance-Bulk-Mode.htm

The bulk mode has improved performance drastically in some scenarios, but as a side effect, the connection order may not work as expected, as you observed.

A possible workaround is, insert a PythonCaller immediately after the reader. The PythonCaller decomposes the "feature table" into individual features, then send them to subsequent connection one by one.

As @nic_ran suggested in this thread, I also hope an option will be added to bulk writers/transformers to disable bulk mode.

https://knowledge.safe.com/questions/102034/fme-20192-a-new-parameter-for-feature-order.html

 

I suppose that the FeatureHolder too supports bulk mode.

You can think that FME internally handles a chunk of features read by a bulk-reader as a Feature Table object (something like a single feature) until it would be decomposed into individual features with a non-bulk-transformer. That is, the FeatureHolder being immediately after the Text File reader would just output the Feature Table object which stores all the text line features.

Hi @lifalin2016, since a few years ago (maybe 2018) FME supports so called "bulk mode" or "feature table", some readers and transformers process and pass every feature at once to subsequent object. The Text File reader supports the bulk mode.

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_QuickTranslator/Workbench/Improving-Performance-Bulk-Mode.htm

The bulk mode has improved performance drastically in some scenarios, but as a side effect, the connection order may not work as expected, as you observed.

A possible workaround is, insert a PythonCaller immediately after the reader. The PythonCaller decomposes the "feature table" into individual features, then send them to subsequent connection one by one.

As @nic_ran suggested in this thread, I also hope an option will be added to bulk writers/transformers to disable bulk mode.

https://knowledge.safe.com/questions/102034/fme-20192-a-new-parameter-for-feature-order.html

 

So the very good question: is there an "Unbulker" transformer for when you don't want this ?

Hi @lifalin2016, since a few years ago (maybe 2018) FME supports so called "bulk mode" or "feature table", some readers and transformers process and pass every feature at once to subsequent object. The Text File reader supports the bulk mode.

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_QuickTranslator/Workbench/Improving-Performance-Bulk-Mode.htm

The bulk mode has improved performance drastically in some scenarios, but as a side effect, the connection order may not work as expected, as you observed.

A possible workaround is, insert a PythonCaller immediately after the reader. The PythonCaller decomposes the "feature table" into individual features, then send them to subsequent connection one by one.

As @nic_ran suggested in this thread, I also hope an option will be added to bulk writers/transformers to disable bulk mode.

https://knowledge.safe.com/questions/102034/fme-20192-a-new-parameter-for-feature-order.html

 

In the FeatureReader Text File parameters there is a setting named "Read Whole File at Once". This looks very much like a bulk settings, but it's set to "No" !?

Hi @lifalin2016, since a few years ago (maybe 2018) FME supports so called "bulk mode" or "feature table", some readers and transformers process and pass every feature at once to subsequent object. The Text File reader supports the bulk mode.

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_QuickTranslator/Workbench/Improving-Performance-Bulk-Mode.htm

The bulk mode has improved performance drastically in some scenarios, but as a side effect, the connection order may not work as expected, as you observed.

A possible workaround is, insert a PythonCaller immediately after the reader. The PythonCaller decomposes the "feature table" into individual features, then send them to subsequent connection one by one.

As @nic_ran suggested in this thread, I also hope an option will be added to bulk writers/transformers to disable bulk mode.

https://knowledge.safe.com/questions/102034/fme-20192-a-new-parameter-for-feature-order.html

 

Yes, as I mentioned already in the answer, my last resort is the PythonCaller. You don't need to customize the script, just insert a PythonCaller onto a connection. It will definitely split a feature table into individual features then output them one by one.

In the FeatureReader Text File parameters there is a setting named "Read Whole File at Once". This looks very much like a bulk settings, but it's set to "No" !?

That parameter controls if the entire text file is read into a single attribute in a single feature rather than line by line, it's not related to bulk mode

This question-of-the-week post may help to give you some answers: https://knowledge.safe.com/questions/104425/question-of-the-week-variables-adjacent-features-a.html

Basically, I really would try to avoid variables where possible. Because (as @takashi mentions) bulk mode changes the order of features, and because order is hard to assess to start with, it's really not the best solution.

What I would try is adjacent feature attributes.

So each row has an attribute for the current info - call it logline for example. You also create an attribute called logstatus.

If logline = starter
        Then you are at the start of a feature. Set logstatus = true
If logline != starter AND featuree-1].logstatus = true
        Then you are inside a feature record. Set logstatus = true
If logline != starter AND featureÂ-1].logstatus = false
        Then you are outside a feature record. Set logstatus = false
If logline = end line
        Then you are at the end of a feature. Set logstatus = false

Then you don't have to worry about passing variables backwards and forwards, or what order events occur in. It's way easier!

Hope this helps

Mark

Reply


Community Stats

30,159
Posts
114,479
Replies
37,803
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.