Skip to main content
Question

Dynamic FeatureWriter and feature counts

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Contributor

I have a scenario where I need to dynamically read in some data (FeatureReader), do a bit of filtering and write it out dynamically with the FeatureWriter and then continue processing.

 

 

Since there's no reader to point to in the Dynamic Schema Definition Schema Sources, I'm using "Schema From Schema Feature". This creates the correct output file with the correct number of features.

 

However, the Summary port feature is including the schema feature in the _total_features_written attribute, so is one more than the number of features recorded in the log, and also outputs it through the Feature Output Port.

 

Is this the desired behaviour?

 

 

6 replies

david_r
Evangelist
  • david_rEvangelist
  • June 1, 2018

What happens if you skip the DataProcessing transformer, i.e. just read-write with nothing in between? Do you still get the same feature counts?

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Author
  • Contributor
  • June 1, 2018
david_r wrote:

What happens if you skip the DataProcessing transformer, i.e. just read-write with nothing in between? Do you still get the same feature counts?

Yes.

 

 

It's the presence of the schema feature. If I use the FeatureWriter dynamically, with the schema source set to a resource and don't attach the schema feature then the _total_features_written count matches the value from the log file.

 

 

Now I can remove the schema feature from the output port by testing for fme_schema_handling = schema_only, but the summary attribute is more concerning.

 

 

If all my FeatureWriters had a schema feature I could just subtract one from the _total_features_written, but I have a mix of static and dynamic writers and keeping tract of which ones should be adjusted will be make future development interesting.

 

 

david_r
Evangelist
  • david_rEvangelist
  • June 1, 2018
jdh wrote:
Yes.

 

 

It's the presence of the schema feature. If I use the FeatureWriter dynamically, with the schema source set to a resource and don't attach the schema feature then the _total_features_written count matches the value from the log file.

 

 

Now I can remove the schema feature from the output port by testing for fme_schema_handling = schema_only, but the summary attribute is more concerning.

 

 

If all my FeatureWriters had a schema feature I could just subtract one from the _total_features_written, but I have a mix of static and dynamic writers and keeping tract of which ones should be adjusted will be make future development interesting.

 

 

Actually, I now remember having to write some Python snippet to "fix" that particular value, probably in FME 2017, before doing some reporting based on it.

 

I would class it as a small bug, consider sending it to Safe.
david_r
Evangelist
  • david_rEvangelist
  • June 20, 2018

In case anybody else stumbles upon this issue, here's some code for a PythonCaller that can be used just after the FeatureWriter to adjust the count values to not include the schema feature:

def FeatureProcessor(feature):
    count_list = feature.getAttribute('_feature_type{}.count')
    adjusted_counts = [int(x)-1 for x in count_list]
    feature.setAttribute('_feature_type{}.count', [str(x) for x in adjusted_counts])

In short, all the values in the list attribute _feature_type{}.count will be subtracted by 1.

taojunabc
Participant
Forum|alt.badge.img+6
  • taojunabcParticipant
  • Participant
  • June 21, 2018
david_r wrote:

In case anybody else stumbles upon this issue, here's some code for a PythonCaller that can be used just after the FeatureWriter to adjust the count values to not include the schema feature:

def FeatureProcessor(feature):
    count_list = feature.getAttribute('_feature_type{}.count')
    adjusted_counts = [int(x)-1 for x in count_list]
    feature.setAttribute('_feature_type{}.count', [str(x) for x in adjusted_counts])

In short, all the values in the list attribute _feature_type{}.count will be subtracted by 1.

If I write 10 features use Dynamic Schema mode, then I get _feature_type{0}.count=10, _feature_type{1}.count=1, both of these results are correct, so I don't think will be It is correct to subtract 1 from these two values.

 

 

What confuses the user is the _total_features_written attribute, which has a value of 11 in this case, but the value itself is correct because 10 features and 1 schema feature are actually entered. When schema source is applied, _total_features_written has a value of 10 because no schema feature is actually entered, so I think FME behaves correctly.

 

 

If you want to get the correct number from summary ports, you can use ListSearcher to search for the exported Shapefile'name in _feature_type{}.name, and then use ListIndexer to get the corresponding _feature_type{}.count.

 

 

taojunabc
Participant
Forum|alt.badge.img+6
  • taojunabcParticipant
  • Participant
  • June 21, 2018

You can use the following method, first add an flag attribute to the input feature, and then add it to the Additional Summary Attributes of the FeatureWriter. You can then use the ListElementFilter to filter out the statistics of the schema so that you can get the count of the feature outside of the schema feature.

Reply


Helpful Members This Week

Community Stats

31,092
Posts
117,873
Replies
38,849
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.