Question

FeatureMerger turns nulls/missing into asian characters

  • 12 February 2024
  • 2 replies
  • 32 views

Badge +2

Hi,

Unfortunately i cannot provide the test data due to confidential data. I also cannot encode/dummify the data because then the issue no longer occurs. It is likely related to the large amount of data being processed. What I can include is the image you see here.

 

My question is: how can the following happen? Is this a bug?

 

I merge two flows, both have a lot of attributes. Requestor is 1.9m features. Supplier is 6k features. Supplier has only in ~1% of cases a value for attributes ‘X’ and ‘Y’ - the other 99% is <missing>. But supplier also has a lot of other data and attributes.

After merging, for attributes x and y a lot more values are found. They are chinese or japanese characters. They are supposed to be numbers or english words.. Also I think they are getting also values from other attributes entirely, but then the actual value instead of random chinese or japanese characters. How is this happening? 

 

I am sure that: the attribute I test is the same (I duplicated the statisticscalculator). I am sure that: the attribute doesnt exist in the requestor flow. 

My FME version is 2023.0.0.3; build 23319.

 


2 replies

Userlevel 4
Badge +25

Very interesting indeed - when you say “both have lots of attributes” how many are we talking here? 

Does the same issue happen with the FeatureJoiner?

Also, I’m not sure if this helps, but can you use a Null attribute mapper to map the missing values in those to null? You can then change the Ignore Nulls setting from No to Yes in the FeatureMerger - This should keep the same behaviour as if they were missing. 

Badge +2

Very interesting indeed - when you say “both have lots of attributes” how many are we talking here? 

Does the same issue happen with the FeatureJoiner?

Also, I’m not sure if this helps, but can you use a Null attribute mapper to map the missing values in those to null? You can then change the Ignore Nulls setting from No to Yes in the FeatureMerger - This should keep the same behaviour as if they were missing. 

 

Hi Matt, yesterday I also discovered that the fix is to use ‘Ignore Nulls = yes’ in the ‘FeatureMerger’. 

Based on your comment i tried the NullAttributeMapper, this also fixes the issue while Ignore Nulls = no.

Also I tested featurejoiner (inner mode): the issue does not occur in this way. So definitely it is something in the FeatureMerger I imagine. 

 

Requestor (left) has only 9 attributes of which 7 exposed. Supplier (right) has 17 attributes of which 13 exposed. Maybe not a whole lot, but at least half of attributes are 100% filled (Some are strings up to 100chars for example).

Reply