Question

FeatureMerger turns nulls/missing into asian characters

  • 12 February 2024
  • 6 replies
  • 116 views
takanaka
Rank
Badge +3

Hi,

Unfortunately i cannot provide the test data due to confidential data. I also cannot encode/dummify the data because then the issue no longer occurs. It is likely related to the large amount of data being processed. What I can include is the image you see here.

 

My question is: how can the following happen? Is this a bug?

 

I merge two flows, both have a lot of attributes. Requestor is 1.9m features. Supplier is 6k features. Supplier has only in ~1% of cases a value for attributes ‘X’ and ‘Y’ - the other 99% is <missing>. But supplier also has a lot of other data and attributes.

After merging, for attributes x and y a lot more values are found. They are chinese or japanese characters. They are supposed to be numbers or english words.. Also I think they are getting also values from other attributes entirely, but then the actual value instead of random chinese or japanese characters. How is this happening? 

 

I am sure that: the attribute I test is the same (I duplicated the statisticscalculator). I am sure that: the attribute doesnt exist in the requestor flow. 

My FME version is 2023.0.0.3; build 23319.

 

6 replies

virtualcitymatt
Rank
Userlevel 5
Badge +32

Very interesting indeed - when you say “both have lots of attributes” how many are we talking here? 

Does the same issue happen with the FeatureJoiner?

Also, I’m not sure if this helps, but can you use a Null attribute mapper to map the missing values in those to null? You can then change the Ignore Nulls setting from No to Yes in the FeatureMerger - This should keep the same behaviour as if they were missing. 

takanaka
Rank
Badge +3

Very interesting indeed - when you say “both have lots of attributes” how many are we talking here? 

Does the same issue happen with the FeatureJoiner?

Also, I’m not sure if this helps, but can you use a Null attribute mapper to map the missing values in those to null? You can then change the Ignore Nulls setting from No to Yes in the FeatureMerger - This should keep the same behaviour as if they were missing. 

 

Hi Matt, yesterday I also discovered that the fix is to use ‘Ignore Nulls = yes’ in the ‘FeatureMerger’. 

Based on your comment i tried the NullAttributeMapper, this also fixes the issue while Ignore Nulls = no.

Also I tested featurejoiner (inner mode): the issue does not occur in this way. So definitely it is something in the FeatureMerger I imagine. 

 

Requestor (left) has only 9 attributes of which 7 exposed. Supplier (right) has 17 attributes of which 13 exposed. Maybe not a whole lot, but at least half of attributes are 100% filled (Some are strings up to 100chars for example).

andrewkb
Rank
Badge

Hi, I had this issue of Chinese characters appearing in my fields.  I narrowed it down to a Tester in my case. Now a Tester shouldn’t change data, yet some features were coming out corrupted. 

I think it happened due to the fact that I had rearranged field order a lot in the workbench.  I got data values from one field appearing in another field without any attribute mapping, as well as Chinese characters and empty strings.  It’s a bug. 

I fixed it by tracing through feature caches to find where it was happening, then replaced the Tester with TestFilter.  I also now try not to mess with field order too much, just reorder at the last step. 

I was using 2022.0. 

virtualcitymatt
Rank
Userlevel 5
Badge +32

Hi, I had this issue of Chinese characters appearing in my fields.  I narrowed it down to a Tester in my case. Now a Tester shouldn’t change data, yet some features were coming out corrupted. 

I think it happened due to the fact that I had rearranged field order a lot in the workbench.  I got data values from one field appearing in another field without any attribute mapping, as well as Chinese characters and empty strings.  It’s a bug. 

I fixed it by tracing through feature caches to find where it was happening, then replaced the Tester with TestFilter.  I also now try not to mess with field order too much, just reorder at the last step. 

I was using 2022.0. 

Crazy - good finding. If you can share the reproduction I can test if it’s still an issues in FME 2024 and submit a bug report for them

DanAtSafe
Rank
Userlevel 3
Badge +13

Hi @andrewkb This problem was fixed in FME 2022 and 2023.  Please submit a ticket  if you still see this issue with FME 2023.2 or 2024.0+

andrewkb
Rank
Badge

So I thought I fixed this but sadly no.  I kept getting nulls and Chinese chars immediately after my file geodatabase reader.  I rebuilt the entire workbench and got the same issue happening.

Found from another post this is a bug with 2022.0 Bulk Mode.  Fix is to use the Decelerator after your reader, then everything works.  

Lucky for me I found this in time, I was close to needing someone to talk me down from a ledge… ha ha

https://community.safe.com/general-10/errors-every-10-000-records-in-fme-desktop-2022-0-how-to-fix-this-20690

ttps://docs.safe.com/fme/2021.0/html/FME_Desktop_Documentation/FME_Workbench/Workbench/Improving-Performance-Bulk-Mode.htm

Reply


Community Stats

29,355
Posts
111,609
Replies
37,018
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.