Skip to main content
Question

FeatureMerger turns nulls/missing into asian characters

  • February 12, 2024
  • 6 replies
  • 290 views
takanaka
Contributor
Forum|alt.badge.img+5

Hi,

Unfortunately i cannot provide the test data due to confidential data. I also cannot encode/dummify the data because then the issue no longer occurs. It is likely related to the large amount of data being processed. What I can include is the image you see here.

Β 

My question is: how can the following happen? Is this a bug?

Β 

I merge two flows, both have a lot of attributes. Requestor is 1.9m features. Supplier is 6k features. Supplier has only in ~1% of casesΒ a value for attributes β€˜X’ and β€˜Y’ - the other 99% is <missing>.Β But supplier also has a lot of other data and attributes.

After merging, for attributes x and y a lot more values are found. They are chinese or japanese characters. They are supposed to be numbers or english words.. Also I think they are getting also values from other attributes entirely, but then the actual value instead of random chinese or japanese characters.Β How is this happening?Β 

Β 

I am sure that: the attribute I test is the same (IΒ duplicated the statisticscalculator). I am sure that: the attribute doesnt exist in the requestor flow.Β 

My FME version is 2023.0.0.3; build 23319.

Β 

6 replies

itsmatt
Celebrity
Forum|alt.badge.img+48
  • itsmattCelebrity
  • Celebrity
  • February 12, 2024

Very interesting indeed - when you say β€œboth have lots of attributes” how many are we talking here?Β 

Does the same issue happen with the FeatureJoiner?

Also, I’m not sure if this helps, but can you use a Null attribute mapper to map the missingΒ values in those to null? You can then change the Ignore Nulls setting from No to Yes in the FeatureMerger - This should keep the same behaviour as if they were missing.Β 

    takanaka
    Contributor
    Forum|alt.badge.img+5
    • takanakaContributor
    • Author
    • Contributor
    • February 13, 2024

    Very interesting indeed - when you say β€œboth have lots of attributes” how many are we talking here?Β 

    Does the same issue happen with the FeatureJoiner?

    Also, I’m not sure if this helps, but can you use a Null attribute mapper to map the missingΒ values in those to null? You can then change the Ignore Nulls setting from No to Yes in the FeatureMerger - This should keep the same behaviour as if they were missing.Β 

    Β 

    Hi Matt, yesterday I also discovered that the fix is to use β€˜Ignore Nulls = yes’ in the β€˜FeatureMerger’.Β 

    Based on your comment i tried the NullAttributeMapper, this also fixes the issue while Ignore Nulls = no.

    Also I tested featurejoiner (inner mode): the issue does not occur in this way. So definitely it is something in the FeatureMerger I imagine.Β 

    Β 

    Requestor (left) has only 9Β attributes of which 7 exposed. Supplier (right) has 17 attributes of which 13 exposed. Maybe not a whole lot, but at least half of attributes are 100% filled (Some are strings up to 100chars for example).

      andrewkb
      Contributor
      Forum|alt.badge.img
      • andrewkbContributor
      • Contributor
      • June 5, 2024

      Hi, I had this issue of Chinese characters appearing in my fields.Β  I narrowed it down to a Tester in my case. Now a Tester shouldn’t change data, yet some features were coming out corrupted.Β 

      I think it happenedΒ due to the fact that I had rearranged field order a lot in the workbench.Β  I got data values from one field appearing in another field without any attribute mapping, as well as Chinese characters and empty strings.Β  It’s a bug.Β 

      I fixed it by tracing through feature caches to find where it was happening, then replaced the Tester with TestFilter.Β  I also now try not to mess with field order too much, just reorder at the last step.Β 

      I was using 2022.0.Β 

      itsmatt
      Celebrity
      Forum|alt.badge.img+48
      • itsmattCelebrity
      • Celebrity
      • June 5, 2024

      Hi, I had this issue of Chinese characters appearing in my fields.Β  I narrowed it down to a Tester in my case. Now a Tester shouldn’t change data, yet some features were coming out corrupted.Β 

      I think it happenedΒ due to the fact that I had rearranged field order a lot in the workbench.Β  I got data values from one field appearing in another field without any attribute mapping, as well as Chinese characters and empty strings.Β  It’s a bug.Β 

      I fixed it by tracing through feature caches to find where it was happening, then replaced the Tester with TestFilter.Β  I also now try not to mess with field order too much, just reorder at the last step.Β 

      I was using 2022.0.Β 

      Crazy - good finding. If you can share the reproduction I can test if it’s still an issues in FME 2024 and submit a bug report for them

      DanAtSafe
      Safer
      Forum|alt.badge.img+23
      • DanAtSafeSafer
      • Safer
      • June 5, 2024

      Hi @andrewkbΒ This problem was fixed in FME 2022 and 2023.Β  Please submit a ticketΒ Β ifΒ you still see this issue with FME 2023.2 or 2024.0+

        andrewkb
        Contributor
        Forum|alt.badge.img
        • andrewkbContributor
        • Contributor
        • June 11, 2024

        So I thought I fixed this but sadly no.Β  I kept getting nulls and Chinese chars immediately after my file geodatabase reader. Β I rebuilt the entire workbench and got the same issue happening.

        Found from another post this is a bug with 2022.0 Bulk Mode.Β  Fix is to use the Decelerator after your reader, then everything works. Β 

        Lucky for me I found this in time, I was close to needing someone to talk me down from a ledge… ha ha

        https://community.safe.com/general-10/errors-every-10-000-records-in-fme-desktop-2022-0-how-to-fix-this-20690

        ttps://docs.safe.com/fme/2021.0/html/FME_Desktop_Documentation/FME_Workbench/Workbench/Improving-Performance-Bulk-Mode.htm

          Community Stats

          32,811
          Posts
          124,414
          Replies
          41,371
          Members
          Terms & ConditionsAccessibility statement
          A product of
          Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.