Skip to main content

I'm reading 328,000+ features from 15 File Geodatabases (separate Readers, identical schema) via a Junction to a Tester which tests whether a particular attribute is null or not. Each time I run the Workspace, I get different results, varying by a few records e.g. 289,431 through the Failed port one time then 289,424 the next time. This then affects a Sorter and 2 DuplicateFilters I have further down the line because I don't want null values going through these.

The Inspector shows the attribute as having a value of <null> in grey so I assume it's a "proper" null rather than a text string of <null> i.e. I assume the Inspector knows the difference. Nonetheless, I've tested for attribute LIKE <null> as per the link below but there aren't any like this.

 

https://knowledge.safe.com/questions/32235/tester-appears-to-be-missing-some-nulls.html

I've tried swapping the Tester round so it's testing for Attribute Is Null with Negate ticked but it doesn't solve the problem.

In any case, my source data is not changing, so why is Tester producing different results each time?

Attribute > 0 appears to be consistent (the non-null values are all integers) so I will use that. I'm still concerned by the inconsistency of null - don't tell me - fixed in FME 2017! ;-)

 

 


I don't know why the Tester would be behaving like this, perhaps you could try using a TestFilter instead?

You could also try removing the Junction and routing the readers directly to the Tester. The Junction should not be affecting anything but it's worth ruling out all possibilities.

Also, to try to determine what the problem is, you could try placing a DuplicateFilter after the tester, to filter out distinct values, then a Logger to see what values actually exist after the Tester. As mentioned in the other article, the Inspector can sometimes show things not exactly as they should appear. The Logger might be a better choice. Of course, that all depends on if the DuplicateRemover is able to determine differences in the values. If DuplicateRemover doesn't work try using a StatisticsCalculator, grouping by your "null" attribute value.


Hi @tim_wood, I'm afraid that it could be a potential bug in a particular version of FME. Which version (build number) of FME are you using? Does the same symptom occur in the latest FME 2017.0 build 17271?

 


Hi @tim_wood

have you tried Attribute Is Null operator in Tester/TestFilter?


Thanks for the suggestions. I'm using FME 2016.1 (Build 16492). Has FME 2017 officially been released yet?

@LenaAtSafe - Attribute Is Null in the Tester was what I did first and what seemed to be giving variable results.

@nic_ran - I tried connecting the Readers directly to the Tester (Attribute Is Null) then each output port of the Tester to a separate Inspector. I ran this twice. Although the same number of records came out of each Tester output port (38717 allegedly not null), 1 record with a null value still ended up in the wrong place each time. But it wasn't the same record!

However, I purged the temporary files and ran it again and this time it matched the results for Attribute > 0 (38716 not null). So I had a look in the FME_TEMP folder and cleared out some other old stuff that was there. Then I re-ran the Workspace and got the correct results. I re-ran it again and this time I got 38720 records supposedly not null but the first 4 in the Inspector were null :-(

I noticed that whereas on the first run, no records went along the not null path for ages (until after over 100,000 records), the second time I got several records going down that path more or less straight away (4 I think but I can't be sure).

Could it be something to do with leaving the Inspector open when re-running the Workspace?

But I did this on the third run and this time, no records went down the not null path and I got the correct results.

For info, the bulk of the records are in the first FGDB to be read (289,000).

TestFilter report to follow when I get a chance.


How about trying an "attribute has value" test instead?


Or the NullAttributeMapper and test for the new value set for multiple conditions ( empty, null, missing, etc)?

Yes FME 2017 is ofiicially released....


Inspector open or not doesn't seem to make a difference.

I think the multiple source FGDBs can also be discounted because I tried it with just the first one and still had issues.

I purged temporary files then tried Attribute has a value. That seems to be more reliable based on the repeat runs so far but I think Attribute is Null should be just as reliable.

I will download 2017 and try that.


fme2016.1.1.0

I had Samish issue where suddenly tester used in 2015 didn't work no more.

I went for testing

NULL,MISSING AND EMPYT. ...just to be sure..lol

First 2016 version seemed to be not able to make up its mind though and even this .

I had one workspace where this did not do anything until I removed the empty test (a.f.a.i. remember)

Lately It seems to be able to run on MISSING AND NULL.


Thanks for the suggestions. I'm using FME 2016.1 (Build 16492). Has FME 2017 officially been released yet?

@LenaAtSafe - Attribute Is Null in the Tester was what I did first and what seemed to be giving variable results.

@nic_ran - I tried connecting the Readers directly to the Tester (Attribute Is Null) then each output port of the Tester to a separate Inspector. I ran this twice. Although the same number of records came out of each Tester output port (38717 allegedly not null), 1 record with a null value still ended up in the wrong place each time. But it wasn't the same record!

However, I purged the temporary files and ran it again and this time it matched the results for Attribute > 0 (38716 not null). So I had a look in the FME_TEMP folder and cleared out some other old stuff that was there. Then I re-ran the Workspace and got the correct results. I re-ran it again and this time I got 38720 records supposedly not null but the first 4 in the Inspector were null :-(

I noticed that whereas on the first run, no records went along the not null path for ages (until after over 100,000 records), the second time I got several records going down that path more or less straight away (4 I think but I can't be sure).

Could it be something to do with leaving the Inspector open when re-running the Workspace?

But I did this on the third run and this time, no records went down the not null path and I got the correct results.

For info, the bulk of the records are in the first FGDB to be read (289,000).

TestFilter report to follow when I get a chance.

FME 2017.0 has already been released. Please download the install from http://www.safe.com/support/support-resources/fme-downloads/

 

May I ask you to run your workspace with FME 2017.0 to confirm whether there is still a problem with Tester testing differently/incorrectly? If FME 2017.0 has the same incorrect (hard to get correct) behaviour as FME 2016, could you please submit a support case? Please add link to this discussion to your problem description (no need to describe everything once again when submitting the case). Ideally, we would need you data - or a data sample that reproduces the wrong behaviour. We definitely want to investigate this problem. Sorry for the inconvenience caused by it - I am glad you caught the wrong output.

 

 


Reply