Question

Different results for Attribute Is Null from Tester each time Workspaces is run

tim_wood
Rank
Badge +7

I'm reading 328,000+ features from 15 File Geodatabases (separate Readers, identical schema) via a Junction to a Tester which tests whether a particular attribute is null or not. Each time I run the Workspace, I get different results, varying by a few records e.g. 289,431 through the Failed port one time then 289,424 the next time. This then affects a Sorter and 2 DuplicateFilters I have further down the line because I don't want null values going through these.

The Inspector shows the attribute as having a value of <null> in grey so I assume it's a "proper" null rather than a text string of <null> i.e. I assume the Inspector knows the difference. Nonetheless, I've tested for attribute LIKE <null> as per the link below but there aren't any like this.

 

https://knowledge.safe.com/questions/32235/tester-appears-to-be-missing-some-nulls.html

I've tried swapping the Tester round so it's testing for Attribute Is Null with Negate ticked but it doesn't solve the problem.

In any case, my source data is not changing, so why is Tester producing different results each time?

10 replies

tim_wood
Rank
Badge +7
Attribute > 0 appears to be consistent (the non-null values are all integers) so I will use that. I'm still concerned by the inconsistency of null - don't tell me - fixed in FME 2017! ;-)

 

 

nic_ran
Rank
Badge +11

I don't know why the Tester would be behaving like this, perhaps you could try using a TestFilter instead?

You could also try removing the Junction and routing the readers directly to the Tester. The Junction should not be affecting anything but it's worth ruling out all possibilities.

Also, to try to determine what the problem is, you could try placing a DuplicateFilter after the tester, to filter out distinct values, then a Logger to see what values actually exist after the Tester. As mentioned in the other article, the Inspector can sometimes show things not exactly as they should appear. The Logger might be a better choice. Of course, that all depends on if the DuplicateRemover is able to determine differences in the values. If DuplicateRemover doesn't work try using a StatisticsCalculator, grouping by your "null" attribute value.

takashi
Rank
Userlevel 2
Badge +17
Hi @tim_wood, I'm afraid that it could be a potential bug in a particular version of FME. Which version (build number) of FME are you using? Does the same symptom occur in the latest FME 2017.0 build 17271?

 

L
Badge

Hi @tim_wood

have you tried Attribute Is Null operator in Tester/TestFilter?

tim_wood
Rank
Badge +7

Thanks for the suggestions. I'm using FME 2016.1 (Build 16492). Has FME 2017 officially been released yet?

@LenaAtSafe - Attribute Is Null in the Tester was what I did first and what seemed to be giving variable results.

@nic_ran - I tried connecting the Readers directly to the Tester (Attribute Is Null) then each output port of the Tester to a separate Inspector. I ran this twice. Although the same number of records came out of each Tester output port (38717 allegedly not null), 1 record with a null value still ended up in the wrong place each time. But it wasn't the same record!

However, I purged the temporary files and ran it again and this time it matched the results for Attribute > 0 (38716 not null). So I had a look in the FME_TEMP folder and cleared out some other old stuff that was there. Then I re-ran the Workspace and got the correct results. I re-ran it again and this time I got 38720 records supposedly not null but the first 4 in the Inspector were null :-(

I noticed that whereas on the first run, no records went along the not null path for ages (until after over 100,000 records), the second time I got several records going down that path more or less straight away (4 I think but I can't be sure).

Could it be something to do with leaving the Inspector open when re-running the Workspace?

But I did this on the third run and this time, no records went down the not null path and I got the correct results.

For info, the bulk of the records are in the first FGDB to be read (289,000).

TestFilter report to follow when I get a chance.

ebygomm
Rank
Userlevel 1
Badge +21

How about trying an "attribute has value" test instead?

itay
Rank
Badge +16

Or the NullAttributeMapper and test for the new value set for multiple conditions ( empty, null, missing, etc)?

Yes FME 2017 is ofiicially released....

tim_wood
Rank
Badge +7

Inspector open or not doesn't seem to make a difference.

I think the multiple source FGDBs can also be discounted because I tried it with just the first one and still had issues.

I purged temporary files then tried Attribute has a value. That seems to be more reliable based on the repeat runs so far but I think Attribute is Null should be just as reliable.

I will download 2017 and try that.

G
Badge +3

fme2016.1.1.0

I had Samish issue where suddenly tester used in 2015 didn't work no more.

I went for testing

NULL,MISSING AND EMPYT. ...just to be sure..lol

First 2016 version seemed to be not able to make up its mind though and even this .

I had one workspace where this did not do anything until I removed the empty test (a.f.a.i. remember)

Lately It seems to be able to run on MISSING AND NULL.

L
Badge

Thanks for the suggestions. I'm using FME 2016.1 (Build 16492). Has FME 2017 officially been released yet?

@LenaAtSafe - Attribute Is Null in the Tester was what I did first and what seemed to be giving variable results.

@nic_ran - I tried connecting the Readers directly to the Tester (Attribute Is Null) then each output port of the Tester to a separate Inspector. I ran this twice. Although the same number of records came out of each Tester output port (38717 allegedly not null), 1 record with a null value still ended up in the wrong place each time. But it wasn't the same record!

However, I purged the temporary files and ran it again and this time it matched the results for Attribute > 0 (38716 not null). So I had a look in the FME_TEMP folder and cleared out some other old stuff that was there. Then I re-ran the Workspace and got the correct results. I re-ran it again and this time I got 38720 records supposedly not null but the first 4 in the Inspector were null :-(

I noticed that whereas on the first run, no records went along the not null path for ages (until after over 100,000 records), the second time I got several records going down that path more or less straight away (4 I think but I can't be sure).

Could it be something to do with leaving the Inspector open when re-running the Workspace?

But I did this on the third run and this time, no records went down the not null path and I got the correct results.

For info, the bulk of the records are in the first FGDB to be read (289,000).

TestFilter report to follow when I get a chance.

FME 2017.0 has already been released. Please download the install from http://www.safe.com/support/support-resources/fme-downloads/

 

May I ask you to run your workspace with FME 2017.0 to confirm whether there is still a problem with Tester testing differently/incorrectly? If FME 2017.0 has the same incorrect (hard to get correct) behaviour as FME 2016, could you please submit a support case? Please add link to this discussion to your problem description (no need to describe everything once again when submitting the case). Ideally, we would need you data - or a data sample that reproduces the wrong behaviour. We definitely want to investigate this problem. Sorry for the inconvenience caused by it - I am glad you caught the wrong output.

 

 

Reply


Community Stats

28,710
Posts
109,486
Replies
36,559
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.