Question

Floating Point expression throwing extraneous display values


Badge +3

An FYI rather than a question:

As others have mentioned previously, comparing geometries with lots of decimals is fraught with danger. I managed to expose a new and odd behaviour though (before I remembered I needed attribute rounders).

 

I have a point filegdb layer, it has attributes that describe the location. These are not automatically updated from the geometry however, so during editing these get out of sync.

A quick coordinate extraction, then an expression that subtracts the attribute from the extracted coordinate and a range filter after that, to accept anything within a metre. When I expected the result of the range filter (and sorted by numeric ascending/descending) I got 6464 points that pass.

That's great, until I scroll through the rows in Inspector. They start off close to -1 and part way through the list, values of -4, -5, -6, -7 start to appear. Wait a minute, how'd they get there?

I then added attribute and coordinate rounders, ran it again and got the same, 6464 points and suddenly, all of those -4, -5, -6 and -7 were gone and were now 0.

So the workbench was running perfectly just what was displaying in Inspector wasn't correct.

I didn't write it out, so I don't know whether the output was correct or not.


3 replies

Badge +2

@bark0s If you could attached a small sample dataset and your workspace, that would help someone in the community try to reproduce your issue and suggest a solution. Also, can you let us know what version of FME you were using when you encountered this issue.

Badge +3

@bark0s If you could attached a small sample dataset and your workspace, that would help someone in the community try to reproduce your issue and suggest a solution. Also, can you let us know what version of FME you were using when you encountered this issue.

Hi Mark,

FME version: FME(R) 2018.1.1.2 (20190121 - Build 18586 - WIN32)

And now I have egg all over my face. FME was working perfectly, I quite simply forgot to expand the width of the column and note the -10 at the end. So although I saw -6.6345 what I was really seeing was -0.000000000066346.

I should have known better than to to doubt FME. My bad. Apologies.

 

 

Badge +2

Hi Mark,

FME version: FME(R) 2018.1.1.2 (20190121 - Build 18586 - WIN32)

And now I have egg all over my face. FME was working perfectly, I quite simply forgot to expand the width of the column and note the -10 at the end. So although I saw -6.6345 what I was really seeing was -0.000000000066346.

I should have known better than to to doubt FME. My bad. Apologies.

 

 

No problem - that's why we're here

Reply