Skip to main content
Question

Floating Point expression throwing extraneous display values


bark0s
Contributor
Forum|alt.badge.img+3

An FYI rather than a question:

As others have mentioned previously, comparing geometries with lots of decimals is fraught with danger. I managed to expose a new and odd behaviour though (before I remembered I needed attribute rounders).

 

I have a point filegdb layer, it has attributes that describe the location. These are not automatically updated from the geometry however, so during editing these get out of sync.

A quick coordinate extraction, then an expression that subtracts the attribute from the extracted coordinate and a range filter after that, to accept anything within a metre. When I expected the result of the range filter (and sorted by numeric ascending/descending) I got 6464 points that pass.

That's great, until I scroll through the rows in Inspector. They start off close to -1 and part way through the list, values of -4, -5, -6, -7 start to appear. Wait a minute, how'd they get there?

I then added attribute and coordinate rounders, ran it again and got the same, 6464 points and suddenly, all of those -4, -5, -6 and -7 were gone and were now 0.

So the workbench was running perfectly just what was displaying in Inspector wasn't correct.

I didn't write it out, so I don't know whether the output was correct or not.

3 replies

Forum|alt.badge.img+2

@bark0s If you could attached a small sample dataset and your workspace, that would help someone in the community try to reproduce your issue and suggest a solution. Also, can you let us know what version of FME you were using when you encountered this issue.


bark0s
Contributor
Forum|alt.badge.img+3
  • Author
  • Contributor
  • March 26, 2019
markatsafe wrote:

@bark0s If you could attached a small sample dataset and your workspace, that would help someone in the community try to reproduce your issue and suggest a solution. Also, can you let us know what version of FME you were using when you encountered this issue.

Hi Mark,

FME version: FME(R) 2018.1.1.2 (20190121 - Build 18586 - WIN32)

And now I have egg all over my face. FME was working perfectly, I quite simply forgot to expand the width of the column and note the -10 at the end. So although I saw -6.6345 what I was really seeing was -0.000000000066346.

I should have known better than to to doubt FME. My bad. Apologies.

 

 


Forum|alt.badge.img+2
bark0s wrote:

Hi Mark,

FME version: FME(R) 2018.1.1.2 (20190121 - Build 18586 - WIN32)

And now I have egg all over my face. FME was working perfectly, I quite simply forgot to expand the width of the column and note the -10 at the end. So although I saw -6.6345 what I was really seeing was -0.000000000066346.

I should have known better than to to doubt FME. My bad. Apologies.

 

 

No problem - that's why we're here


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings