Skip to main content
Solved

What is causing a difference in attribute handling with the AttributeManager (Arithmetic Editor) (2)?

  • March 20, 2020
  • 3 replies
  • 49 views
E
Forum|alt.badge.img

Hello,

Some time ago I had troubles with an unwanted outcome of my AttributeManager, see here:

https://knowledge.safe.com/questions/107194/what-is-causing-a-difference-in-attribute-handling.html

 

Now I have a similar issue. I have a table with 224 records, but for 3 records the outcome of the AttributeManger is not what I want and what I would expect.

It goes wrong for these three records:

This table goes into an AttributeManager where I multiply the values in the 'top' column with 1000.

I use this statement: @int(@Value(top)*1000)

 

The outcome is as follows:

Does anyone know why it does not result in these values:

LocDepth_IDtopB49F0423-8.118110B50D0035-2.012010B50D0036-4.024020

 

And why it goes wrong for only 3 out of 224 records?

 

Thanks in advance!

Eva

Best answer by ebygomm

I suspect it's a floating point issue as described here

https://knowledge.safe.com/questions/108749/-16.html

8.11 * 1000 is actually 8109.999999999999 so when you then use int on it you get 8109 instead of the expected 8110

If all your inputs only have two decimal places you could remove the decimal point and add a 0 on the end with the text editor.

Or you could try rounding before the int function, although that might introduce it's own problems elsewhere

@int(@round(@Value(number)*1000))
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

3 replies

ebygomm
Influencer
Forum|alt.badge.img+44
  • ebygommInfluencer
  • Influencer
  • 3427 replies
  • Best Answer
  • March 20, 2020

I suspect it's a floating point issue as described here

https://knowledge.safe.com/questions/108749/-16.html

8.11 * 1000 is actually 8109.999999999999 so when you then use int on it you get 8109 instead of the expected 8110

If all your inputs only have two decimal places you could remove the decimal point and add a 0 on the end with the text editor.

Or you could try rounding before the int function, although that might introduce it's own problems elsewhere

@int(@round(@Value(number)*1000))
mark2atsafe
Safer
Forum|alt.badge.img+56
  • mark2atsafeSafer
  • Safer
  • 2554 replies
  • March 20, 2020

I suspect it's a floating point issue as described here

https://knowledge.safe.com/questions/108749/-16.html

8.11 * 1000 is actually 8109.999999999999 so when you then use int on it you get 8109 instead of the expected 8110

If all your inputs only have two decimal places you could remove the decimal point and add a 0 on the end with the text editor.

Or you could try rounding before the int function, although that might introduce it's own problems elsewhere

@int(@round(@Value(number)*1000))

If you rounded to (say) 7 decimal places, then it shouldn't introduce other problems, should it? You'd get rid of the excess decimals by rounding up (or down) but wouldn't affect any other numbers. 

So:

@int(@round(@value(number)*1000),7)

Basically round off to greater than your own precision.

FME Evangelist to the Rich and Famous!!
E
Forum|alt.badge.img
  • eva218
  • Author
  • 21 replies
  • March 23, 2020

Thank your for your replies. Both work!

I will use @int(@round(@value(number)*1000),7), so hopefully it will not introduce problems elsewhere.

 

 

Community Stats

32,076
Posts
121,798
Replies
39,707
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.