Skip to main content
Solved

What is causing a difference in attribute handling with the AttributeManager (Arithmetic Editor) (2)?

E
Forum|alt.badge.img

Hello,

Some time ago I had troubles with an unwanted outcome of my AttributeManager, see here:

https://knowledge.safe.com/questions/107194/what-is-causing-a-difference-in-attribute-handling.html

 

Now I have a similar issue. I have a table with 224 records, but for 3 records the outcome of the AttributeManger is not what I want and what I would expect.

It goes wrong for these three records:

This table goes into an AttributeManager where I multiply the values in the 'top' column with 1000.

I use this statement: @int(@Value(top)*1000)

 

The outcome is as follows:

Does anyone know why it does not result in these values:

LocDepth_IDtopB49F0423-8.118110B50D0035-2.012010B50D0036-4.024020

 

And why it goes wrong for only 3 out of 224 records?

 

Thanks in advance!

Eva

Best answer by ebygomm

I suspect it's a floating point issue as described here

https://knowledge.safe.com/questions/108749/-16.html

8.11 * 1000 is actually 8109.999999999999 so when you then use int on it you get 8109 instead of the expected 8110

If all your inputs only have two decimal places you could remove the decimal point and add a 0 on the end with the text editor.

Or you could try rounding before the int function, although that might introduce it's own problems elsewhere

@int(@round(@Value(number)*1000))
View original
Did this help you find an answer to your question?

3 replies

ebygomm
Influencer
Forum|alt.badge.img+38
  • ebygommInfluencer
  • Influencer
  • Best Answer
  • March 20, 2020

I suspect it's a floating point issue as described here

https://knowledge.safe.com/questions/108749/-16.html

8.11 * 1000 is actually 8109.999999999999 so when you then use int on it you get 8109 instead of the expected 8110

If all your inputs only have two decimal places you could remove the decimal point and add a 0 on the end with the text editor.

Or you could try rounding before the int function, although that might introduce it's own problems elsewhere

@int(@round(@Value(number)*1000))
mark2atsafe
Safer
Forum|alt.badge.img+44
  • mark2atsafeSafer
  • Safer
  • March 20, 2020
ebygomm wrote:

I suspect it's a floating point issue as described here

https://knowledge.safe.com/questions/108749/-16.html

8.11 * 1000 is actually 8109.999999999999 so when you then use int on it you get 8109 instead of the expected 8110

If all your inputs only have two decimal places you could remove the decimal point and add a 0 on the end with the text editor.

Or you could try rounding before the int function, although that might introduce it's own problems elsewhere

@int(@round(@Value(number)*1000))

If you rounded to (say) 7 decimal places, then it shouldn't introduce other problems, should it? You'd get rid of the excess decimals by rounding up (or down) but wouldn't affect any other numbers. 

So:

@int(@round(@value(number)*1000),7)

Basically round off to greater than your own precision.

E
Forum|alt.badge.img
  • eva218
  • Author
  • March 23, 2020

Thank your for your replies. Both work!

I will use @int(@round(@value(number)*1000),7), so hopefully it will not introduce problems elsewhere.

 

 

Reply


Helpful Members This Week

Community Stats

31,341
Posts
118,874
Replies
38,944
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.