1.145 as an input returns 1.15
Any ideas why, am I missing something?
FME 2014 SP2 (20140623 - Build 14339 - WIN32)
Question
Attribute rounder - 0.145
+44- Influencer
- 3427 replies
I have an attribute rounder which is rounding a figure to 2 decimal places with Round-off Direction set to nearest but when using 0.145 as an input is returning 0.14
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
- 8392 replies
Hi,
rounding is surprisingly complicated. Your observation is due to inaccuracies when translating between decimal and binary floating point numbers.
See the note here for more info: https://docs.python.org/2/library/functions.html#round
This is of course not specific to FME and it is a pretty well known problem within computer science, see e.g. http://floating-point-gui.de/
For your case, you can try to work around it using an ExpressionEvaluator with a "manual" rounding:
@round(@Value(input)*100)/100.0
This gives me a result of 0.15.
David
- 8392 replies
To add a bit to the specific case of 0.145:
0.145 (decimal, what you see) = 00111110 00010100 01111010 11100001 (binary, what the computer stores)
When you convert the binary representation back to decimal, however, you will see that the conversion introduced a slight "rounding error", since not all decimal numbers can be accurately represented in binary. In this case, we get 0.14499999582767487
Now, if you round 0.14499999582767487 to the nearest two decimals you will get 0.14.
So 0.14 is the correct answer to the computer, although maybe not the one the user expected :-)
Hope this makes sense.
David
+44- Influencer
- 3427 replies
Ah, my favourite number! This 'trick' doesn't work anymore, at least for 0.145 specifically, it returns 0.14.
@round(@Value(input)*100)/100.0I know there were some changes with rounding with python 3, not sure if it could be related