Question

Attribute rounder - 0.145

  • 13 October 2014
  • 3 replies
  • 11 views
ebygomm
Rank
Userlevel 1
Badge +10
I have an attribute rounder which is rounding a figure to 2 decimal places with Round-off Direction set to nearest but when using 0.145 as an input is returning 0.14

 

 

1.145 as an input returns 1.15

 

 

Any ideas why, am I missing something?

 

 

FME 2014 SP2 (20140623 - Build 14339 - WIN32)

3 replies

david_r
Rank
Userlevel 4
Hi,

 

 

rounding is surprisingly complicated. Your observation is due to inaccuracies when translating between decimal and binary floating point numbers.

 

 

See the note here for more info: https://docs.python.org/2/library/functions.html#round

 

 

This is of course not specific to FME and it is a pretty well known problem within computer science, see e.g. http://floating-point-gui.de/

 

 

For your case, you can try to work around it using an ExpressionEvaluator with a "manual" rounding:

 

 

@round(@Value(input)*100)/100.0

 

 

This gives me a result of 0.15.

 

 

David
david_r
Rank
Userlevel 4
To add a bit to the specific case of 0.145:

 

 

0.145 (decimal, what you see) = 00111110 00010100 01111010 11100001 (binary, what the computer stores)

 

 

When you convert the binary representation back to decimal, however, you will see that the conversion introduced a slight "rounding error", since not all decimal numbers can be accurately represented in binary. In this case, we get 0.14499999582767487

 

 

Now, if you round 0.14499999582767487 to the nearest two decimals you will get 0.14.

 

 

So 0.14 is the correct answer to the computer, although maybe not the one the user expected :-)

 

 

Hope this makes sense.

 

 

David
ebygomm
Rank
Userlevel 1
Badge +10

Ah, my favourite number! This 'trick' doesn't work anymore, at least for 0.145 specifically, it returns 0.14.

@round(@Value(input)*100)/100.0

The following will return 0.15.

(@round((@round((@Value(Value)*1000))/10)))/100

 I know there were some changes with rounding with python 3, not sure if it could be related

Reply


Community Stats

28,652
Posts
109,285
Replies
36,527
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.