Skip to main content

Hello,

I have a problem with the calcul of a sum (expression evaluator, statistic calculator..).

The result is not correct.
I make this calcul : 10000+10000+1+100+0.1+0.01 and the result is 20101.109999999997 .


I have this problem in fme 2018 and 2017 but in 2015 the result is correct .


Have you got any idea to solve my problem?
Thanks a lot.

It's due to floating point precision, more info here: https://floating-point-gui.de/

You can use the AttributeRounder to, let's say 5 decimals, which will return 20101.11.

Why it works differently in FME 2015 I cannot really say. Perhaps someone from Safe has a theory...


It's due to floating point precision, more info here: https://floating-point-gui.de/

You can use the AttributeRounder to, let's say 5 decimals, which will return 20101.11.

Why it works differently in FME 2015 I cannot really say. Perhaps someone from Safe has a theory...

I remember that the Table View in FME Data Inspector in old versions - maybe FME 2015 and earlier - did not display floating point numbers with the full precision, unlike the Feature Information window.

 

 


Thank you very much @david_r for your answer. I didn't know the floating point precision. Now yes !
Thank you too for the proposition of used the attributerounder with 5 decimals (I had chosen 2 decimals).

Thank you too @takashi indeed the result seem good only in the table view.

Reply