+12
Is this expected behaviour?
I am extracting a 32 bit real value of "7.35" using python using feature.getAttribute in 2018.1/2019.1 (and probably every other one). It is returning 7.349999904632568 as a 64 Bit Real . I get that this calculating using 64 bit, so that's where it is coming from, but interested that it is performing that calc on an existing known number.
The workaround was to use feature.getAttributeAsType(<attribute>, 11) to treat as a string
Example attached
It's strictly a representational error, i.e. a floating point rounding issue. You can e.g. use Python to check for this:
>>> from decimal import Decimal
>>> Decimal(7.35)
Decimal('7.3499999999999996447286321199499070644378662109375')As you can see, there is no exact way to represent 7.35 in binary, it will always be an approximation.
See also: https://floating-point-gui.de/
For even more information: https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.