Skip to main content
Question

32 bit real attribute via python getAttribute

  • July 18, 2019
  • 1 reply
  • 3 views

todd_davis
Influencer
Forum|alt.badge.img+22

Is this expected behaviour?

I am extracting a 32 bit real value of "7.35" using python using feature.getAttribute in 2018.1/2019.1 (and probably every other one). It is returning 7.349999904632568 as a 64 Bit Real . I get that this calculating using 64 bit, so that's where it is coming from, but interested that it is performing that calc on an existing known number.

The workaround was to use feature.getAttributeAsType(<attribute>, 11) to treat as a string

Example attached

1 reply

david_r
Celebrity
  • July 18, 2019

It's strictly a representational error, i.e. a floating point rounding issue. You can e.g. use Python to check for this:

>>> from decimal import Decimal
>>> Decimal(7.35)
Decimal('7.3499999999999996447286321199499070644378662109375')

As you can see, there is no exact way to represent 7.35 in binary, it will always be an approximation.

See also: https://floating-point-gui.de/

For even more information: https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/ 


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings