Skip to main content
Question

32 bit real attribute via python getAttribute

  • July 18, 2019
  • 1 reply
  • 10 views

todd_davis
Influencer
Forum|alt.badge.img+23

Is this expected behaviour?

I am extracting a 32 bit real value of "7.35" using python using feature.getAttribute in 2018.1/2019.1 (and probably every other one). It is returning 7.349999904632568 as a 64 Bit Real . I get that this calculating using 64 bit, so that's where it is coming from, but interested that it is performing that calc on an existing known number.

The workaround was to use feature.getAttributeAsType(<attribute>, 11) to treat as a string

Example attached

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

1 reply

david_r
Celebrity
  • July 18, 2019

It's strictly a representational error, i.e. a floating point rounding issue. You can e.g. use Python to check for this:

>>> from decimal import Decimal
>>> Decimal(7.35)
Decimal('7.3499999999999996447286321199499070644378662109375')

As you can see, there is no exact way to represent 7.35 in binary, it will always be an approximation.

See also: https://floating-point-gui.de/

For even more information: https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/