Question

GDB Float to ArcGISPortal Double - floating point precision issues

ebygomm
Rank
Userlevel 1
Badge +10

I'm reading in data from a filegdb which has an attribute of type float. I'm then writing the data to an ArcGIS Portal featureservice which has an attribute type of double.

 

The data is read into fme with a type of 32 bit real and a value of 77.56, when this ends up in the Portal featureservice this ends up with a value of 77.55999756

 

Now I think I understand where the 77.55999756 comes from as if you convert 77.56 to 32 bits of binary and back again you get 77.55999755859375

 

But what I'd really like is for the double field to end up with a value of 77.56 (77.56000000000000227373675443232059478759765625) .

 

Now i can achieve this by converting the attribute to a string and then to a 64 bit real value in FME before sending as an update to the writer. This then returns a value of 77.56 when i read the data back into FME which matches the value in the gdb. It also matches what would happen if I went into the portal front end and manually added the 77.56 value

 

The question is, is this the right way to go about this? Is there something I'm missing

@David davidr​  @mark2atsafe​ 

 

 

 

4 replies

david_r
Rank
Userlevel 4

I'm not quite sure there's a right or a wrong way to do this, but perhaps using string operations isn't the most efficient.

I would perhaps rather apply the actual precision of the dataset, e.g. if the data is precise to four digits, I would do

>>> round(77.55999755859375, 4)
77.56

This works since Python is using 64-bit floats internally, which should map nicely to your featureservice.

ebygomm
Rank
Userlevel 1
Badge +10

I'm not quite sure there's a right or a wrong way to do this, but perhaps using string operations isn't the most efficient.

I would perhaps rather apply the actual precision of the dataset, e.g. if the data is precise to four digits, I would do

>>> round(77.55999755859375, 4)
77.56

This works since Python is using 64-bit floats internally, which should map nicely to your featureservice.

Rounding doesn't help here as the issue appears to be the internal FME datatype of 32 bit real

imageIf I write the following values to fields where the datatype is double, e.g. my portal feature service or an ArcSDE instance, when i read the data back in again this is what you see

image 

david_r
Rank
Userlevel 4

Rounding doesn't help here as the issue appears to be the internal FME datatype of 32 bit real

imageIf I write the following values to fields where the datatype is double, e.g. my portal feature service or an ArcSDE instance, when i read the data back in again this is what you see

image 

The result that you're showing is exactly what I'd expect. Although I believe that "77.56" has been rounded up from 77.56000000000000227373675443232059478759765625 for display purposes.

Either way, you cannot represent 77.56 exactly using binary, it's always going to be an approximation. While increasing the resolution (number of bits) will certainly improve the approximation, it'll never be 100% exact.

>>> import decimal
>>> value=77.56
>>> print(decimal.Decimal.from_float(value))
77.56000000000000227373675443232059478759765625

 

ebygomm
Rank
Userlevel 1
Badge +10

Rounding doesn't help here as the issue appears to be the internal FME datatype of 32 bit real

imageIf I write the following values to fields where the datatype is double, e.g. my portal feature service or an ArcSDE instance, when i read the data back in again this is what you see

image 

I guess what i'm asking is, is it expected that when the internal FME dataype is 32 bit real, the approximation reflects the 32 bit value when writing to a double field, instead of using the approximation of the 64 bit value which the double field can handle?

 

In terms of why this matters in my use case, if i have a gdb, convert the floats to strings and send the data to my portal featureservice then if i then use a change detector there are no reported changes. If i send the floats as is (with an fme data type of 32 bit real), the change detector says a lot of the values are different :-)

Reply


Community Stats

28,670
Posts
109,366
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.