I am currently comparing FME and PostgreSQL/PostGIS when transforming some line geometry. In each scenario I need to round the coordinates to 3 decimal places. This has thrown up a anomaly when rounding coordinates ending .4999. For example, the coordinate 104269.57149999999 will be rounded to 104269.571 by ST_ReducePrecision but 104269.572 by the FME CoordinateRounder.
I believe this is due to different rounding techniques like round-to-even or half-round-up. Can anyone explain this behaviour? Is there any “right” answer?
Thank you,
David
Page 1 / 1
I believe it’s hitting a precision limit for geometry in FME, in testing I’m seeing it will round anything which is above 15 digits long.
Use a Creator, and paste in 104269.57149999999, inspect the feature created, you’ll see the coordinate precision in the geometry has been rounded. Try paste the full coordinate back in and remove 9’s until it stops being rounded.
It’s easier to see this if you instead try create an X coordinate of 1234567890.123456 (16 digits), which gets created as 1234567890.12346
So 104269.57149999999 is too long, instead of truncating the extra 9’s, it is being rounded when read into FME, turning it to 104269.5715. Then the CoordinateRounder is rounding to 104269.572
To ensure you always get the same result, you might have to read the geometry after applying ST_ReducePrecision, or read with that in a SQLExecutor.
That might not be your issue exactly - I actually stumbled on a regression with the Creator Version 6, the Creator is rounding the coordinates. It works fine in an older version of FME (2021)
I also used your example value of 104269.57149999999 in a coordinate, and using the coordinate rounder it gave me expected results. (104269.571)
Have you inspected the data directly before the CoordinateRounder?
The 499999 certainly seems like there’s some kind of floating point precision issue. Ironically the correct answer based on the original data would probably be 572. But of course consistency is important.
My guess is that somewhere along the way (probably in the reader) the value is getting rounded to 9 decimal places - this would result in the 5715 value, which in turn will get rounded up.
This is I think essentially echos what @ctredinnick said
After Matt and Chris, have been explaining why the difference is occurring. I am coming on a different angle and specifically around whether you need to round geometries at all.
The ChangeDetector transformer allows for the “Vector Tolerance” when comparing geometry. I certainly use this over trying to round coordinates from different sources, as formats and tables can do some fantastically weird thing with precision.
Thank you for all the replies. To add a little more context, the geometry transformation is to extract the first and last 5m of a line unless the line is less than 10m at which point we extract 49% (to prevent overlapping). As a result we get some geometry with a lot of decimal places but our standard is to round the geometry to 3dp.
I have set the Vector Tolerance in the change detector to 0.01 to get around this issue which has been accepted. My question was to try understand why there is a difference and if there is anything that can be done. I though maybe ST_QuantizeCoordinates could be used somewhere to replicate the truncating FME appears to be doing.