Skip to main content
Solved

Spatial filter not identifying identical lines.

C
Forum|alt.badge.img
  • cj

Hi,

I am doing an update task where I need to update dataset A with the new lines from dataset B. Dataset B includes all of the lines that are already in dataset A plus a number of new lines. I am using a spatial filter to filter out just those lines in dataset B that are not in dataset A. In the spatial filter I am using "Filter Equals Candidate" as the test. My expected result is that the new lines would fail that test and appear at the Failed port, however all lines fail.

When I do a select by location in ArcGIS with the selection method set to "identical to the source layer feature" and reverse the selection I get the correct new lines.

I can get the spatial filter to output the data in the correct way (based on my test in ArcMap) but only if I add a coordinate rounder to both input features with the precisions set to 3. This however changes the output data making it inaccurate.

Is there anyway I can achieve what I need to do using the spatial filter without introducing the inaccuracy from the spatial filter.

Dataset A = FGDB

Dataset B = shapefiles

Best answer by mark2atsafe

I strongly suspect that FGDB is a floating point format and Shapefile is integer-based. When FME converts the integer shapefile values to floating point (FME works in floating point), then there is a little rounding error. It's not a bug, just the nature of binary maths on a computer.

For example, coordinates that were (eg) 12345.6 in the Shapefile could now be 12345.5999999999 and so the coordinates don't match the FGDB (which is floating point already and so not affected).

Other products will hide this by rounding the values automatically. FME doesn't, so you would have to round the values off with a CoordinateRounder transformer. It shouldn't produce any more inaccuracies than other software. The rounding error is normally at the 12th decimal place (or something like that), way more precise than most spatial data. It's assumed you know the precision you want your data to be, and rounding it off to that point won't affect it.

However... In the future, FME is getting a Tolerance parameter directly in many of the transformers. There are about 15 of these transformers in 2018:

No tolerance means you want the existing behaviour. Custom Value means you are going to enter a rounding value (just as in the CoordinateRounder). That will help in this situation but also be useful where the data is bad in itself.

Automatic means that FME is going to round the coordinates automatically. The exact mechanism I don't know off hand, but it will basically bring FME into line with tools that automatically round coordinates.

I hope that helps. I don't think the SpatialFilter is getting this behaviour in 2018, but I expect it will in a future release. I'm going to be writing up a blog post about this so keep an eye out for it on blog.safe.com

Mark

View original
Did this help you find an answer to your question?

5 replies

erik_jan
Contributor
Forum|alt.badge.img+18
  • erik_janContributor
  • Contributor
  • January 8, 2018

Did you have a look at the ChangeDetector transformer? That is built for this kind of comparison.

And you have to keep in mind that different data formats can have different tolerances for coordinates. Usually the CoordinateRounder transformer is used to match the tolerances (number of decimals on the coordinates).

lau
Forum|alt.badge.img+3
  • lau
  • January 8, 2018

If using a CoordinatesRounder makes the job, you can store your geometries into an attribute BEFORE any CoordinatesRounder (GeometryExtractor) and restore them after your spatial filter with a GeometryReplacer

mark2atsafe
Safer
Forum|alt.badge.img+44
  • mark2atsafeSafer
  • Safer
  • Best Answer
  • January 8, 2018

I strongly suspect that FGDB is a floating point format and Shapefile is integer-based. When FME converts the integer shapefile values to floating point (FME works in floating point), then there is a little rounding error. It's not a bug, just the nature of binary maths on a computer.

For example, coordinates that were (eg) 12345.6 in the Shapefile could now be 12345.5999999999 and so the coordinates don't match the FGDB (which is floating point already and so not affected).

Other products will hide this by rounding the values automatically. FME doesn't, so you would have to round the values off with a CoordinateRounder transformer. It shouldn't produce any more inaccuracies than other software. The rounding error is normally at the 12th decimal place (or something like that), way more precise than most spatial data. It's assumed you know the precision you want your data to be, and rounding it off to that point won't affect it.

However... In the future, FME is getting a Tolerance parameter directly in many of the transformers. There are about 15 of these transformers in 2018:

No tolerance means you want the existing behaviour. Custom Value means you are going to enter a rounding value (just as in the CoordinateRounder). That will help in this situation but also be useful where the data is bad in itself.

Automatic means that FME is going to round the coordinates automatically. The exact mechanism I don't know off hand, but it will basically bring FME into line with tools that automatically round coordinates.

I hope that helps. I don't think the SpatialFilter is getting this behaviour in 2018, but I expect it will in a future release. I'm going to be writing up a blog post about this so keep an eye out for it on blog.safe.com

Mark

mark2atsafe
Safer
Forum|alt.badge.img+44
  • mark2atsafeSafer
  • Safer
  • January 8, 2018
mark2atsafe wrote:

I strongly suspect that FGDB is a floating point format and Shapefile is integer-based. When FME converts the integer shapefile values to floating point (FME works in floating point), then there is a little rounding error. It's not a bug, just the nature of binary maths on a computer.

For example, coordinates that were (eg) 12345.6 in the Shapefile could now be 12345.5999999999 and so the coordinates don't match the FGDB (which is floating point already and so not affected).

Other products will hide this by rounding the values automatically. FME doesn't, so you would have to round the values off with a CoordinateRounder transformer. It shouldn't produce any more inaccuracies than other software. The rounding error is normally at the 12th decimal place (or something like that), way more precise than most spatial data. It's assumed you know the precision you want your data to be, and rounding it off to that point won't affect it.

However... In the future, FME is getting a Tolerance parameter directly in many of the transformers. There are about 15 of these transformers in 2018:

No tolerance means you want the existing behaviour. Custom Value means you are going to enter a rounding value (just as in the CoordinateRounder). That will help in this situation but also be useful where the data is bad in itself.

Automatic means that FME is going to round the coordinates automatically. The exact mechanism I don't know off hand, but it will basically bring FME into line with tools that automatically round coordinates.

I hope that helps. I don't think the SpatialFilter is getting this behaviour in 2018, but I expect it will in a future release. I'm going to be writing up a blog post about this so keep an eye out for it on blog.safe.com

Mark

Actually I think it's the other way around in terms of format. eg Shapefile returns an x value of 492182.717 whereas FGDB returns 492182.7170000002 - but the issue is still the same that one is being rounded and you can see that rounding off to 3dp won't hurt my data.

 

C
Forum|alt.badge.img
  • cj
  • Author
  • February 11, 2018

I ended up using the UpdateDetector with a GeometryExtractor and GeometryReplacer either side of the CoordinateRounder - worked a treat, thanks.

 

Reply


Helpful Members This Week

Community Stats

31,341
Posts
118,864
Replies
38,931
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.