I load files into a PostGIS DB. When I get new files that update certain areas, I'd like to do change detection. I've tried a bunch of different ways to try to compare the file to the data already in the DB. Is there a trick to this? Seems everything I try give either all new or updated. I know that there is mostly the same polies with very few updates. Is there a trick if the datasources are DB against file? Any help greatly appreciated.
Page 1 / 1
I find it helps to use CoordinateRounder on both data sources (say 3 decimals for projected data, maybe 6 or 7 for geographic) and if dates are being compared then also convert to a fixed precision format like %Y%m%d%H%M%S
Hi @planomax
There are a few reasons why features might not match and various things you can do to check.
Firstly, yes, check the coordinates to see if one of these are getting rounded. If one dataset is stored as floating point, and one as integers, when FME reads the data the integers can be converted to floats and induce a minor "error". As Bruce says, the CoordinateRounder is one way to solve that.
Experiment with the Match Geometry parameter and the Lenient Matching parameter. Sometimes (for example) a file dataset might contain two consecutive duplicate points. A database often deletes those extra vertices automatically. So lenient matching might cause those to match, when they otherwise wouldn't.
Also try the tolerance parameter. For more information on that, see this blog post: https://www.safe.com/blog/2019/01/change-detection-2019-fmeevangelist184/
Of course - and this might be a bit obvious - but the datasets are in the same coordinate system, right?
Which version of FME are you using, by the way? 2019 has better tolerance matching than 2018 did.
@planomax If you could include a small sample dataset of original & update then we can take a look why this isn't working for you. Rounding both sets of coordinates as @bruceharold suggests can really help. The ChangeDetector in 2019 has more matching options than older versions.
If you use the CRCCalculator to convert the rounded geometries to a CRC code, then the change detection will be a lot faster as you'll be comparing two attributes rather then geometries (assuming you don't want any leniency in the geometry match)
What are you using for Update Detection Key Attribute @planomax? These attributes must be persistent between the two datasets. Otherwise, most will be "new data", with a few unintentional matching attributes as "updated data".