+1Hi all,
I have a workspace, where I am detecting the hange between my original data in an SDE and revised data coming from a public source.
I'm using the fme_db_operation to push Inserts, Updates and Deletes to my SDE. However it apparantly just keeps adding the data as duplicates.
I have 6 exactly same polygons for each feature, with the only difference being a Version ID...
Any thoughts?
Best answer by nielsgerrits
Hi Mark,
First of all, thank you for your input. I have read through everything that I can find related to the change detection, including the article you refer to, and still no luck. Hopefully I'm just missing some silly setting or something :-)
I have attached a zip file containing my fmw file and a file gdb containing one of my layers (the original).
Within my workspace, the first part downloads and extracts a file gdb from the public source, which my feature reader reads. The paths will have to be updated to a local path, but other than that, it should run straight through.
I have made 4 small attribute changes in the Original dataset, så it should, theoretically, just update those 4 features. Thanx :-)
With regards to the coordinateRounder - if my Update Detection Key Attribute is set and matches the original and revised data, then even changes in the coordinates should still be parsed as Updates, and not Delete all the original and Insert all the revised data, shouldn't it?
With regards to the CoordinateRounder - if my Update Detection Key Attribute is set and matches the original and revised data, then even changes in the coordinates should still be parsed as Updates, and not Delete all the original and Insert all the revised data, shouldn't it?
This is true, but as you can see in the inspector, the format of the ID "OBJEKT_ID" is different in the 2 sets.
GEN_ARTSFUND_LN_GDB.gdb
Danske Grunddata Test.gdb
As these differ the ChangeDetector sees 81 inserts and 81 deletes instead of 81 updates.
So if you add an AttributeCreator where you set OBJEKT_ID = {@Value(OBJEKT_ID)} in the stream from GEN_ARTSFUND_LN_GDB.gdb it returns 81 updates.
Next, if you add the changelist you can see all the differences.
If you drill down to geometry, you see the coordinates differ from one to another. The first point of OBJEKT_ID {01FEE9E4-945C-443D-9EAF-A29F84AC66B8}
GEN_ARTSFUND_LN_GDB.gdb
Danske Grunddata Test.gdb
But with {57CAEB1E-064E-437C-A6E3-7669E8BDFD53} it is the other way around:
GEN_ARTSFUND_LN_GDB.gdb
Danske Grunddata Test.gdb
This is why @Mark Stoakes suggests you to add CoordinateRounders.
I personally prefer to check for changes with rounded geometry and then restore the original geometry. This can easily be done with the GeometryExtractor (geometry to attribute) before the CoordinateRounder and a GeometryReplacer (attribute to geometry) after the ChangeDetector, to prevent putting rounded data in my source database.
I also see you have added fme_db_operation manually, but the ChangeDetector generates this attribute so you don't have to do that.
In the Writer, make sure you set Feature Operation to fme_db_operation and Match Columns to OBJECTID. In the sample workspace it is set to Insert.
First you have to check what the detected changes are. You can do this by generating a detailed changes list in the ChangeDetector:
If the changes are about geometry, comparing data from SDE with a non SDE source is always tricky because in SDE geometry is snapped to a grid. One thing you can try is to snap the non SDE data to the same grid before changedetection. You can do this using the ArcSDEGridSnapper. Also see this article.
+1First you have to check what the detected changes are. You can do this by generating a detailed changes list in the ChangeDetector:
If the changes are about geometry, comparing data from SDE with a non SDE source is always tricky because in SDE geometry is snapped to a grid. One thing you can try is to snap the non SDE data to the same grid before changedetection. You can do this using the ArcSDEGridSnapper. Also see this article.
I'm still fairly new to FME, so forgive me if my questions are a bit daft! :-)
What will that list give me?
Shouldn't it be straight forward with just connecting the Updated, Deleted and Inserted to the writer, perhaps running it through an AttributeCreator first before the writer?
I'm still fairly new to FME, so forgive me if my questions are a bit daft! :-)
What will that list give me?
Shouldn't it be straight forward with just connecting the Updated, Deleted and Inserted to the writer, perhaps running it through an AttributeCreator first before the writer?
The list will give you the reason why the ChangeDetector concluded the feature which is now duplicate is marked as inserted / new.
+1The funny thing is that it does not seem to update anything at any point - it merely deletes the old and inserts the new.. I have checked my settings, and the Update Detection Key attribute and it is identical. So it should be able to read the changes to the data.
Or at least just insert a few and delete a few records, instead of deleting everything and inserting the new.
I cant see that what should be changes in the writer:
@larsec Can you export a small part of your data to a File Geodb and include that and your workspace and the 'public' data in an attachment? That would give the community something to work with.
If you are trying to match geometry then, as @nielsgerrits suggests, this can be tricky. Try a CoordinateRounder in front of the ChangeDetector inputs. Use the Feature Caching / Partial Runs to carefully inspect the features and compare them - including the coordinates . You might find some tips in the links in this article
+1@larsec Can you export a small part of your data to a File Geodb and include that and your workspace and the 'public' data in an attachment? That would give the community something to work with.
If you are trying to match geometry then, as @nielsgerrits suggests, this can be tricky. Try a CoordinateRounder in front of the ChangeDetector inputs. Use the Feature Caching / Partial Runs to carefully inspect the features and compare them - including the coordinates . You might find some tips in the links in this article
Hi Mark,
First of all, thank you for your input. I have read through everything that I can find related to the change detection, including the article you refer to, and still no luck. Hopefully I'm just missing some silly setting or something :-)
I have attached a zip file containing my fmw file and a file gdb containing one of my layers (the original).
Within my workspace, the first part downloads and extracts a file gdb from the public source, which my feature reader reads. The paths will have to be updated to a local path, but other than that, it should run straight through.
I have made 4 small attribute changes in the Original dataset, så it should, theoretically, just update those 4 features. Thanx :-)
With regards to the coordinateRounder - if my Update Detection Key Attribute is set and matches the original and revised data, then even changes in the coordinates should still be parsed as Updates, and not Delete all the original and Insert all the revised data, shouldn't it?
Hi Mark,
First of all, thank you for your input. I have read through everything that I can find related to the change detection, including the article you refer to, and still no luck. Hopefully I'm just missing some silly setting or something :-)
I have attached a zip file containing my fmw file and a file gdb containing one of my layers (the original).
Within my workspace, the first part downloads and extracts a file gdb from the public source, which my feature reader reads. The paths will have to be updated to a local path, but other than that, it should run straight through.
I have made 4 small attribute changes in the Original dataset, så it should, theoretically, just update those 4 features. Thanx :-)
With regards to the coordinateRounder - if my Update Detection Key Attribute is set and matches the original and revised data, then even changes in the coordinates should still be parsed as Updates, and not Delete all the original and Insert all the revised data, shouldn't it?
With regards to the CoordinateRounder - if my Update Detection Key Attribute is set and matches the original and revised data, then even changes in the coordinates should still be parsed as Updates, and not Delete all the original and Insert all the revised data, shouldn't it?
This is true, but as you can see in the inspector, the format of the ID "OBJEKT_ID" is different in the 2 sets.
GEN_ARTSFUND_LN_GDB.gdb
Danske Grunddata Test.gdb
As these differ the ChangeDetector sees 81 inserts and 81 deletes instead of 81 updates.
So if you add an AttributeCreator where you set OBJEKT_ID = {@Value(OBJEKT_ID)} in the stream from GEN_ARTSFUND_LN_GDB.gdb it returns 81 updates.
Next, if you add the changelist you can see all the differences.
If you drill down to geometry, you see the coordinates differ from one to another. The first point of OBJEKT_ID {01FEE9E4-945C-443D-9EAF-A29F84AC66B8}
GEN_ARTSFUND_LN_GDB.gdb
Danske Grunddata Test.gdb
But with {57CAEB1E-064E-437C-A6E3-7669E8BDFD53} it is the other way around:
GEN_ARTSFUND_LN_GDB.gdb
Danske Grunddata Test.gdb
This is why @Mark Stoakes suggests you to add CoordinateRounders.
I personally prefer to check for changes with rounded geometry and then restore the original geometry. This can easily be done with the GeometryExtractor (geometry to attribute) before the CoordinateRounder and a GeometryReplacer (attribute to geometry) after the ChangeDetector, to prevent putting rounded data in my source database.
I also see you have added fme_db_operation manually, but the ChangeDetector generates this attribute so you don't have to do that.
In the Writer, make sure you set Feature Operation to fme_db_operation and Match Columns to OBJECTID. In the sample workspace it is set to Insert.
+1Hi Mark,
First of all, thank you for your input. I have read through everything that I can find related to the change detection, including the article you refer to, and still no luck. Hopefully I'm just missing some silly setting or something :-)
I have attached a zip file containing my fmw file and a file gdb containing one of my layers (the original).
Within my workspace, the first part downloads and extracts a file gdb from the public source, which my feature reader reads. The paths will have to be updated to a local path, but other than that, it should run straight through.
I have made 4 small attribute changes in the Original dataset, så it should, theoretically, just update those 4 features. Thanx :-)
With regards to the coordinateRounder - if my Update Detection Key Attribute is set and matches the original and revised data, then even changes in the coordinates should still be parsed as Updates, and not Delete all the original and Insert all the revised data, shouldn't it?
HeHe total newbie - didn't realize that the curly braces actually where perceived as a change - I focused solely on the data itself in the OBJECT_ID.. Then it makes a lot more sence why ereything is seen as new data.
With regards to the changelist - I simply dont know how to add it and where to find it afterwards. HSould I just write changelist in the List Name? I haven't had any training in FME, so I'm learning as I go along. :-)
The comment about the fme_db_operation - does that mean that I dont have to have the three attributecreators with Update, Insert and Delete, and just connect the Change Detector and the writer directly?
Thank you for your quick response Niels :-)
Hi Mark,
First of all, thank you for your input. I have read through everything that I can find related to the change detection, including the article you refer to, and still no luck. Hopefully I'm just missing some silly setting or something :-)
I have attached a zip file containing my fmw file and a file gdb containing one of my layers (the original).
Within my workspace, the first part downloads and extracts a file gdb from the public source, which my feature reader reads. The paths will have to be updated to a local path, but other than that, it should run straight through.
I have made 4 small attribute changes in the Original dataset, så it should, theoretically, just update those 4 features. Thanx :-)
With regards to the coordinateRounder - if my Update Detection Key Attribute is set and matches the original and revised data, then even changes in the coordinates should still be parsed as Updates, and not Delete all the original and Insert all the revised data, shouldn't it?
Most of us started this way, no problem :)
Yes, just add the listname. Then add an Inspector transformer to the ChangeDetector Updated outputport. Then you get a view with the result for every run. Select one of the features in this view and look for the list in the feature information window.
+1Hi Mark,
First of all, thank you for your input. I have read through everything that I can find related to the change detection, including the article you refer to, and still no luck. Hopefully I'm just missing some silly setting or something :-)
I have attached a zip file containing my fmw file and a file gdb containing one of my layers (the original).
Within my workspace, the first part downloads and extracts a file gdb from the public source, which my feature reader reads. The paths will have to be updated to a local path, but other than that, it should run straight through.
I have made 4 small attribute changes in the Original dataset, så it should, theoretically, just update those 4 features. Thanx :-)
With regards to the coordinateRounder - if my Update Detection Key Attribute is set and matches the original and revised data, then even changes in the coordinates should still be parsed as Updates, and not Delete all the original and Insert all the revised data, shouldn't it?
Hi Niels,
Thank you for your valuable input so far.
I have a follow up question to the changelist. I have either excluded or fixed all the differences in the changelist but one.
Apparently there is a field called geometryObject - not something I can see anywhere, but it is the only thing left. However the Original and Revised are both empty.. So why does it come up in the changelist? (see attachment).
I have added an AttributeExposer to see if I can find the field, but no luck. Do you have any bright ideas? :-)
Thanks.
Hi Mark,
First of all, thank you for your input. I have read through everything that I can find related to the change detection, including the article you refer to, and still no luck. Hopefully I'm just missing some silly setting or something :-)
I have attached a zip file containing my fmw file and a file gdb containing one of my layers (the original).
Within my workspace, the first part downloads and extracts a file gdb from the public source, which my feature reader reads. The paths will have to be updated to a local path, but other than that, it should run straight through.
I have made 4 small attribute changes in the Original dataset, så it should, theoretically, just update those 4 features. Thanx :-)
With regards to the coordinateRounder - if my Update Detection Key Attribute is set and matches the original and revised data, then even changes in the coordinates should still be parsed as Updates, and not Delete all the original and Insert all the revised data, shouldn't it?
Hej Lars, geometryObject means the detected change is in the geometry, not in an attribute.
What I would do is isolate the feature with this ID with a Tester in both streams (Original and Revised) before feeding it into the ChangeDetector. Then add Inspectors to both Testers Passed outputports. After a run, select both features in the Data Inspector and switch between them back and forth, looking at the geometry, at the bottom in the Feature Information.
You probably see differences in coordinate precision, which the ChangeDetector identifies as a modification.
Now you have to choose what kind of strategy you want to follow:
+1Hi Mark,
First of all, thank you for your input. I have read through everything that I can find related to the change detection, including the article you refer to, and still no luck. Hopefully I'm just missing some silly setting or something :-)
I have attached a zip file containing my fmw file and a file gdb containing one of my layers (the original).
Within my workspace, the first part downloads and extracts a file gdb from the public source, which my feature reader reads. The paths will have to be updated to a local path, but other than that, it should run straight through.
I have made 4 small attribute changes in the Original dataset, så it should, theoretically, just update those 4 features. Thanx :-)
With regards to the coordinateRounder - if my Update Detection Key Attribute is set and matches the original and revised data, then even changes in the coordinates should still be parsed as Updates, and not Delete all the original and Insert all the revised data, shouldn't it?
Hej Niels,
Awesome advise! I found the "BUG" - I had added a Coordinate rounder to my reivsed dataset but not my original, so there where extra decimals that triggered the change. Thanx :-)
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
