Hello I just started learning FME. What I need to do is to find any difference in attributes between two versions of the same feature class. In details, I have a copy of a FGDB feature from one vendor and another copy from another vendor. I want to create FME workspace to be able to find any difference between these two versions and write the difference to a csv file. Both versions have the same schema and I am only interested in the attribute difference at this point. The feature class has a GlobalID which is unique in both versions. The output csv should be able to give info like new records from version 1, new records from version 2, attribute difference for Field 1 for record {aud1123ksfs34400}, and so on. Currently I have a python script for this purpose, but my new manager wants to use a FME job to automate this. What type of reader, transformer and writer I should use to achieve this purpose? Please help!
+10- Contributor
I think you can use a serie of ChangeDetectors for this.
First compare both featureclasses on GlobalID only to find the new records in each featureclass. Then compare the Unchanges output again on all attributes and geometry.
Use a FGDB reader and Csv writer for in and output
I think you can use a serie of ChangeDetectors for this.
First compare both featureclasses on GlobalID only to find the new records in each featureclass. Then compare the Unchanges output again on all attributes and geometry.
Use a FGDB reader and Csv writer for in and output
Thanks koen. what is the name for Transformer NEW_in_A and New_in_B in your diagram?
+10- Contributor
Thanks koen. what is the name for Transformer NEW_in_A and New_in_B in your diagram?
In my exampe it's an attributecreator, but those transformers are irrelevant, thats just for the picture.
It's the ChangeDetectors who do the work.
+5There are various transformers you can use for this. The ChangeDetector is the most obvious one.
The Matcher and DuplicateFilter are others you can use, though you then need to test the value of fme_feature_type to see where each original feature came from.
Finally, if performance is important, you could try the FeatureMerger. Technically this transformer is for merging data on a common key, but obviously if there is no merge then there are no matching attributes, so it is a test of sorts and way faster transformer than the ChangeDetector.
So, lots of options, but the ChangeDetector is what I suggest to try first.
+5There are various transformers you can use for this. The ChangeDetector is the most obvious one.
The Matcher and DuplicateFilter are others you can use, though you then need to test the value of fme_feature_type to see where each original feature came from.
Finally, if performance is important, you could try the FeatureMerger. Technically this transformer is for merging data on a common key, but obviously if there is no merge then there are no matching attributes, so it is a test of sorts and way faster transformer than the ChangeDetector.
So, lots of options, but the ChangeDetector is what I suggest to try first.
Oh - also see this blog post - tip #6 covers some of what I've said but in more detail.
I think you can use a serie of ChangeDetectors for this.
First compare both featureclasses on GlobalID only to find the new records in each featureclass. Then compare the Unchanges output again on all attributes and geometry.
Use a FGDB reader and Csv writer for in and output
I have tested the above workspace. I can get added, deleted, and attribute-modified records. But for attribute-modified records, is there a way to know which attribute or attributes have been changed for each feature? for instance, for feature with an EventID of {120oui11000}, field A changed from "12345" to "54321", field B changed from "On" to "Off", and so on.
There are various transformers you can use for this. The ChangeDetector is the most obvious one.
The Matcher and DuplicateFilter are others you can use, though you then need to test the value of fme_feature_type to see where each original feature came from.
Finally, if performance is important, you could try the FeatureMerger. Technically this transformer is for merging data on a common key, but obviously if there is no merge then there are no matching attributes, so it is a test of sorts and way faster transformer than the ChangeDetector.
So, lots of options, but the ChangeDetector is what I suggest to try first.
thanks Mark for your response. do you know if there is a way in FME to capture specific attribute changes? For instance, for feature with an EventID of {120oui11000}, field A changed from "12345" to "54321", field B changed from "On" to "Off", and so on.
+5thanks Mark for your response. do you know if there is a way in FME to capture specific attribute changes? For instance, for feature with an EventID of {120oui11000}, field A changed from "12345" to "54321", field B changed from "On" to "Off", and so on.
I think to pick up individual changes you would need to use a ChangeDetector for each attribute, so you could find out which specific attribute is different. To be honest that's not very efficient so I will file an enhancement request with our developers to ask if they can tag features with which component is different. Doesn't do much for you in the short term, but it would help in the future.
Alternatively, you could try using the AttributeSplitter before the ChangeDetector. That would give you a separate feature per attribute, so when a feature fails to match, you know which attribute it is and therefore which attribute is different. (it's like pratap's schema reader idea, just a little easier to implement).
Hope that helps.
I think to pick up individual changes you would need to use a ChangeDetector for each attribute, so you could find out which specific attribute is different. To be honest that's not very efficient so I will file an enhancement request with our developers to ask if they can tag features with which component is different. Doesn't do much for you in the short term, but it would help in the future.
Alternatively, you could try using the AttributeSplitter before the ChangeDetector. That would give you a separate feature per attribute, so when a feature fails to match, you know which attribute it is and therefore which attribute is different. (it's like pratap's schema reader idea, just a little easier to implement).
Hope that helps.
Appreciate your response, Mark!
Hi @fmenewbie
we improved ChangeDetector in FME 2019. Now your problem can be solved with a single ChangeDetector that creates report with all changes between the original dataset and the revised dataset.
Please give FME 2019 beta a try.
Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.