Skip to main content
Question

New to FME, question regarding change detector

  • January 31, 2017
  • 16 replies
  • 148 views
C

I am trying to compare the differences between 2 csv files. They each have the same format but one contains an extra row of data. I have them linked as readers to the original and revised options of the transformer and 3 csv output writers for unchanged, added and deleted. I only get output for unchanged and the csv file is blank. Am I missing something? My search on the forums here says it should be very straight forward..

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

16 replies

A
Forum|alt.badge.img

I would suggest that you use DuplicateFilter Transformer (doc). You may need to put Sorter if front of that. Specify key attributes to compare/find duplicates, then changes will be output via a 'unique' port. To differentiate b/w files, you could set an attribute 'Source' or expose fme_dataset (which I believe will = to filename).

I hope it helps.

Sasha

itay
Supporter
Forum|alt.badge.img+18
  • itaySupporter
  • Supporter
  • January 31, 2017

from the look of it you have not set the transformer parameters, hence the yellow color of the ... button on the thransformer.

C
  • crow132
  • Author
  • January 31, 2017

I would suggest that you use DuplicateFilter Transformer (doc). You may need to put Sorter if front of that. Specify key attributes to compare/find duplicates, then changes will be output via a 'unique' port. To differentiate b/w files, you could set an attribute 'Source' or expose fme_dataset (which I believe will = to filename).

I hope it helps.

Sasha

I'm using v.2013 so it doesn't have the DuplicateFilter transformer.

 

I used sorter and DuplicateRemover and got 2 csv output files but both are blank..

 

C
  • crow132
  • Author
  • January 31, 2017

from the look of it you have not set the transformer parameters, hence the yellow color of the ... button on the thransformer.

Ah you're right. Just changed that but now I'm not getting any output files?

 

 

jdh
Contributor
Forum|alt.badge.img+40
  • jdhContributor
  • Contributor
  • January 31, 2017
Ah you're right. Just changed that but now I'm not getting any output files?

 

 

It looks like you accidentally disabled the transformer, It's grey instead of blue.

 

 

C
  • crow132
  • Author
  • February 1, 2017
It looks like you accidentally disabled the transformer, It's grey instead of blue.

 

 

I did indeed. Enabled it then ran but only got unchanged output this time and was still blank.

 

 

takashi
Celebrity
  • takashiCelebrity
  • February 1, 2017

Hi @crow132, I would recommend you to investigate how the CSV reader has read the source tables. Connect two Inspector transformers to the reader feature types for each and run. You can then see the contents of the tables read by the reader with FME Data Inspector (or FME Universal Viewer in 2013?).

Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
C
  • crow132
  • Author
  • February 1, 2017

Hi @crow132, I would recommend you to investigate how the CSV reader has read the source tables. Connect two Inspector transformers to the reader feature types for each and run. You can then see the contents of the tables read by the reader with FME Data Inspector (or FME Universal Viewer in 2013?).

I've ran it with the inspectors attached but the output shows me nothing. The readers do pick up the field names (name, job and school) so not sure what I'm missing?

 

takashi
Celebrity
  • takashiCelebrity
  • February 1, 2017
I've ran it with the inspectors attached but the output shows me nothing. The readers do pick up the field names (name, job and school) so not sure what I'm missing?

 

FME Universal Viewer was not good to inspect non-spatial features. Change inspection application to FME Data Inspector, and then retry.

 

To change the option, open the FME Options dialog (Workbench menu: Tools > FME Options ) and check "Use FME Data Inspector".

 

Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
C
  • crow132
  • Author
  • February 1, 2017
FME Universal Viewer was not good to inspect non-spatial features. Change inspection application to FME Data Inspector, and then retry.

 

To change the option, open the FME Options dialog (Workbench menu: Tools > FME Options ) and check "Use FME Data Inspector".

 

Ok so I changed that and this is the output. Stil get a blank csv output file even though the reader is reading the input files. Each of those tabs down the bottom return 1 row from the input csv files.

 

 

takashi
Celebrity
  • takashiCelebrity
  • February 1, 2017
I've ran it with the inspectors attached but the output shows me nothing. The readers do pick up the field names (name, job and school) so not sure what I'm missing?

 

Looks like the reader has read the tables correctly. I guess you have set the three attributes (name, job, school) to the Group By parameter in the Inspector. If you clear the parameter, entire tables appear in the Table View and you can compare the two tables.

 

Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
C
  • crow132
  • Author
  • February 1, 2017
Looks like the reader has read the tables correctly. I guess you have set the three attributes (name, job, school) to the Group By parameter in the Inspector. If you clear the parameter, entire tables appear in the Table View and you can compare the two tables.

 

Yes that worked, I can see the full tables of the two input csv files. However I'm still getting blank output files for deleted/unchanged/added. I'm thinking (like with python) that because each output has more than one (unchanged 5 and the others 2) output link after the transformer that they're overwriting themselves and this turns up with blank csvs (well it happens with python I've found). Would that be correct?

 

 

takashi
Celebrity
  • takashiCelebrity
  • February 1, 2017
I've ran it with the inspectors attached but the output shows me nothing. The readers do pick up the field names (name, job and school) so not sure what I'm missing?

 

Next, inspect the output features from the ChangeDetector with the Inspectors as well.

 

I expect that:

 

  • The output features from the ChangeDetector have your desired attributes,
  • but the CSV writer won't write them into the destination files.
If so, the schema setting of writer feature types could be wrong. The writer feature type setting should look like this.

 

Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
C
  • crow132
  • Author
  • February 1, 2017
Next, inspect the output features from the ChangeDetector with the Inspectors as well.

 

I expect that:

 

  • The output features from the ChangeDetector have your desired attributes,
  • but the CSV writer won't write them into the destination files.
If so, the schema setting of writer feature types could be wrong. The writer feature type setting should look like this.

 

Ahhh yes you are correct. Those are blank for all 3 writers. If I manually added in their names (name, job, school) my output CSVs come out correctly with visible data. Although I shouldn't have to type the user attribute names out, surely there is an easier way, especially with lots of attributes? In the format parameters tab I tried inherit from writer, yes and append from the source dataset but all came up with blank csvs.

 

 

takashi
Celebrity
  • takashiCelebrity
  • February 1, 2017
Ahhh yes you are correct. Those are blank for all 3 writers. If I manually added in their names (name, job, school) my output CSVs come out correctly with visible data. Although I shouldn't have to type the user attribute names out, surely there is an easier way, especially with lots of attributes? In the format parameters tab I tried inherit from writer, yes and append from the source dataset but all came up with blank csvs.

 

 

Yes, there is a way not to type individual attributes manually, and the FME Knowledge Center provides a good tutorial.

 

Tutorial: Dynamic Workflows

 

Although the articles in the tutorial are created with FME 2016+ and details are different from FME 2013, basic concept is common to every version. Good luck!
Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
david_r
Celebrity
  • david_rCelebrity
  • February 2, 2017
FME Universal Viewer was not good to inspect non-spatial features. Change inspection application to FME Data Inspector, and then retry.

 

To change the option, open the FME Options dialog (Workbench menu: Tools > FME Options ) and check "Use FME Data Inspector".

 

Agreed. As far as I've understood it, the Universal Viewer will no longer be available starting FME 2017.

Community Stats

32,221
Posts
122,361
Replies
39,916
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.