Skip to main content
Question

New to FME, question regarding change detector

C

I am trying to compare the differences between 2 csv files. They each have the same format but one contains an extra row of data. I have them linked as readers to the original and revised options of the transformer and 3 csv output writers for unchanged, added and deleted. I only get output for unchanged and the csv file is blank. Am I missing something? My search on the forums here says it should be very straight forward..

16 replies

A
Forum|alt.badge.img

I would suggest that you use DuplicateFilter Transformer (doc). You may need to put Sorter if front of that. Specify key attributes to compare/find duplicates, then changes will be output via a 'unique' port. To differentiate b/w files, you could set an attribute 'Source' or expose fme_dataset (which I believe will = to filename).

I hope it helps.

Sasha

itay
Supporter
Forum|alt.badge.img+17
  • itaySupporter
  • Supporter
  • January 31, 2017

from the look of it you have not set the transformer parameters, hence the yellow color of the ... button on the thransformer.

C
  • crow132
  • Author
  • January 31, 2017
alexandervstepa wrote:

I would suggest that you use DuplicateFilter Transformer (doc). You may need to put Sorter if front of that. Specify key attributes to compare/find duplicates, then changes will be output via a 'unique' port. To differentiate b/w files, you could set an attribute 'Source' or expose fme_dataset (which I believe will = to filename).

I hope it helps.

Sasha

I'm using v.2013 so it doesn't have the DuplicateFilter transformer.

 

I used sorter and DuplicateRemover and got 2 csv output files but both are blank..

 

C
  • crow132
  • Author
  • January 31, 2017
itay wrote:

from the look of it you have not set the transformer parameters, hence the yellow color of the ... button on the thransformer.

Ah you're right. Just changed that but now I'm not getting any output files?

 

 

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Contributor
  • January 31, 2017
crow132 wrote:
Ah you're right. Just changed that but now I'm not getting any output files?

 

 

It looks like you accidentally disabled the transformer, It's grey instead of blue.

 

 

C
  • crow132
  • Author
  • February 1, 2017
jdh wrote:
It looks like you accidentally disabled the transformer, It's grey instead of blue.

 

 

I did indeed. Enabled it then ran but only got unchanged output this time and was still blank.

 

 

takashi
Evangelist
  • takashiEvangelist
  • February 1, 2017

Hi @crow132, I would recommend you to investigate how the CSV reader has read the source tables. Connect two Inspector transformers to the reader feature types for each and run. You can then see the contents of the tables read by the reader with FME Data Inspector (or FME Universal Viewer in 2013?).

C
  • crow132
  • Author
  • February 1, 2017
takashi wrote:

Hi @crow132, I would recommend you to investigate how the CSV reader has read the source tables. Connect two Inspector transformers to the reader feature types for each and run. You can then see the contents of the tables read by the reader with FME Data Inspector (or FME Universal Viewer in 2013?).

I've ran it with the inspectors attached but the output shows me nothing. The readers do pick up the field names (name, job and school) so not sure what I'm missing?

 

takashi
Evangelist
  • takashiEvangelist
  • February 1, 2017
crow132 wrote:
I've ran it with the inspectors attached but the output shows me nothing. The readers do pick up the field names (name, job and school) so not sure what I'm missing?

 

FME Universal Viewer was not good to inspect non-spatial features. Change inspection application to FME Data Inspector, and then retry.

 

To change the option, open the FME Options dialog (Workbench menu: Tools > FME Options ) and check "Use FME Data Inspector".

 

C
  • crow132
  • Author
  • February 1, 2017
takashi wrote:
FME Universal Viewer was not good to inspect non-spatial features. Change inspection application to FME Data Inspector, and then retry.

 

To change the option, open the FME Options dialog (Workbench menu: Tools > FME Options ) and check "Use FME Data Inspector".

 

Ok so I changed that and this is the output. Stil get a blank csv output file even though the reader is reading the input files. Each of those tabs down the bottom return 1 row from the input csv files.

 

 

takashi
Evangelist
  • takashiEvangelist
  • February 1, 2017
crow132 wrote:
I've ran it with the inspectors attached but the output shows me nothing. The readers do pick up the field names (name, job and school) so not sure what I'm missing?

 

Looks like the reader has read the tables correctly. I guess you have set the three attributes (name, job, school) to the Group By parameter in the Inspector. If you clear the parameter, entire tables appear in the Table View and you can compare the two tables.

 

C
  • crow132
  • Author
  • February 1, 2017
takashi wrote:
Looks like the reader has read the tables correctly. I guess you have set the three attributes (name, job, school) to the Group By parameter in the Inspector. If you clear the parameter, entire tables appear in the Table View and you can compare the two tables.

 

Yes that worked, I can see the full tables of the two input csv files. However I'm still getting blank output files for deleted/unchanged/added. I'm thinking (like with python) that because each output has more than one (unchanged 5 and the others 2) output link after the transformer that they're overwriting themselves and this turns up with blank csvs (well it happens with python I've found). Would that be correct?

 

 

takashi
Evangelist
  • takashiEvangelist
  • February 1, 2017
crow132 wrote:
I've ran it with the inspectors attached but the output shows me nothing. The readers do pick up the field names (name, job and school) so not sure what I'm missing?

 

Next, inspect the output features from the ChangeDetector with the Inspectors as well.

 

I expect that:

 

  • The output features from the ChangeDetector have your desired attributes,
  • but the CSV writer won't write them into the destination files.
If so, the schema setting of writer feature types could be wrong. The writer feature type setting should look like this.

 

C
  • crow132
  • Author
  • February 1, 2017
takashi wrote:
Next, inspect the output features from the ChangeDetector with the Inspectors as well.

 

I expect that:

 

  • The output features from the ChangeDetector have your desired attributes,
  • but the CSV writer won't write them into the destination files.
If so, the schema setting of writer feature types could be wrong. The writer feature type setting should look like this.

 

Ahhh yes you are correct. Those are blank for all 3 writers. If I manually added in their names (name, job, school) my output CSVs come out correctly with visible data. Although I shouldn't have to type the user attribute names out, surely there is an easier way, especially with lots of attributes? In the format parameters tab I tried inherit from writer, yes and append from the source dataset but all came up with blank csvs.

 

 

takashi
Evangelist
  • takashiEvangelist
  • February 1, 2017
crow132 wrote:
Ahhh yes you are correct. Those are blank for all 3 writers. If I manually added in their names (name, job, school) my output CSVs come out correctly with visible data. Although I shouldn't have to type the user attribute names out, surely there is an easier way, especially with lots of attributes? In the format parameters tab I tried inherit from writer, yes and append from the source dataset but all came up with blank csvs.

 

 

Yes, there is a way not to type individual attributes manually, and the FME Knowledge Center provides a good tutorial.

 

Tutorial: Dynamic Workflows

 

Although the articles in the tutorial are created with FME 2016+ and details are different from FME 2013, basic concept is common to every version. Good luck!
david_r
Celebrity
  • david_rCelebrity
  • February 2, 2017
takashi wrote:
FME Universal Viewer was not good to inspect non-spatial features. Change inspection application to FME Data Inspector, and then retry.

 

To change the option, open the FME Options dialog (Workbench menu: Tools > FME Options ) and check "Use FME Data Inspector".

 

Agreed. As far as I've understood it, the Universal Viewer will no longer be available starting FME 2017.

Reply


Helpful Members This Week

Community Stats

31,477
Posts
119,329
Replies
39,092
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.