Skip to main content
Solved

Remove duplicates as simply as possible

  • December 17, 2020
  • 6 replies
  • 26 views

acusig
Participant
Forum|alt.badge.img+1

Hello,

 

After struggling with several transformers, I can't figure out how to simply remove duplicates from 2 identical Excel files with exact same column names, etc ... but with different numbers of rows.

 

First file contains 3130 entries and second one 2926. What I want is to be able to extract the 204 missing lines to check their differences.

 

In my small brain, a simple transformer that checks missing entries between two columns and group them in an inspector should exists but it seems not to be complicated enough ... ;o)

 

Thanks for any help

Best answer by david_r

Here it is.

The problem is that the primary keys aren't called the same in both tables. As it is currently configured, the ChangeDetector will try to match based on the combination of both ID and Fuld_ID on both tables. Since one table only has ID and the other only has Fuld_ID, there are no matches.

Try renaming "Fuld_ID" to just "ID" on the "Data_Final" input before the ChangeDetector. You should then remove "Fuld_ID" from the ChangeDetector settings.

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

6 replies

david_r
Celebrity
  • 8392 replies
  • December 17, 2020

Did you try the ChangeDetector?


acusig
Participant
Forum|alt.badge.img+1
  • Author
  • Participant
  • 11 replies
  • December 17, 2020

Just tried but no way. I always get 3130 "deleted" and 2926 "inserted" whatever parameters I changed.


david_r
Celebrity
  • 8392 replies
  • December 17, 2020

Just tried but no way. I always get 3130 "deleted" and 2926 "inserted" whatever parameters I changed.

Can you share a screen dump of ChangeDetector configuration.


acusig
Participant
Forum|alt.badge.img+1
  • Author
  • Participant
  • 11 replies
  • December 17, 2020

Here it is.


david_r
Celebrity
  • 8392 replies
  • Best Answer
  • December 17, 2020

Here it is.

The problem is that the primary keys aren't called the same in both tables. As it is currently configured, the ChangeDetector will try to match based on the combination of both ID and Fuld_ID on both tables. Since one table only has ID and the other only has Fuld_ID, there are no matches.

Try renaming "Fuld_ID" to just "ID" on the "Data_Final" input before the ChangeDetector. You should then remove "Fuld_ID" from the ChangeDetector settings.


acusig
Participant
Forum|alt.badge.img+1
  • Author
  • Participant
  • 11 replies
  • December 17, 2020

It's working, thanks a lot for your help !

 

My bad, I changed the field's name to ease selection from two different sets but didn't thought (nor understood from docs) an exact match was needed for the transformer to work.