Skip to main content
Solved

Remove duplicates as simply as possible

  • December 17, 2020
  • 6 replies
  • 14 views

acusig
Participant
Forum|alt.badge.img+1

Hello,

 

After struggling with several transformers, I can't figure out how to simply remove duplicates from 2 identical Excel files with exact same column names, etc ... but with different numbers of rows.

 

First file contains 3130 entries and second one 2926. What I want is to be able to extract the 204 missing lines to check their differences.

 

In my small brain, a simple transformer that checks missing entries between two columns and group them in an inspector should exists but it seems not to be complicated enough ... ;o)

 

Thanks for any help

Best answer by david_r

acusig wrote:

Here it is.

The problem is that the primary keys aren't called the same in both tables. As it is currently configured, the ChangeDetector will try to match based on the combination of both ID and Fuld_ID on both tables. Since one table only has ID and the other only has Fuld_ID, there are no matches.

Try renaming "Fuld_ID" to just "ID" on the "Data_Final" input before the ChangeDetector. You should then remove "Fuld_ID" from the ChangeDetector settings.

View original
Did this help you find an answer to your question?

6 replies

david_r
Celebrity
  • December 17, 2020

Did you try the ChangeDetector?


acusig
Participant
Forum|alt.badge.img+1
  • Author
  • Participant
  • December 17, 2020

Just tried but no way. I always get 3130 "deleted" and 2926 "inserted" whatever parameters I changed.


david_r
Celebrity
  • December 17, 2020
acusig wrote:

Just tried but no way. I always get 3130 "deleted" and 2926 "inserted" whatever parameters I changed.

Can you share a screen dump of ChangeDetector configuration.


acusig
Participant
Forum|alt.badge.img+1
  • Author
  • Participant
  • December 17, 2020

Here it is.


david_r
Celebrity
  • Best Answer
  • December 17, 2020
acusig wrote:

Here it is.

The problem is that the primary keys aren't called the same in both tables. As it is currently configured, the ChangeDetector will try to match based on the combination of both ID and Fuld_ID on both tables. Since one table only has ID and the other only has Fuld_ID, there are no matches.

Try renaming "Fuld_ID" to just "ID" on the "Data_Final" input before the ChangeDetector. You should then remove "Fuld_ID" from the ChangeDetector settings.


acusig
Participant
Forum|alt.badge.img+1
  • Author
  • Participant
  • December 17, 2020

It's working, thanks a lot for your help !

 

My bad, I changed the field's name to ease selection from two different sets but didn't thought (nor understood from docs) an exact match was needed for the transformer to work.

 


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings