Skip to main content
Question

Iterate over a list of attribute names to compare in 2 datasets

  • August 7, 2023
  • 1 reply
  • 281 views

Forum|alt.badge.img

Dear All,

 

I am facing the following issue. I have 2 datasets (which are quite large) that contain a series of common attributes. I have a 3d table that stores the attribute names that I would like to compare (no geometry comparison).

 

What I would like to do sounds very simple (sorry, I am newbie to FME...) :

  1. loading the 2 data, add prefix, merging them based on a key attribute (they will then have attributes like source1_col1, source2_col1, source1_col2, source2_col2, etc.)
  2. load the data that contain the column names I want to compare (e.g. col1, col2, etc.)
  3. iterate over col1, col2, etc. and for each of this value use TestFilter testing sth like "@Value(source1_$(column)) = @Value(source2_$(column))", write the unfiltered in an excel file on "$column" sheet)

 

I have tried thousand things (custom transformer, ChangeDetector, workspace runner), but could not achieve exactly what I would like to do.

 

One solution works rather well : I have a parent workspace that loads the column names, and I use a WorkspaceRunner that takes as input each of the feature (=column name) and in which I load the 2 datasets and performs the comparison. The issue here is that it is not efficient at all -> for each column name, the 2 datasets are loaded each time.

Ideally, I would like to pass the 2 (already) loaded dataset to the WorkspaceRunner but so far as I know, this is not possible ??

Alternatively, I've tried to use a CustomTransformer which takes as input the 2 already loaded datasets, but then I don't know how to pass the column names inside the CustomTransformer ??

I was thinking about using a loop inside the CustomTransformer (and pass the column names as input to the CustomTransformer for the loop), but I did not succeed. Would it be possible to do it in this way ??

 

Or maybe my way to approach the problem is not a good one at all ?

 

In brief, I am quite desperate because I was initially thinking that it would be very simple to do, but I spend hours without obtaining significant results... So if any of you would have suggestions, I will be very grateful 🙂

 

Thanks in advance

 

NB : I have attached some of my trials and toy data

1 reply

virtualcitymatt
Celebrity
Forum|alt.badge.img+35

You might want to check out the AttributeExploder. What this will do is explode your attribtues into a list of attribute names (_attr_name) and attribute values (_attr_value).

 

You can then compare these with the Matcher. You'll just need to make sure that you skip a lot of the FME created attributes and only test the ones you want to test.

 

In the Matcher we can test where _attr_name, _attr_value, and "betpar"? are all the same and make sure that fme_dataset is different (to be sure we don't get matches from withing the same dataset).

 

Here's an example - I'm not 100% if it gets you what you want but I think at least the AttributeExploder is something worth looking into. image 

 


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings