Skip to main content
  • EN

Dear All,

 

I am facing the following issue. I have 2 datasets (which are quite large) that contain a series of common attributes. I have a 3d table that stores the attribute names that I would like to compare (no geometry comparison).

 

What I would like to do sounds very simple (sorry, I am newbie to FME...) :

  1. loading the 2 data, add prefix, merging them based on a key attribute (they will then have attributes like source1_col1, source2_col1, source1_col2, source2_col2, etc.)
  2. load the data that contain the column names I want to compare (e.g. col1, col2, etc.)
  3. iterate over col1, col2, etc. and for each of this value use TestFilter testing sth like "@Value(source1_$(column)) = @Value(source2_$(column))", write the unfiltered in an excel file on "$column" sheet)

 

I have tried thousand things (custom transformer, ChangeDetector, workspace runner), but could not achieve exactly what I would like to do.

 

One solution works rather well : I have a parent workspace that loads the column names, and I use a WorkspaceRunner that takes as input each of the feature (=column name) and in which I load the 2 datasets and performs the comparison. The issue here is that it is not efficient at all -> for each column name, the 2 datasets are loaded each time.

Ideally, I would like to pass the 2 (already) loaded dataset to the WorkspaceRunner but so far as I know, this is not possible ??

Alternatively, I've tried to use a CustomTransformer which takes as input the 2 already loaded datasets, but then I don't know how to pass the column names inside the CustomTransformer ??

I was thinking about using a loop inside the CustomTransformer (and pass the column names as input to the CustomTransformer for the loop), but I did not succeed. Would it be possible to do it in this way ??

 

Or maybe my way to approach the problem is not a good one at all ?

 

In brief, I am quite desperate because I was initially thinking that it would be very simple to do, but I spend hours without obtaining significant results... So if any of you would have suggestions, I will be very grateful 🙂

 

Thanks in advance

 

NB : I have attached some of my trials and toy data

You might want to check out the AttributeExploder. What this will do is explode your attribtues into a list of attribute names (_attr_name) and attribute values (_attr_value).

 

You can then compare these with the Matcher. You'll just need to make sure that you skip a lot of the FME created attributes and only test the ones you want to test.

 

In the Matcher we can test where _attr_name, _attr_value, and "betpar"? are all the same and make sure that fme_dataset is different (to be sure we don't get matches from withing the same dataset).

 

Here's an example - I'm not 100% if it gets you what you want but I think at least the AttributeExploder is something worth looking into. image 

 

Reply


Community Stats

30,112
Posts
114,305
Replies
37,745
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.