Skip to main content
Question

Data filtering

  • February 8, 2023
  • 3 replies
  • 24 views

fmesafe.podpora
Contributor
Forum|alt.badge.img+11

Hello. You don't know which transformer to use. I tried testfilter, but there you can't combine two lines and featuremerger, it's also not quite ideal. I need to filter data in a database with a condition:

- all data with the same uuid

- a point that has the same value of validfrom and validto

validto changes to a value of -1 second down

The result should be validfrom 20220908000000 and validto 20221005235959 in the first line

In the second line validfrom 20221006000000 and validto 99991231235959

Thank you very much.

3 replies

hkingsbury
Celebrity
Forum|alt.badge.img+63
  • Celebrity
  • 1620 replies
  • February 8, 2023

I think the key to solving your problem is to get the two features with the same uuid into one feature. A rough order of operations to do this is:

  1. for the first set of input features, prefix "validfrom" and "validto" with "a_"
  2. for the second set of input features, prefix "validfrom" and "validto" with "b_"
  3. using an aggregator, aggregate the two sets of feautres, grouping by the uuid and making sure you merge attributes from all features
  4. you can now use a tester to remove uuids that don't have two "parts" - set the tester to pass features that have both "a_validfrom" and "b_validfrom" values
  5. you can now do an subsequent tests against the "a_" and "b_" values
  6. assuming you want the original features, you can then take the features from the previous step, just keep the uuid and use these with a featuremerger (as the supplier) and pass the original input features into the requestor port, merging on the uuid

fmesafe.podpora
Contributor
Forum|alt.badge.img+11
  • Author
  • Contributor
  • 136 replies
  • February 9, 2023

Hi, that could be the solution, thank you very much for the detailed description, I will try it tomorrow and let you know.


fmesafe.podpora
Contributor
Forum|alt.badge.img+11
  • Author
  • Contributor
  • 136 replies
  • February 15, 2023

I think the key to solving your problem is to get the two features with the same uuid into one feature. A rough order of operations to do this is:

  1. for the first set of input features, prefix "validfrom" and "validto" with "a_"
  2. for the second set of input features, prefix "validfrom" and "validto" with "b_"
  3. using an aggregator, aggregate the two sets of feautres, grouping by the uuid and making sure you merge attributes from all features
  4. you can now use a tester to remove uuids that don't have two "parts" - set the tester to pass features that have both "a_validfrom" and "b_validfrom" values
  5. you can now do an subsequent tests against the "a_" and "b_" values
  6. assuming you want the original features, you can then take the features from the previous step, just keep the uuid and use these with a featuremerger (as the supplier) and pass the original input features into the requestor port, merging on the uuid

Hi,

Sorry for the late reply, but I was trying to test your tutorial today. I've tried different aggregator settings but can't seem to come up with a solution. I must be doing something wrong somewhere. I am attaching the WS test. Thank you