I have a StatisticsCalculator that determines the min and max time values in a data set. If I feed the time.min and time.max into a TestFilter to filter out values from another table which are outside of the time.min and time.max range, the filter doesn't filter properly but if I use the actual time value is does. What do I need to do to get the TestFilter to work with the FeatureAttribute? 
TestFilter doesn't filter properly when using FeatureAttribute but does work when using the actual value of the FeatureAttribute
Userlevel 5
+29
- Celebrity
- 1108 replies
-
16 February 2023
From what I can see in the screenshots, it seems fine. Can you share some sample data and the workbench?
Yes. I will share. I made a reduced version and still have the same issue. I just need to figure out how to package the data and workflow so I can upload it.
I'm not sure if these files will work...
Userlevel 5
+29
- Celebrity
- 1108 replies
-
16 February 2023
Those files worked fine. I've figure out what the problem is.
One thing to remember about FME is that it (generally) operates on a feature by feature level. So (most) operations occur on each feature in isolation. In the workbench you shared, you're passing 14825 features into your test filter(s) however only one of those features has the time.min/max attributes. That one feature is the one from the statisticscalculator
The remaining features do not have those attributes.
Passing two connections, or streams of data, into one port does not do a join/merge/aggregate. The tester is running the statement across all features in isolation and because all but one feature doesn't have the two attributes, you're getting the wrong result.
What you need to do first, is merge the two data streams together, putting the time.min/max attributes onto the other features.
There's many ways to do this, the way I would do it, is using the featuremerger. the time data goes into the Requestor port, and the min/max goes into the Supplier port (the requestor requests data from the supplier).
Because you're only passing in one value to the supplier and you want that on all requestors, set the join to 1:1.
Then, if you inspect the data coming out of the merged port, you can see all features have the min/max attribute and will therefore be processed correctly by the testfilter
Thank you for taking a look at this issue. I'm still not completely clear on the issue though - I thought the filter would take on the numeric value of "time.min" but it doesn't seem that is what is happening or is suppose to happen.
It is interesting how this has gone full circle for me. For this case, I was using the TestFilter to try to remove records outside of the min/max values to reduce the volume of records before using a FeatureMerger just as you propose to join the records. I had sucessfully use the method you propose to join records in the past but now, in some cases, the records for evaluation were getting so large the FeatureMerger was creating millions of records which was very, very slow.
Someone had suggested using InlineQuerier but I wasn't that familiar with the SQL proposed to do so. I think I'll give the InlineQuerier a go again to see if that will work fast enough with large records
My previous post on this issue:
Thank you,
Gil
Userlevel 5
+29
- Celebrity
- 1108 replies
-
19 February 2023
Thank you for taking a look at this issue. I'm still not completely clear on the issue though - I thought the filter would take on the numeric value of "time.min" but it doesn't seem that is what is happening or is suppose to happen.
It is interesting how this has gone full circle for me. For this case, I was using the TestFilter to try to remove records outside of the min/max values to reduce the volume of records before using a FeatureMerger just as you propose to join the records. I had sucessfully use the method you propose to join records in the past but now, in some cases, the records for evaluation were getting so large the FeatureMerger was creating millions of records which was very, very slow.
Someone had suggested using InlineQuerier but I wasn't that familiar with the SQL proposed to do so. I think I'll give the InlineQuerier a go again to see if that will work fast enough with large records
My previous post on this issue:
Thank you,
Gil
It does take the value, but it only looks at one feature at a time. So only one of the features had the time.min/max values on it. The featuremerger in my example is transferring that value from the one features onto the other features that don't have that value
Userlevel 5
+29
- Celebrity
- 1108 replies
-
19 February 2023
Thank you for taking a look at this issue. I'm still not completely clear on the issue though - I thought the filter would take on the numeric value of "time.min" but it doesn't seem that is what is happening or is suppose to happen.
It is interesting how this has gone full circle for me. For this case, I was using the TestFilter to try to remove records outside of the min/max values to reduce the volume of records before using a FeatureMerger just as you propose to join the records. I had sucessfully use the method you propose to join records in the past but now, in some cases, the records for evaluation were getting so large the FeatureMerger was creating millions of records which was very, very slow.
Someone had suggested using InlineQuerier but I wasn't that familiar with the SQL proposed to do so. I think I'll give the InlineQuerier a go again to see if that will work fast enough with large records
My previous post on this issue:
Thank you,
Gil
Re the "slowness" of the featuremerger. It sounds like you could optimise the order of your operation. This will likely give a considerable performance increase.
By default, a featuremerger is a blocking transformer (https://community.safe.com/s/article/blocking-transformers-and-the-flow-of-features). It waits for all features to get to the input ports before processing. You can drastically speed this up be setting the featuremerger to "suppliers first". This however ONLY WORKS if your supplier reach the featuremerger FIRST. If requestors start hitting the featuremerger and there are no suppliers, they have no feautres to join to so will pass through and come out the unmerged port.
Have a read through the featruemerger documentation and that should provide some more details - https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Transformers/Transformers/featuremerger.htm
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.