A workspace has an aggregator that is aggregating features using multiple group by attributes. One of the group by attributes is ordered, but the others are not. Any suggestions on how this could be processed so that the ordered feature could be considered, rather than the aggregator holding on to everything?
Question
Making the aggregator less 'blocking' when using multiple group by attributes
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
- 7843 replies
I think the only way is to sort the features by all the Group By attributes beforehand and set Yes to the Input is Ordered by Group parameter in the Aggregator.
+60- 2940 replies
I think the only way is to sort the features by all the Group By attributes beforehand and set Yes to the Input is Ordered by Group parameter in the Aggregator.
Or read the source data in sessions grouped by, process and aggregate the features grouped by. But not sure if this works with your data / workflow.
Or temporary write features and then read features in sessions by group by attribute. But depending on volume and storing solutions maybe to much I/O.
+44- Influencer
- 3429 replies
I think the only way is to sort the features by all the Group By attributes beforehand and set Yes to the Input is Ordered by Group parameter in the Aggregator.
Does that not just move the blocking problem to the sorter?
- 7843 replies
Does that not just move the blocking problem to the sorter?
I don't think you can avoid the blocking problem. However, the performance could be better if you inserted a Sorter before the Aggregator, so I think it's worth to try.
+20- Safer
- 3719 replies
I see the issue and it is a tricky one. Once one of the attributes changes, you'd like to flush out everything in the aggregator because you know that one attribute was ordered. The only trick I could think of was to make use of custom transformers. If you make a custom xformer of just an aggregator, and then turn on its "parallel group by" to have just the one attribute you know is ordered, and indicate that it is ordered, then you *should* get what you want. I didn't try but see below and attached. Let us know if this works.
+44- Influencer
- 3429 replies
I see the issue and it is a tricky one. Once one of the attributes changes, you'd like to flush out everything in the aggregator because you know that one attribute was ordered. The only trick I could think of was to make use of custom transformers. If you make a custom xformer of just an aggregator, and then turn on its "parallel group by" to have just the one attribute you know is ordered, and indicate that it is ordered, then you *should* get what you want. I didn't try but see below and attached. Let us know if this works.
I'll take a look at that option tomorrow, I'm trying to remove the requirement to use a workspace runner and the only issue now is the aggregator.
+44- Influencer
- 3429 replies
I'll take a look at that option tomorrow, I'm trying to remove the requirement to use a workspace runner and the only issue now is the aggregator.
Putting the aggregator inside the custom transfomer actually adds 30 seconds to the time for half a million features. The only difference i saw in output was that the features came out the aggregator ordered by the parallel group by attribute
+20- Safer
- 3719 replies
Putting the aggregator inside the custom transfomer actually adds 30 seconds to the time for half a million features. The only difference i saw in output was that the features came out the aggregator ordered by the parallel group by attribute