A workspace has an aggregator that is aggregating features using multiple group by attributes. One of the group by attributes is ordered, but the others are not. Any suggestions on how this could be processed so that the ordered feature could be considered, rather than the aggregator holding on to everything?
Making the aggregator less 'blocking' when using multiple group by attributes
Userlevel 2
+17
- Contributor
- 7538 replies
-
6 November 2018
I think the only way is to sort the features by all the Group By attributes beforehand and set Yes to the Input is Ordered by Group parameter in the Aggregator.
Userlevel 6
+33
- Celebrity
- 2347 replies
-
6 November 2018
I think the only way is to sort the features by all the Group By attributes beforehand and set Yes to the Input is Ordered by Group parameter in the Aggregator.
Or read the source data in sessions grouped by, process and aggregate the features grouped by. But not sure if this works with your data / workflow.
Or temporary write features and then read features in sessions by group by attribute. But depending on volume and storing solutions maybe to much I/O.
Userlevel 1
+21
- Contributor
- 3079 replies
-
6 November 2018
I think the only way is to sort the features by all the Group By attributes beforehand and set Yes to the Input is Ordered by Group parameter in the Aggregator.
Does that not just move the blocking problem to the sorter?
Userlevel 2
+17
- Contributor
- 7538 replies
-
6 November 2018
Does that not just move the blocking problem to the sorter?
I don't think you can avoid the blocking problem. However, the performance could be better if you inserted a Sorter before the Aggregator, so I think it's worth to try.
Userlevel 4
+13
- Contributor
- 3725 replies
-
6 November 2018
I see the issue and it is a tricky one. Once one of the attributes changes, you'd like to flush out everything in the aggregator because you know that one attribute was ordered. The only trick I could think of was to make use of custom transformers. If you make a custom xformer of just an aggregator, and then turn on its "parallel group by" to have just the one attribute you know is ordered, and indicate that it is ordered, then you *should* get what you want. I didn't try but see below and attached. Let us know if this works.
Userlevel 1
+21
- Contributor
- 3079 replies
-
6 November 2018
I see the issue and it is a tricky one. Once one of the attributes changes, you'd like to flush out everything in the aggregator because you know that one attribute was ordered. The only trick I could think of was to make use of custom transformers. If you make a custom xformer of just an aggregator, and then turn on its "parallel group by" to have just the one attribute you know is ordered, and indicate that it is ordered, then you *should* get what you want. I didn't try but see below and attached. Let us know if this works.
I'll take a look at that option tomorrow, I'm trying to remove the requirement to use a workspace runner and the only issue now is the aggregator.
Userlevel 1
+21
- Contributor
- 3079 replies
-
9 November 2018
I'll take a look at that option tomorrow, I'm trying to remove the requirement to use a workspace runner and the only issue now is the aggregator.
Putting the aggregator inside the custom transfomer actually adds 30 seconds to the time for half a million features. The only difference i saw in output was that the features came out the aggregator ordered by the parallel group by attribute
Userlevel 4
+13
- Contributor
- 3725 replies
-
10 November 2018
Putting the aggregator inside the custom transfomer actually adds 30 seconds to the time for half a million features. The only difference i saw in output was that the features came out the aggregator ordered by the parallel group by attribute
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.