Released
Sorter may have Group By and Parallel Processing Level Parameters
When the user needs to sort huge number of features grouping by an attribute value, it could be performed more efficiently if the Sorter had the Group By and Parallel Processing Level parameters. In the expected Sorter, when the user has set an attribute to the Group By parameter but not set sort keys, it should work to just make the order of features by group. If it could be done efficiently in conjunction with parallel processing, the performance of a following transformer that has the "Input is Ordered by Group" parameter could also be improved. e.g. Aggregator.
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+59- Safer
Already filed in our system as PR#19404. I've increased the priority and
posted a link to this page so we are notified if and when it is
completed. Note that you can do this now by adding the Sorter into a custom transformer. You can give the transformer a group-by and use parallel processing.
Thanks for your comment. I know the tip, but Workbench sometimes crashes when running a workspace containing a custom transformer that publishes parallel processing parameters. In my observation, the condition to crash is - Parallel Process By: <an attribute>, Parallel Processing Level: No Parallelism. Try with Workbench 2015.1.3.
I suspect that there is an issue on memory leak, and have reported a related issue to your support team before - C96706 (PR#56018). Fixed in 2016?
sorter-groupby-parallelprocessing-b15573.fmw
+22- Enthusiast
I would like to see this. It would be great, though I don't know how easy, if the sorter could sort and release some features batch by batch rather than blocking everything all at once. I have some huge processes to run and would be interested to see how best to do this - fairly new to big data!
Login to the community
No account yet? Create an account
An FME Account is required to contribute
LoginEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.