Question

Ordered input and Parallelism

J
Badge

FME 2016 (but happens in 2017 too).

A simple workflow:

Input -> Dissolver -> Aggregator ->Output

Both the Dissolver and Aggregator are set to group by an ID, the features are ordered by that ID, and the transformers are set to know this. Parallelism is off for the Dissolver. The result is that features flow smoothly through the workflow, and there is no caching of features at any point. Exactly as desired.

However, if I then change the Dissolver to use Parallelism (any level) (to see if I get more speed), and change nothing else, this flowing stops. Now, all features appear to be cached by the Dissolver after the dissolving process has finished by the child threads, and are only released to the Aggregator at the end of the workflow.

Anyone else experience this? Is this the expected behaviour? I can't see anything in the docs about the Input Ordered and Parallelism being mutually exclusive, but maybe I missed it.

6 replies

erik_jan
Rank
Userlevel 2
Badge +12

Could it be that the features, coming from the Dissolver, are no longer ordered by ID as some parallel processes take longer than others?

Adding a Sorter before the Aggregator could solve that.

J
Badge

Could it be that the features, coming from the Dissolver, are no longer ordered by ID as some parallel processes take longer than others?

Adding a Sorter before the Aggregator could solve that.

Good thought, but I'd then expect an error from the Aggregator about not-in-order features, because the features should still leave the Dissolver as soon as it has finished with them.

 

Tested with the Sorter and as expected, no difference. The Dissolver is storing everything, I think before it even *starts* processing.

 

 

erik_jan
Rank
Userlevel 2
Badge +12

Does the same happen if you wrap the Dissolver as an Embedded Custom Transformer and set the parallel processing for the custom transformer instead?

J
Badge

Does the same happen if you wrap the Dissolver as an Embedded Custom Transformer and set the parallel processing for the custom transformer instead?

Works (as expected?) if the Dissolver is wrapped into a custom transformer and that parallel processing is used. That doesn't seem sensible.

 

G
Badge +3

If you create a custom transformer and put the dissolver in that you get the parallel processing option also in the advanced section in the navigator panel of the custom.

I tried it, it then kind of staccato outputs the objects per group.

sorted ).

Final output will still be after it finishes though.

Actually without the custom , it processed objects sin the same way, with different staccato output values.

Process time was the same, no gain.

Dissolving without parallel was actually 5 seconds faster (39+ vs 35+ secs on 1668149 objects into 10 groups)

If you want output per group I guess you need to use a workspace caller.

Dissolver doesn't seem to benefit from parallel processing.

takashi
Rank
Userlevel 2
Badge +17

Hi @jonathan, the child threads will be launched for each group in the input order, but it's not guaranteed that they will finish in the same order. I therefore think it's intentional that the Dissolver waits for completion of all child threads before starting output, in order to output the resulting features in the same order as input.

It's just my guess. Hope someone from Safe elaborates this.

Reply


Community Stats

28,670
Posts
109,366
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.