Skip to main content
  • EN

Today I ran into a problem executing workspaces containing multiple versions of a custom transformer, which has parallel processing 'turned on'. I was wondering if someone has some thoughts on the matter I found.

I am using FME 2016.1 build 16494.

With all the instances of the transformer embedded the workspace runs fine, and reacts to changes in the parallel processing level (and the amount of groups the data is split into) quite like one would expect.

When I have the same custom transformers 'linked', the whole thing crashes and / or does not function at all.

As soon as the linked transformer is read, it requests for the amount of threads regulated by the default value of the Parallel Processing Level of the Public parameter I use to set the level. The value of the parameter in the workspace from which the transformer is set does not seem to affect this initialisation-value.

When set to moderate it means every transformer requests four threads, with six instances embedded leads to disaster on my machine. When set to 'no parallel processing' as the default-value in the transformer, the linked transformer requests 0 threads according to the log on startup, effectively making the workspace halt.

When I set the default level to minimal, and run the workspace it starts off, but eventually fails in some Windows error message. Regardless of the actual settings I apply through the published parameter for the level of parallel processing.

As said, embedding the transformers makes it work great again, and I have workspaces with single instances of linked custom transformers having parallel processing enabled behaving as expected. Only when I use multiple instances of the same linked transformer, things go bad.

Anyone some thoughts?

Kind regards,

Martin

Sounds like a case for safe support.

 

 

I have encountered issues with parallel processing in custom transformers when the number of groups becomes to high, which does lead to a windows error, but that should not be affected by whether the custom transformer is linked or embedded.

 

 

Not an answer per-se, but what I have found out is that sometimes these threads seem to compete against the same log-files. So when I minimize the logging levels (to a very minimum) it seems to work with much less error. Try turning off logging "completely" - yes, its not easy to debug afterwards.... Especially if you are using HTTPCallers you have to turn of Information due to the large number of logs per request. Will you attend the FMEUC in Vancouver? Would be great with a chat regarding this as Parallell Processing is quite handy, when it works as expected!

I have had the same problem with linked CT. I droped the parallel processing parameter, and sett all to default moderate, turned off the logging (thx to tips from Sigtill) and get it to work again.

Not an answer per-se, but what I have found out is that sometimes these threads seem to compete against the same log-files. So when I minimize the logging levels (to a very minimum) it seems to work with much less error. Try turning off logging "completely" - yes, its not easy to debug afterwards.... Especially if you are using HTTPCallers you have to turn of Information due to the large number of logs per request. Will you attend the FMEUC in Vancouver? Would be great with a chat regarding this as Parallell Processing is quite handy, when it works as expected!

Great tip @sigtill - Disabling logging did the trick for me running into a crash with the following during PP with more groups than usual

 

|FATAL |Precondition failed at c:\\code\\fme\\foundation\\framework\\engine\\stffeatimp.cpp:6899. Expression: false

Reply


Community Stats

30,159
Posts
114,480
Replies
37,806
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.