Are features ran through pythonCaller sequentially or with multi-threading ?

jeroenstiers wrote:

Sequentially

I was afraid so !

 

vchalmel wrote:

I was afraid so !

 

Actually, you can do multi-threading in a PythonCaller. I've once built something that uses a queueing system which spawns a couple of worker threads that deal with the incoming features. However, the features will always come in sequentially, like Jeroen said.

 

Note that this kind of multi-threading is only beneficial if you have a lot of features and the workers do short and simple tasks (e.g. querying some online API, where most of the processing will take place server-side), otherwise there will not be a noticeable performance gain or even a degradation. If the tasks become more CPU intensive, you need multi-processing, which spawns new processes (that can also be multi-threaded) which can make use of more than 1 CPU core. So far, I haven't been able to make that work in FME (because that creates new "orphaned" FMESessions and those require a license?), but my guess is that it should be possible using Python 3+ perhaps? So far, I always used 2.7...

 

 

geosander wrote:

Actually, you can do multi-threading in a PythonCaller. I've once built something that uses a queueing system which spawns a couple of worker threads that deal with the incoming features. However, the features will always come in sequentially, like Jeroen said.

 

Note that this kind of multi-threading is only beneficial if you have a lot of features and the workers do short and simple tasks (e.g. querying some online API, where most of the processing will take place server-side), otherwise there will not be a noticeable performance gain or even a degradation. If the tasks become more CPU intensive, you need multi-processing, which spawns new processes (that can also be multi-threaded) which can make use of more than 1 CPU core. So far, I haven't been able to make that work in FME (because that creates new "orphaned" FMESessions and those require a license?), but my guess is that it should be possible using Python 3+ perhaps? So far, I always used 2.7...

 

 

If you would have to run a pythoncode that requires multi-processing I would suggest to do it outside of FME. Just write your features to a file, call a .py file via the syscaller and read the result into FME again.