Everything is in the title.
Solved
Are features ran through pythonCaller sequentially or with multi-threading ?
Best answer by jeroenstiers
Sequentially
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+7I was afraid so !
Note that this kind of multi-threading is only beneficial if you have a lot of features and the workers do short and simple tasks (e.g. querying some online API, where most of the processing will take place server-side), otherwise there will not be a noticeable performance gain or even a degradation. If the tasks become more CPU intensive, you need multi-processing, which spawns new processes (that can also be multi-threaded) which can make use of more than 1 CPU core. So far, I haven't been able to make that work in FME (because that creates new "orphaned" FMESessions and those require a license?), but my guess is that it should be possible using Python 3+ perhaps? So far, I always used 2.7...
+7Actually, you can do multi-threading in a PythonCaller. I've once built something that uses a queueing system which spawns a couple of worker threads that deal with the incoming features. However, the features will always come in sequentially, like Jeroen said.
Note that this kind of multi-threading is only beneficial if you have a lot of features and the workers do short and simple tasks (e.g. querying some online API, where most of the processing will take place server-side), otherwise there will not be a noticeable performance gain or even a degradation. If the tasks become more CPU intensive, you need multi-processing, which spawns new processes (that can also be multi-threaded) which can make use of more than 1 CPU core. So far, I haven't been able to make that work in FME (because that creates new "orphaned" FMESessions and those require a license?), but my guess is that it should be possible using Python 3+ perhaps? So far, I always used 2.7...