So I have a big job to process and I’m trying to work out some theoretical numbers for the client (a rough estimate on the processing time range depending on various factors). I want to leverage the fmeworker limit of 16 parallel threads (I’m assuming a machine with essentially unlimited resources) because the fme.exe limit is 8.
I have a batch job of many tiles with many rasters and features which all need to be processed at once. The easy thing to do is to just throw it to the workspace runner but the license only gives me 7-8 parallel threads (which is probably more than enough to be fair, most likely CPU will not be the bottle neck).
Anyways for fun I did a test to see what the limit was on the number of parallel groups to send through a custom transformer when parallel processing was on. I pretty quickly got the following error:
An error has occurred. Check the logfile above for details
f_6 (TransformFact): Transformer 'ThreadLimitTester': Unable to create worker; try reducing the parallelism level (currently 'EXTREME') or the number of groups (7663 seen so far, 1000 considered borderline, 10000 close to upper limit in best possible scenario)
I thought this was a nice error - “1000” considered borderline. This error came weather or not to process when the group changed or to wait until all received.
Has anyone run a job with 10s or 100’s of thousands of groups at all? I assume that each groups will take 1 - 2 mins to process. The more parallel threads the better here.
I can definitely design it in a way where I use a mixture of workspace runner plus custom transformer - but I don’t make it more complicated than it needs to be.
Unfortunately it seems like FME Server/FME Flow is not an option