The number of concurrent processes is indeed limited by license level and machine architecture and it has been my understanding that if there’s more than can be handled in one go they’ll just have to wait. 

Each group is communicates with the main process over an ephemeral port. The port range is limited and I think this is what is the limiting factor although I’m not 100% on that at all.

When groups are ordered and you are processing by group I would hope the FME would just reuse ports, however, it’s possible that this isn’t happening and this is why this issue is possing up. I’ve run into the same issue are what you’ve hit and due to time pressure I was forces to use a WorkspaceRunner instead I’m not really sure if this is an option for you though.

 

​@virtualcitymatt thanks for that answer. I was considering the WorkspaceRunner but parallel processing is faster (if it works). I was able to “batch” my parallel processing calls to reduce the number but you end up losing visibility of what’s going on inside the custom transformer. It would be nice if FME would just handle it and not go beyond it’s own limits and die!

Interesting observation on the port range.

​@redgeographics I thought FME would just wait as well if the limit got exceeded but that’s not what’s happening for me. FME is just going beyond its limits and crashing.

Yeah, I feel that that shouldn’t happen really.