Skip to main content
Solved

Understanding parallel processing in FME Form

nic_ran
Contributor
Forum|alt.badge.img+16

I’m using FME 2023.2.

I thought I understood parallel processing but clearly there are some intricacies that I don’t understand and I’d appreciate any insights people can provide.

I have a custom transformer that processes Lidar tiles and takes some time to run, so I enabled parallel processing and saw some significant performance improvements with my test set of 20 tiles.

However, when I enable the full data set of 10,000+ files I get an error:

2025-03-28 11:26:41|  63.6|  0.0|ERROR |Could not bind port to socket
2025-03-28 11:26:41|  63.6|  0.0|ERROR |Failed to create transporter
2025-03-28 11:26:41|  63.6|  0.0|ERROR |TransporterSender (TransporterFactory): An error has occurred. Check the logfile above for details
2025-03-28 11:26:41|  63.6|  0.0|ERROR |An error has occurred. Check the logfile above for details
2025-03-28 11:26:41|  63.6|  0.0|ERROR |f_25 (TransformFact): Transformer 'ProcessSingleTile': Unable to create worker; try reducing the parallelism level (currently 'MODERATE') or the number of groups (7521 seen so far, 1000 considered borderline, 10000 close to upper limit in best possible scenario)

So, what I don’t understand is, why does FME try to establish over 7000 processes? I thought the number of processes was limited based on the machine architecture (# cpu’s) and the license level?

Best answer by virtualcitymatt

Each group is communicates with the main process over an ephemeral port. The port range is limited and I think this is what is the limiting factor although I’m not 100% on that at all.

When groups are ordered and you are processing by group I would hope the FME would just reuse ports, however, it’s possible that this isn’t happening and this is why this issue is possing up. I’ve run into the same issue are what you’ve hit and due to time pressure I was forces to use a WorkspaceRunner instead I’m not really sure if this is an option for you though.

 

View original
danilo_fme
crutledge
2 people upvoted this
  • danilo_fme
    danilo_fme
  • crutledge
    crutledge
Did this help you find an answer to your question?

5 replies

redgeographics
Celebrity
Forum|alt.badge.img+49

The number of concurrent processes is indeed limited by license level and machine architecture and it has been my understanding that if there’s more than can be handled in one go they’ll just have to wait. 

danilo_fme
1 person upvoted this
  • danilo_fme
    danilo_fme
virtualcitymatt
Celebrity
Forum|alt.badge.img+34

Each group is communicates with the main process over an ephemeral port. The port range is limited and I think this is what is the limiting factor although I’m not 100% on that at all.

When groups are ordered and you are processing by group I would hope the FME would just reuse ports, however, it’s possible that this isn’t happening and this is why this issue is possing up. I’ve run into the same issue are what you’ve hit and due to time pressure I was forces to use a WorkspaceRunner instead I’m not really sure if this is an option for you though.

 

redgeographics
danilo_fme
2 people upvoted this
  • redgeographics
    redgeographics
  • danilo_fme
    danilo_fme
nic_ran
Contributor
Forum|alt.badge.img+16
  • nic_ranContributor
  • Author
  • Contributor
  • April 2, 2025

​@virtualcitymatt thanks for that answer. I was considering the WorkspaceRunner but parallel processing is faster (if it works). I was able to “batch” my parallel processing calls to reduce the number but you end up losing visibility of what’s going on inside the custom transformer. It would be nice if FME would just handle it and not go beyond it’s own limits and die!

 

Interesting observation on the port range.

nic_ran
Contributor
Forum|alt.badge.img+16
  • nic_ranContributor
  • Author
  • Contributor
  • April 2, 2025

​@redgeographics I thought FME would just wait as well if the limit got exceeded but that’s not what’s happening for me. FME is just going beyond its limits and crashing.

redgeographics
Celebrity
Forum|alt.badge.img+49
nic_ran wrote:

​@redgeographics I thought FME would just wait as well if the limit got exceeded but that’s not what’s happening for me. FME is just going beyond its limits and crashing.

Yeah, I feel that that shouldn’t happen really.

Reply


Helpful Members This Week

Community Stats

30,983
Posts
117,416
Replies
38,765
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.