Skip to main content
Solved

Performance Scaling


pkno
Contributor
Forum|alt.badge.img+5
  • Contributor

Hello everyone,

I recently got my computer running FME Form replaced and made some observations regarding performance that I wanted to lay out since I am not sure that everything runs as it should.

I am testing with a big workspace with close to 1000 transformers.

Coming from an aging Intel i5 6th Gen with 32 gigs of RAM and now sporting an Intel i7 14700-K with 128GB of RAM I was expecting some substantial performance uplifts.

The observation I made is this:

Yes, when I only read and write locally I am seeing a good boost in performance (a little over 2 times faster). However, in production this workspace is meant to write to a PostgreSQL database over the network. In this scenario my new computer is very close in performance to the old machine, even getting beaten by a few % in overall processing time (writing to the same database over the network). 

Here is the numbers.

  New Computer Old Computer
Writing around 430.000 Features to database over network 1hr 39m 1hr 29m
Writing around 430.000 Features to GeoPackage locally 12m 41s 26m 52s

 

I realize that writing over the network can bottleneck overall performance but considering the large amounts of processing I was expecting some kind of uplift, certainly not degradation.

Does this seem plausible or did I miss something obvious? Can the difference in writing speeds over the network attributed to variations in network speeds (both computers are tethered to the same network via cable but the sockets may be handled differently on the backend I am not sure on this) 

Since I just installed FME on the new computer I was wondering if I maybe missed some setting that I may have switched in the past.

Any suggestions what I can try out or what changes I can make?

 

Thanks you all

 

 

 

Best answer by hkingsbury

There are lots of considerations to make re performance. The writing is always going to be the slowest part, especially via a network.

In regards to the performance decrease, I doubt you’d be able to prove the network, database and everything else involved in writing was identical between the two tests. You’re only looking at ~10% difference there.

If you’re chasing speed/performance, it’s very possible that gains can be made in your workspace. things like:

  • Optimizing order of operations
  • Reducing data volumes during processing (removing unneeded attributes)
  • Offloading joins and filters to source databases (if applicable)
  • Reducing blocking transformers
  • Fine tuning transformers
  • Reducing the need to break out of bulk mode

 

View original
Did this help you find an answer to your question?

3 replies

hkingsbury
Celebrity
Forum|alt.badge.img+53
  • Celebrity
  • Best Answer
  • April 10, 2025

There are lots of considerations to make re performance. The writing is always going to be the slowest part, especially via a network.

In regards to the performance decrease, I doubt you’d be able to prove the network, database and everything else involved in writing was identical between the two tests. You’re only looking at ~10% difference there.

If you’re chasing speed/performance, it’s very possible that gains can be made in your workspace. things like:

  • Optimizing order of operations
  • Reducing data volumes during processing (removing unneeded attributes)
  • Offloading joins and filters to source databases (if applicable)
  • Reducing blocking transformers
  • Fine tuning transformers
  • Reducing the need to break out of bulk mode

 


pkno
Contributor
Forum|alt.badge.img+5
  • Author
  • Contributor
  • April 11, 2025

Thank you for your answer,

to follow up on this. It seems to me that performance optimizations on the workspace itself will only be able to shave off some % from the actual processing which seems to only amount to less than 13 minutes (based on the local writing speeds) with the majority of the time needed beeing determined by network-/database performance. 

Following this train of thought, I shouldnt be able to improve overall speed when wirting to the database by that much by reworking the workspace.

The only points I can see making much of a difference in this case are  

  • Reducing the need to break out of bulk mode
  • Offloading joins and filters to source databases (if applicable)

Am I right in that assessment?   

 


hkingsbury
Celebrity
Forum|alt.badge.img+53
  • Celebrity
  • April 11, 2025

When chasing performance it can be difficult for someone with no context or limited knowledge of the data/process to give anything more than generic advice! That being said, those two points you’ve highlighted are generally pretty good starting points and are likely to give the best ‘return on investment’


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings