I think this question has been asked before, but the links in its answer were broken so I’m unsure what the answer was.
If anyone is able to put me in the right direction that would be great
Basically I have a complicated workspace doing a few things on large files. In order to make it more efficient I would like to know if there is a reasonably easy way to get a rough indication how long each individual transformer was running in total. Then I can get an idea of where bottle necks are, or things I could improve.
Currently I just look through the log to see if something takes a while, but that's tedious. Is there a better way? Is it possible to get some sort of summary how long each transformer takes to run?
I do run the workspace on FME server/flow, though I do test it locally as well.
In here there is a section on performance profiling:
Additional Performance Tuning Tips
In FME Workbench, you can run a profile of your FME Workspace to obtain a detailed log of how much time is spent in each underlying factory or function. Use Tools -> Edit Header to add the directive: FME_PROFILE_RESULT_CSV i.e:
FME_PROFILE_RESULT_CSV C:\TEMP\profile.csv
Notes:
The profile directive is not supported on FME Flow
Don't forget to remove the profile directive for your production workspace since profiling also has an impact on performance!
It's sometimes worth using a Performance Monitoring tool (such as PerfMon) to log the CPU and memory usage of a process.
It’s not quite a breakdown per transformer I don’t think, however, it does give you an indicator on which factories are using the time.
You’re right in that it’s a bit of tedious process, and even going through the log doesn’t always give you a good insight into where the bottle necks are but it is really important still.
In general going through this article series is great and will give you some excellent insights in how fme works
In here there is a section on performance profiling:
Additional Performance Tuning Tips
In FME Workbench, you can run a profile of your FME Workspace to obtain a detailed log of how much time is spent in each underlying factory or function. Use Tools -> Edit Header to add the directive: FME_PROFILE_RESULT_CSV i.e:
FME_PROFILE_RESULT_CSV C:\TEMP\profile.csv
Notes:
The profile directive is not supported on FME Flow
Don't forget to remove the profile directive for your production workspace since profiling also has an impact on performance!
It's sometimes worth using a Performance Monitoring tool (such as PerfMon) to log the CPU and memory usage of a process.
It’s not quite a breakdown per transformer I don’t think, however, it does give you an indicator on which factories are using the time.
You’re right in that it’s a bit of tedious process, and even going through the log doesn’t always give you a good insight into where the bottle necks are but it is really important still.
In general going through this article series is great and will give you some excellent insights in how fme works
Just remember to remove the FME_PROFILE_RESULT_CSV setting before publishing the workspace to production, as it can introduce quite a lot of overhead on each execution.
Also, make sure you’re only working on local files. Directly working on large files on a network drive can be catastrophic for performance.
We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.
