Here is another curiosity from the log files on Server A - after we tried adjusting the FME_ENGINE_MEMORY_REDLINE to 0.4
33 INFORM 0.0 0.1 Today at 12:25:39 FME_ENGINE_MEMORY_REDLINE is set to 0.4
34 INFORM 0.0 0.1 Today at 12:25:39 FME Configuration: Process limits are 32.00 GB of physical memory and 128.00 TB of address space
35 INFORM 0.0 0.1 Today at 12:25:39 FME Configuration: Start freeing memory when process usage exceeds 33.64 MB of memory or 102.40 TB of address space
36 INFORM 0.0 0.1 Today at 12:25:39 FME Configuration: Stop freeing memory when process usage is below 32.29 MB of memory and 76.80 TB of address space
Hi @malteohlers - This is a large topic and will no doubt yield a lot of responses. If I were debugging this I'd start with ensuring I was happy with the setup of the workspace in FME Desktop, the key thing I'd be looking to ensure was that the Peak memory usage was as low as possible and that I was never seeing the statement "optimizing resource usage" in the log (that's never good as it means you're paging memory).
There is a lot of suggestions here:
https://knowledge.safe.com/articles/579/performance-tuning-fme.html?smartspace=performance-tuning_2
...and a couple of our webinars here:
https://youtu.be/JTIevi4tHhI?list=PLPTavfHRhhqt2pBXYXF4dUgnz0t_008Vs -
Tips for performance tweaking your FME processes
and
https://www.youtube.com/watch?v=kyDCzhC40cA&index;=3&list;=PLPTavfHRhhqt2pBXYXF4dUgnz0t_008Vs - Tuning FME for speedy data processing
Once you've got the workspace as tight as possible, test the process on FME Desktop on the same machine that FME Server is installed on. This will then include environmental issues in the test related to the machine setup. If you then promote the process to FME Server then and get a different result there is hopefully something to see as the main culprit.
As far as memory redline is concerned with FME Server Safe Software did have some issues with the third-party memory management library they use back in 2015 which forced them to impose a memory cap on 64bit FME. However that has now been largely resolved, though I believe there is still memory cap, albeit quite a high one. Hopefully someone from Safe will clarify that here.
I believe the FME_ENGINE_MEMORY_REDLINE parameter allows you to remove that cap, as explained in
http://fmepedia.safe.com/articles/How_To/How-to-control-FME-Server-Engine-Memory-usage
Safe have previously told me:
"Increasing the memory limit should give you better performance if the resource manager never kicks in, but will give you significantly worse performance if it does kick in. If you want to go all out, you can set FME_ENGINE_MEMORY_REDLINE to 1.0 and the resource manager will never kick in and you'll likely get the best performance, but you may run out of memory."
To apply this value to a workspace in FME Desktop you’ll need to add that redline setting to your workspace header. To do the same in Server you'll need to edit the fmeEngineConfig.txt file.I hope that helps, good luck. Dave
Ok, so we have partly solved the problem for now.
But there seems to be something strange going on with the engine memory allocation function.
As I had showed the engines would be allowed to take up 96GB of 32GB of memory:
2016-08-29 15:46:20| 0.2| 0.0|INFORM|FME Configuration: Process limits are 32.00 GB of
physical memory and 128.00 TB of address space2016-08-29 15:46:20| 0.2| 0.0|INFORM|FME Configuration: Start
freeing memory when process usage exceeds 96.00 GB of memory or 128.00 TB of address space
2016-08-29 15:46:20| 0.2|
0.0|INFORM|FME Configuration: Stop freeing
memory when process usage is below 72.00 GB of memory
and 96.00 TB of address space
We have now changed the FME_ENGINE_MEMORY_REDLINE to 0.08 in the fmeEngineConfig file.
(We struggled a bit with this - also because we where tying to set the parameter via the header).
Now we get a more usable per engine memory allocation:
33
INFORM
0.0
0.2
Today at 12:23:01
FME_ENGINE_MEMORY_REDLINE is set to 0.08
34
INFORM
0.0
0.2
Today at 12:23:01
FME Configuration: Process limits are 32.00 GB of physical memory and 128.00 TB of address space
35
INFORM
0.0
0.2
Today at 12:23:01
FME Configuration: Start freeing memory when process usage exceeds 15.36 GB of memory or 20.48 TB of address space
36
INFORM
0.0
0.2
Today at 12:23:01
FME Configuration: Stop freeing memory when process usage is below 11.52 GB of memory and 15.36 TB of address space
We might have to tweak this parameter further to allow for multiple engiens to run at the same time.
I'm also considering useing Job Routing and job tacking to configure a "Highmemory" engine and some lower ones.
We are still confused about why we would see numbers in the MB (not GB) order on our other server with the standard FME_ENGINE_MEMORY_REDLINE 0.5:
33 INFORM 0.0 0.2 Today at 10:45:23 FME Configuration: Process limits are 8.00 GB of physical memory and 128.00 TB of address space
34 INFORM 0.0 0.2 Today at 10:45:23 FME Configuration: Start freeing memory when process usage exceeds 39.74 MB of memory or 128.00 TB of address space
35 INFORM 0.0 0.2 Today at 10:45:23 FME Configuration: Stop freeing memory when process usage is below 38.15 MB of memory and 96.00 TB of address space
Is it potentially a bug we are are seeing or is this expected behavior?
Hi @malteohlers - This is a large topic and will no doubt yield a lot of responses. If I were debugging this I'd start with ensuring I was happy with the setup of the workspace in FME Desktop, the key thing I'd be looking to ensure was that the Peak memory usage was as low as possible and that I was never seeing the statement "optimizing resource usage" in the log (that's never good as it means you're paging memory).
There is a lot of suggestions here:
https://knowledge.safe.com/articles/579/performance-tuning-fme.html?smartspace=performance-tuning_2
...and a couple of our webinars here:
https://youtu.be/JTIevi4tHhI?list=PLPTavfHRhhqt2pBXYXF4dUgnz0t_008Vs -
Tips for performance tweaking your FME processes
and
https://www.youtube.com/watch?v=kyDCzhC40cA&index;=3&list;=PLPTavfHRhhqt2pBXYXF4dUgnz0t_008Vs - Tuning FME for speedy data processing
Once you've got the workspace as tight as possible, test the process on FME Desktop on the same machine that FME Server is installed on. This will then include environmental issues in the test related to the machine setup. If you then promote the process to FME Server then and get a different result there is hopefully something to see as the main culprit.
As far as memory redline is concerned with FME Server Safe Software did have some issues with the third-party memory management library they use back in 2015 which forced them to impose a memory cap on 64bit FME. However that has now been largely resolved, though I believe there is still memory cap, albeit quite a high one. Hopefully someone from Safe will clarify that here.
I believe the FME_ENGINE_MEMORY_REDLINE parameter allows you to remove that cap, as explained in
http://fmepedia.safe.com/articles/How_To/How-to-control-FME-Server-Engine-Memory-usage
Safe have previously told me:
"Increasing the memory limit should give you better performance if the resource manager never kicks in, but will give you significantly worse performance if it does kick in. If you want to go all out, you can set FME_ENGINE_MEMORY_REDLINE to 1.0 and the resource manager will never kick in and you'll likely get the best performance, but you may run out of memory."
To apply this value to a workspace in FME Desktop you’ll need to add that redline setting to your workspace header. To do the same in Server you'll need to edit the fmeEngineConfig.txt file.I hope that helps, good luck. Dave
Hi Dave,
Thanks a lot for you input and link to great resources! It's greatly apriciated!
We will go though all of if and try and optimize or work spaces even more - but with the current input datasets we need to process there is also only so much we can do about the memory usage.
But as you can see in my comment below - we partly solved the problem (of the server running of memory) but tuning the FME_ENGINE_MEMORY_REDLINE parameter quite heavily. But we are still surprised by the different way this parameter (appears to) works on out two server (0.5 resulting in 300% of physical memory or 0.5% of physical memory).
Now we'll continue or work with parallelization and other interesting stuff - so it's probable not the last time you'll hear from me
Hi. I can certainly chime in on one oddity in the log files. I too had messages like:
Process limits are 24.00 GB of physical memory and 8.00 TB of address space
Start freeing memory when process usage exceeds 71.99 GB of memory or 8.00 TB of address space
Stop freeing memory when process usage is below 53.99 GB of memory and 6.00 TB of address space;
...and wondered how FME could possibly use 72gb of memory when my machine only has 24gb installed.
One of our developers told me:
There are three terms we are using in those messages:
- physical memory: The amount of RAM your machine has.
- memory: The amount of memory the process is using. This includes both data in RAM and data paged to disk by the operating system, which is why it can be greater than physical memory.
- address space: The amount of memory the process currently has available to it. Requesting more memory may cause this to increase. A program runs out of memory when this grows too large.
Note that the process limit message refers to "physical memory", which is different than the more generic "memory" in the start/stop messages.
I know it looks misleading, but I don't want to have them make those messages so complex they have no meaning. Anyway, tweaking the redline until the start/stop memory messages are below the physical memory shouldn't be required, and I don't know that I'd recommend it.
In fact, I'm really not sure I would touch the redline setting at all, at least not until I'd ruled out every other possible problem. Can you send me the full log files for a) the workspace run on Desktop, and b) the workspace run on Server w/o any redline alterations? That way I can check to see if there are any other obvious differences. Either post them here or email me (mark.ireland @ safe.com)
The message in Mb not Gb is interesting. I suspect that 8Gb of memory on a server is just not going to cut it to "process relatively large data sets" and "run multiple workspaces in parallel". I mean, there is the operating system to consider and you're running Windows Server which I suspect is a memory hog by itself. I'm running Windows 7 and even with everything except this web browser shut off, the system memory use is over 10Gb. So it might be that the log is correct and FME really does only have a few Mb left to play with. Can you check the Task Manager and see what free memory is actually available?
I hope this helps. In general Desktop and Server are both using the same underlying engine, and given the same parameters on the same system should give pretty much the same results. I assume you are using 64-bit FME on Desktop as well as Server?
Mark Ireland
Product Evangelist
Safe Software
Update: on the server Malte mentions above we've had to disable swapping entirely (for now). We noticed we were running out of space on C: and so increased it, twice. Eventually we figured out the machine was using >50GB swap (server has 48GB RAM). Performance was of course very bad.
After disabling swap, the FME engine processes behave better (far better performance), but now we run into out of memory issues when the AIXM Writer -- having collected all the features to write -- starts to generate the XML file, which of course fails the entire job. :(
So either we have swap, that appears to be consumed up to whatever size we limit it to, or (and?) then the FME engine will sometimes fail because of out-or-memory in the final steps.
To top it all off, we see hangs and failures of both the FME Server Core and Application Server "services" with restarts and interrupted running jobs as a result.
Any suggestions most welcome.
Wild idea: is there a way to trigger the FME engine garbage collection just before the Writer starts? I'm sure a fairly large part of the 10-12GB those engine processes consume is not "active" data at that point.
Potentially less wild idea: should we have our workspaces write to FFS intially, and then have another workspace do the final FFS->AIXM step?
Best regards,
Håkan
These problems are always awkward. I'd suggest running a few features through the process with BreakPoints turned on to try to identify where the features are bottlenecking. Clearly if you can avoid the bottleneck you'll be able to get features free-flowing through the process direct to the writer.
As you mention, if practical you could split the process, write to FFS at the end of process 1 and read from FFS in process 2, using a WorkspaceRunner to chain the processes. Alternatively, you could try caching all the data to disk part way through thus dumping it from memory, a Recorder would allow you to do this in 'record' mode and then a Sampler to sample off a single trigger feature that you pass into a Player to read back the contents of the FFS file you've just writen out. This is much simpler with the FeatureReader and FeatureWriter these days, but I'm not sure if in your build you have the FeatureWriter.
I think this one perhaps needs escalating to your Reseller or direct to Safe as a support ticket.
@hakan_olsson
I'm wondering if you would be able to send us a reproduction of this workflow (data, workspaces)? It would be wonderful to reproduce this in-house at Safe and see if we can make improvements. Please create a case at https://www.safe.com/support/ with any information that you feel would be helpful.
I think the workarounds suggested by @1spatialdave are wonderful and should help. Sorry to hear you have encountered these problems.
Brian
