Hi
I've run into something similar, but on a physical server. The server in question had 32GB of physical RAM, but 64-bit FME would always start "optimizing memory" (swapping to disk, really) around the 12GB mark, which made the workspace take forever to complete. Increasing the physical memory to 64GB changed nothing, FME would still start optimizing at 12GB, regardless. Does this look similar to what you're observing?
Here's what Safe said about it:
We made a change in FME 2015 64 bit to limit the amount
of memory usage before stashing to disk to 12 GB. We did an intensive profiling of the memory manager in
FME on 64 bit computers, and found that the time required to swap memory to
disk increased by a large amount once memory usage passed 12GB. This change will slow down the processing for those
workspace that need less than the installed amount of RAM, but will speed up
those which still had to cache to disk. We are planning an update to the memory manager
for FME 2016
Trying again using the latest FME 2016 beta could be interesting, however I do not know if it includes the new memory manager or not. Maybe @daleatsafe could shed some light on this?
Also consider modifying the FME_ENGINE_MEMORY_REDLINE in the workspace header,
see this article.
David
Hi
I've run into something similar, but on a physical server. The server in question had 32GB of physical RAM, but 64-bit FME would always start "optimizing memory" (swapping to disk, really) around the 12GB mark, which made the workspace take forever to complete. Increasing the physical memory to 64GB changed nothing, FME would still start optimizing at 12GB, regardless. Does this look similar to what you're observing?
Here's what Safe said about it:
We made a change in FME 2015 64 bit to limit the amount
of memory usage before stashing to disk to 12 GB. We did an intensive profiling of the memory manager in
FME on 64 bit computers, and found that the time required to swap memory to
disk increased by a large amount once memory usage passed 12GB. This change will slow down the processing for those
workspace that need less than the installed amount of RAM, but will speed up
those which still had to cache to disk. We are planning an update to the memory manager
for FME 2016
Trying again using the latest FME 2016 beta could be interesting, however I do not know if it includes the new memory manager or not. Maybe @daleatsafe could shed some light on this?
Also consider modifying the FME_ENGINE_MEMORY_REDLINE in the workspace header,
see this article.
David
Thanks for your answer, but I assume it does not apply to my problem.
I tested FME_ENGINE_MEMORY_REDLINE in FME 2015 Desktop, but after editing the header, saving, viewing header again, the line is gone. I assume this is not working anymore in FME Desktop.
Thanks for your answer, but I assume it does not apply to my problem.
I tested FME_ENGINE_MEMORY_REDLINE in FME 2015 Desktop, but after editing the header, saving, viewing header again, the line is gone. I assume this is not working anymore in FME Desktop.
I can't say why the setting disappears from your headers, but I've successfully used this option in FME 2015 Desktop before. It is still there.
I can't say why the setting disappears from your headers, but I've successfully used this option in FME 2015 Desktop before. It is still there.
It works when I add it as first line, thanks. But it has no effect on hypervisor, it will not unblock more memory.
It works when I add it as first line, thanks. But it has no effect on hypervisor, it will not unblock more memory.
I'd be quite surprised if the hypervisor comes into play here. What value did you use for the memory redline?
I can't say why the setting disappears from your headers, but I've successfully used this option in FME 2015 Desktop before. It is still there.
sorry for replying here, "reply" command is missing on your last comment.
I tried some numbers for memory redline, 0.1, 0.5, 0.75, 1. The log file contained the correct values of the limits in GB, but the process starts freeing memory much quicker at just some MBs instead of 4 GB when using 0.5 for example. As I updated my original post, only the value of 1 helped to increasing memory consumption before swapping, but this lead to out of memory error (which was very clear to me when I started the translation to see was will happen).
If you don't get any better responses I think you should log this one with the FME support team (www.safe.com/support) - they'll be able to talk with the developers and find out what might have changed and what we can do to help
We definitely have changed the memory management for FME 2016. Now the engine will use more memory. So as you increase the memory on your system FME 2016 will leverage that.
In all FME's we do ask the machine how much physical RAM it has and then work out our plan from there. Sounds like in the Hypervisor case we may be getting mislead, so we'll have to check on that.
FME 2015.1.3 has a revised memory management scheme on 64 bit, as does FME 2016. The Hypervisor may still be goofing us up, but we've updated things so we will take more that 12 GB of RAM if it is there.
In all FME's we do ask the machine how much physical RAM it has and then work out our plan from there. Sounds like in the Hypervisor case we may be getting mislead, so we'll have to check on that.
FME 2015.1.3 has a revised memory management scheme on 64 bit, as does FME 2016. The Hypervisor may still be goofing us up, but we've updated things so we will take more that 12 GB of RAM if it is there.
Thanks for clarification, setting minimum RAM for virtual machine seems to solve the problem. Thumbs up for FME :-)
