Solved

Why doesn't FME 2015 (64bit) use more memory on a virtual machine with dynamic memory management?

offermann82
I am running fme 2015 (64bit) on a virtual machine. The Hyper-V server works with dynamic memory management, this leads to nearly 80% memory "blocked" by the hypervisor. In theory, the hypervisor should unblock more memory to the virtual machine, when a process like FME needs more memory. But instead, the FME optimizes its memory before and this leads to FME.exe process running with a limit of 200 MB on a virtual machine with 8 GB RAM.

 

This only occurs using FME 2015, I did not have this behaviour running FME 2014. May be the Resource Manager improvements in 2015 are a problem?

 

 

UPDATE following @david_r 's suggestion, the fme.exe process gets more memory when using "FME_ENGINE_MEMORY_REDLINE 1", but a smaller value than 1 does not work to get more memory. Let's see what happens when the process hits the ceiling...

 

 

UPDATE Our IT has risen the minimum RAM of the virtual machine to 4 GB, this seems to solve the problem. The FME process is now using more RAM without swapping.
icon

Best answer by fmelizard 9 November 2015, 19:42

View original

11 replies

david_r
Rank
Userlevel 4
Hi

 

 

I've run into something similar, but on a physical server. The server in question had 32GB of physical RAM, but 64-bit FME would always start "optimizing memory" (swapping to disk, really) around the 12GB mark, which made the workspace take forever to complete. Increasing the physical memory to 64GB changed nothing, FME would still start optimizing at 12GB, regardless. Does this look similar to what you're observing?

 

 

Here's what Safe said about it:

 

 

We made a change in FME 2015 64 bit to limit the amount of memory usage before stashing to disk to 12 GB. We did an intensive profiling of the memory manager in FME on 64 bit computers, and found that the time required to swap memory to disk increased by a large amount once memory usage passed 12GB. This change will slow down the processing for those workspace that need less than the installed amount of RAM, but will speed up those which still had to cache to disk. We are planning an update to the memory manager for FME 2016

 

Trying again using the latest FME 2016 beta could be interesting, however I do not know if it includes the new memory manager or not. Maybe @daleatsafe could shed some light on this?

 

 

Also consider modifying the FME_ENGINE_MEMORY_REDLINE in the workspace header, see this article.

 

 

David

 

offermann82
Hi

 

 

I've run into something similar, but on a physical server. The server in question had 32GB of physical RAM, but 64-bit FME would always start "optimizing memory" (swapping to disk, really) around the 12GB mark, which made the workspace take forever to complete. Increasing the physical memory to 64GB changed nothing, FME would still start optimizing at 12GB, regardless. Does this look similar to what you're observing?

 

 

Here's what Safe said about it:

 

 

We made a change in FME 2015 64 bit to limit the amount of memory usage before stashing to disk to 12 GB. We did an intensive profiling of the memory manager in FME on 64 bit computers, and found that the time required to swap memory to disk increased by a large amount once memory usage passed 12GB. This change will slow down the processing for those workspace that need less than the installed amount of RAM, but will speed up those which still had to cache to disk. We are planning an update to the memory manager for FME 2016

 

Trying again using the latest FME 2016 beta could be interesting, however I do not know if it includes the new memory manager or not. Maybe @daleatsafe could shed some light on this?

 

 

Also consider modifying the FME_ENGINE_MEMORY_REDLINE in the workspace header, see this article.

 

 

David

 

Thanks for your answer, but I assume it does not apply to my problem.

 

I tested FME_ENGINE_MEMORY_REDLINE in FME 2015 Desktop, but after editing the header, saving, viewing header again, the line is gone. I assume this is not working anymore in FME Desktop.
david_r
Rank
Userlevel 4

Thanks for your answer, but I assume it does not apply to my problem.

 

I tested FME_ENGINE_MEMORY_REDLINE in FME 2015 Desktop, but after editing the header, saving, viewing header again, the line is gone. I assume this is not working anymore in FME Desktop.

I can't say why the setting disappears from your headers, but I've successfully used this option in FME 2015 Desktop before. It is still there.

offermann82

I can't say why the setting disappears from your headers, but I've successfully used this option in FME 2015 Desktop before. It is still there.

It works when I add it as first line, thanks. But it has no effect on hypervisor, it will not unblock more memory.

david_r
Rank
Userlevel 4

It works when I add it as first line, thanks. But it has no effect on hypervisor, it will not unblock more memory.

I'd be quite surprised if the hypervisor comes into play here. What value did you use for the memory redline?

offermann82

I can't say why the setting disappears from your headers, but I've successfully used this option in FME 2015 Desktop before. It is still there.

sorry for replying here, "reply" command is missing on your last comment.

I tried some numbers for memory redline, 0.1, 0.5, 0.75, 1. The log file contained the correct values of the limits in GB, but the process starts freeing memory much quicker at just some MBs instead of 4 GB when using 0.5 for example. As I updated my original post, only the value of 1 helped to increasing memory consumption before swapping, but this lead to out of memory error (which was very clear to me when I started the translation to see was will happen).

mark2atsafe
Rank
Userlevel 4
Badge +25
If you don't get any better responses I think you should log this one with the FME support team (www.safe.com/support) - they'll be able to talk with the developers and find out what might have changed and what we can do to help

 

donatsafe
Rank
We definitely have changed the memory management for FME 2016. Now the engine will use more memory. So as you increase the memory on your system FME 2016 will leverage that.
fmelizard
Rank
Userlevel 4
Badge +13

In all FME's we do ask the machine how much physical RAM it has and then work out our plan from there. Sounds like in the Hypervisor case we may be getting mislead, so we'll have to check on that.

FME 2015.1.3 has a revised memory management scheme on 64 bit, as does FME 2016. The Hypervisor may still be goofing us up, but we've updated things so we will take more that 12 GB of RAM if it is there.

offermann82

In all FME's we do ask the machine how much physical RAM it has and then work out our plan from there. Sounds like in the Hypervisor case we may be getting mislead, so we'll have to check on that.

FME 2015.1.3 has a revised memory management scheme on 64 bit, as does FME 2016. The Hypervisor may still be goofing us up, but we've updated things so we will take more that 12 GB of RAM if it is there.

Thanks for clarification, setting minimum RAM for virtual machine seems to solve the problem. Thumbs up for FME :-)

fmelizard
Rank
Userlevel 4
Badge +13

In all FME's we do ask the machine how much physical RAM it has and then work out our plan from there. Sounds like in the Hypervisor case we may be getting mislead, so we'll have to check on that.

FME 2015.1.3 has a revised memory management scheme on 64 bit, as does FME 2016. The Hypervisor may still be goofing us up, but we've updated things so we will take more that 12 GB of RAM if it is there.

Great news!

Reply


Community Stats

28,661
Posts
109,324
Replies
36,528
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.