Adjusting Memory Resources

Thanks, its the first time ever working with this much data and is super frustrating when almost done reading file and it fails.

That isnt a bad idea. I never thought of it. It is actually about 40 shape files that i read into one merged file. i could take one at a time, but i was wanting to do it in one shot

Hi,

Its good that you have option to divide the source dataset/workspace.

After applying this change, still if you have same problem then probably you have to lookout some blocking transformers which are used and review them

or

some features are with invalid geometry which are effecting the translation.

Pratap

Definitely provide a bit more detail as suggested by Dale, as a conflation of Shape files isn't that heavy a job, unless you are also doing a geospatial operation that you haven't exposed? For the reference of others though there is a hack that can produce some successful results with memory management as long as you accept the health warnings.

In FME there is a memory resource manager dealt with using the directive:

FME_ENGINE_MEMORY_REDLINE <factor>

The automatic resource manager determines the optimal total memory the FME Engine process should use. It also dynamically allocates this total memory optimally to the algorithms within FME requesting it.

The FME_ENGINE_MEMORY_REDLINE directive is a hint to the FME Engine on how aggressive it should be in consuming memory. It takes a value between 0 and 1 (0.5 is the default value). For more aggressive memory usage, a value above 0.5 should be used. For safer memory usage, a value below 0.5 should be used. The risk in being too aggressive is the process running out of memory or the machine thrashing. The risk in being to conservative is that the process will take longer to complete.

To change the default behaviour, you will need to change the FME_ENGINE_MEMORY_REDLINE setting, as explained in:

http://fmepedia.safe.com/articles/How_To/How-to-control-FME-Server-Engine-Memory-usage

As explained by one of the developers working on memory related 'stuff' at Safe "Increasing the memory limit should give you better performance if the resource manager never kicks in, but will give you significantly worse performance if it does kick in. If you want to go all out, you can set FME_ENGINE_MEMORY_REDLINE to 1.0 and the resource manager will never kick in and you'll likely get the best performance, but you may run out of memory."

To apply this value to a workspace in FME Desktop you’ll need to add that redline setting to your workspace header, if you go to this article:

https://knowledge.safe.com/questions/2273/fatal-insufficient-memory-available-error-code-was.html

...and scroll to when Gerhard chimes in, he explains what you’ll need to do. Anecdotally another customer of ours who recently tweaked this header setting a bit discovered that:

“I don’t have a comparison to see how increasing the memory changed performance to another workspace, but the log file shows that I was able to use more RAM. By watching my task manager, I can confirm the RAM usage matched what the log said it would do. I also fiddled with the redline level a bit and found that increasing to 1.0 releases more memory than the machine has, I am not sure where it gets 8tb from, but 0.99 will let me use just about all of the 40gb I have…though I never will leave some in reserve for Windows 7 to consume. I think this will really help us run multiple workspaces at once while managing resources to allow our modelling simulations to run at the same time.”

I hope that helps you out a bit. Cheers Dave.

cc. @erik_jan

FME 2016 includes a lot of effort around memory utilisation, so in theory there may be some light at the end of the tunnel.

Some related comments here https://knowledge.safe.com/questions/19814/why-doesnt-fme-2015-64bit-use-more-memory-on-a-vir.html.