In my experience FME is not always using the FME_TEMP setting. Changing the TEMP environment setting to a large and fast drive can sometimes help too.
In my experience FME is not always using the FME_TEMP setting. Changing the TEMP environment setting to a large and fast drive can sometimes help too.
Thanks, its the first time ever working with this much data and is super frustrating when almost done reading file and it fails.
Maybe off-topic, but it would also help to not read the whole dataset at once. If possible dividing the dataset in smaller portions and looping through these using the WorkSpaceRunner has worked miraculous for me a lot of times.
I do not know if this is a possible scenario for you.
Maybe off-topic, but it would also help to not read the whole dataset at once. If possible dividing the dataset in smaller portions and looping through these using the WorkSpaceRunner has worked miraculous for me a lot of times.
I do not know if this is a possible scenario for you.
That isnt a bad idea. I never thought of it. It is actually about 40 shape files that i read into one merged file. i could take one at a time, but i was wanting to do it in one shot
That isnt a bad idea. I never thought of it. It is actually about 40 shape files that i read into one merged file. i could take one at a time, but i was wanting to do it in one shot
You could use the Directory and File Path Reader to get the names of the Shape files and add those in a WorkSpaceRunner using Published parameters. That will cut down the needed resources by a lot and still allows you to run one workspace spawning off subworkspaces.
Hi,
Its good that you have option to divide the source dataset/workspace.
After applying this change, still if you have same problem then probably you have to lookout some blocking transformers which are used and review them
or
some features are with invalid geometry which are effecting the translation.
Pratap
It would be helpful to see a pic of your workspace a full log. I wouldn't expect that merging 40 shape files would be a problem, depending on the definition of "merge". Also is it 64 or 32 bit that you are running with?
Definitely provide a bit more detail as suggested by Dale, as a conflation of Shape files isn't that heavy a job, unless you are also doing a geospatial operation that you haven't exposed? For the reference of others though there is a hack that can produce some successful results with memory management as long as you accept the health warnings.
In FME there is a memory resource manager dealt with using the directive:
FME_ENGINE_MEMORY_REDLINE <factor>
The automatic resource manager determines the optimal total memory the FME Engine process should use. It also dynamically allocates this total memory optimally to the algorithms within FME requesting it.
The FME_ENGINE_MEMORY_REDLINE directive is a hint to the FME Engine on how aggressive it should be in consuming memory. It takes a value between 0 and 1 (0.5 is the default value). For more aggressive memory usage, a value above 0.5 should be used. For safer memory usage, a value below 0.5 should be used. The risk in being too aggressive is the process running out of memory or the machine thrashing. The risk in being to conservative is that the process will take longer to complete.
To change the default behaviour, you will need to change the FME_ENGINE_MEMORY_REDLINE setting, as explained in:
http://fmepedia.safe.com/articles/How_To/How-to-control-FME-Server-Engine-Memory-usage
As explained by one of the developers working on memory related 'stuff' at Safe "Increasing the memory limit should give you better performance if the resource manager never kicks in, but will give you significantly worse performance if it does kick in. If you want to go all out, you can set FME_ENGINE_MEMORY_REDLINE to 1.0 and the resource manager will never kick in and you'll likely get the best performance, but you may run out of memory."
To apply this value to a workspace in FME Desktop you’ll need to add that redline setting to your workspace header, if you go to this article:
https://knowledge.safe.com/questions/2273/fatal-insufficient-memory-available-error-code-was.html
...and scroll to when Gerhard chimes in, he explains what you’ll need to do.
Anecdotally another customer of ours who recently tweaked this header setting a bit discovered that:
“I don’t have a comparison to see how increasing the memory changed performance to another workspace, but the log file shows that I was able to use more RAM. By watching my task manager, I can confirm the RAM usage matched what the log said it would do. I also fiddled with the redline level a bit and found that increasing to 1.0 releases more memory than the machine has, I am not sure where it gets 8tb from, but 0.99 will let me use just about all of the 40gb I have…though I never will leave some in reserve for Windows 7 to consume. I think this will really help us run multiple workspaces at once while managing resources to allow our modelling simulations to run at the same time.”
I hope that helps you out a bit. Cheers Dave.
cc. @erik_jan
In my experience FME is not always using the FME_TEMP setting. Changing the TEMP environment setting to a large and fast drive can sometimes help too.
More details here https://knowledge.safe.com/articles/176/fme-temp-environment-variable.html
Definitely provide a bit more detail as suggested by Dale, as a conflation of Shape files isn't that heavy a job, unless you are also doing a geospatial operation that you haven't exposed? For the reference of others though there is a hack that can produce some successful results with memory management as long as you accept the health warnings.
In FME there is a memory resource manager dealt with using the directive:
FME_ENGINE_MEMORY_REDLINE <factor>
The automatic resource manager determines the optimal total memory the FME Engine process should use. It also dynamically allocates this total memory optimally to the algorithms within FME requesting it.
The FME_ENGINE_MEMORY_REDLINE directive is a hint to the FME Engine on how aggressive it should be in consuming memory. It takes a value between 0 and 1 (0.5 is the default value). For more aggressive memory usage, a value above 0.5 should be used. For safer memory usage, a value below 0.5 should be used. The risk in being too aggressive is the process running out of memory or the machine thrashing. The risk in being to conservative is that the process will take longer to complete.
To change the default behaviour, you will need to change the FME_ENGINE_MEMORY_REDLINE setting, as explained in:
http://fmepedia.safe.com/articles/How_To/How-to-control-FME-Server-Engine-Memory-usage
As explained by one of the developers working on memory related 'stuff' at Safe "Increasing the memory limit should give you better performance if the resource manager never kicks in, but will give you significantly worse performance if it does kick in. If you want to go all out, you can set FME_ENGINE_MEMORY_REDLINE to 1.0 and the resource manager will never kick in and you'll likely get the best performance, but you may run out of memory."
To apply this value to a workspace in FME Desktop you’ll need to add that redline setting to your workspace header, if you go to this article:
https://knowledge.safe.com/questions/2273/fatal-insufficient-memory-available-error-code-was.html
...and scroll to when Gerhard chimes in, he explains what you’ll need to do.
Anecdotally another customer of ours who recently tweaked this header setting a bit discovered that:
“I don’t have a comparison to see how increasing the memory changed performance to another workspace, but the log file shows that I was able to use more RAM. By watching my task manager, I can confirm the RAM usage matched what the log said it would do. I also fiddled with the redline level a bit and found that increasing to 1.0 releases more memory than the machine has, I am not sure where it gets 8tb from, but 0.99 will let me use just about all of the 40gb I have…though I never will leave some in reserve for Windows 7 to consume. I think this will really help us run multiple workspaces at once while managing resources to allow our modelling simulations to run at the same time.”
I hope that helps you out a bit. Cheers Dave.
cc. @erik_jan
FME 2016 includes a lot of effort around memory utilisation, so in theory there may be some light at the end of the tunnel.
Some related comments here https://knowledge.safe.com/questions/19814/why-doesnt-fme-2015-64bit-use-more-memory-on-a-vir.html.
