Skip to main content
  • EN

Hi Guys

 

 

I currently have a Dynamic Schema workbench with a custom-built reprojection transformer along with the following transformers that are allowing me to rename the output files and place them in a new relative folder structure: FilenamePartExtractor, StringReplacer and StringConcatenator.

 

 

The workbench works correctly on my first lot of files which are quite small in size, but when it comes up against files that are 100MB to 700MB in size, it says "ResourceManager: Optimizing Memory Usage" and stalls, and eventually crashes with a "Fatal Error" and other messages relating to memory. 

 

 

I have tried writing to a different network drive from the input files, and have also purged my temporary files, but neither has helped. 

 

 

My custom-built reprojection transformer is very complex and I am guessing that this may be causing the memory problem on the large files, but am wondering if using Dynamic Schema also affects memory.  If so, does anyone have any advice on how to fix this?

 

 

Thanks
Things I can suggest are as follows:

 

 

Environment:

 

  • Make sure your temp files are being written to a fast local disk with plenty of space. Use the FME_TEMP env variable to direct the temp files.
http://fmepedia.safe.com/articles/FAQ/FME-TEMP-environment-variable

 

 

  • Make sure you have some allocated disk space 'virtual memory'.
  • Ideally use 32bit FME on a 64bit OS to ensure 4GB of RAM can be addressed. You can also consider trying 64bit FME on a 64bit OS if you have lots of RAM available as this config will allow you address all the RAM you have. This assumes the formats you are using are available on the 64bit platform. 
http://www.safe.com/fme/format-search/#showPlatforms=true

 

http://www.safe.com/downloads

 

 

Workspace:
  • Remove any attributes that are not required in your target or by the process as soon as you can (AttributeKeeper/AttributeRemover) this reduces the memory load hugely if you have lots that are being carried unneccessarily through your process.
  • Remove geometry (GeometryRemover) if it's not needed or coerce it to something less "heavy" if you can. For example if you have polygons but you only actually need the centroid, replace the geometry with it's centroid.
  • Make sure that any blocking transformers are appropriately positioned. These chaps force features into memory. Some of these have parameters that can turn them into less memory hungry beasts if you can sort your features before they get to the transformer. For example the Aggregator has a parameter "Input is Ordered by Group".
  • Make sure you only carry out heavy operations on features to which the process applies, so route features around parts of your workspace if you can appropriately filter them.
  • Consider moving your process to a batch run if you can't avoid any of the above. In this case use a workspace containing a WorkspaceRunner to call your master workspace many times on batches of your data. The WorkspaceRunner is awesome and frequently saves the day for me.
Hope that helps, Dave
I also should have mentioned that much of what I describe and more is detailed here:

 

 

http://fmepedia.safe.com/articles/How_To/Performance-Tuning-FME

 

 

You should also take a look at these Evangelist articles:

 

 

http://evangelism.safe.com/fmeevangelist94/

 

http://evangelism.safe.com/fmeevangelist79/

 

 

The first highlights the benefits of the recent inovation of parallel processing. This is hugely useful if you can chunk your data up into groups as it allows you to concurrently process it on different cores. The second details a couple of the different dat joining methods so if you are doing this you might find the FeatureMerger is not always the best bet.

 

 

Good luck.

Reply


Community Stats

30,096
Posts
114,255
Replies
37,726
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.