Skip to main content
  • EN
Hi,

 

 

I have hundreds of datasets that I need to process.  I attempted last night to batch process 8 of these datasets overnight.  However, coming back in the office this morning I found that it is saying it is out of memory, and that none of the datasets processed at all.

 

 

I was hoping to get some advice on improving memory capacity so that I am able to batch process a lot of files at a time.  Unfortunately I don't have many admin rights on the company computer so it will be difficult to change many of the settings.  Keeping this in mind, if you have some simple and effective solutions for assisting in memory capacity, it would be greatly appreciated.

 

 

I don't have much experience in fiddling with memory and caching settings, so a simple and clear explanation would be great.

 

 

Thanks,
Hi,

 

 

what does the log tell you and what does the workspace do? Which version of FME? What kind of server?

 

 

There is normally little use of adjusting memory and cache settings, unless you have thoroughly reviewed your workspace and processing methodologies first.

 

 

David
Hi David,

 

 

I am on FME Desktop 2009.  The log file was saying something like "out of memory" and "program terminating" unfortunately I don't have the logfile still to know the exact wording.

 

 

The workspace converts an ESRI Ascii Grid to XYZ, then does a datum transformation, and then outputs the XYZ into an Access Database.  The secondary output is a bounding extent box of the transformed dataset in DGN format.

 

Primary output:

  • Input: ESRI Ascii Grid --> RasterCellCoercer --> Custom built datum transformer --> CoordinateConcatenator --> AttributeSplitter --> ListExploder --> AttributeSplitter --> Output: ACCESS database.

I also have a secondary output in the same Workspace:

  • Input: ESRI Ascii Grid --> RasterPropertiesExtractor --> 2DBoxReplacer --> Custom built datum transformer --> Output: DGN

 

Thanks
Hi,

 

 

FME2009 is really(!) old, I highly recommend upgrading.

 

 

Remember that you can install several versions of FME on the same server for testing purposes, just remember to install them into separate directories.

 

 

Without the log file it is difficult to say what is going on.

 

 

David
Hi,

 

 

It's difficult to specify the problem, but it can be said that the ListExploder tends to consume large memory because it copies geometry per every list element. If your purpose is to extract coordinate values from point geometries, consider using the CoordinateExtractor instead. However, if you need to get all vertices of lines or polygons, you still have to use the CoordinateConcatenator, AttributeSplitter and ListExploder as long as you use FME 2009 (The Copper of newer FME can do that efficiently!).

 

When you use the ListExploder, consider removing geometries before exploding list by the GeometryRemover in order to reduce amount of memory consuming.

 

  Takashi
Thanks Takashi,

 

 

I wanted to try CoordinateExtractor instead, except I can't find it when I search the list of Transformers.  Is it available with FME 2009?

 

 

Thanks,
It sounds strange. I believe there is the CoordinateExtractor in FME 2009...

 

 

Takashi
Hi Takashi,

 

 

the CoordinateExtractor was actually called the CoordinateFetcher in FME 2009.

 

 

David
Ah, I forgot that! Thanks for the correction. And sorry about that, aquamarine.

 

 

Takashi

Reply


Community Stats

29,896
Posts
113,589
Replies
37,532
Members
Terms & ConditionsCookie settings