Skip to main content
Solved

Raster Mosaicker Insufficient memory

bjudes
Enthusiast
Forum|alt.badge.img+10

I'm currently mosaicking 600 geotiffs, these amount to 400mb. Everytime the writer gets to between 80%-90% completed I get the following error:

FATAL |Insufficient memory available -- error code was 2 - please read the FME Help section 'Adjusting Memory Resources' for workarounds.

Out of memory. Program terminating. Please read the FME Help section 'Adjusting Memory Resources' for workarounds.

I have run several others mosaicks with geotiffs in the same workbench except smaller amounts (between 150mb-350mb) and there has been no issues. The smaller mosaicks also seem to complete at a much quicker rate. Anything seemingly over the 350/400mb mark seems to crash the program.

I have FME 2017.1 (64bit). I have 8gb of ram.

Thanks very much for any help!!

Best answer by david_r

The problem with some GeoTIFFs is that they compress really well on file, but when processing them with FME they have to be decompressed to memory, which can make the memory consumption explode. This means that even if your 600 GeoTIFFs "only" take up 400 MB on disk, they could potentially consume a huge amount of memory when decompressed into FME raster features.

As a test, you could try taking a single GeoTIFF and write it with compression turned off to get a feel for how big it will be in memory.

In short: You may have to find a way to group your files to avoid mosaicking all 600 files at once.

View original
1 Attachments
Did this help you find an answer to your question?

17 replies

david_r
Evangelist
  • david_rEvangelist
  • November 1, 2017

Since you're using 64-bit FME, the first thing I'd do is verify the size of your virtual memory paging file. Make sure you set it to resize automatically:

redgeographics
Celebrity
Forum|alt.badge.img+48

One thing that comes to mind is the 4GB file size limit on GeoTIFFs. There's a parameter ("Enable BigTIFF support" on the writer feature type, it defaults to no, can you check whether you have that set to yes?

bjudes
Enthusiast
Forum|alt.badge.img+10
  • bjudesEnthusiast
  • Author
  • Enthusiast
  • November 1, 2017
redgeographics wrote:

One thing that comes to mind is the 4GB file size limit on GeoTIFFs. There's a parameter ("Enable BigTIFF support" on the writer feature type, it defaults to no, can you check whether you have that set to yes?

I've enabled bigtiff already!!

 

I've attached the workbench, if that helps, thanks!!

 

geotiff2mosaic.fmw

 

 

david_r
Evangelist
  • david_rEvangelist
  • November 1, 2017

You could also try setting the Snapping Type in the RasterMosaicker to "Offset" rather than "Resample" and see if that makes a difference in memory consumption.

bjudes
Enthusiast
Forum|alt.badge.img+10
  • bjudesEnthusiast
  • Author
  • Enthusiast
  • November 1, 2017
david_r wrote:

Since you're using 64-bit FME, the first thing I'd do is verify the size of your virtual memory paging file. Make sure you set it to resize automatically:

The virtual memory paging file is set to "Automatically". Please see attached image. Thanks for the help thus far.

 

 

 

bjudes
Enthusiast
Forum|alt.badge.img+10
  • bjudesEnthusiast
  • Author
  • Enthusiast
  • November 1, 2017
david_r wrote:

You could also try setting the Snapping Type in the RasterMosaicker to "Offset" rather than "Resample" and see if that makes a difference in memory consumption.

I'm currently running with "Offset" rather than "Resample", it appears to be still using as much memory. I'm getting the following warnings in the log file:

 

 

2017-11-01 12:20:55|1450.0| 2.2|WARN |ResourceManager: Detected system low memory state using 0 bytes of system available memory

 

2017-11-01 12:20:55|1450.0| 0.0|WARN |ResourceManager: Reducing process memory usage. Start freeing memory at: 22.86 GB

 

 

I will leave it running but I don't think there has been any improvement, thanks for the help.

 

 

david_r
Evangelist
  • david_rEvangelist
  • Best Answer
  • November 1, 2017

The problem with some GeoTIFFs is that they compress really well on file, but when processing them with FME they have to be decompressed to memory, which can make the memory consumption explode. This means that even if your 600 GeoTIFFs "only" take up 400 MB on disk, they could potentially consume a huge amount of memory when decompressed into FME raster features.

As a test, you could try taking a single GeoTIFF and write it with compression turned off to get a feel for how big it will be in memory.

In short: You may have to find a way to group your files to avoid mosaicking all 600 files at once.

bjudes
Enthusiast
Forum|alt.badge.img+10
  • bjudesEnthusiast
  • Author
  • Enthusiast
  • November 1, 2017
david_r wrote:

The problem with some GeoTIFFs is that they compress really well on file, but when processing them with FME they have to be decompressed to memory, which can make the memory consumption explode. This means that even if your 600 GeoTIFFs "only" take up 400 MB on disk, they could potentially consume a huge amount of memory when decompressed into FME raster features.

As a test, you could try taking a single GeoTIFF and write it with compression turned off to get a feel for how big it will be in memory.

In short: You may have to find a way to group your files to avoid mosaicking all 600 files at once.

I know from working with this data previously that they are quite large uncompressed.

 

 

Does this have anything to do with the physical ram on the computer, would getting more ram improve the situation?

 

 

Otherwise, is grouping them into smaller clusters the only option?

 

 

Thanks very much for your help.

 

 

B

 

 

david_r
Evangelist
  • david_rEvangelist
  • November 1, 2017
bjudes wrote:
I know from working with this data previously that they are quite large uncompressed.

 

 

Does this have anything to do with the physical ram on the computer, would getting more ram improve the situation?

 

 

Otherwise, is grouping them into smaller clusters the only option?

 

 

Thanks very much for your help.

 

 

B

 

 

What kind of number are we talking about if you take a single uncompressed raster and multiply by 600?
bjudes
Enthusiast
Forum|alt.badge.img+10
  • bjudesEnthusiast
  • Author
  • Enthusiast
  • November 1, 2017
david_r wrote:
What kind of number are we talking about if you take a single uncompressed raster and multiply by 600?
Somewhere in the region of 40gb for the 600 files. :)

 

 

david_r
Evangelist
  • david_rEvangelist
  • November 1, 2017
bjudes wrote:
Somewhere in the region of 40gb for the 600 files. :)

 

 

You'll have to verify this with Safe, but I suspect that since there is no Group By on the RasterMosaicker, FME will try to read all of the 60GB into memory to resolve the mosaic. Which is about the double of what you have currently available.

 

 

You could potentially increase the page file to 128 GB or so if you have the space for it, but it would be horrible from a performance standpoint as disks are an order of magnitude slower than RAM.
bjudes
Enthusiast
Forum|alt.badge.img+10
  • bjudesEnthusiast
  • Author
  • Enthusiast
  • November 1, 2017
david_r wrote:

The problem with some GeoTIFFs is that they compress really well on file, but when processing them with FME they have to be decompressed to memory, which can make the memory consumption explode. This means that even if your 600 GeoTIFFs "only" take up 400 MB on disk, they could potentially consume a huge amount of memory when decompressed into FME raster features.

As a test, you could try taking a single GeoTIFF and write it with compression turned off to get a feel for how big it will be in memory.

In short: You may have to find a way to group your files to avoid mosaicking all 600 files at once.

Is there a correlation between the amount of virtual memory I can allocate and the physical memory I already have?

 

Thank you!

 

 

david_r
Evangelist
  • david_rEvangelist
  • November 1, 2017
bjudes wrote:
Is there a correlation between the amount of virtual memory I can allocate and the physical memory I already have?

 

Thank you!

 

 

It depends entirely on your OS, but I believe that in e.g. Windows 10 you can specify as much as 16TB of virtual memory regardless of your physical memory, but you'll have to look it up or ask on an appropriate forum.
bjudes
Enthusiast
Forum|alt.badge.img+10
  • bjudesEnthusiast
  • Author
  • Enthusiast
  • November 1, 2017
david_r wrote:
It depends entirely on your OS, but I believe that in e.g. Windows 10 you can specify as much as 16TB of virtual memory regardless of your physical memory, but you'll have to look it up or ask on an appropriate forum.
Thanks very much for all your help, you were very helpful!! :)

 

 

david_r
Evangelist
  • david_rEvangelist
  • November 1, 2017
bjudes wrote:
Thanks very much for all your help, you were very helpful!! :)

 

 

Glad that I could be of help even if there was no easy solution.
bjudes
Enthusiast
Forum|alt.badge.img+10
  • bjudesEnthusiast
  • Author
  • Enthusiast
  • November 3, 2017

Just an update to this for anyone interested. I upgraded my ram from 8gb to 16gb and managed to get the mosaic to complete. B

david_r
Evangelist
  • david_rEvangelist
  • November 3, 2017
bjudes wrote:

Just an update to this for anyone interested. I upgraded my ram from 8gb to 16gb and managed to get the mosaic to complete. B

Excellent! Thanks for sharing.

Reply


Helpful Members This Week

Community Stats

30,942
Posts
117,256
Replies
38,742
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.