Solved

Raster Mosaicker Insufficient memory

bjudes
Rank
Badge +3

I'm currently mosaicking 600 geotiffs, these amount to 400mb. Everytime the writer gets to between 80%-90% completed I get the following error:

FATAL |Insufficient memory available -- error code was 2 - please read the FME Help section 'Adjusting Memory Resources' for workarounds.

Out of memory. Program terminating. Please read the FME Help section 'Adjusting Memory Resources' for workarounds.

I have run several others mosaicks with geotiffs in the same workbench except smaller amounts (between 150mb-350mb) and there has been no issues. The smaller mosaicks also seem to complete at a much quicker rate. Anything seemingly over the 350/400mb mark seems to crash the program.

I have FME 2017.1 (64bit). I have 8gb of ram.

Thanks very much for any help!!

icon

Best answer by david_r 1 November 2017, 13:48

View original
1 Attachment

17 replies

david_r
Rank
Userlevel 4

Since you're using 64-bit FME, the first thing I'd do is verify the size of your virtual memory paging file. Make sure you set it to resize automatically:

redgeographics
Rank
Userlevel 4
Badge +25

One thing that comes to mind is the 4GB file size limit on GeoTIFFs. There's a parameter ("Enable BigTIFF support" on the writer feature type, it defaults to no, can you check whether you have that set to yes?

bjudes
Rank
Badge +3

One thing that comes to mind is the 4GB file size limit on GeoTIFFs. There's a parameter ("Enable BigTIFF support" on the writer feature type, it defaults to no, can you check whether you have that set to yes?

I've enabled bigtiff already!!

 

I've attached the workbench, if that helps, thanks!!

 

geotiff2mosaic.fmw

 

 

david_r
Rank
Userlevel 4

You could also try setting the Snapping Type in the RasterMosaicker to "Offset" rather than "Resample" and see if that makes a difference in memory consumption.

bjudes
Rank
Badge +3

Since you're using 64-bit FME, the first thing I'd do is verify the size of your virtual memory paging file. Make sure you set it to resize automatically:

The virtual memory paging file is set to "Automatically". Please see attached image. Thanks for the help thus far.

 

 

 

bjudes
Rank
Badge +3

You could also try setting the Snapping Type in the RasterMosaicker to "Offset" rather than "Resample" and see if that makes a difference in memory consumption.

I'm currently running with "Offset" rather than "Resample", it appears to be still using as much memory. I'm getting the following warnings in the log file:

 

 

2017-11-01 12:20:55|1450.0| 2.2|WARN |ResourceManager: Detected system low memory state using 0 bytes of system available memory

 

2017-11-01 12:20:55|1450.0| 0.0|WARN |ResourceManager: Reducing process memory usage. Start freeing memory at: 22.86 GB

 

 

I will leave it running but I don't think there has been any improvement, thanks for the help.

 

 

david_r
Rank
Userlevel 4

The problem with some GeoTIFFs is that they compress really well on file, but when processing them with FME they have to be decompressed to memory, which can make the memory consumption explode. This means that even if your 600 GeoTIFFs "only" take up 400 MB on disk, they could potentially consume a huge amount of memory when decompressed into FME raster features.

As a test, you could try taking a single GeoTIFF and write it with compression turned off to get a feel for how big it will be in memory.

In short: You may have to find a way to group your files to avoid mosaicking all 600 files at once.

bjudes
Rank
Badge +3

The problem with some GeoTIFFs is that they compress really well on file, but when processing them with FME they have to be decompressed to memory, which can make the memory consumption explode. This means that even if your 600 GeoTIFFs "only" take up 400 MB on disk, they could potentially consume a huge amount of memory when decompressed into FME raster features.

As a test, you could try taking a single GeoTIFF and write it with compression turned off to get a feel for how big it will be in memory.

In short: You may have to find a way to group your files to avoid mosaicking all 600 files at once.

I know from working with this data previously that they are quite large uncompressed.

 

 

Does this have anything to do with the physical ram on the computer, would getting more ram improve the situation?

 

 

Otherwise, is grouping them into smaller clusters the only option?

 

 

Thanks very much for your help.

 

 

B

 

 

david_r
Rank
Userlevel 4
I know from working with this data previously that they are quite large uncompressed.

 

 

Does this have anything to do with the physical ram on the computer, would getting more ram improve the situation?

 

 

Otherwise, is grouping them into smaller clusters the only option?

 

 

Thanks very much for your help.

 

 

B

 

 

What kind of number are we talking about if you take a single uncompressed raster and multiply by 600?
bjudes
Rank
Badge +3
What kind of number are we talking about if you take a single uncompressed raster and multiply by 600?
Somewhere in the region of 40gb for the 600 files. :)

 

 

david_r
Rank
Userlevel 4
Somewhere in the region of 40gb for the 600 files. :)

 

 

You'll have to verify this with Safe, but I suspect that since there is no Group By on the RasterMosaicker, FME will try to read all of the 60GB into memory to resolve the mosaic. Which is about the double of what you have currently available.

 

 

You could potentially increase the page file to 128 GB or so if you have the space for it, but it would be horrible from a performance standpoint as disks are an order of magnitude slower than RAM.
bjudes
Rank
Badge +3

The problem with some GeoTIFFs is that they compress really well on file, but when processing them with FME they have to be decompressed to memory, which can make the memory consumption explode. This means that even if your 600 GeoTIFFs "only" take up 400 MB on disk, they could potentially consume a huge amount of memory when decompressed into FME raster features.

As a test, you could try taking a single GeoTIFF and write it with compression turned off to get a feel for how big it will be in memory.

In short: You may have to find a way to group your files to avoid mosaicking all 600 files at once.

Is there a correlation between the amount of virtual memory I can allocate and the physical memory I already have?

 

Thank you!

 

 

david_r
Rank
Userlevel 4
Is there a correlation between the amount of virtual memory I can allocate and the physical memory I already have?

 

Thank you!

 

 

It depends entirely on your OS, but I believe that in e.g. Windows 10 you can specify as much as 16TB of virtual memory regardless of your physical memory, but you'll have to look it up or ask on an appropriate forum.
bjudes
Rank
Badge +3
It depends entirely on your OS, but I believe that in e.g. Windows 10 you can specify as much as 16TB of virtual memory regardless of your physical memory, but you'll have to look it up or ask on an appropriate forum.
Thanks very much for all your help, you were very helpful!! :)

 

 

david_r
Rank
Userlevel 4
Thanks very much for all your help, you were very helpful!! :)

 

 

Glad that I could be of help even if there was no easy solution.
bjudes
Rank
Badge +3

Just an update to this for anyone interested. I upgraded my ram from 8gb to 16gb and managed to get the mosaic to complete. B

david_r
Rank
Userlevel 4

Just an update to this for anyone interested. I upgraded my ram from 8gb to 16gb and managed to get the mosaic to complete. B

Excellent! Thanks for sharing.

Reply


Community Stats

28,634
Posts
109,179
Replies
36,513
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.