Since you're using 64-bit FME, the first thing I'd do is verify the size of your virtual memory paging file. Make sure you set it to resize automatically:
One thing that comes to mind is the 4GB file size limit on GeoTIFFs. There's a parameter ("Enable BigTIFF support" on the writer feature type, it defaults to no, can you check whether you have that set to yes?
One thing that comes to mind is the 4GB file size limit on GeoTIFFs. There's a parameter ("Enable BigTIFF support" on the writer feature type, it defaults to no, can you check whether you have that set to yes?
I've enabled bigtiff already!!
I've attached the workbench, if that helps, thanks!!
geotiff2mosaic.fmw
You could also try setting the Snapping Type in the RasterMosaicker to "Offset" rather than "Resample" and see if that makes a difference in memory consumption.
Since you're using 64-bit FME, the first thing I'd do is verify the size of your virtual memory paging file. Make sure you set it to resize automatically:
The virtual memory paging file is set to "Automatically". Please see attached image. Thanks for the help thus far.
You could also try setting the Snapping Type in the RasterMosaicker to "Offset" rather than "Resample" and see if that makes a difference in memory consumption.
I'm currently running with "Offset" rather than "Resample", it appears to be still using as much memory. I'm getting the following warnings in the log file:
2017-11-01 12:20:55|1450.0| 2.2|WARN |ResourceManager: Detected system low memory state using 0 bytes of system available memory
2017-11-01 12:20:55|1450.0| 0.0|WARN |ResourceManager: Reducing process memory usage. Start freeing memory at: 22.86 GB
I will leave it running but I don't think there has been any improvement, thanks for the help.
The problem with some GeoTIFFs is that they compress really well on file, but when processing them with FME they have to be decompressed to memory, which can make the memory consumption explode. This means that even if your 600 GeoTIFFs "only" take up 400 MB on disk, they could potentially consume a huge amount of memory when decompressed into FME raster features.
As a test, you could try taking a single GeoTIFF and write it with compression turned off to get a feel for how big it will be in memory.
In short: You may have to find a way to group your files to avoid mosaicking all 600 files at once.
The problem with some GeoTIFFs is that they compress really well on file, but when processing them with FME they have to be decompressed to memory, which can make the memory consumption explode. This means that even if your 600 GeoTIFFs "only" take up 400 MB on disk, they could potentially consume a huge amount of memory when decompressed into FME raster features.
As a test, you could try taking a single GeoTIFF and write it with compression turned off to get a feel for how big it will be in memory.
In short: You may have to find a way to group your files to avoid mosaicking all 600 files at once.
I know from working with this data previously that they are quite large uncompressed.
Does this have anything to do with the physical ram on the computer, would getting more ram improve the situation?
Otherwise, is grouping them into smaller clusters the only option?
Thanks very much for your help.
B
I know from working with this data previously that they are quite large uncompressed.
Does this have anything to do with the physical ram on the computer, would getting more ram improve the situation?
Otherwise, is grouping them into smaller clusters the only option?
Thanks very much for your help.
B
What kind of number are we talking about if you take a single uncompressed raster and multiply by 600?
What kind of number are we talking about if you take a single uncompressed raster and multiply by 600?
Somewhere in the region of 40gb for the 600 files. :)
Somewhere in the region of 40gb for the 600 files. :)
You'll have to verify this with Safe, but I suspect that since there is no Group By on the RasterMosaicker, FME will try to read all of the 60GB into memory to resolve the mosaic. Which is about the double of what you have currently available.
You could potentially increase the page file to 128 GB or so if you have the space for it, but it would be horrible from a performance standpoint as disks are an order of magnitude slower than RAM.
The problem with some GeoTIFFs is that they compress really well on file, but when processing them with FME they have to be decompressed to memory, which can make the memory consumption explode. This means that even if your 600 GeoTIFFs "only" take up 400 MB on disk, they could potentially consume a huge amount of memory when decompressed into FME raster features.
As a test, you could try taking a single GeoTIFF and write it with compression turned off to get a feel for how big it will be in memory.
In short: You may have to find a way to group your files to avoid mosaicking all 600 files at once.
Is there a correlation between the amount of virtual memory I can allocate and the physical memory I already have?
Thank you!
Is there a correlation between the amount of virtual memory I can allocate and the physical memory I already have?
Thank you!
It depends entirely on your OS, but I believe that in e.g. Windows 10 you can specify as much as 16TB of virtual memory regardless of your physical memory, but you'll have to look it up or ask on an appropriate forum.
It depends entirely on your OS, but I believe that in e.g. Windows 10 you can specify as much as 16TB of virtual memory regardless of your physical memory, but you'll have to look it up or ask on an appropriate forum.
Thanks very much for all your help, you were very helpful!! :)
Thanks very much for all your help, you were very helpful!! :)
Glad that I could be of help even if there was no easy solution.
Just an update to this for anyone interested. I upgraded my ram from 8gb to 16gb and managed to get the mosaic to complete. B
Just an update to this for anyone interested. I upgraded my ram from 8gb to 16gb and managed to get the mosaic to complete. B
Excellent! Thanks for sharing.
