Question

Best method to reproject Large raster

  • 17 September 2019
  • 7 replies
  • 38 views
kieranmg
Badge +3

Hi All,

I have 15 ECW files with a total of 20GB~ and another one large area ECW of 44GB.

I also have the tif tiles for all of these images, about a total of 200GB and 600GB respectively

Goal:

Long term goal is mosaic 15 files on top of the 1 large file and reproject to use in GIS viewer.

Short term goal I would settle for having only the large image reprojected!(and let the App look after the rest on-the-fly)

 

I've only used the ECW files in my attempts so far to mosiac and reproject but the workspace slows dramatically it seems half way through. Then begins to show something like 'retry attempt 2' while processing

-ECW writer: Completed 50.83% of intermediate processing

Assuming at this point i'm running low on RAM. Using workbench 2019, 64bit, 32GB RAM and 1 TB of space

I've since doubled my memory to 64, without really any understanding as to what i need.

 

Hoping for some direction as to what resources i need to complete this job (without waiting weeks), or pointers as to a better method, i.e. using the TIFs instead, or somehow breaking the images up further.

 

Appreciate any advice given

7 replies

featuremichael
Rank
Badge +6

Hi,

i think this process can not be handled by your Computer.

Just 4 Info:

i used a 35MB ECW-File -> and a Reprojector and wrote a FFS-File.

35MB File -> 1,5 GB Ram usage and a FFS-File with 373MB (To Store the Data to Write as ECW-File)

I think u need a good server to perform this LARGE amount of Data.
david_r
Rank
Userlevel 4

Reprojecting rasters can be extremely time- and memory-consuming.

This very much depends on the type of transformation that's needed, but sometimes you can get away with using e.g. the Offsetter or the Affiner to rapidly get a suitable approximation.

Alternatively you will probably have to split up your rasters and reproject them in smaller chunks (Tiler) before merging them back together (RasterMosaicker) in the end.

 

G
Badge +3

@kieranmg

As @david_r says.

I would tile the process and use a workspace caller to prevent to much memory hogging.

Workspace caller set to wait, give it space to clear memory.

 

For the mosaicking I would do this as well but after using extents (rasterextentcoercer) to set up groups.

So you could prevent mosaicking everything at once.

helmoet
Badge +8

Hi @kieranmg, I would opt for the suggestion of david_r, create a grid that clips your raster into smaller blocks and reproject the smaller blocks. After that you could try and mosaik them together. You might want to consider using a virtual raster format (VRT) in stead of one big raster file.

itay
Rank
Badge +16

Hi @kieranmg,

Is a large raster necessary? can you avoid reprojecting+ mosaicking ? If your goal is to display the raster(s) in a GIS viewer, you could consider servicing the tiles.

That way you only need to reproject the tiles, which is more efficient.

sigtill
Rank
Badge +21

Which software do you use to read the raster? You mention "GIS viewer" . Which Desktop software or Online viewer should show the data?

The real SigTill
kieranmg
Badge +3

Thanks all, seems most feedback suggest tiling before reprojecting. I've failed this approach in the past as i had issues with black/white collars as ecw doesn't support nodata, adding the alpha band solves this but my s/w (PB spectrum spatial) doesn't support alpha band ecw yet. As suggested maybe another format is needed. Food for thought, thanks again for all your suggestions, theres 101 ways to skin a cat so it can be difficult for a newcomer to find the 'best' way.

 

Also, perhaps a point of reference for anyone thinking of doing something similar, i needed the following to reproject one large ecw of 44GB

w2016 server - 64GB-RAM - Disk space didn't seem to be a problem

 

FME Session Duration: 3 days 3 hours 17 minutes 45.4 seconds. (CPU: 354403.7s user, 168235.2s system)

END - ProcessID: 912, peak process memory usage: 59863728 kB, current process memory usage: 186432 kB

Reply


Community Stats

28,681
Posts
109,410
Replies
36,542
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.