Question

How to deal with thousands of TIFF files?

  • 4 April 2013
  • 9 replies
  • 42 views

Badge +2

Hello all,

I would like to know if there is a way to ease the rendering of many tif files. Individually, they are really small, a few kbites but added togheter.. pff.

I tried creating overviews with gdalwrap but it does a bad jos in preserving aspect. Is there a way to maybe handle these files with FME Desktop?

The main goal would be to display them on a map and I would like to find out the best way to do that if we are dealing with so many..

 

Cheers,


9 replies

Userlevel 4
Hi Robert,

 

 

perhaps you could benefit from reducing the numbers by combining the rasters into larger ones.

 

 

The RasterBandCombiner might then be the one you're looking for, particularly with a Group by attribute.

 

 

David
Badge +2

oh, ok. I will try this one out and I will get back with the results as soon as I can.

Thank you David!

Userlevel 4
Badge +13
Hi,

 

 

Are the rasters overlapping? if so consider not using raster which are totally overlapped by other rasters and lowering the number by mosaicking them always helps. Keep in mind that you might have to resample if the rasters have different resolutions.

 

Hope this helps.
Badge +2

Hello Itay,

No, all my geotiff files have the same format, and none of them overlap. It s just the fact that there are too many and every time - at every zoom level to be more precise (using gdal tileindex) it has to read all the files in order to render them... 

I was thinking of creating rasters which should contain more tiff files, so it would have to read less files.. But I have no idea how to deal with such an amount of TIFF files.. 

I have to try out several ways I guess

Userlevel 4
Badge +13
Hi Robert,

 

 

In that case I would use the raster mosaicker to create bigger (but less) rasters.

 

 

I would extract the tiles bounding boxes and create a common ID (based on afeature if available or visually selecting tiles) after that you can use the mosaicker set on a  group by (your common ID)

 

Hope this helps.
Badge +14
Yep, the RasterMosaiker will glue all your tiles together. You could also do some grouping here based on an ID or filename too if there are too many tiles for 1 image. That said a mid flow RasterCheckPointer will take the data out of memory and pin it to disk prior to you running the RasterTiler across your large image to chop it up into more manageable and fewer tiles!

 

 

Good luck.
Userlevel 4
Badge +25
Hi Robert,

 

I'm not really clear what the specific case is here. If you are looking for an alternative method for indexing than gdaltindex then in FME there is a transformer called the WebMapTiler. That transformer will tile your data in a way that is suitable for use in a mapping application. It has numerous controls over the quality and lets you choose different index types. It will be slow - this is not a simple task - but it certainly shouldn't be worse than gdal.

 

 

If you want to pre-process your data in a way that will speed up gdaltindex then you could reduce the resolution with a RasterResampler in FME. This will reduce quality, but the resulting files should be smaller and easier to handle.

 

 

Hope this is useful

 

 

Mark Ireland

 

Product Evangelist

 

Safe Software Inc
Badge +2

Hello all,

Thanks for the feedback, I am sure it will pan out in the end... 

@mark2atsafe: No, I am not looking for an alternative for gdaltileindex..I am happy with how it works. I was looking for an alternative to somehow reduce the number of records in the tileindex.

 

 

You see, if in the tileindex I have 10,000 records, at every zoom level, in will search for all of these files, so I am sending the webmapping tool (vie gdaltileindex) to look for 10,000 different paths at each zoom level which takes a lot of time.I want to gain some by maybe adding some of those files into a single raster. (maybe a raster for each of the folders which contain hundreds of tiff files?). I would like to preserv the quality of the tiff files as best as I can, and maybe use a LZW compresion.

I will have to try the options listed here at home so I will not mess up the whole database. Maybe on a folder to start with, and then with the rest of them.

Will get back to you as soon as I have some results.

rasterege's tiff image viewer vb.net can  do all the trick: zoom convert and compress. it is a container for many images at one time and process in batch.  you can combine them into one big file.i

Reply