• Home
  • Forums
  • FME Form
  • Data
  • NetCDF writer - would like to output one file with 100+ bands and multiple variables, but as compact as possible
Question

NetCDF writer - would like to output one file with 100+ bands and multiple variables, but as compact as possible

  • 5 May 2020
  • 1 reply
  • 7 views
dhavanv2
Rank
Badge +3

Hello, I am new with the NetCDF format so apologies if there is something obvious I am missing.

I have a bunch of GeoTIFFs which I am reading in and extracting some variable information from. Then I use a RasterBandCombiner to go from about 150 raster features to 1 with each of it now a band. When I write this out as a NetCDF file, it is about 85 MB. All the Geotiffs combined were about 220 MB.

A colleague of mine has tried this using Python (gdal, netcdf4), and is able to translate all the data into the netcdf file that is only 3 MB.

It would be cool to have this workflow set up in FME for subsequent use. So I was wondering if there are any parameters or transformers in FME that might help with this? Is it possible to only write out the non-spatial data? I tried using a GeometryRemover, but the writer gave me a fatal error which said it requires raster geometry. Is this why the file output from FME is so much larger in size - the geometry?

I also tried bringing in the 3MB file in to FME and writing it back out again and it was 85MB without me doing anything to it.

Can anyone think of what might be going on? Any help would be appreciated! Cheers. Sorry I can't share the data, but I'm only after ideas if anyone has tried this before

1 reply

david_r
Rank
Userlevel 4

NetCDF is a fairly complex format with a lot of features, not all supported by FME. In addition there are different conventions that you can add on top of the format to further complicate things, e.g. http://cfconventions.org/. Without more details about the contents of your file, it is difficult to say why your file is so much bigger.

But I think the most important thing to check is the band interpretation and how they translate into the NetCDF. Meaning that 150 bands of 32-bit floating point values (Real32) will of course take up a lot more space than 150 bands of 8-bit integer (Int8) values. There's even a mechanism in NetCDF with offsets and scale factors that can help you store e.g. floating point values in integers.

In FME, look specifically for the RasterBandInterpretationCoercer and RasteInterpretationCoercer transformers.

You may also want to use the 'ncdump' utility on your colleagues and your own NetCDF file to compare exactly what is going on. That utility can be a life saver when working with NetCDF.

Reply


Community Stats

28,669
Posts
109,364
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.