Solved

reading netcdf 4

  • 14 December 2016
  • 9 replies
  • 18 views
T
Badge

I am unable to read a netcdf4 file into fme (2015, 16 or 17).

I get the following error:

NETCDF reader: `C:/Users/phtr/Desktop/TCI_ref_to_month_europe_2000_v1.nc' not recognized as a supported file format.

NETCDF reader: Failed to open the dataset 'C:/Users/phtr/Desktop/TCI_ref_to_month_europe_2000_v1.nc'. Please ensure source data is valid and correct reader is selected

A fatal error has occurred. Check the logfile above for details

Failed to obtain any schemas from reader 'NETCDF' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information

If I convert the input to netcdf 3, FME can read it OK.

Is anyone else having this issue?  Is it a know limitation of FME?

Many thanks

icon

Best answer by jdh 14 December 2016, 16:23

View original

9 replies

jdh
Rank
Badge +22

Per the documentation FME supports up to version 3.6.2 of netCDF.

I suggest contacting support and requesting it and voting on the idea https://knowledge.safe.com/idea/28450/update-to-netcdf-format.html

david_r
Rank
Userlevel 4

@jdh is right, NetCDF 4 isnt't supported by FME and it's a known limitation.

Another, more peculiar limitation, is that FME cannot read NetCDF files >2GB on the Windows edition, although the Mac and Linux editions can, apparently.

It would be great if the NetCDF libraries could be upgraded soon.

scyphers
Rank
Badge +5

@jdhI I am strongly considering making a workaround for NetCDf4 data using either nccopy.exe as a third party tool or this python script to save netCDf4 as netCDF_classic (v3) to then read it through.

I suggested it as my #fmeuc hackathon topic/group at the very least we should be able to wrap up a workaround process to make the data readable. As there is some netCDF4 data i would very much like to tackle with FME.

the python recipe i have found is here : https://gist.github.com/guziy/8543562#file-converter-py it will require numpy, netCDF4 dependencies to make it work. And you can expect up to 8x bloat of your netcdf4 data moving to netcdf_classic so i hope you can acheive <2gb limits in your input data. If you have opportunity to come to the User Conference let solve it together, and post something in the FMEHUB!

david_r
Rank
Userlevel 4

@jdhI I am strongly considering making a workaround for NetCDf4 data using either nccopy.exe as a third party tool or this python script to save netCDf4 as netCDF_classic (v3) to then read it through.

I suggested it as my #fmeuc hackathon topic/group at the very least we should be able to wrap up a workaround process to make the data readable. As there is some netCDF4 data i would very much like to tackle with FME.

the python recipe i have found is here : https://gist.github.com/guziy/8543562#file-converter-py it will require numpy, netCDF4 dependencies to make it work. And you can expect up to 8x bloat of your netcdf4 data moving to netcdf_classic so i hope you can acheive <2gb limits in your input data. If you have opportunity to come to the User Conference let solve it together, and post something in the FMEHUB!

Great idea, I'd be glad to join, just let me know.

 

A few years ago I already did something similar using the Python GDAL bindings to access the individual cell values in a classic NetCDF without having to coerce it into individual features first. The performance gains were pretty spectacular compared to using the RasterCellCoercer.

 

tono
Badge

Any news regarding support for Netcdf4 format?

david_r
Rank
Userlevel 4

Any news regarding support for Netcdf4 format?

NetCDF4 is supported starting with FME 2018.
tono
Badge
NetCDF4 is supported starting with FME 2018.
I have tried Sentinel 5p data with FME(R) 2018.1.0.0 (20180606 - Build 18493 - WIN64) and i get this error:

 

"NETCDF reader: Opening file 'C:/Temp/data/S5P_NRTI_L2__NO2.nc'

 

NETCDF reader: C:/Temp/data/S5P_NRTI_L2__NO2.nc is a netCDF file, but without any dimensions >= 2.

 

NETCDF reader: `C:/Temp/data/S5P_NRTI_L2__NO2.nc' exists but could not be opened. File contents may be invalid, or the file format may not be supported

 

NETCDF reader: Failed to open the dataset 'C:/Temp/data/S5P_NRTI_L2__NO2.nc'. Please ensure source data is valid and correct reader is selected

 

A fatal error has occurred. Check the logfile above for details

 

Failed to obtain any schemas from reader 'NETCDF' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information

 

FME Session Duration: 0.3 seconds. (CPU: 0.1s user, 0.0s system)

 

END - ProcessID: 12744, peak process memory usage: 48604 kB, current process memory usage: 43448 kB

 

Failed to obtain any schemas from reader 'NETCDF' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information

 

Program Terminating

 

Translation FAILED."

 

 

Any clue about fail?

 

Thanks in advance

 

 

 

david_r
Rank
Userlevel 4
I have tried Sentinel 5p data with FME(R) 2018.1.0.0 (20180606 - Build 18493 - WIN64) and i get this error:

 

"NETCDF reader: Opening file 'C:/Temp/data/S5P_NRTI_L2__NO2.nc'

 

NETCDF reader: C:/Temp/data/S5P_NRTI_L2__NO2.nc is a netCDF file, but without any dimensions >= 2.

 

NETCDF reader: `C:/Temp/data/S5P_NRTI_L2__NO2.nc' exists but could not be opened. File contents may be invalid, or the file format may not be supported

 

NETCDF reader: Failed to open the dataset 'C:/Temp/data/S5P_NRTI_L2__NO2.nc'. Please ensure source data is valid and correct reader is selected

 

A fatal error has occurred. Check the logfile above for details

 

Failed to obtain any schemas from reader 'NETCDF' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information

 

FME Session Duration: 0.3 seconds. (CPU: 0.1s user, 0.0s system)

 

END - ProcessID: 12744, peak process memory usage: 48604 kB, current process memory usage: 43448 kB

 

Failed to obtain any schemas from reader 'NETCDF' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information

 

Program Terminating

 

Translation FAILED."

 

 

Any clue about fail?

 

Thanks in advance

 

 

 

Unfortunately I have no experience with Sentinel data, my suggestion would be to send the NetCDF together with a demo workspace and log file to Safe support.

 

Since the NetCDF 4 support is so recent, there's always the possibility that there's an edge case that needs to be taken into account.
tono
Badge
Unfortunately I have no experience with Sentinel data, my suggestion would be to send the NetCDF together with a demo workspace and log file to Safe support.

 

Since the NetCDF 4 support is so recent, there's always the possibility that there's an edge case that needs to be taken into account.
Done. Case sendt. Example data can be downloaded here: https://s5phub.copernicus.eu/dhus/#/home

Reply


Community Stats

28,681
Posts
109,409
Replies
36,542
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.