Question

How to convert NetCDF data to csv/excel in FME compared to the python code (included)?

  • 16 February 2023
  • 3 replies
  • 38 views

Dear Altruists, I am trying to convert NetCDF evaporation data to further processing, e.g. aggregate by month or calculating average value etc. However, I am struggling in the basic steps to unfold the multidimensional variables. When I am reading the data in NetCDF reader, it returned some attributes where I cannot see the lat/long or date file. But after using AttributeExposer to get the netcdf_subdataset_name; there I see lat, long and evaporation (apet) name but not sure how to get the values of these variables. I am given a python code in which the developer wrote a function to process the .nc data. Can anyone kindly tell me what FME transformers I need to use the replicate the python process as below:

 

def funt_1(fileName, points, var):

  nc_fid = netCDF4.Dataset(fileName, 'r')

  prbc = nc_fid.variables[var]

  lats, lons = nc_fid.variables['lat'], nc_fid.variables['lon']

  time = nc_fid.variables['time'][:]

  # Convert time to Julian

  Y = int(nc_fid.variables['time'].units.split(" ")[-1].split("-")[0])

  M = int(nc_fid.variables['time'].units.split(" ")[-1].split("-")[1])

  D = int(nc_fid.variables['time'].units.split(" ")[-1].split("-")[2])

 

  DT = [dt.date(Y, M, D) + dt.timedelta(t) for t in time]

  DT = pd.DataFrame(DT);

  DT.columns = ['DateTime']

  datecol = DT.apply(lambda x: x.DateTime.strftime('%Y/%m/%d'), axis=1)

 

  rows=[]

  for i in range(0, prbc.shape[0]):

   if var == 'pr-bc':

      nar_lats = lats[:][prbc[i, :, :].mask==False]

      nar_lons = lons[:][prbc[i, :, :].mask==False]

      prbc_um = prbc[i, :, :][prbc[i, :, :].mask==False]

      narclim_swath = geometry.SwathDefinition(nar_lons, nar_lats)

    else:

      narclim_swath = geometry.GridDefinition(lons=lons[:], lats=lats[:])

      prbc_um = prbc[i, :, :]

 

    # Define some sample points

    my_lons = np.array(points.Lon)

    my_lats = np.array(points.Lat)

    swath = geometry.SwathDefinition(lons=my_lons, lats=my_lats)

 

    wf = lambda r: 1/r**2

    result = pyresample.kd_tree.resample_custom(narclim_swath, prbc_um,

                          swath,

                          radius_of_influence=60000,

                          neighbours=3,

                          weight_funcs=wf,

                          fill_value=np.nan)

 

    row = pd.DataFrame({'Date':datecol[i], 'Lon':points.Lon,

              'Lat': points.Lat, '%s' % var : result})

    rows.append(row)

  return pd.concat(rows)


3 replies

Userlevel 5
Badge +25

You can use the RasterCellCoercer to generate a point for each pixel, which will hold a list of all the band values of that pixel. That should get you going I think.

You can use the RasterCellCoercer to generate a point for each pixel, which will hold a list of all the band values of that pixel. That should get you going I think.

Hi @Hans van der Maarel​ Thanks a lot. After running the transformer, I exposed the attributes with band values but not sure what do these values refer to. Is there anyway to know the meaning of these band values?

Userlevel 5
Badge +25

Hi @Hans van der Maarel​ Thanks a lot. After running the transformer, I exposed the attributes with band values but not sure what do these values refer to. Is there anyway to know the meaning of these band values?

That should be covered in the dataset's metadata.

Reply