Solved

Unable to use ListSearcher on NetCDF Readers netcdf_coordinate_variable list

  • 13 January 2023
  • 2 replies
  • 3 views

Badge +1

I'm encountering an issue reading NetCDF files I got from here:

https://dap.ceda.ac.uk/badc/cmip6/data/CMIP6/CMIP/AS-RCEC/TaiESM1/historical/r1i1p1f1/day/pr/gn/latest/

 

When attempting to use a ListSearcher to search the "netcdf_coordinate_variable" list for an element with axis = T, I get no matches, even though its the first element.

I also get no results using a ListExploder and a Tester to test the axis attribute.

In fact the only attribute from the list that will match is "name".

 

Looking at the list there are 2 elements, and the second element only has a "name" attribute and no others.

I could see why this might trip up the ListSearcher, but I don't understand why the ListExploder + Tester method also fails to match?

 

This is with FME 2022.1.2 on MacOS. I've attached a workspace (data linked above) and a screen shot showing the list has an element with an axis value of "T".

image

icon

Best answer by david_r 16 January 2023, 11:29

View original

2 replies

Userlevel 4

Weirdly enough, FME is adding a zero-byte at the end of the X, Y and T variable names in this particular NetCDF. I suspect this is a bug, because this does not happen when reading the NetCDF from Python. You can see this special character if you use the BinaryEncoder on the variable name ("axis"), you'll get "5400" rather than just "54" (upper case T). I would consider opening a support ticket with your FME reseller/Safe Software.

You can work around this issue by using the following regex in the Tester or the ListSearcher:

^T.*$

Alternatively, you can use a StringSearcher to remove any zero-bytes from the variable name before further processing:

image

Badge +1

Weirdly enough, FME is adding a zero-byte at the end of the X, Y and T variable names in this particular NetCDF. I suspect this is a bug, because this does not happen when reading the NetCDF from Python. You can see this special character if you use the BinaryEncoder on the variable name ("axis"), you'll get "5400" rather than just "54" (upper case T). I would consider opening a support ticket with your FME reseller/Safe Software.

You can work around this issue by using the following regex in the Tester or the ListSearcher:

^T.*$

Alternatively, you can use a StringSearcher to remove any zero-bytes from the variable name before further processing:

image

Thanks, the BinaryEncoder is what I was looking for but was suffering from "last thing on a Friday" haze!

I've modified your suggested RegEx pattern to use the unicode regex and make the search a bit more explicit:

^T\p{Cc}*$

Though there is another FME quirk, as you should just be able to use the RegEx below, which works in a StringSearcher, but it throws an error in the ListSearcher:

^T\p{C}*$

The error is in the ListSearcher is:

libc++abi: terminating with uncaught exception of type boost::wrapexcept<boost::regex_error>: Escape sequence was neither a valid property nor a valid character class name.  The error occurred while parsing the regular expression: '^T(\p{C}>>>HERE>>>)*'.

I've finally used a python caller to strip out the unprintable characters in the resulting attibutes, borrowing code from here.

 

FME looks like Its added the same "00" (Null) character to all of the list attributes for element 0 that don't also exist on element 1 (i.e. every attrib. except "name"). I haven't tested this theory yet.

Reply