Question

Problem with Python module xarray used in FME Flow 2023.2.2

  • 8 April 2024
  • 3 replies
  • 43 views

Badge +1

Hello everyone,

I have a problem with FME Flow 2023.2.2 and the module xarray 2024.2.0. I need xarray for reading netCDF data and have created a Python script for this in FME Form 2023.2.2. This also runs without any problems. However, when I upload this workspace to FME Flow and start it there, I always get the message that the run has failed.
After some research I found out that FME Flow has problems executing the command xarray.open_dataset(). I'll post the script and the logs of FME Form and Flow:

import fmeobjects
import xarray as xr


class FeatureCreator(object):
"""Template Class Interface:
When using this class, make sure its name is set as the value of the 'Class to Process Features'
transformer parameter.
"""

def __init__(self):
"""Base constructor for class members."""
pass

def input(self, feature: fmeobjects.FMEFeature):

print("Assign file")
file = '/dafis/OpMod/t_file_fine/t_file_fine_2024040300_006_012.nc'
print ('File: ' + file)
print("Load Dataset")
ds = xr.open_dataset(file)
print('Finished Load Dataset')
print(xr.show_versions())

newFeature = fmeobjects.FMEFeature()
self.pyoutput(newFeature)

Here is the Log from FME Form:

Assign file
File: /dafis/OpMod/t_file_fine/t_file_fine_2024040300_006_012.nc
Load Dataset
E:\Python_3_9\lib\site-packages\xarray\conventions.py:286: SerializationWarning: variable 'temp' has multiple fill values {-31111, -32222} defined, decoding all values to NaN.
  var = coder.decode(var, name=name)
E:\Python_3_9\lib\site-packages\xarray\conventions.py:286: SerializationWarning: variable 'salt' has multiple fill values {-31111, -32222} defined, decoding all values to NaN.

  var = coder.decode(var, name=name)
Finished Load Dataset

Log from FME Form

 

Here the Log from FME Flow:

54 2024-4-8 10:15:40 | Assign file
55 2024-4-8 10:15:40 | File: /dafis/OpMod/t_file_fine/t_file_fine_2024040300_006_012.nc
56 2024-4-8 10:15:40 | Load Dataset
Log from FME Flow

 

As you can see, it simply aborts when executing open_dataset(). Does anyone know this problem? Is there a way for FME to output the error. Even with try - catch no error is output. xarray was installed with pip on the server.


3 replies

Userlevel 5

At first glance it looks like a version or requirement mismatch.

Installing packages with a lot of dependencies, like xarray, can be tricky since they might depend on several other modules that might already be part of the FME installation, but in a different and sometimes not-fully-compatible version.

Did you specify the --target option when doing the pip install? Did you get any warnings during the installation of xarray on either Flow or Form? Are you certain that FME Flow and Form are using the same version of xarray and all dependencies?

I would recommend doing a pip list on both FME Form and Flow to see what differs, e.g.

c:\program files\fme\fme.exe python -m pip list

and

c:\program files\fme flow\server\fme\fme.exe python -m pip list

Also check the sys.path entries on both platforms so see if there are several versions of the same package installed in different locations of the search path. Numpy is a pretty common culprit since it’s bundled with FME inside FME_HOME, but many packages require a different version. The default behavior of FME is then to install a second version of numpy in the user’s plugin folder, which can create weird behavior since you cannot easily control which package loads which version of numpy.

Badge +1

We did not use  --target for installation. The comparison with both tests has also shown that the two module packages are the same and use the same versions.
Since FME Flow is installed on Linux, my colleague tried the import once with the installed interpreter and the other time with the FME Flow Python interpreter. With the former, the import of xarray and also fmeobjects worked without any problems, with the latter the script was aborted. However, the error message 

fme: /io/netcdf-c-4.9.2/libdispatch/drc.c:174: ncrc_setrchome: Assertion `ncg && ncg->home' failed

was displayed.


We now have the following questions:
Can the default interpreter in FME FLOW be modified similarly to FME Form or at least for the runtime of the Python script?
If there is a problem with the NetCDF module from FME and the netCDF library for Python, can the import of the FME module be prevented?

Userlevel 1
Badge +6

Hi @manuel_h ,

If you run workspaces from FME Flow that reference Python, the FME Engines must know which Python interpreter to use. Workspaces built with FME Form tell FME which version of Python is required to run Python code through the Python Compatibility workspace parameter.

The Python interpreter that the FME Engines actually use is the one that most closely matches the Python Compatibility workspace parameter, which, by default, is FME’s latest supported version of Python 3. To change the default Python interpreter used by the FME Engines, see Changing the Python Interpreter (Windows only).

If you have a custom python module you need to use on FME Flow that is not part of the Python standard library, use the Upload tool on the Resources page of the FME Flow Web User Interface, and upload it to the folder under Engine\Plugins\python that corresponds to the Python release compatible with the module (for example, python36). Please see this documentation for more information. 

 

Could you elaborate on what you mean by can the import of the FME module be prevented?

 

Thank you!

Kezia

Reply