Skip to main content
Question

HDF5 configuration for FME Server 2019.0 on Ubuntu 18.04


martinkoch
Supporter
Forum|alt.badge.img+18

I try to install, among others, the keras modules for use in the PythonCallers etc...

I installed them doing

sudo /opt/fme-engine-2019/fme python -m pip install --target=/opt/fmeserver/resources/engine/Plugins/Python keras keras_metrics tensorflow scipy

and reorder the sys.path so all stuff in this Plugins directory comes before all other python modules. This normally works to override versions of packages and such.

When doing

import keras

the engine running the workspace terminates instantly.

Now when I do

/opt/fme-engine-2019/fme python
>> import keras

I get a very long summary of the HDF5 configuration from which I pasted the interesting bits below...

Strangely I do not have the libhdf5-100 package installed on my system, and installing this (version 1.10) does not solve the problem.

Any idea's how to go about this? No wonder the engine craches when a python-command results in an enormous amount of text being spewed out to STD-OUT.

Warning! ***HDF5 library version mismatched error***
The HDF5 header files used to compile this application do not match the version used by the HDF5 library to which this application is linked
...
...
You should recompile the application or check your shared library relatedsettings such as 'LD_LIBRARY_PATH'.You can, at your own risk, disable this warning by setting the environmentvariable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.Setting it to 2 or higher will suppress the warning messages totally.
Headers are 1.10.4, library is 1.8.19

            SUMMARY OF THE HDF5 CONFIGURATION
            =================================
General Information:
-------------------
                 HDF5 Version: 1.8.19
                 Configured on: Wed Jul  5 09:54:00 PDT 2017
                 Configured by: safe@cfl-linux
                 Configure mode: production
                 Host system: x86_64-unknown-linux-gnu
                 Uname information: Linux cfl-linux 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
                 Byte sex: little-endian
                 Libraries: static, shared
                 Installation point: /home/safe/code/fme-2018.0/3rd/redistributable/free_use/hdf5/hdf5-1.8.19/hdf5…...
.....
....
Bye...
Aborted (core dumped)

 

 

 

 

3 replies

martinkoch
Supporter
Forum|alt.badge.img+18
  • Author
  • Supporter
  • September 12, 2019

I started struggling again with this, on FME Server 2019.1.2, hoping this would use a newer HFD5.

Firstly, I got my system-python3 to load a trained model of one of my collegues. This works.

Now for FME, loading it with the

/opt/fme-engine-2019./fme python

I  found out, FME on Linux uses the system-python, so changing the python-interpreter to a custom one did not do anything.

Disabeling the hdf5-check with

os.environ['HDF5_DISABLE_VERSION_CHECK']='2'

 resulted in a succesfull 'import keras', but a segmentation-fault when loading the trained model.

By shuffling sys.path, I can highly influence the libraries loaded, but not the hdf5-one. Is this older (pré 1.10.4) version used by FME internally, so I can't override it?

Please, anyone any advise or help?


jovitaatsafe
Safer
Forum|alt.badge.img+11

Hi @martinkoch,

Sorry to hear that you've run into this issue! One of our devs noticed your question and has filed an issue (FMEENGINE-59889) about a library conflict between the python library keras which depends on libhdf5, and the one bundled up in FME. I'll let you know as soon as the issue has been addressed!


martinkoch
Supporter
Forum|alt.badge.img+18
  • Author
  • Supporter
  • September 17, 2019
jovitaatsafe wrote:

Hi @martinkoch,

Sorry to hear that you've run into this issue! One of our devs noticed your question and has filed an issue (FMEENGINE-59889) about a library conflict between the python library keras which depends on libhdf5, and the one bundled up in FME. I'll let you know as soon as the issue has been addressed!

Great!!!!!!

I now made a workaround, in which I wrapped our model-call in a python-script with parameters and writing to a file. I run this with a SystemCaller (using the Ubuntu system-python) and read back the output with a featureReader.

Looking forward to embedding it all neatly again in a PythonCaller.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings