***Note from Migration:***
Original Title was: Enhance FME Server on Linux so PythonCaller scripts can use GDAL / OGR and associated libraries
The PythonCaller transformer provides a powerful way of extending what can be done with FME, when there is no native transformer available. A key set of tools for working with geospatial data in Python are the GDAL libraries and Python bindings. However, at present PythonCaller transformers that use GDAL (or OGR, or various associated libraries) do not work in FME Server on Linux. This means that one of the main uses for the PythonCaller - to extend the capability of FME itself - breaks.
Problem
My scenario is that I am using a Python-based approach to calculating raster zonal statistics via an FME workflow. The FME processing creates or supplies the zone geometries and then the Python code loads the raster files directly from disk via the GDAL library, calculates the zonal statistics, and outputs them on FME features. This enables rapid calculation of zonal statistics on huge numbers (tens of thousands) of raster files, with FME helping to manage the results. This works very well in FME Desktop, but in Server on Linux it fails because the GDAL library in Python is not able to load the raster files.
The issue can be demonstrated very simply. Running the following code in a Python interpreter on Windows, or on the system interpreter on the Linux machine, will return a number indicating how many formats GDAL can read:
>>> from osgeo import gdal >>> print gdal.GetDriverCount() 207
The number will vary depending on what exactly is installed. However, running this code inside a PythonCaller or PythonCreator in a workbench published to FME Server on linux, outputs a value of 0, indicating that GDAL cannot read any data formats.
I think this fails because FME has its own built-in copy of some parts of GDAL and this isn't meant to be used other than for FME itself, so it's probably been compiled without that stuff built in. However instead of removing the unnecessary symbols from the FME-supplied library, they're there but crippled. Because Python runs within the FME process, the Python code also gets presented with the FME copy of the GDAL library when it tries to load it, not the system one, and therefore can't read any GDAL data.
Possible Solution
I think there's at least two approaches that could be taken to fixing this:
- Customising the FME-internal version of GDAL a bit more so that the functions (symbols) it provides don't conflict with the original GDAL ones. So when the Python GDAL library tries to call the native GDAL functions, it gets the correct system-installed ones, and stuff that's internal to FME stays that way and doesn't "accidentally" get called by a user's code. I don't think that I as a user should be able to have my Python code accidentally call things that are supposed to be internal to the code of FME. (Note, the same issue I'm describing here used to also happen with the SQLite library, which also didn't work in Python code on Server, but that has now been fixed in recent versions of Server which no longer have an FME-specific version of that library. So change is possible here!)
- Increasing the separation of the Python environment that runs within a PythonCaller or PythonCreator. I don't know how or if this could be done. However since on linux FME uses the system Python interpreter, then I think it's reasonable to expect Python code that works at the command line, to also work when called via FME.
