Skip to main content

​Main question

Can anyone advise me about the installation of the GDAL Python module and its implications on FME's Python interpretations? Can anyone share their experiences in using a different Python module within FME other than FME's Python interpreter or ArcPy?

 

Background info

Currently, it is not directly supported to calculate percentile values in the *StatisticsCalculator transformers of FME (e.g. also not in the RasterStatisticsCalculator). See also this Idea on the FME community.

In order to still do so, I thought about using a Python script/module to perform this step. Specifically, I would like to extract the 95% percentile of pixel values in a GeoTIFF (those pixel values represent absolute height values), for portions of the GeoTIFF that overlap/intersect with 2D polygon features of building extents. 

 

Python modules likely suitable for the task of calculating 'zonal statistics over a raster'

One might use the 'Zonal Statistics As Table (Raster Analysis)' from ArcGIS Pro, which relies on the ArcPy Python module, but this requires Esri licenses, and these are quite scarce in our organization (and by floating license not always available).

Then I also found out about the rasterstats Python module, which doesn't rely on an Esri license. So I wanted to give this a try. However, this module seems to require the GDAL module/library of Python, and I'm not sure if it is recommended to install/link that module/library with FME's Python interpreter. See also below 

 

Installing the rasterstats Python module and requirement on GDAL

To install a Python packages/modules for FME Desktop, I had a look this documentation page of Safe for this. I used the following command for this in command prompt:

fme.exe python -m pip install rasterstats --target "C:\Users\<USERNAME>\OneDrive - <ORGANIZATION>\FME\Plugins\Python"

That resulted in an error:

A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable

Now at the rasterstats page there is some info on how to install it here, particularly for windows it is described to follow the rasterio installation, in which under the 'Source Distributions' section, subsection 'Windows', it is described how/where to download a binary distribution of GDAL (here), and how to install it. 

 

Only, when I look a bit further, I found this tread on the FME community. I am a bit thrown off by david_r comment "I do seem to remember, however, that getting the gdal Python bindings to work properly on Windows was a royal pain, but hopefully it's gotten better by now.", and that he described/hinted to using a stand-alone Python interpreter, whereas I would like to the Python interpreters of FME. 

I'm not sure however if doing so would interfere with how FME handles geometric operations... Therefore, before doing so I wanted to ask for some advice on this...

 

Some closing remarks/info

  • The underlying goal/task is actually identical to this thread on the FME community (there they try to achieve the same with an Rcaller). 
  • I actually managed to make this work using FME only, by using a RasterCellCoercer to get individual points for each raster, creating a list of pixel values for each building, sorting the list using a ListSorter, extracting the appropriate list element(s) for the percentile value, and then setting the percentile value (using manual interpolation between two list element values).
  • ​As such, I'm not looking for advice on how to do this in FME - since that approach is functional already. 
  • You might wonder "Then why still try the Python approach?". The scope of this question reaches further than just this specific Rasterstats functionality: it is part of an effort to experiment with using Python more inside of FME workspaces, so that we can extend FME with Python's power whenever necessary. 

 

Some general comments about using external Python modules in FME as I don't have much personal experience with the modules you mentioned.

 

It's possible to use external Python modules within FME and there are some users who do so within their workflows. Users can either install the module using an external Python interpreter or the FME's bundled interpreter. At the very least, need to ensure the Python version and bit-version (if applicable for the Python module in question) matches. But it can get more complicated as the number of load-time dependencies increases.

 

Note the custom Python interpreter option in Windows only (sorry, macOS and Linux users).

 


Hi @debbiatsafe​,

Thanks for your comments about using external Python modules in FME. Good to hear that this is possible.

However, my question is mainly focussed on the use of GDAL, where "GDAL is a translator library for raster and vector geospatial data formats" .

 

I'm worried that installing the GDAL library might result in conflicts in how FME translates raster and vector data. So I'm curious to hear if anyone has any experience with this GDAL module and can tell me if this worry of mine is legitimate or not (if not I think I can just try it out and if neccessary perform a reinstall of FME).


Hi @debbiatsafe​,

Thanks for your comments about using external Python modules in FME. Good to hear that this is possible.

However, my question is mainly focussed on the use of GDAL, where "GDAL is a translator library for raster and vector geospatial data formats" .

 

I'm worried that installing the GDAL library might result in conflicts in how FME translates raster and vector data. So I'm curious to hear if anyone has any experience with this GDAL module and can tell me if this worry of mine is legitimate or not (if not I think I can just try it out and if neccessary perform a reinstall of FME).

Hi @thijsknapen​ 

I asked our development team about this and they responded it would depend on the operating system.

 

On Windows, they do not expect issues due to the way GDAL is built (as a static library). However, FME installs a GDAL shared library on macOS and Linux and there have been a previous report of library conflicts for Linux.

 

I hope this information helps.


Hi @thijsknapen​ 

I asked our development team about this and they responded it would depend on the operating system.

 

On Windows, they do not expect issues due to the way GDAL is built (as a static library). However, FME installs a GDAL shared library on macOS and Linux and there have been a previous report of library conflicts for Linux.

 

I hope this information helps.

Hi @debbiatsafe (Safer)​ 

Thanks for asking the development team about this. That answer definitely helps!

 

Good to know that there are dependencies/considerations on which operation is being used. My organization is using windows (also for FME Server), so I guess I can safely give it a try.


Reply