Question

Python compatibility issues while upgrading to FME Workbench 2021.2.1 and ArcGIS Pro 2.9

  • 10 January 2022
  • 4 replies
  • 7 views

After upgrading FME Workbench to 2021.2.1 our arcgis pro 2.9 environment doesn’t cooperate well anymore with Python 3.7+ scripting.

 

For example, for some of our FME services we’ve installed Geopandas, which worked fine with ArcGIS Pro<2.6 and FME <= 2021.0, but not anymore with FME Python 3.7 and ArcGIS Pro 2.9. To fix this we’ve tried several things with a dummy script (attachment):

  • FME ESRI ArcGIS Python 3.7 (Python compatibility ESRI ArcGIS Python 3.7) succeeds
  • Clone ArcGIS Pro 2.9 with Scripting compatibility FME ESRI ArcGIS Python 3.7 succeeds
  • Clone ArcGIS Pro 2.9 with geopandas>0.8 and Scripting compatibility Python 3.7 executed in FME Workbench fails
  • Clone ArcGIS Pro 2.9 with geopandas>0.8 and Conda forge Numpy 1.20.3 executed in a Conda python prompt succeeds

How can we have a ArcGIS Pro 2.9 environment with Geopandas working smoothly together with FME Python 3.7 in FME Workbench?


4 replies

Badge +4

Be sure to install your dependencies anywhere except from any fme system folder. feks: fme python -m pip install geopandas -t C:\\my_dependencies. in your fme script use import sys, sys.path = [r'c:\\my_dependencies'] + sys.path import geopands after fixing sys.path. this will make sure it import from your dependencies before looking at system packages. fme comes bundled with numpy which is of an older version.

Userlevel 2
Badge +17

Hi @geojanmov​ 

Your attached test workspace has a Python Compatibility parameter setting of Python 3.7+. Since you are calling arcpy in this workspace, the Python Compatibility parameter must be Esri ArcGIS Python 3.7. Otherwise, it will fail with ModuleNotFoundError.

 

I also want to mention if a workspace's Python Compatibility is set to Esri ArcGIS Python 3.x, FME can only use the default Conda environment that comes with ArcPro (ie. arcgispro-py3). To access modules installed in a cloned environment, try inserting the path of the cloned environment's site-packages directory to the PATH variable. For certain modules, you may also need to add the bin directory as well.

@Paal Pedersen​ , @debbiatsafe​ , thanks for your responses!

 

Last weeks we've been busy with testing your options (and more). Unfortunately we've not found the satifying solution yet. The following we've tried:

  • We've installed our dependendies outside the fme system folder, like @Paal Pedersen​ suggested. After some trial and error we fixed our challenge with geopandas (0.9.0) and FME Python 3.7 Numpy version. Thanks!
  • We still encounter difficulties with the suggestion of @debbiatsafe​: when importing pyproj from the site-packages folder using Esri ArcGIS Python 3.7 Python Compatibility and our conda environment (mamba create -n miniArcpy python=3.7 arcpy fiona geopandas shapely gdal proj-data -c conda-forge -c esri). See attached FME workspace (numpyArcpyTest.fmw) for the calls in the PythonCaller

 

Greetings,

 

Jan

Userlevel 2
Badge +17

@Paal Pedersen​ , @debbiatsafe​ , thanks for your responses!

 

Last weeks we've been busy with testing your options (and more). Unfortunately we've not found the satifying solution yet. The following we've tried:

  • We've installed our dependendies outside the fme system folder, like @Paal Pedersen​ suggested. After some trial and error we fixed our challenge with geopandas (0.9.0) and FME Python 3.7 Numpy version. Thanks!
  • We still encounter difficulties with the suggestion of @debbiatsafe​: when importing pyproj from the site-packages folder using Esri ArcGIS Python 3.7  Python Compatibility and our conda environment (mamba create -n miniArcpy python=3.7 arcpy fiona geopandas shapely gdal proj-data -c conda-forge -c esri). See attached FME workspace (numpyArcpyTest.fmw) for the calls in the PythonCaller

 

Greetings,

 

Jan

Hi @geojanmov​ 

I tried to replicate your environment based on the information you have shared. I did run into the following error when running your test workspace. It is caused from importing pyproj

DLL load failed: The specified module could not be found

I'm not certain this is the error you are seeing as you did not provide a log file. If it is this error, try adding this path C:\Users\<user>\Miniconda3\envs\miniarcpy\Library\bin to the PATH environment variable and then restart Workbench (this is important). Based on my tests, simply appending/inserting this path within the PythonCaller did not work.

 

In case your error is different, I'll also share my debugging steps. Started off comparing environments where the script does run (eg. Miniconda) with FME. Key items I looked at were PATH values and environment variables. My next step was use Process Monitor to see which specific DLL was failing to load (filter by Process Name is fme.exe, start capture, run workspace). On my test machine, this was proj_8_2.dll. I then used Process Monitor to find the directory where Miniconda3 loaded this DLL (filter by Process Name is python.exe and Path ends with proj_8_2.dll). 

 

I hope this information helps.

Reply