Solved

Running Python in FME

  • 19 September 2017
  • 16 replies
  • 40 views
davideagle
Rank
Badge +14

I've got a few questions around the use of Python within an FME2017+ process and the possible scenarios.

There seems to be a few ways in which I might need configure some Python to run in FME and I'm trying to understand all of the options. There are quite a few articles on the Knowledge Center but a few are getting a little long in the tooth now and several seem to suggest that things have changed in recent versions of FME and I think that's because there were some fairly significant underlying changes around 2012 or so.

  • FME installs a build of Python that allows you to run some scripts. It seems to install 2.7 and 3.4 and 3.5 libraries (as per the Help>Options) but I can only find \\fmepython27 and \\fmepython35. Is 3.4 part of 3.5? Presumably if I use the option 'FME Python 2.7' it's using the FME installed Python interpreter?
  • If I wanted to create a Python script that FME and another application like ArcGIS needed to use, presumably I'd need to change the FME options to 'use custom interpreter' and then point it at one that had actually been installed by ArcGIS?
  • If I had a really old Python script that I didn't want to upgrade, presumably I could install the old interpreter and again, point FME to the right place using the 'custom interpreter' option?

Have I missed out any scenarios? What about if I wanted to call custom Python libraries or functions that FME hadn't installed. Are the Python libraries that FME installs by default a subset of a much fuller Python installation that I should really consider installing and pointing FME at most of the time, assuming I was going to be doing lots of Python development?

Thanks for the help FME Python Gurus.

icon

Best answer by david_r 19 September 2017, 11:48

View original

16 replies

david_r
Rank
Userlevel 4

Hi Dave

Personally I have two different setups I use, depending on the scenario. This is based on discussions with Safe over the years, but just as you do, I wonder if the basic premise is about to change, or has already changed subtly with FME 2017. It'd be great if someone from Safe could tell us if these "best practices" are still valid onwards.

Scenario 1: The workspace needs to integrate with ArcGIS arcpy, and I may or may not need custom modules

  • Change the Python interpreter to the one installed with ArcGIS as per this article (dating from FME 2012!).
  • Set the environment variable PYTHONHOME and point it to the ArcGIS Python, e.g. C:\\Python27\\ArcGIS10.2

     

    This is to prevent PATH confusions in case there are several Python installations on the same machine.
  • All needed 3rd party modules are installed in the ArcGIS Python setup

Scenario 2: There is no integration with ArcGIS arcpy, but I need custom modules

  • Download and install a full Python setup from python.org, making sure to use the 32-bit version for 32-bit FME etc. Whether to use Python 2.7 or 3.x is a personal choice, but if I have the choice I'll go for Python 3.x since 2.7 is deprecated.
  • Change the Python interpreter to the one I just installed as per this article (dating from FME 2012!).
  • Set the environment variable PYTHONHOME and point it to the newly installed Python, e.g. C:\\Python35
  • Install all 3rd party modules in the newly installed Python

The main takeaway here is that I never ever install or modify anything in the FME default Python interpreter. The moment I need to change change anything at all, I switch to a full Python installation, either one supplied by ArcGIS, or a default setup from python.org.

It'd be interesting to hear what others are doing.

david_r
Rank
Userlevel 4

Hi Dave

Personally I have two different setups I use, depending on the scenario. This is based on discussions with Safe over the years, but just as you do, I wonder if the basic premise is about to change, or has already changed subtly with FME 2017. It'd be great if someone from Safe could tell us if these "best practices" are still valid onwards.

Scenario 1: The workspace needs to integrate with ArcGIS arcpy, and I may or may not need custom modules

  • Change the Python interpreter to the one installed with ArcGIS as per this article (dating from FME 2012!).
  • Set the environment variable PYTHONHOME and point it to the ArcGIS Python, e.g. C:\\Python27\\ArcGIS10.2

     

    This is to prevent PATH confusions in case there are several Python installations on the same machine.
  • All needed 3rd party modules are installed in the ArcGIS Python setup

Scenario 2: There is no integration with ArcGIS arcpy, but I need custom modules

  • Download and install a full Python setup from python.org, making sure to use the 32-bit version for 32-bit FME etc. Whether to use Python 2.7 or 3.x is a personal choice, but if I have the choice I'll go for Python 3.x since 2.7 is deprecated.
  • Change the Python interpreter to the one I just installed as per this article (dating from FME 2012!).
  • Set the environment variable PYTHONHOME and point it to the newly installed Python, e.g. C:\\Python35
  • Install all 3rd party modules in the newly installed Python

The main takeaway here is that I never ever install or modify anything in the FME default Python interpreter. The moment I need to change change anything at all, I switch to a full Python installation, either one supplied by ArcGIS, or a default setup from python.org.

It'd be interesting to hear what others are doing.

Just to add that I realize that I'm not answering your exact question, but I hope the above can inspire some discussion and hopefully also help us get some updated guidelines from Safe.
davideagle
Rank
Badge +14

Hi Dave

Personally I have two different setups I use, depending on the scenario. This is based on discussions with Safe over the years, but just as you do, I wonder if the basic premise is about to change, or has already changed subtly with FME 2017. It'd be great if someone from Safe could tell us if these "best practices" are still valid onwards.

Scenario 1: The workspace needs to integrate with ArcGIS arcpy, and I may or may not need custom modules

  • Change the Python interpreter to the one installed with ArcGIS as per this article (dating from FME 2012!).
  • Set the environment variable PYTHONHOME and point it to the ArcGIS Python, e.g. C:\\Python27\\ArcGIS10.2

     

    This is to prevent PATH confusions in case there are several Python installations on the same machine.
  • All needed 3rd party modules are installed in the ArcGIS Python setup

Scenario 2: There is no integration with ArcGIS arcpy, but I need custom modules

  • Download and install a full Python setup from python.org, making sure to use the 32-bit version for 32-bit FME etc. Whether to use Python 2.7 or 3.x is a personal choice, but if I have the choice I'll go for Python 3.x since 2.7 is deprecated.
  • Change the Python interpreter to the one I just installed as per this article (dating from FME 2012!).
  • Set the environment variable PYTHONHOME and point it to the newly installed Python, e.g. C:\\Python35
  • Install all 3rd party modules in the newly installed Python

The main takeaway here is that I never ever install or modify anything in the FME default Python interpreter. The moment I need to change change anything at all, I switch to a full Python installation, either one supplied by ArcGIS, or a default setup from python.org.

It'd be interesting to hear what others are doing.

Thank you David, that's really helpful and excellent detail. I agree there is definitely a need for some best practice suggestions and it would be super interesting to hear other people's approaches. Thanks again.

 

 

daveatsafe
Rank
Userlevel 2
Badge +17

Hi @1spatialdave, @david_r,

The latest docs on setting the Python version you want to use in FME 2017 can be found at this page.

The big change for FME 2017 is that the desired version of Python to use is now stored in the workspace, and can be set in the Navigator under Workspace Parameters - Scripting - Python Compatibility.

The setting Tools - FME Options - Translation - Python Interpreter is still used, but it now controls the default setting for a new workspaces. The setting in the workspace is what is used, and if the Python version chosen is not available the workspace will fail with an error.

Both settings now use a picklist to select the Python version, so there is no longer any need to search for a Python DLL, in most cases.

I agree with David that the best way to use 3rd party modules is to install them in a full version of Python, and configure FME to use that full version of Python instead of its own.

The FMEOptions setting has an option not found in the workspace - Use Custom Interpreter. This option will prompt for the Python DLL to use, and can be used to tell FME to run the full version of Python 2.7 or 3.5 instead of its own. In this case, use the workspace setting that corresponds to the Python version installed.

david_r
Rank
Userlevel 4

Hi @1spatialdave, @david_r,

The latest docs on setting the Python version you want to use in FME 2017 can be found at this page.

The big change for FME 2017 is that the desired version of Python to use is now stored in the workspace, and can be set in the Navigator under Workspace Parameters - Scripting - Python Compatibility.

The setting Tools - FME Options - Translation - Python Interpreter is still used, but it now controls the default setting for a new workspaces. The setting in the workspace is what is used, and if the Python version chosen is not available the workspace will fail with an error.

Both settings now use a picklist to select the Python version, so there is no longer any need to search for a Python DLL, in most cases.

I agree with David that the best way to use 3rd party modules is to install them in a full version of Python, and configure FME to use that full version of Python instead of its own.

The FMEOptions setting has an option not found in the workspace - Use Custom Interpreter. This option will prompt for the Python DLL to use, and can be used to tell FME to run the full version of Python 2.7 or 3.5 instead of its own. In this case, use the workspace setting that corresponds to the Python version installed.

Hi @DaveAtSafe (we're all "Dave" here, it seems!)

 

 

Thanks for sharing the details, much appreciated. Just a question: how does FME "map" the workspace-specific interpreter version to a physical version?

 

 

A couple of examples:

 

  1. Let's say I have Python 3.5 installed both in 32-bit and 64-bit versions. Will 32- and 64-bit FME each be able to automatically choose the right one?
  2. Let's say I have both the ArcGIS Python 2.7 and a stand-alone Python 2.7 and I tell FME to use "Python 2.7", will FME be able to tell them apart automatically? Or do we still have to use PYTHONHOME to direct the python27.dll to the right installation?
J
This question is quite timely as I had some FME Python drama this week.  I have an existing workspace which I run on FME Desktop 2015 and FME Server 2015. 
A startup Python script is used to call an Oracle stored procedure to record some information.  At the end a shutdown Python script is used to call the same Oralce stored procedure to record whether the workspace completed successfully. The Python scripts use the cx_Oracle package to connect to the Oracle database.  In FME 2015 it looks like the cx_Oracle package was shipped by default (with both Desktop and Server), so at that stage we didn't have to do anything special to get it work. 
But cx_Oracle doesn't seem to be included in FME 2017.  When I run the workspace on FME 2017 I get these errors: 
ERROR |Python Exception <ImportError>: No module named cx_Oracle 
ERROR |Error executing string `import cx_Oracle
For FME Desktop 2017, I tried following these instructions: https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Workbench/Workbench/Installing-Python-Packages.htm But I just couldn't get FME to recognise cx_Oracle.  Following the advise in this post, I installed standalone Python 3.5.4 from python.org (making sure I used a 64bit Python install to match my FME).  I then setup cx_Oracle using: 
python -m pip install cx_Oracle --upgrade
In the workspace I set the Python Compatibility to "Python 2.7 or 3.4+" and in Tools > FME Options > Translation I set the "Preferred Python Interpreter" to my standalone python35.dll

The Python script then ran OK on FME Desktop.

For FME Serer 2017, I tried following these instructions... https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Workbench/Workbench/Importing_Custom_Python_Modules_FMEServer.htm ... to upload cx_Oracle.cp35-win_amd64.pyd as a Resource, but FME still didn't seem to find cx_Oracle.  I then went with the standalone Python setup on the FME Server, just as I did on FME Desktop.
The Tools > FME Options > Translation "Preferred Python Interpreter" setting doesn't seem to apply to FME Server, so to get it to use the non-default Python interpreter I had to follow these instructions  https://docs.safe.com/fme/html/FME_Server_Documentation/Content/AdminGuide/Using-Python-with-FME_Server.htm
C:\Progra~1\FMEServer\Server\fme\fme.exe APPLY_SETTINGS SYSTEM "Python/Use Custom Python" true
C:\Progra~1\FMEServer\Server\fme\fme.exe APPLY_SETTINGS SYSTEM "Python/Python Interpreter" c:/Python35/python35.dll 
Then the Python script ran ok on FME Server.
L
Badge

@john_otoole

Regarding cx_Oracle (and this is working for other python packages), this is what I've been told to do by SAFE support:

1. visit https://pypi.python.org/pypi/cx_Oracle

2. Download the appropriate Python Wheel file (whl).

3. Open the .whl with 7zip and grab the pyd out of it.

4. Drop it into the site-packages dir of the FME install (may have to create the site-packages folder if not existing).

I did this for psycopg2 except that I copied the psycopg2 folder to the fmepython27\\Lib of my FME installation and it worked.

david_r
Rank
Userlevel 4
This question is quite timely as I had some FME Python drama this week.  I have an existing workspace which I run on FME Desktop 2015 and FME Server 2015. 
A startup Python script is used to call an Oracle stored procedure to record some information.  At the end a shutdown Python script is used to call the same Oralce stored procedure to record whether the workspace completed successfully. The Python scripts use the cx_Oracle package to connect to the Oracle database.  In FME 2015 it looks like the cx_Oracle package was shipped by default (with both Desktop and Server), so at that stage we didn't have to do anything special to get it work. 
But cx_Oracle doesn't seem to be included in FME 2017.  When I run the workspace on FME 2017 I get these errors: 
ERROR |Python Exception <ImportError>: No module named cx_Oracle 
ERROR |Error executing string `import cx_Oracle
For FME Desktop 2017, I tried following these instructions: https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Workbench/Workbench/Installing-Python-Packages.htm But I just couldn't get FME to recognise cx_Oracle.  Following the advise in this post, I installed standalone Python 3.5.4 from python.org (making sure I used a 64bit Python install to match my FME).  I then setup cx_Oracle using: 
python -m pip install cx_Oracle --upgrade
In the workspace I set the Python Compatibility to "Python 2.7 or 3.4+" and in Tools > FME Options > Translation I set the "Preferred Python Interpreter" to my standalone python35.dll

The Python script then ran OK on FME Desktop.

For FME Serer 2017, I tried following these instructions... https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Workbench/Workbench/Importing_Custom_Python_Modules_FMEServer.htm ... to upload cx_Oracle.cp35-win_amd64.pyd as a Resource, but FME still didn't seem to find cx_Oracle.  I then went with the standalone Python setup on the FME Server, just as I did on FME Desktop.
The Tools > FME Options > Translation "Preferred Python Interpreter" setting doesn't seem to apply to FME Server, so to get it to use the non-default Python interpreter I had to follow these instructions  https://docs.safe.com/fme/html/FME_Server_Documentation/Content/AdminGuide/Using-Python-with-FME_Server.htm
C:\Progra~1\FMEServer\Server\fme\fme.exe APPLY_SETTINGS SYSTEM "Python/Use Custom Python" true
C:\Progra~1\FMEServer\Server\fme\fme.exe APPLY_SETTINGS SYSTEM "Python/Python Interpreter" c:/Python35/python35.dll 
Then the Python script ran ok on FME Server.
Excellent, thanks for sharing the details.
A
Badge +1

Hi Dave

Personally I have two different setups I use, depending on the scenario. This is based on discussions with Safe over the years, but just as you do, I wonder if the basic premise is about to change, or has already changed subtly with FME 2017. It'd be great if someone from Safe could tell us if these "best practices" are still valid onwards.

Scenario 1: The workspace needs to integrate with ArcGIS arcpy, and I may or may not need custom modules

  • Change the Python interpreter to the one installed with ArcGIS as per this article (dating from FME 2012!).
  • Set the environment variable PYTHONHOME and point it to the ArcGIS Python, e.g. C:\\Python27\\ArcGIS10.2

     

    This is to prevent PATH confusions in case there are several Python installations on the same machine.
  • All needed 3rd party modules are installed in the ArcGIS Python setup

Scenario 2: There is no integration with ArcGIS arcpy, but I need custom modules

  • Download and install a full Python setup from python.org, making sure to use the 32-bit version for 32-bit FME etc. Whether to use Python 2.7 or 3.x is a personal choice, but if I have the choice I'll go for Python 3.x since 2.7 is deprecated.
  • Change the Python interpreter to the one I just installed as per this article (dating from FME 2012!).
  • Set the environment variable PYTHONHOME and point it to the newly installed Python, e.g. C:\\Python35
  • Install all 3rd party modules in the newly installed Python

The main takeaway here is that I never ever install or modify anything in the FME default Python interpreter. The moment I need to change change anything at all, I switch to a full Python installation, either one supplied by ArcGIS, or a default setup from python.org.

It'd be interesting to hear what others are doing.

Hi,

 

 

I recently installed Anaconda because I need access to 3rd party modules such as Pandas and Matplotlib and due to IT restrictions I couldn't install these modules separately.

 

 

I can't work from these questions and threads the exact paths I need to use when setting my 'Custom Interpreter' in FME, and for the PYTHONHOME environment variable for use with Anaconda.

 

 

Can someone please advise what paths to use ?

 

 

I am using FME Desktop 2017.0 64 bit with Python 3.x

 

 

Thanks,

 

 

david_r
Rank
Userlevel 4
Hi,

 

 

I recently installed Anaconda because I need access to 3rd party modules such as Pandas and Matplotlib and due to IT restrictions I couldn't install these modules separately.

 

 

I can't work from these questions and threads the exact paths I need to use when setting my 'Custom Interpreter' in FME, and for the PYTHONHOME environment variable for use with Anaconda.

 

 

Can someone please advise what paths to use ?

 

 

I am using FME Desktop 2017.0 64 bit with Python 3.x

 

 

Thanks,

 

 

Sounds like you need to go for Scenario 2.

 

I have no experience integrating with Anaconda, however, so I can't really help you there.
A
Badge +1
Hi,

 

 

I recently installed Anaconda because I need access to 3rd party modules such as Pandas and Matplotlib and due to IT restrictions I couldn't install these modules separately.

 

 

I can't work from these questions and threads the exact paths I need to use when setting my 'Custom Interpreter' in FME, and for the PYTHONHOME environment variable for use with Anaconda.

 

 

Can someone please advise what paths to use ?

 

 

I am using FME Desktop 2017.0 64 bit with Python 3.x

 

 

Thanks,

 

 

Actually I worked out what the issue was! The version of Python that I was using (FME 2017.0 only uses Python version up to 3.5 and the version of Anaconda that I had installed used Python 3.6. Therefore, by installing an earlier version of Anaconda that uses Python 3.5 I was able to access the Python modules that I required within the PythonCaller transformer.

 

david_r
Rank
Userlevel 4
Actually I worked out what the issue was! The version of Python that I was using (FME 2017.0 only uses Python version up to 3.5 and the version of Anaconda that I had installed used Python 3.6. Therefore, by installing an earlier version of Anaconda that uses Python 3.5 I was able to access the Python modules that I required within the PythonCaller transformer.

 

Excellent! Thanks for sharing the solution.
R

Hi,

The option to set ESRI ArcGIS Desktop (Python 2.7) does not work, becuase ESRI install a 32-bit and 64-bit of Python 2.7. In my case my extension are installed the 32-bit version, but FME picks up the 64-bit version for ESRI

Using Python interpreter from `C:\\Windows\\system32\\python27.dll' with PYTHONHOME `C:\\Program Files\\Python27\\ArcGISx6410.5'

for example an import error Python Exception <ImportError>: No module named suds.client

This worked before in 2016 and 2014

david_r
Rank
Userlevel 4

Hi,

The option to set ESRI ArcGIS Desktop (Python 2.7) does not work, becuase ESRI install a 32-bit and 64-bit of Python 2.7. In my case my extension are installed the 32-bit version, but FME picks up the 64-bit version for ESRI

Using Python interpreter from `C:\\Windows\\system32\\python27.dll' with PYTHONHOME `C:\\Program Files\\Python27\\ArcGISx6410.5'

for example an import error Python Exception <ImportError>: No module named suds.client

This worked before in 2016 and 2014

If you're running the 64-bit version of FME, you have no other choice but to use th 64-bit version of Python, you cannot use 32-bit Python. Could such a mixup be the cause here?

 

 

tim_wood
Rank
Badge +7

This seems like a good thread to resurrect rather than post a new topic.

I have a machine with the following software installed:

ArcGIS 10.3.1 + Python 2.7 (in a separate folder with 2 sub-folders - "ArcGIS10.3" and "ArcGISx6410.3")

FME 2018.1.0.1 (64-bit and 32-bit)

FME 2019.1.0.0 (64-bit)

When I installed FME 2019, I chose not to install Python 2.7.

So does this mean I have Python 3.x available to FME 2019? Or do I have to install it myself?

If Python 3.x is available to FME, will having it installed affect using Python in ArcGIS 10.3.1?

I have not yet needed to use ArcPy in any Python script in FME.

david_r
Rank
Userlevel 4

This seems like a good thread to resurrect rather than post a new topic.

I have a machine with the following software installed:

ArcGIS 10.3.1 + Python 2.7 (in a separate folder with 2 sub-folders - "ArcGIS10.3" and "ArcGISx6410.3")

FME 2018.1.0.1 (64-bit and 32-bit)

FME 2019.1.0.0 (64-bit)

When I installed FME 2019, I chose not to install Python 2.7.

So does this mean I have Python 3.x available to FME 2019? Or do I have to install it myself?

If Python 3.x is available to FME, will having it installed affect using Python in ArcGIS 10.3.1?

I have not yet needed to use ArcPy in any Python script in FME.

FME comes with its own Python 3 interpreter(s), so no need to install anything else unless you e.g. need to add custom modules.

Reply


Community Stats

28,731
Posts
109,530
Replies
36,565
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.