Question

Importing Python Libraries to FME Cloud

  • 27 April 2018
  • 7 replies
  • 3 views
cwarren
Badge

I'm having some trouble when trying to upload Python libraries to FME Cloud and I couldn't find any documentation over it. I have tested that the workspace successfully imports the PyPDF2 python library on my desktop (windows 10). I then followed the directions on the post that is linked below and I am still unable to have the workspace on FME cloud import PyPDF2. I'm still fairly new to Python and haven't had any experience with Linux, so I'm not sure what I might be doing wrong.

My steps

  • Created a virtual machine with ubuntu (32 bit. No option for 64)
  • Downloaded the source code for PyPDF2 from: https://github.com/mstamy2/PyPDF2
  • Compiled the library on the virtual machine and copied the folder created in the dest-packages folder to FME Cloud folder located at Resources>Engine>Plugins>Python>Python27. (no files in site-packages)

Reference to Post: https://knowledge.safe.com/questions/4164/python-libraries-on-fme-cloud.html?childToView=69192#comment-69192

Any help on what I may be doing wrong is greatly appreciated!

7 replies

claudeatsafe

Hi madwarren,

FME Cloud is 64bit, so you'll have to find a way to compile the library in Ubuntu 16.04 64bit. VirtualBox is an open-source tool which should allow you to create 64bit VMs (https://www.virtualbox.org/ with image http://releases.ubuntu.com/16.04/ubuntu-16.04.4-desktop-amd64.iso). If this still does not work, please let us know.

Claude

cwarren
Badge

Hi madwarren,

FME Cloud is 64bit, so you'll have to find a way to compile the library in Ubuntu 16.04 64bit. VirtualBox is an open-source tool which should allow you to create 64bit VMs (https://www.virtualbox.org/ with image http://releases.ubuntu.com/16.04/ubuntu-16.04.4-desktop-amd64.iso). If this still does not work, please let us know.

Claude

@claudevessaz Ok. I was thinking the 32/64 bit might have been the issue. I was able to install the 64bit Ubuntu on another pc using VirtualBox, re-compiled the library, and uploaded it to FME Cloud. I'm still receiving the error that it is unable to import the library at run-time though.
G
Badge

Hi @madwarren

could you confirm in the log file that FME Engine is loading indeed python 2.7 and not a newer version of python? You dropped the library in the Python27 folder so it won't be loading when the FME Engine is using python 3.x. You can set your workspace to load python 2.7 via the Python Compatibility parameter in FME Desktop.

jatoxa
Rank
Badge +2

I know this an old question but I couldn't find any newer info on the matter.

 

1. Is the above approach still valid?

 

2. Are there any issues regarding versions of libraries interfering with pre-installed libraries? For instance, I recently tried uploading libraries that were dependent on a different Numpy version and I couldn't get it to work. The libraries were compiled on a 64 bit Ubuntu version.

 

 

 

kailinatsafe
Rank
Userlevel 3
Badge +13

I know this an old question but I couldn't find any newer info on the matter.

 

1. Is the above approach still valid?

 

2. Are there any issues regarding versions of libraries interfering with pre-installed libraries? For instance, I recently tried uploading libraries that were dependent on a different Numpy version and I couldn't get it to work. The libraries were compiled on a 64 bit Ubuntu version.

 

 

 

Hello @jatoxa​, It might be beneficial for you to post a new question, to further describe the issue you’re facing. Please feel free to link this as a reference however!

 

Nonetheless, yes, this approach is should still be valid for FME Flow Hosted. A couple things to note about your second question. If you upload Python libraries that require a newer version of numpy than the one natively available with FME Flow/Flow Hosted, you will need to also upload the newer numpy library on FME Flow Hosted. The steps should go as follow:

  1. Upload the required libraries including the newer numpy library to Resources for the appropriate Python version
  2. Edit your workspace (eg. PythonCaller or other) to use/point to the newer numpy version (see the third option from @debbiatsafe's comment here)
  3. Republish the workspace to FME Flow Hosted

 

Hope this helps, Kailin.

jatoxa
Rank
Badge +2

Hello @jatoxa​, It might be beneficial for you to post a new question, to further describe the issue you’re facing. Please feel free to link this as a reference however!

 

Nonetheless, yes, this approach is should still be valid for FME Flow Hosted. A couple things to note about your second question. If you upload Python libraries that require a newer version of numpy than the one natively available with FME Flow/Flow Hosted, you will need to also upload the newer numpy library on FME Flow Hosted. The steps should go as follow:

  1. Upload the required libraries including the newer numpy library to Resources for the appropriate Python version
  2. Edit your workspace (eg. PythonCaller or other) to use/point to the newer numpy version (see the third option from @debbiatsafe's comment here)
  3. Republish the workspace to FME Flow Hosted

 

Hope this helps, Kailin.

Hi Kailin,

I got it to work as expected after compiling the libraries on an Ubuntu machine with roughly the same specs, versions etc.

Thank you for the help!

 

E

I got it to work as expected after compiling the libraries on an Ubuntu machine with roughly the same specs, versions etc. Water Supply in Dubai

 

 

Reply


Community Stats

28,692
Posts
109,433
Replies
36,544
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.