Skip to main content
Archived

Allow Different FME Cloud Python Interpreter

Related products:FME Form
  • November 7, 2016
  • 12 replies
  • 145 views
P

Similar to https://knowledge.safe.com/content/idea/19407/update-to-python-3x.html

FME Cloud currently loads python v. 2.7.6, however quite a few use cases exist that require 2.7.9+ (or even better 3.x - but that appears to be on the roadmap). Our issue involves SNI limitations with the urllib library that are hampering our ability to connect to SSL enabled REST endpoints. It would be great to have more control over the interpreter in the cloud similar to what we expect in Desktop and Server versions.


Thanks!

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

12 replies

stewartatsafe
Safer
Forum|alt.badge.img+10

We just checked and Ubuntu 14.04, which is what FME Cloud runs on, does actually have Python 3.4 installed. You just have to get the FME Engine to use it. To do this you need to set it in FME Workbench before you upload it to FME Server. In FME Workbench, open up FME Options, click on the Translation section and set the preferred python interpreter to FME Python 3.4. Now publish this workspace to FME Cloud.

Things get a little bit tricky now. Engines only load the Python interpreter the first time they start up. The way the engine decides which interpreter to load is whichever interpreter is defined in the first workspace that runs on each engine.

So if you want to run multiple workspaces on the same FME Server and some of them use Python 2.7.6 and others 3.4 then you are going to have a problem. If all of your workspaces are 3.4 then just update the interpreter in all of them, restart the FME Server and publish them up there.

Note if you are using formats that are written in Python, they may not behave correctly in 2016 when using Python 3.4.

sigtill
Supporter
Forum|alt.badge.img+25
  • sigtillSupporter
  • Supporter
  • March 20, 2018
@stewartharper / this is a large issue now. We have started to move some workspaces to 36 and we have some on 27 which cant be upgraded. I have contacted support - but I just wanted to add information on this also.

 

The real SigTill
david_r
Celebrity
  • david_rCelebrity
  • March 20, 2018
I agree, we've stumbled upon this as well. Our workaround was to tell FME Server to restart each engine after each translation, but it's far from optimal.
sigtill
Supporter
Forum|alt.badge.img+25
  • sigtillSupporter
  • Supporter
  • March 20, 2018
Hm, cant find the documentation for this, I know I have seen it somewhere. Do you remember?

 

I was thinking about that - but that WILL create a delay I assume?

 

 

The real SigTill
sigtill
Supporter
Forum|alt.badge.img+25
  • sigtillSupporter
  • Supporter
  • March 21, 2018
Feel free to vote for this idea: https://knowledge.safe.com/questions/66459/make-it-easier-not-to-mix-different-python-version.html?

 

 

The real SigTill
stewartatsafe
Safer
Forum|alt.badge.img+10
Glad to see you have an idea. As you alluded to in the idea, this dev work would need to be done on FME Server and then there shouldn't be any modifications required when we come to deploy on FME Cloud.
david_r
Celebrity
  • david_rCelebrity
  • March 22, 2018
You'll have to modify these two lines in fmeServerConfig.txt and restart:

 

MAX_TRANSACTION_RESULT_SUCCESSES=1 MAX_TRANSACTION_RESULT_FAILURES=1 This will force FME Server to restart each engine after every translation. And yes, there's a small time penalty for this as the engine takes a second or two to start, which is why I don't like this workaround very much. But it does work ;-)

 

Documentation:

 

sigtill
Supporter
Forum|alt.badge.img+25
  • sigtillSupporter
  • Supporter
  • March 22, 2018

david_r : I wonder what happens if you have a Custom Transformer in Parallell mode - will it consider each running process a transaction or not, and if you have a jobsubmitter ... Well, thanks for the workaround.

The real SigTill
G
Forum|alt.badge.img

With FME Server 2018.0 on FME Cloud, you could also create 2 different Job Queues, one for Python 2.7 and one for Python 3.x and assign engines to both of them. If you route all jobs using a specific version of python to the same engine (e.g. single jobs, or a repository) you would not need to restart the engine. Also on FME Cloud, the pricing is not tied to the number of engines so usually it does not matter if you need a few more engines just for proper routing.

sigtill
Supporter
Forum|alt.badge.img+25
  • sigtillSupporter
  • Supporter
  • April 30, 2018

It seems that when you use parallell processing on a custom transformer it fires up:

FME Configuration: Command line arguments are `C:\\Program Files\\FMEServer\\Server\\fme\\fmeworker' `d:\\FMETEMP\\fmeengines\\localhost_Engine4\\childProcMap1525074270579_31724.fme' `-WORKER_CAPABILITY' `247L1228U1' `LOG_FILENAME' `D:\\FMEServerSystemShare\\resources\\logs\\engine\\current\\jobs\\2000\\job_2228_ffs\\job_2228_7.log' `-LOG_PREPEND_ID' `7' `--SOURCE_PORT_OUT' `54529' `--SOURCE_PORT_IN' `54528'

However the fmeworker.exe process does not inherit the python-interpreter. It starts the default 2.7 :(

The real SigTill
sigtill
Supporter
Forum|alt.badge.img+25
  • sigtillSupporter
  • Supporter
  • April 30, 2018
This workaround does not work when a workspace failes within a Custom Transformer in Parallell mode (as it is the fmeworker.exe failing - not fme.exe) - so it will not restart these processes.

 

 

The real SigTill
sigtill
Supporter
Forum|alt.badge.img+25
  • sigtillSupporter
  • Supporter
  • April 30, 2018
This workaround does not work when a workspace failes within a Custom Transformer in Parallell mode (as it is the fmeworker.exe failing - not fme.exe) - so it will not restart these processes.

 

The real SigTill

Community Stats

32,753
Posts
124,208
Replies
41,264
Members
Terms & ConditionsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.