Question

FME Server and tranformers from Shared FME Folders ?

lifalin2016
Rank
Userlevel 1
Badge +22

Hi list.

I have built a number of custom transformers, using FMX and Python, and storing them in the shared folders structure. This works (apart from some kinks) well in Desktop.

Now I've built a workspace with such a transformers, published the workspace to an FME Server, and tried to run the workspace as a job.

And it fails with: Python Exception <ModuleNotFoundError>: No module named 'Xxxxx'

How do I tell FME Server to look for the custom transformers in my shared folders. They're located on a network drive, so it has access.

Cheers

-- Cheers, Lars I.

19 replies

lifalin2016
Rank
Userlevel 1
Badge +22

I would have expected to find some setting in the web interface, but so far I haven't been able to locate it.

-- Cheers, Lars I.
david_r
Rank
Userlevel 4

You'll probably need to add the shared folder to the Python search path, e.g. add this a the top of your script, before importing the custom modules:

import sys
sys.path.append(r'c:\my\shared\folder')

 

lifalin2016
Rank
Userlevel 1
Badge +22

You'll probably need to add the shared folder to the Python search path, e.g. add this a the top of your script, before importing the custom modules:

import sys
sys.path.append(r'c:\my\shared\folder')

 

This isn't necessary in Desktop, so it's not a good work-around, imho. And I fear that it might very well add problems to the flow between desktop and server.

-- Cheers, Lars I.
david_r
Rank
Userlevel 4

Did you try copying the Python modules into the dedicated resources folder?

david_r
Rank
Userlevel 4

This isn't necessary in Desktop, so it's not a good work-around, imho. And I fear that it might very well add problems to the flow between desktop and server.

FME Desktop does it for you, that's why it works. But FME Server does not have the same notion of shared folders as Desktop, it has the concept of shared resources which works a bit differently in this regard.

lifalin2016
Rank
Userlevel 1
Badge +22

Did you try copying the Python modules into the dedicated resources folder?

If the FME Server is unable to utilize FME Shared Folders, I will consider copying. But again, it's not a good work-around, having to maintain identical codes in several places. FME 2020 desktop already has problems in handling custom Python transformers in shared folders correctly, which I've made support aware of.

-- Cheers, Lars I.
lifalin2016
Rank
Userlevel 1
Badge +22

FME Desktop does it for you, that's why it works. But FME Server does not have the same notion of shared folders as Desktop, it has the concept of shared resources which works a bit differently in this regard.

The whole point of having "shared" resources, is that they can be shared. And I would very much expect FME Server to also be able to use such shared content.

But the configuration files for FME Server are very complex, so I won't guess on where and how to implement it myself, but I'm confident that Safe - as always - can find a way to do it.

-- Cheers, Lars I.
david_r
Rank
Userlevel 4

The whole point of having "shared" resources, is that they can be shared. And I would very much expect FME Server to also be able to use such shared content.

But the configuration files for FME Server are very complex, so I won't guess on where and how to implement it myself, but I'm confident that Safe - as always - can find a way to do it.

You have to distinguish between a shared folder and a Python search path, they're not the same.

You can add your own shared resources in FME Server (without even touching the config files), but they won't automatically be added to the search path of the Python interpreter.

See: http://docs.safe.com/fme/html/FME_Server_Documentation/WebUI/Manage_Resource_Connections.htm

david_r
Rank
Userlevel 4

If the FME Server is unable to utilize FME Shared Folders, I will consider copying. But again, it's not a good work-around, having to maintain identical codes in several places. FME 2020 desktop already has problems in handling custom Python transformers in shared folders correctly, which I've made support aware of.

FME Server can of course use shared resources, but it does not automaticall add them all to the Python search path.

virtualcitymatt
Rank
Userlevel 4
Badge +26

I feel your pain on this one a bit but we use FME Cloud so shared folders don't work anyway as we're not in the same network. It would be nice to have the option to share the same functionality as desktop

 

 

One method could be to actually use the FME Server engine plugins folder as you're default public share for desktop. I haven't tried it but it could be the fix you're after.

 

 

For python I think adding to the PYTHONPATH environment variable on the FME Server should do the trick. That doesn't help for custom formats and transformers though. They still need to be copied.

 

 

The way that I do it on FME Cloud is rather manual and I don't think very maintainable or will scale easily. Thankfully so far we don't work with too many custom tools at this points.

 

 

 

david_r
Rank
Userlevel 4

I feel your pain on this one a bit but we use FME Cloud so shared folders don't work anyway as we're not in the same network. It would be nice to have the option to share the same functionality as desktop

 

 

One method could be to actually use the FME Server engine plugins folder as you're default public share for desktop. I haven't tried it but it could be the fix you're after.

 

 

For python I think adding to the PYTHONPATH environment variable on the FME Server should do the trick. That doesn't help for custom formats and transformers though. They still need to be copied.

 

 

The way that I do it on FME Cloud is rather manual and I don't think very maintainable or will scale easily. Thankfully so far we don't work with too many custom tools at this points.

 

 

 

I agree that setting PYTHONPATH on the server could be a good solution, while avoiding hard-coding paths in the workspace scripts.

lifalin2016
Rank
Userlevel 1
Badge +22

You have to distinguish between a shared folder and a Python search path, they're not the same.

You can add your own shared resources in FME Server (without even touching the config files), but they won't automatically be added to the search path of the Python interpreter.

See: http://docs.safe.com/fme/html/FME_Server_Documentation/WebUI/Manage_Resource_Connections.htm

Well, it may come down to a search path underneath it all, but as FME Desktop have an official setting (in FME Options) to add shared folder paths, I would expect FME Server to have a similar option or capability.

I'm looking for at solution here, not a work-around. Would anyone from Safe like chip in on this ? @markatsafe ? @daleatsafe ?

-- Cheers, Lars I.
virtualcitymatt
Rank
Userlevel 4
Badge +26

Well, it may come down to a search path underneath it all, but as FME Desktop have an official setting (in FME Options) to add shared folder paths, I would expect FME Server to have a similar option or capability.

I'm looking for at solution here, not a work-around. Would anyone from Safe like chip in on this ? @markatsafe ? @daleatsafe ?

Maybe you can create an idea. I'd vote on it

virtualcitymatt
Rank
Userlevel 4
Badge +26

I did some digging into the config files and saw the following line in

 

C:\\Program Files\\FMEServer\\Server\\fmeEngineConfig.txt

 

 

FME_SHARED_RESOURCE_DIR "!FME_SERVER_ROOT!/resources/engine"

 

 

I suspect that you could just change this, restart FME Server and give it a try.
steveatsafe
Rank
Badge +11

You'll probably need to add the shared folder to the Python search path, e.g. add this a the top of your script, before importing the custom modules:

import sys
sys.path.append(r'c:\my\shared\folder')

 

Hi @lifalin2016,

 

Thanks for posting this question.

There's been a lot of changes in trying to make the FME engine get the right interpreter loaded and then comes the problem of loading new modules like this. As @david_r correctly points out, the quick solution is to copy the mods to the appropriate python folder in plugins. For more on this visit this link (I'm sure you are aware of this - and this is our recommended approach): https://docs.safe.com/fme/html/FME_Server_Documentation/AdminGuide/Using-Python-with-FME_Server.htm.

I appreciate the duplication of files and maintainability by using our recommended approach. But you can also appreciate the problem if a change is made to the python mods that break FME Server Jobs for some reason that is not found in FME Desktop. In my opinion, there's a good reason for keeping the modules as a controlled copy of your source files that you might be managing in Python/FME Desktop. I assume you know the risk. And it is possible, you are actually managing the new modules away from FME Desktop as well... but just want to make sure we flush out all the facts here for why you need this behaviour to be different in FME Server and not make use of the Shared Resources and the plugins folder.

So now the question becomes, is it possible to add a shared folder to the search path for the python interpreter in use or add the search path to the FME Server engine for python visibility in some other way? And I do not know the answer to this at this time. It could be possible, but I suspect this would involve a hack. So I'll have to take this back to the team to see what might possible now, or what could be added to the product in the future.

I like the idea of a new "IDEA" as this will allow all you awesome Experts to chime in for why this feature will benefit your requirements.

SteveatSafe
david_r
Rank
Userlevel 4

Hi @lifalin2016,

 

Thanks for posting this question.

There's been a lot of changes in trying to make the FME engine get the right interpreter loaded and then comes the problem of loading new modules like this. As @david_r correctly points out, the quick solution is to copy the mods to the appropriate python folder in plugins. For more on this visit this link (I'm sure you are aware of this - and this is our recommended approach): https://docs.safe.com/fme/html/FME_Server_Documentation/AdminGuide/Using-Python-with-FME_Server.htm.

I appreciate the duplication of files and maintainability by using our recommended approach. But you can also appreciate the problem if a change is made to the python mods that break FME Server Jobs for some reason that is not found in FME Desktop. In my opinion, there's a good reason for keeping the modules as a controlled copy of your source files that you might be managing in Python/FME Desktop. I assume you know the risk. And it is possible, you are actually managing the new modules away from FME Desktop as well... but just want to make sure we flush out all the facts here for why you need this behaviour to be different in FME Server and not make use of the Shared Resources and the plugins folder.

So now the question becomes, is it possible to add a shared folder to the search path for the python interpreter in use or add the search path to the FME Server engine for python visibility in some other way? And I do not know the answer to this at this time. It could be possible, but I suspect this would involve a hack. So I'll have to take this back to the team to see what might possible now, or what could be added to the product in the future.

I like the idea of a new "IDEA" as this will allow all you awesome Experts to chime in for why this feature will benefit your requirements.

Thanks for the great reply. I'll just add that I completely agree about keeping FME Desktop and Server as separate as possible. Since the Desktop users aren't bound by the Server security model, it would be a very bad idea to let them have free range on Python modules shared with the Server instance.

lifalin2016
Rank
Userlevel 1
Badge +22

Hi @lifalin2016,

 

Thanks for posting this question.

There's been a lot of changes in trying to make the FME engine get the right interpreter loaded and then comes the problem of loading new modules like this. As @david_r correctly points out, the quick solution is to copy the mods to the appropriate python folder in plugins. For more on this visit this link (I'm sure you are aware of this - and this is our recommended approach): https://docs.safe.com/fme/html/FME_Server_Documentation/AdminGuide/Using-Python-with-FME_Server.htm.

I appreciate the duplication of files and maintainability by using our recommended approach. But you can also appreciate the problem if a change is made to the python mods that break FME Server Jobs for some reason that is not found in FME Desktop. In my opinion, there's a good reason for keeping the modules as a controlled copy of your source files that you might be managing in Python/FME Desktop. I assume you know the risk. And it is possible, you are actually managing the new modules away from FME Desktop as well... but just want to make sure we flush out all the facts here for why you need this behaviour to be different in FME Server and not make use of the Shared Resources and the plugins folder.

So now the question becomes, is it possible to add a shared folder to the search path for the python interpreter in use or add the search path to the FME Server engine for python visibility in some other way? And I do not know the answer to this at this time. It could be possible, but I suspect this would involve a hack. So I'll have to take this back to the team to see what might possible now, or what could be added to the product in the future.

I like the idea of a new "IDEA" as this will allow all you awesome Experts to chime in for why this feature will benefit your requirements.

Thanks Steve.

The separate approach only makes sense, if you publish workspaces to FME Server with untested transformers. I never use untested transformers for production, and I do test them thoroughly before committing them for production use.

So I don't see a problem with FME Server using shared resources like Python based transformers. Isn't that really what "shared" really entails ?

Besides, a failing transformer will only make the workspace fail, not crash the server. In the end, it's all down to fme.exe in both desktop and server, isn't it ?

I'll see about promoting this as an idea, but I fear that very few use FME Server _and_ create their own tranformers, and so acknowledge the issue.

-- Cheers, Lars I.
lifalin2016
Rank
Userlevel 1
Badge +22

I did some digging into the config files and saw the following line in

 

C:\\Program Files\\FMEServer\\Server\\fmeEngineConfig.txt

 

 

FME_SHARED_RESOURCE_DIR "!FME_SERVER_ROOT!/resources/engine"

 

 

I suspect that you could just change this, restart FME Server and give it a try.

I did look at the config files, but is a little scared to tinker with them, and wanted someone at Safe to look at the possibilities.

Do you know if this setting can use multiple folders, or is it replacement ?

-- Cheers, Lars I.
virtualcitymatt
Rank
Userlevel 4
Badge +26

I did look at the config files, but is a little scared to tinker with them, and wanted someone at Safe to look at the possibilities.

Do you know if this setting can use multiple folders, or is it replacement ?

Not sure if you can add multiple paths here. I assume it's just a replacement. Definitely create a backup of the file! - if you are scared to test it out you couuld install a trial FME Server on your local machine and tinker away as much as you like until you are confident to make the switch.

 

 

I really think this would work for custom transforms and formats - I'm not sure if it would work for python, that might be controlled by another variable, but setting an env variable will sort that out if it doesn't work.

 

 

Back in the day there was a lot of tinkering in those config files.

Reply


Community Stats

28,694
Posts
109,434
Replies
36,544
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.