Solved

Portable FME Flow/Form Python environment

1 year ago
July 12, 2023
5 replies
304 views

+12

vlroyrenn
Supporter
62 replies

The FME Python environment is a bit weird, to say the least, which makes it difficult to use in various contexts. It's a Windows Python embedded installation, with the bundled and user-installed libraries in non-standard paths. As stated by the Python doc linked above, this installation doesn't officially support using Pip for package installation (due to the sysconfig paths likely being invalid). For dependency management, I would honestly much prefer using a virtual environment, but the python311 installation bundled with FME 2023 doesn't include the venv module, so that's not even an option.

At that point, I would rather just create a separate, regular python installation, manage the packages myself and tell FME to use that instead of the included one, but then I would need to move all the compiled FME libraries (and the DLLs they depend on)to that environment, and I don't see a list of that anywhere. Is there any version of the FME Python libs that can be installed to a fresh Python environment, or are the only ones supported the ones bundled with FME, to be used by path injection around a separate Python environment like ArcGIS?

Best answer by vlroyrenn

Disregard everything I said above regarding copying libraries and search path shenanigans and just use fme-packager config-env in any fresh Python venv to provide access to FME-specific libraries, including fmeobjects.

It took me that long to stumble upon it, but it’s official and a lot cleaner and easier to setup than what I was up to last year. It’s obviously not mentionned anywhere that’s not related to FME Transformer Designer, and has never been talked about once in the community forums, but it seems to be the one correct solution to this issue.

View original

Did this help you find an answer to your question?

david_r
8314 replies
1 year ago
July 12, 2023

I can only say that I wholeheartedly agree about the weirdness. I've spent countless hours trying to work around the specifics, which thankfully, has gotten somewhat better in the later versions, I would guess around 2021.

If you do something too fancy with e.g. path injection, you do risk running into a lot of issues with the PYTHONPATH that is automatically modified by FME on each execution. Also, I'm not sure how much luck you'll have moving dll files around, personally I've given up on trying to do something too clever.

You can of course still use pip with FME, e.g. to upgrade the "requests" module that is installed with FME:

fme.exe python -m pip install requests --upgrade requests --target C:\Apps\FME2022\python

Note, however, that you risk that such upgrades can conflict with other modules that FME depend on, and in worst case it will make certain functionality or transformers that depend on Python to stop working.

I actually think ArcGIS has a better solution to their Python integration than FME, where it's a full, stand-alone Python installation using venv. It would be fantastic if FME could have something similar.

+12

vlroyrenn
Author
Supporter
62 replies
1 year ago
July 12, 2023

I wrote a script to try and resolve DLL paths to get some kind of list and get an idea of just how much inter-dependency there is between all these libs. There's a lot.

Just to be able to import fmeobjects (which FME automatically imports as part of running fmesite), you need at least all of these libraries in your search path just for the thing to import:

{'ADVAPI32.dll': None,
 'AUTHZ.dll': None,
 'COMCTL32.dll': None,
 'COMDLG32.dll': None,
 'CRYPT32.dll': None,
 'DNSAPI.dll': None,
 'IPHLPAPI.DLL': None,
 'KERNEL32.dll': None,
 'MPR.dll': None,
 'MSVCP140.dll': None,
 'MSVCP140_1.dll': None,
 'NETAPI32.dll': None,
 'Normaliz.dll': None,
 'ODBC32.dll': None,
 'OLEAUT32.dll': None,
 'PSAPI.DLL': None,
 'Qt6Core5Compat_fme.dll': Path('FMEFlow/Server/fme/Qt6Core5Compat_fme.dll'),
 'Qt6Core_fme.dll': Path('FMEFlow/Server/fme/Qt6Core_fme.dll'),
 'Qt6Network_fme.dll': Path('FMEFlow/Server/fme/Qt6Network_fme.dll'),
 'Qt6Sql_fme.dll': Path('FMEFlow/Server/fme/Qt6Sql_fme.dll'),
 'RPCRT4.dll': None,
 'SHELL32.dll': None,
 'SHFOLDER.dll': None,
 'SHLWAPI.dll': None,
 'Secur32.dll': None,
 'USER32.dll': None,
 'USERENV.dll': None,
 'VCRUNTIME140.dll': Path('FMEFlow/Server/fme/fmepython311/VCRUNTIME140.dll'),
 'VCRUNTIME140_1.dll': Path('FMEFlow/Server/fme/fmepython311/VCRUNTIME140_1.dll'),
 'VERSION.dll': None,
 'WINHTTP.dll': None,
 'WINMM.dll': None,
 'WINTRUST.dll': None,
 'WLDAP32.dll': None,
 'WS2_32.dll': None,
 'WSOCK32.dll': None,
 'api-ms-win-core-path-l1-1-0.dll': None,
 'api-ms-win-core-synch-l1-2-0.dll': Path('FMEFlow/Server/fme/api-ms-win-core-synch-l1-2-0.dll'),
 'api-ms-win-crt-conio-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-conio-l1-1-0.dll'),
 'api-ms-win-crt-convert-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-convert-l1-1-0.dll'),
 'api-ms-win-crt-environment-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-environment-l1-1-0.dll'),
 'api-ms-win-crt-filesystem-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-filesystem-l1-1-0.dll'),
 'api-ms-win-crt-heap-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-heap-l1-1-0.dll'),
 'api-ms-win-crt-locale-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-locale-l1-1-0.dll'),
 'api-ms-win-crt-math-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-math-l1-1-0.dll'),
 'api-ms-win-crt-multibyte-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-multibyte-l1-1-0.dll'),
 'api-ms-win-crt-process-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-process-l1-1-0.dll'),
 'api-ms-win-crt-runtime-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-runtime-l1-1-0.dll'),
 'api-ms-win-crt-stdio-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-stdio-l1-1-0.dll'),
 'api-ms-win-crt-string-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-string-l1-1-0.dll'),
 'api-ms-win-crt-time-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-time-l1-1-0.dll'),
 'api-ms-win-crt-utility-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-utility-l1-1-0.dll'),
 'bcrypt.dll': None,
 'boost_locale-vc141-mt-x64-1_67.dll': Path('FMEFlow/Server/fme/boost_locale-vc141-mt-x64-1_67.dll'),
 'boost_system-vc141-mt-x64-1_67.dll': Path('FMEFlow/Server/fme/boost_system-vc141-mt-x64-1_67.dll'),
 'csutils.dll': Path('FMEFlow/Server/fme/csutils.dll'),
 'dhcpcsvc.DLL': None,
 'esriutils.dll': Path('FMEFlow/Server/fme/esriutils.dll'),
 'fme.dll': Path('FMEFlow/Server/fme/fme.dll'),
 'fme_stringconverter.dll': Path('FMEFlow/Server/fme/fme_stringconverter.dll'),
 'fme_usagestats.dll': Path('FMEFlow/Server/fme/fme_usagestats.dll'),
 'fme_webservices.dll': Path('FMEFlow/Server/fme/fme_webservices.dll'),
 'fmecomm.dll': Path('FMEFlow/Server/fme/fmecomm.dll'),
 'fmeobjects.pyd': Path('FMEFlow/Server/fme/fmeobjects/python311/fmeobjects.pyd'),
 'fmepython311.dll': Path('FMEFlow/Server/fme/fmepython311.dll'),
 'fmeserverapi.dll': Path('FMEFlow/Server/fme/fmeserverapi.dll'),
 'fmetransform.dll': Path('FMEFlow/Server/fme/fmetransform.dll'),
 'fmeutil.dll': Path('FMEFlow/Server/fme/fmeutil.dll'),
 'geos_fme.dll': Path('FMEFlow/Server/fme/geos_fme.dll'),
 'iconv_fme.dll': Path('FMEFlow/Server/fme/iconv_fme.dll'),
 'icudt_fme.dll': Path('FMEFlow/Server/fme/icudt_fme.dll'),
 'icuin_fme.dll': Path('FMEFlow/Server/fme/icuin_fme.dll'),
 'icuuc_fme.dll': Path('FMEFlow/Server/fme/icuuc_fme.dll'),
 'jpeg62.dll': Path('FMEFlow/Server/fme/jpeg62.dll'),
 'libarchive_fme.dll': Path('FMEFlow/Server/fme/libarchive_fme.dll'),
 'libcrypto_fme.dll': Path('FMEFlow/Server/fme/libcrypto_fme.dll'),
 'libcurl_fme.dll': Path('FMEFlow/Server/fme/libcurl_fme.dll'),
 'libfmeuicore.dll': Path('FMEFlow/Server/fme/libfmeuicore.dll'),
 'libfmeuihub.dll': Path('FMEFlow/Server/fme/libfmeuihub.dll'),
 'libfmeuiparamdef.dll': Path('FMEFlow/Server/fme/libfmeuiparamdef.dll'),
 'libfmeuiparammodel.dll': Path('FMEFlow/Server/fme/libfmeuiparammodel.dll'),
 'libfmeuiserver.dll': Path('FMEFlow/Server/fme/libfmeuiserver.dll'),
 'libfmeuixformdef.dll': Path('FMEFlow/Server/fme/libfmeuixformdef.dll'),
 'liblz4.dll': Path('FMEFlow/Server/fme/liblz4.dll'),
 'liblzma_fme.dll': Path('FMEFlow/Server/fme/liblzma_fme.dll'),
 'libssh2_fme.dll': Path('FMEFlow/Server/fme/libssh2_fme.dll'),
 'libssl_fme.dll': Path('FMEFlow/Server/fme/libssl_fme.dll'),
 'libxml2_fme.dll': Path('FMEFlow/Server/fme/libxml2_fme.dll'),
 'mpfr_fme.dll': Path('FMEFlow/Server/fme/mpfr_fme.dll'),
 'mpir_fme.dll': Path('FMEFlow/Server/fme/mpir_fme.dll'),
 'nghttp2_fme.dll': Path('FMEFlow/Server/fme/nghttp2_fme.dll'),
 'ogcwktutil_fme.dll': Path('FMEFlow/Server/fme/ogcwktutil_fme.dll'),
 'ole32.dll': None,
 'pluginutils.dll': Path('FMEFlow/Server/fme/pluginutils.dll'),
 'proj_fme.dll': Path('FMEFlow/Server/fme/proj_fme.dll'),
 'projutil_fme.dll': Path('FMEFlow/Server/fme/projutil_fme.dll'),
 'python311.dll': Path('FMEFlow/Server/fme/fmepython311/python311.dll'),
 'rptree.dll': Path('FMEFlow/Server/fme/rptree.dll'),
 'sqlite3_fme.dll': Path('FMEFlow/Server/fme/sqlite3_fme.dll'),
 'stk.dll': Path('FMEFlow/Server/fme/stk.dll'),
 'tbb.dll': Path('FMEFlow/Server/fme/tbb.dll'),
 'tbbmalloc.dll': Path('FMEFlow/Server/fme/tbbmalloc.dll'),
 'tcl_fme.dll': Path('FMEFlow/Server/fme/tcl_fme.dll'),
 'threadlockmgr.dll': Path('FMEFlow/Server/fme/threadlockmgr.dll'),
 'tiff_fme.dll': Path('FMEFlow/Server/fme/tiff_fme.dll'),
 'unrar_fme.dll': Path('FMEFlow/Server/fme/unrar_fme.dll'),
 'xerces-c_3_2_fme.dll': Path('FMEFlow/Server/fme/xerces-c_3_2_fme.dll'),
 'zlib_fme.dll': Path('FMEFlow/Server/fme/zlib_fme.dll')}

And a few extra DLLs that are optionally loaded and might needed for all the features to be available.

'DWrite.dll': None,
  'GDI32.dll': None,
  'GTransTF.dll': None,
  'NCSEcw.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/NCSEcw.dll'),
  'OCI.dll': None,
  'WININET.dll': None,
  'dbgeng.dll': None,
  'dedit.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/dedit.dll'),
  'fmepythonpip.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/fmepythonpip.dll'),
  'imagehlp.dll': None,
  'meshsimplificationutil.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/meshsimplificationutil.dll'),
  'opennurbs_fme.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/opennurbs_fme.dll'),
  'pcache.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/pcache.dll'),
  'solidvalidation.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/solidvalidation.dll'),
  'straightskeleton.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/straightskeleton.dll'),
  'trianglesurface.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/trianglesurface.dll'),
  'trnapi.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/trnapi.dll')

The DLLs with "None" written next to them are not in the FME Python search path, so they're expected to be in the Windows system path (most of these are standard Windows API DLLs, but some like OCI.dll are third-party).

At any rate, that's a lot of DLLs to carry around by hand just to get the basic features working, and I was certainly hoping there wouldn't be that many.

david_r
8314 replies
1 year ago
July 13, 2023

vlroyrenn wrote:

I wrote a script to try and resolve DLL paths to get some kind of list and get an idea of just how much inter-dependency there is between all these libs. There's a lot.

Just to be able to import fmeobjects (which FME automatically imports as part of running fmesite), you need at least all of these libraries in your search path just for the thing to import:

{'ADVAPI32.dll': None,
 'AUTHZ.dll': None,
 'COMCTL32.dll': None,
 'COMDLG32.dll': None,
 'CRYPT32.dll': None,
 'DNSAPI.dll': None,
 'IPHLPAPI.DLL': None,
 'KERNEL32.dll': None,
 'MPR.dll': None,
 'MSVCP140.dll': None,
 'MSVCP140_1.dll': None,
 'NETAPI32.dll': None,
 'Normaliz.dll': None,
 'ODBC32.dll': None,
 'OLEAUT32.dll': None,
 'PSAPI.DLL': None,
 'Qt6Core5Compat_fme.dll': Path('FMEFlow/Server/fme/Qt6Core5Compat_fme.dll'),
 'Qt6Core_fme.dll': Path('FMEFlow/Server/fme/Qt6Core_fme.dll'),
 'Qt6Network_fme.dll': Path('FMEFlow/Server/fme/Qt6Network_fme.dll'),
 'Qt6Sql_fme.dll': Path('FMEFlow/Server/fme/Qt6Sql_fme.dll'),
 'RPCRT4.dll': None,
 'SHELL32.dll': None,
 'SHFOLDER.dll': None,
 'SHLWAPI.dll': None,
 'Secur32.dll': None,
 'USER32.dll': None,
 'USERENV.dll': None,
 'VCRUNTIME140.dll': Path('FMEFlow/Server/fme/fmepython311/VCRUNTIME140.dll'),
 'VCRUNTIME140_1.dll': Path('FMEFlow/Server/fme/fmepython311/VCRUNTIME140_1.dll'),
 'VERSION.dll': None,
 'WINHTTP.dll': None,
 'WINMM.dll': None,
 'WINTRUST.dll': None,
 'WLDAP32.dll': None,
 'WS2_32.dll': None,
 'WSOCK32.dll': None,
 'api-ms-win-core-path-l1-1-0.dll': None,
 'api-ms-win-core-synch-l1-2-0.dll': Path('FMEFlow/Server/fme/api-ms-win-core-synch-l1-2-0.dll'),
 'api-ms-win-crt-conio-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-conio-l1-1-0.dll'),
 'api-ms-win-crt-convert-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-convert-l1-1-0.dll'),
 'api-ms-win-crt-environment-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-environment-l1-1-0.dll'),
 'api-ms-win-crt-filesystem-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-filesystem-l1-1-0.dll'),
 'api-ms-win-crt-heap-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-heap-l1-1-0.dll'),
 'api-ms-win-crt-locale-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-locale-l1-1-0.dll'),
 'api-ms-win-crt-math-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-math-l1-1-0.dll'),
 'api-ms-win-crt-multibyte-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-multibyte-l1-1-0.dll'),
 'api-ms-win-crt-process-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-process-l1-1-0.dll'),
 'api-ms-win-crt-runtime-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-runtime-l1-1-0.dll'),
 'api-ms-win-crt-stdio-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-stdio-l1-1-0.dll'),
 'api-ms-win-crt-string-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-string-l1-1-0.dll'),
 'api-ms-win-crt-time-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-time-l1-1-0.dll'),
 'api-ms-win-crt-utility-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-utility-l1-1-0.dll'),
 'bcrypt.dll': None,
 'boost_locale-vc141-mt-x64-1_67.dll': Path('FMEFlow/Server/fme/boost_locale-vc141-mt-x64-1_67.dll'),
 'boost_system-vc141-mt-x64-1_67.dll': Path('FMEFlow/Server/fme/boost_system-vc141-mt-x64-1_67.dll'),
 'csutils.dll': Path('FMEFlow/Server/fme/csutils.dll'),
 'dhcpcsvc.DLL': None,
 'esriutils.dll': Path('FMEFlow/Server/fme/esriutils.dll'),
 'fme.dll': Path('FMEFlow/Server/fme/fme.dll'),
 'fme_stringconverter.dll': Path('FMEFlow/Server/fme/fme_stringconverter.dll'),
 'fme_usagestats.dll': Path('FMEFlow/Server/fme/fme_usagestats.dll'),
 'fme_webservices.dll': Path('FMEFlow/Server/fme/fme_webservices.dll'),
 'fmecomm.dll': Path('FMEFlow/Server/fme/fmecomm.dll'),
 'fmeobjects.pyd': Path('FMEFlow/Server/fme/fmeobjects/python311/fmeobjects.pyd'),
 'fmepython311.dll': Path('FMEFlow/Server/fme/fmepython311.dll'),
 'fmeserverapi.dll': Path('FMEFlow/Server/fme/fmeserverapi.dll'),
 'fmetransform.dll': Path('FMEFlow/Server/fme/fmetransform.dll'),
 'fmeutil.dll': Path('FMEFlow/Server/fme/fmeutil.dll'),
 'geos_fme.dll': Path('FMEFlow/Server/fme/geos_fme.dll'),
 'iconv_fme.dll': Path('FMEFlow/Server/fme/iconv_fme.dll'),
 'icudt_fme.dll': Path('FMEFlow/Server/fme/icudt_fme.dll'),
 'icuin_fme.dll': Path('FMEFlow/Server/fme/icuin_fme.dll'),
 'icuuc_fme.dll': Path('FMEFlow/Server/fme/icuuc_fme.dll'),
 'jpeg62.dll': Path('FMEFlow/Server/fme/jpeg62.dll'),
 'libarchive_fme.dll': Path('FMEFlow/Server/fme/libarchive_fme.dll'),
 'libcrypto_fme.dll': Path('FMEFlow/Server/fme/libcrypto_fme.dll'),
 'libcurl_fme.dll': Path('FMEFlow/Server/fme/libcurl_fme.dll'),
 'libfmeuicore.dll': Path('FMEFlow/Server/fme/libfmeuicore.dll'),
 'libfmeuihub.dll': Path('FMEFlow/Server/fme/libfmeuihub.dll'),
 'libfmeuiparamdef.dll': Path('FMEFlow/Server/fme/libfmeuiparamdef.dll'),
 'libfmeuiparammodel.dll': Path('FMEFlow/Server/fme/libfmeuiparammodel.dll'),
 'libfmeuiserver.dll': Path('FMEFlow/Server/fme/libfmeuiserver.dll'),
 'libfmeuixformdef.dll': Path('FMEFlow/Server/fme/libfmeuixformdef.dll'),
 'liblz4.dll': Path('FMEFlow/Server/fme/liblz4.dll'),
 'liblzma_fme.dll': Path('FMEFlow/Server/fme/liblzma_fme.dll'),
 'libssh2_fme.dll': Path('FMEFlow/Server/fme/libssh2_fme.dll'),
 'libssl_fme.dll': Path('FMEFlow/Server/fme/libssl_fme.dll'),
 'libxml2_fme.dll': Path('FMEFlow/Server/fme/libxml2_fme.dll'),
 'mpfr_fme.dll': Path('FMEFlow/Server/fme/mpfr_fme.dll'),
 'mpir_fme.dll': Path('FMEFlow/Server/fme/mpir_fme.dll'),
 'nghttp2_fme.dll': Path('FMEFlow/Server/fme/nghttp2_fme.dll'),
 'ogcwktutil_fme.dll': Path('FMEFlow/Server/fme/ogcwktutil_fme.dll'),
 'ole32.dll': None,
 'pluginutils.dll': Path('FMEFlow/Server/fme/pluginutils.dll'),
 'proj_fme.dll': Path('FMEFlow/Server/fme/proj_fme.dll'),
 'projutil_fme.dll': Path('FMEFlow/Server/fme/projutil_fme.dll'),
 'python311.dll': Path('FMEFlow/Server/fme/fmepython311/python311.dll'),
 'rptree.dll': Path('FMEFlow/Server/fme/rptree.dll'),
 'sqlite3_fme.dll': Path('FMEFlow/Server/fme/sqlite3_fme.dll'),
 'stk.dll': Path('FMEFlow/Server/fme/stk.dll'),
 'tbb.dll': Path('FMEFlow/Server/fme/tbb.dll'),
 'tbbmalloc.dll': Path('FMEFlow/Server/fme/tbbmalloc.dll'),
 'tcl_fme.dll': Path('FMEFlow/Server/fme/tcl_fme.dll'),
 'threadlockmgr.dll': Path('FMEFlow/Server/fme/threadlockmgr.dll'),
 'tiff_fme.dll': Path('FMEFlow/Server/fme/tiff_fme.dll'),
 'unrar_fme.dll': Path('FMEFlow/Server/fme/unrar_fme.dll'),
 'xerces-c_3_2_fme.dll': Path('FMEFlow/Server/fme/xerces-c_3_2_fme.dll'),
 'zlib_fme.dll': Path('FMEFlow/Server/fme/zlib_fme.dll')}

And a few extra DLLs that are optionally loaded and might needed for all the features to be available.

'DWrite.dll': None,
  'GDI32.dll': None,
  'GTransTF.dll': None,
  'NCSEcw.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/NCSEcw.dll'),
  'OCI.dll': None,
  'WININET.dll': None,
  'dbgeng.dll': None,
  'dedit.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/dedit.dll'),
  'fmepythonpip.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/fmepythonpip.dll'),
  'imagehlp.dll': None,
  'meshsimplificationutil.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/meshsimplificationutil.dll'),
  'opennurbs_fme.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/opennurbs_fme.dll'),
  'pcache.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/pcache.dll'),
  'solidvalidation.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/solidvalidation.dll'),
  'straightskeleton.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/straightskeleton.dll'),
  'trianglesurface.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/trianglesurface.dll'),
  'trnapi.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/trnapi.dll')

At any rate, that's a lot of DLLs to carry around by hand just to get the basic features working, and I was certainly hoping there wouldn't be that many.

That's an impressive list, although I'm not very surprised. I have the impression that fmeobjects is just a relatively thin interface on top of the C++ libraries that make up the fundamental FME engine.

For what it's worth, here are my own current guidelines for developing Python with FME when the script is non-trivial:

Develop the entire functionality in a proper IDE, e.g. Pycharm, without any references to fmeobjects whatsoever. Generalize everything into proper first-order Python classes without relying on the FME feature/object model. Write unit tests.
Try to stick with the standard library as much as possible, or if necessary, only install minimalist packages for specific functionality. Try very hard to avoid monoliths such as Anaconda, Geopandas, etc. as they can be tricky to make work with FME due to dependencies.
In your workspace (e.g. PythonCaller) import the above mentioned libraries and write the least amount of lines possible, only what's necessary to instantiate and interact with the business objects using the incoming feature.

The idea is to fully separate FME and the business logic code, making it easy to debug, extend, refactor and automate testing of the business objects without relying on FME.

+12

vlroyrenn
Author
Supporter
62 replies
1 year ago
July 13, 2023

david_r wrote:

For what it's worth, here are my own current guidelines for developing Python with FME when the script is non-trivial:

Develop the entire functionality in a proper IDE, e.g. Pycharm, without any references to fmeobjects whatsoever. Generalize everything into proper first-order Python classes without relying on the FME feature/object model. Write unit tests.
Try to stick with the standard library as much as possible, or if necessary, only install minimalist packages for specific functionality. Try very hard to avoid monoliths such as Anaconda, Geopandas, etc. as they can be tricky to make work with FME due to dependencies.
In your workspace (e.g. PythonCaller) import the above mentioned libraries and write the least amount of lines possible, only what's necessary to instantiate and interact with the business objects using the incoming feature.

The idea is to fully separate FME and the business logic code, making it easy to debug, extend, refactor and automate testing of the business objects without relying on FME.

Developping functionnalities as importable, unit-tested, FME-independent modules is an interesting practice that I hadn't considered before and might be tempted to use for transformers that handle saner amounts of data (see my previous question regarding dataframes, since I'm currently handling too much data for pure Python to handle this with a reasonnable performance level). I like that it encourages making your transformer code portable. Still, I can't currently avoid pulling dependencies, whether FME-vendored ones or user-installed ones, and at the very least being able to use the same environment that FME would have access to at runtime would be very beneficial.

So I spent some time yesterday trying to do just that.

***

As it turns out, given the way FME loads Python interpreters, whether built-in or custom, FME can't make use of the environment constructed below and will fail to include the modules from either site-packages in its module search path, because the virtual environment is activated by python.exe, not python3.dll. This would only be useful for developping PythonCaller and PythonCreator (and possibly scripted user parameters) scripts outside of FME Workbench. The user-installed dependencies would still need to be installed in the native locations for both of these in order to run user scripts there.

Nevertheless, it turns out that making a standalone Python install that's compatible with the embedded version is possible and the result works raher well, so for good measure, here's how you can create one yourself for local developpment.

Install Python 3.11 (preferably without pip) in a system location. Uncheck adding it to PATH and the environment, as well as py launcher. The locatino where you've installed is one I'm going to call `MYPYTHON_FME_RO` from now on.
Copy everything from `FME2023/python` (minus the directories that start with "python3"), `FME2023/python/python311` and `FME2023\fmeobjects\python311` to `MYPYTHON_FME_RO/Lib/site-packages`. There shouldn't be anything in site-packages unless you asked the installer to install Pip, but if there's something in there, you can safely remove it. If you're copying the files from FME Server, these are instead located in `FMEFlow\Server\fme`.
To make sure the DLLs from FME are on the DLL search path, add a `MYPYTHON_FME_RO/Lib/site-packages/sitecustomize.py` file that contains `import os;os.add_dll_directory(r"D:\Program Files\FME2023")` (replace that path with whatever FME Flow/FME Form directory you've been using copying things from so far, the one that contains fme.exe and all the DLLs I mentionned in my previous reply)
A good idea is to also add a `MYPYTHON_FME_RO/Lib/EXTERNALLY-MANAGED` file (not in `site-packages`, this goes in `Lib` directly) to your new Python environment to prevent Pip from adding new packages to it. The contents should look like this:

; This is an INI file following the PEP 668 specification. It's pretty new.
[externally-managed]
Error=This is a copy of the FME Python environment, please do not alter it.

`MYPYTHON_FME_RO` is intended to be a read-only copy of the FME libs, so we're not going to be touching anything in there anymore. To avoid interfences, the user-installed libraries will be installed in a separate, virtual environment. Simply run `MYPYTHON_FME_RO\python.exe -mvenv --system-site-packages "MYPYTHON_RW"`, with `MYPYTHON_RW` being a new, distinct path from `MYPYTHON_FME_RO`.
When your IDE of choice asks you what interpreter you would like to use, specify `MYPYTHON_RW/Scripts/python.exe`.

The DLL list I posted yesterday establishes that every non-system DLL needed by fmeobjects is in FMEFlow\Server\fme, so adding that to the DLL import path using sitecustomize instead of moving these by hand definitely seems like the most manageable option.

You can install any library you want in the `MYPYTHON_RW` virtual environment, and pip will resolve dependencies in accordance with the library versions supplied by FME. I wouldn't reccomend installing any upgraded versions of the included libraries, though, because while Pip will raise dependency warnings if such an upgrade were to break the requirements of another FME-supplied lib that's not being upgraded, it will still install the upgrade and break the other libraries. That is because Python will load the modules from `MYPYTHON_RW` in priority, including the ones being imported by modules located in `MYPYTHON_FME_RO` that expect to see the versions that are also in `MYPYTHON_FME_RO`.

Most crucially, though, is that you can run `import fmeobjects` from within this environment and it won't raise any errors. Pycharm and VSCode will both recognize and work with everything in this environment out of the box (something I haven't managed to do with Pycharm, despite the tutorial), with no additionnal configuration needed.

+12

vlroyrenn
Author
Supporter
62 replies
Best Answer
7 months ago
August 26, 2024

Reply

Rich Text Editor, editor1

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

Cookie settings

We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.

Basic
Functional

Normal
Functional + analytics

Complete
Functional + analytics + social media + embedded videos

Portable FME Flow/Form Python environment

5 replies

Reply

Helpful Members This Week

Recently Solved Questions

WorkspaceReader - Find annotation linked to transformers

Linear Referencing Speed along line / Event CSV and Line Geometry

Reading and IFC-file, reproject it and write back to new IFC-file

ERROR : INCLUDE -- failed to evaluate TCL expression

Deployment parameters in Flow 2024.1.1 app not working

Community Stats

Latest FME

Cookie policy

Cookie settings

Reply

Related Topics

list of 2024 r1icon

Issue After Upgrade to 2024 R1: Aggregate Validation Error on POLine+baseReceivedQtyicon

Substitution lists with GIsicon

Reading GI Values from Graph Code 2024r1icon

PXAutomation not in 2024R1 or was it renamed?icon

Helpful Members This Week

Recently Solved Questions

WorkspaceReader - Find annotation linked to transformers

Linear Referencing Speed along line / Event CSV and Line Geometry

Reading and IFC-file, reproject it and write back to new IFC-file

ERROR : INCLUDE -- failed to evaluate TCL expression

Deployment parameters in Flow 2024.1.1 app not working

Popular Tags

Community Stats

Latest FME

Sign up

An FME Account is required to contribute

Login to the community

An FME Account is required to contribute

Scanning file for viruses.

This file cannot be downloaded

Cookie policy

Cookie settings