I can only say that I wholeheartedly agree about the weirdness. I've spent countless hours trying to work around the specifics, which thankfully, has gotten somewhat better in the later versions, I would guess around 2021.
If you do something too fancy with e.g. path injection, you do risk running into a lot of issues with the PYTHONPATH that is automatically modified by FME on each execution. Also, I'm not sure how much luck you'll have moving dll files around, personally I've given up on trying to do something too clever.
You can of course still use pip with FME, e.g. to upgrade the "requests" module that is installed with FME:
fme.exe python -m pip install requests --upgrade requests --target C:\Apps\FME2022\python
Note, however, that you risk that such upgrades can conflict with other modules that FME depend on, and in worst case it will make certain functionality or transformers that depend on Python to stop working.
I actually think ArcGIS has a better solution to their Python integration than FME, where it's a full, stand-alone Python installation using venv. It would be fantastic if FME could have something similar.
I wrote a script to try and resolve DLL paths to get some kind of list and get an idea of just how much inter-dependency there is between all these libs. There's a lot.
Just to be able to import fmeobjects (which FME automatically imports as part of running fmesite), you need at least all of these libraries in your search path just for the thing to import:
{'ADVAPI32.dll': None,
'AUTHZ.dll': None,
'COMCTL32.dll': None,
'COMDLG32.dll': None,
'CRYPT32.dll': None,
'DNSAPI.dll': None,
'IPHLPAPI.DLL': None,
'KERNEL32.dll': None,
'MPR.dll': None,
'MSVCP140.dll': None,
'MSVCP140_1.dll': None,
'NETAPI32.dll': None,
'Normaliz.dll': None,
'ODBC32.dll': None,
'OLEAUT32.dll': None,
'PSAPI.DLL': None,
'Qt6Core5Compat_fme.dll': Path('FMEFlow/Server/fme/Qt6Core5Compat_fme.dll'),
'Qt6Core_fme.dll': Path('FMEFlow/Server/fme/Qt6Core_fme.dll'),
'Qt6Network_fme.dll': Path('FMEFlow/Server/fme/Qt6Network_fme.dll'),
'Qt6Sql_fme.dll': Path('FMEFlow/Server/fme/Qt6Sql_fme.dll'),
'RPCRT4.dll': None,
'SHELL32.dll': None,
'SHFOLDER.dll': None,
'SHLWAPI.dll': None,
'Secur32.dll': None,
'USER32.dll': None,
'USERENV.dll': None,
'VCRUNTIME140.dll': Path('FMEFlow/Server/fme/fmepython311/VCRUNTIME140.dll'),
'VCRUNTIME140_1.dll': Path('FMEFlow/Server/fme/fmepython311/VCRUNTIME140_1.dll'),
'VERSION.dll': None,
'WINHTTP.dll': None,
'WINMM.dll': None,
'WINTRUST.dll': None,
'WLDAP32.dll': None,
'WS2_32.dll': None,
'WSOCK32.dll': None,
'api-ms-win-core-path-l1-1-0.dll': None,
'api-ms-win-core-synch-l1-2-0.dll': Path('FMEFlow/Server/fme/api-ms-win-core-synch-l1-2-0.dll'),
'api-ms-win-crt-conio-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-conio-l1-1-0.dll'),
'api-ms-win-crt-convert-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-convert-l1-1-0.dll'),
'api-ms-win-crt-environment-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-environment-l1-1-0.dll'),
'api-ms-win-crt-filesystem-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-filesystem-l1-1-0.dll'),
'api-ms-win-crt-heap-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-heap-l1-1-0.dll'),
'api-ms-win-crt-locale-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-locale-l1-1-0.dll'),
'api-ms-win-crt-math-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-math-l1-1-0.dll'),
'api-ms-win-crt-multibyte-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-multibyte-l1-1-0.dll'),
'api-ms-win-crt-process-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-process-l1-1-0.dll'),
'api-ms-win-crt-runtime-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-runtime-l1-1-0.dll'),
'api-ms-win-crt-stdio-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-stdio-l1-1-0.dll'),
'api-ms-win-crt-string-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-string-l1-1-0.dll'),
'api-ms-win-crt-time-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-time-l1-1-0.dll'),
'api-ms-win-crt-utility-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-utility-l1-1-0.dll'),
'bcrypt.dll': None,
'boost_locale-vc141-mt-x64-1_67.dll': Path('FMEFlow/Server/fme/boost_locale-vc141-mt-x64-1_67.dll'),
'boost_system-vc141-mt-x64-1_67.dll': Path('FMEFlow/Server/fme/boost_system-vc141-mt-x64-1_67.dll'),
'csutils.dll': Path('FMEFlow/Server/fme/csutils.dll'),
'dhcpcsvc.DLL': None,
'esriutils.dll': Path('FMEFlow/Server/fme/esriutils.dll'),
'fme.dll': Path('FMEFlow/Server/fme/fme.dll'),
'fme_stringconverter.dll': Path('FMEFlow/Server/fme/fme_stringconverter.dll'),
'fme_usagestats.dll': Path('FMEFlow/Server/fme/fme_usagestats.dll'),
'fme_webservices.dll': Path('FMEFlow/Server/fme/fme_webservices.dll'),
'fmecomm.dll': Path('FMEFlow/Server/fme/fmecomm.dll'),
'fmeobjects.pyd': Path('FMEFlow/Server/fme/fmeobjects/python311/fmeobjects.pyd'),
'fmepython311.dll': Path('FMEFlow/Server/fme/fmepython311.dll'),
'fmeserverapi.dll': Path('FMEFlow/Server/fme/fmeserverapi.dll'),
'fmetransform.dll': Path('FMEFlow/Server/fme/fmetransform.dll'),
'fmeutil.dll': Path('FMEFlow/Server/fme/fmeutil.dll'),
'geos_fme.dll': Path('FMEFlow/Server/fme/geos_fme.dll'),
'iconv_fme.dll': Path('FMEFlow/Server/fme/iconv_fme.dll'),
'icudt_fme.dll': Path('FMEFlow/Server/fme/icudt_fme.dll'),
'icuin_fme.dll': Path('FMEFlow/Server/fme/icuin_fme.dll'),
'icuuc_fme.dll': Path('FMEFlow/Server/fme/icuuc_fme.dll'),
'jpeg62.dll': Path('FMEFlow/Server/fme/jpeg62.dll'),
'libarchive_fme.dll': Path('FMEFlow/Server/fme/libarchive_fme.dll'),
'libcrypto_fme.dll': Path('FMEFlow/Server/fme/libcrypto_fme.dll'),
'libcurl_fme.dll': Path('FMEFlow/Server/fme/libcurl_fme.dll'),
'libfmeuicore.dll': Path('FMEFlow/Server/fme/libfmeuicore.dll'),
'libfmeuihub.dll': Path('FMEFlow/Server/fme/libfmeuihub.dll'),
'libfmeuiparamdef.dll': Path('FMEFlow/Server/fme/libfmeuiparamdef.dll'),
'libfmeuiparammodel.dll': Path('FMEFlow/Server/fme/libfmeuiparammodel.dll'),
'libfmeuiserver.dll': Path('FMEFlow/Server/fme/libfmeuiserver.dll'),
'libfmeuixformdef.dll': Path('FMEFlow/Server/fme/libfmeuixformdef.dll'),
'liblz4.dll': Path('FMEFlow/Server/fme/liblz4.dll'),
'liblzma_fme.dll': Path('FMEFlow/Server/fme/liblzma_fme.dll'),
'libssh2_fme.dll': Path('FMEFlow/Server/fme/libssh2_fme.dll'),
'libssl_fme.dll': Path('FMEFlow/Server/fme/libssl_fme.dll'),
'libxml2_fme.dll': Path('FMEFlow/Server/fme/libxml2_fme.dll'),
'mpfr_fme.dll': Path('FMEFlow/Server/fme/mpfr_fme.dll'),
'mpir_fme.dll': Path('FMEFlow/Server/fme/mpir_fme.dll'),
'nghttp2_fme.dll': Path('FMEFlow/Server/fme/nghttp2_fme.dll'),
'ogcwktutil_fme.dll': Path('FMEFlow/Server/fme/ogcwktutil_fme.dll'),
'ole32.dll': None,
'pluginutils.dll': Path('FMEFlow/Server/fme/pluginutils.dll'),
'proj_fme.dll': Path('FMEFlow/Server/fme/proj_fme.dll'),
'projutil_fme.dll': Path('FMEFlow/Server/fme/projutil_fme.dll'),
'python311.dll': Path('FMEFlow/Server/fme/fmepython311/python311.dll'),
'rptree.dll': Path('FMEFlow/Server/fme/rptree.dll'),
'sqlite3_fme.dll': Path('FMEFlow/Server/fme/sqlite3_fme.dll'),
'stk.dll': Path('FMEFlow/Server/fme/stk.dll'),
'tbb.dll': Path('FMEFlow/Server/fme/tbb.dll'),
'tbbmalloc.dll': Path('FMEFlow/Server/fme/tbbmalloc.dll'),
'tcl_fme.dll': Path('FMEFlow/Server/fme/tcl_fme.dll'),
'threadlockmgr.dll': Path('FMEFlow/Server/fme/threadlockmgr.dll'),
'tiff_fme.dll': Path('FMEFlow/Server/fme/tiff_fme.dll'),
'unrar_fme.dll': Path('FMEFlow/Server/fme/unrar_fme.dll'),
'xerces-c_3_2_fme.dll': Path('FMEFlow/Server/fme/xerces-c_3_2_fme.dll'),
'zlib_fme.dll': Path('FMEFlow/Server/fme/zlib_fme.dll')}
And a few extra DLLs that are optionally loaded and might needed for all the features to be available.
'DWrite.dll': None,
'GDI32.dll': None,
'GTransTF.dll': None,
'NCSEcw.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/NCSEcw.dll'),
'OCI.dll': None,
'WININET.dll': None,
'dbgeng.dll': None,
'dedit.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/dedit.dll'),
'fmepythonpip.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/fmepythonpip.dll'),
'imagehlp.dll': None,
'meshsimplificationutil.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/meshsimplificationutil.dll'),
'opennurbs_fme.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/opennurbs_fme.dll'),
'pcache.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/pcache.dll'),
'solidvalidation.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/solidvalidation.dll'),
'straightskeleton.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/straightskeleton.dll'),
'trianglesurface.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/trianglesurface.dll'),
'trnapi.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/trnapi.dll')
The DLLs with "None" written next to them are not in the FME Python search path, so they're expected to be in the Windows system path (most of these are standard Windows API DLLs, but some like OCI.dll are third-party).
At any rate, that's a lot of DLLs to carry around by hand just to get the basic features working, and I was certainly hoping there wouldn't be that many.
I wrote a script to try and resolve DLL paths to get some kind of list and get an idea of just how much inter-dependency there is between all these libs. There's a lot.
Just to be able to import fmeobjects (which FME automatically imports as part of running fmesite), you need at least all of these libraries in your search path just for the thing to import:
{'ADVAPI32.dll': None,
'AUTHZ.dll': None,
'COMCTL32.dll': None,
'COMDLG32.dll': None,
'CRYPT32.dll': None,
'DNSAPI.dll': None,
'IPHLPAPI.DLL': None,
'KERNEL32.dll': None,
'MPR.dll': None,
'MSVCP140.dll': None,
'MSVCP140_1.dll': None,
'NETAPI32.dll': None,
'Normaliz.dll': None,
'ODBC32.dll': None,
'OLEAUT32.dll': None,
'PSAPI.DLL': None,
'Qt6Core5Compat_fme.dll': Path('FMEFlow/Server/fme/Qt6Core5Compat_fme.dll'),
'Qt6Core_fme.dll': Path('FMEFlow/Server/fme/Qt6Core_fme.dll'),
'Qt6Network_fme.dll': Path('FMEFlow/Server/fme/Qt6Network_fme.dll'),
'Qt6Sql_fme.dll': Path('FMEFlow/Server/fme/Qt6Sql_fme.dll'),
'RPCRT4.dll': None,
'SHELL32.dll': None,
'SHFOLDER.dll': None,
'SHLWAPI.dll': None,
'Secur32.dll': None,
'USER32.dll': None,
'USERENV.dll': None,
'VCRUNTIME140.dll': Path('FMEFlow/Server/fme/fmepython311/VCRUNTIME140.dll'),
'VCRUNTIME140_1.dll': Path('FMEFlow/Server/fme/fmepython311/VCRUNTIME140_1.dll'),
'VERSION.dll': None,
'WINHTTP.dll': None,
'WINMM.dll': None,
'WINTRUST.dll': None,
'WLDAP32.dll': None,
'WS2_32.dll': None,
'WSOCK32.dll': None,
'api-ms-win-core-path-l1-1-0.dll': None,
'api-ms-win-core-synch-l1-2-0.dll': Path('FMEFlow/Server/fme/api-ms-win-core-synch-l1-2-0.dll'),
'api-ms-win-crt-conio-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-conio-l1-1-0.dll'),
'api-ms-win-crt-convert-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-convert-l1-1-0.dll'),
'api-ms-win-crt-environment-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-environment-l1-1-0.dll'),
'api-ms-win-crt-filesystem-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-filesystem-l1-1-0.dll'),
'api-ms-win-crt-heap-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-heap-l1-1-0.dll'),
'api-ms-win-crt-locale-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-locale-l1-1-0.dll'),
'api-ms-win-crt-math-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-math-l1-1-0.dll'),
'api-ms-win-crt-multibyte-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-multibyte-l1-1-0.dll'),
'api-ms-win-crt-process-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-process-l1-1-0.dll'),
'api-ms-win-crt-runtime-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-runtime-l1-1-0.dll'),
'api-ms-win-crt-stdio-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-stdio-l1-1-0.dll'),
'api-ms-win-crt-string-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-string-l1-1-0.dll'),
'api-ms-win-crt-time-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-time-l1-1-0.dll'),
'api-ms-win-crt-utility-l1-1-0.dll': Path('FMEFlow/Server/fme/api-ms-win-crt-utility-l1-1-0.dll'),
'bcrypt.dll': None,
'boost_locale-vc141-mt-x64-1_67.dll': Path('FMEFlow/Server/fme/boost_locale-vc141-mt-x64-1_67.dll'),
'boost_system-vc141-mt-x64-1_67.dll': Path('FMEFlow/Server/fme/boost_system-vc141-mt-x64-1_67.dll'),
'csutils.dll': Path('FMEFlow/Server/fme/csutils.dll'),
'dhcpcsvc.DLL': None,
'esriutils.dll': Path('FMEFlow/Server/fme/esriutils.dll'),
'fme.dll': Path('FMEFlow/Server/fme/fme.dll'),
'fme_stringconverter.dll': Path('FMEFlow/Server/fme/fme_stringconverter.dll'),
'fme_usagestats.dll': Path('FMEFlow/Server/fme/fme_usagestats.dll'),
'fme_webservices.dll': Path('FMEFlow/Server/fme/fme_webservices.dll'),
'fmecomm.dll': Path('FMEFlow/Server/fme/fmecomm.dll'),
'fmeobjects.pyd': Path('FMEFlow/Server/fme/fmeobjects/python311/fmeobjects.pyd'),
'fmepython311.dll': Path('FMEFlow/Server/fme/fmepython311.dll'),
'fmeserverapi.dll': Path('FMEFlow/Server/fme/fmeserverapi.dll'),
'fmetransform.dll': Path('FMEFlow/Server/fme/fmetransform.dll'),
'fmeutil.dll': Path('FMEFlow/Server/fme/fmeutil.dll'),
'geos_fme.dll': Path('FMEFlow/Server/fme/geos_fme.dll'),
'iconv_fme.dll': Path('FMEFlow/Server/fme/iconv_fme.dll'),
'icudt_fme.dll': Path('FMEFlow/Server/fme/icudt_fme.dll'),
'icuin_fme.dll': Path('FMEFlow/Server/fme/icuin_fme.dll'),
'icuuc_fme.dll': Path('FMEFlow/Server/fme/icuuc_fme.dll'),
'jpeg62.dll': Path('FMEFlow/Server/fme/jpeg62.dll'),
'libarchive_fme.dll': Path('FMEFlow/Server/fme/libarchive_fme.dll'),
'libcrypto_fme.dll': Path('FMEFlow/Server/fme/libcrypto_fme.dll'),
'libcurl_fme.dll': Path('FMEFlow/Server/fme/libcurl_fme.dll'),
'libfmeuicore.dll': Path('FMEFlow/Server/fme/libfmeuicore.dll'),
'libfmeuihub.dll': Path('FMEFlow/Server/fme/libfmeuihub.dll'),
'libfmeuiparamdef.dll': Path('FMEFlow/Server/fme/libfmeuiparamdef.dll'),
'libfmeuiparammodel.dll': Path('FMEFlow/Server/fme/libfmeuiparammodel.dll'),
'libfmeuiserver.dll': Path('FMEFlow/Server/fme/libfmeuiserver.dll'),
'libfmeuixformdef.dll': Path('FMEFlow/Server/fme/libfmeuixformdef.dll'),
'liblz4.dll': Path('FMEFlow/Server/fme/liblz4.dll'),
'liblzma_fme.dll': Path('FMEFlow/Server/fme/liblzma_fme.dll'),
'libssh2_fme.dll': Path('FMEFlow/Server/fme/libssh2_fme.dll'),
'libssl_fme.dll': Path('FMEFlow/Server/fme/libssl_fme.dll'),
'libxml2_fme.dll': Path('FMEFlow/Server/fme/libxml2_fme.dll'),
'mpfr_fme.dll': Path('FMEFlow/Server/fme/mpfr_fme.dll'),
'mpir_fme.dll': Path('FMEFlow/Server/fme/mpir_fme.dll'),
'nghttp2_fme.dll': Path('FMEFlow/Server/fme/nghttp2_fme.dll'),
'ogcwktutil_fme.dll': Path('FMEFlow/Server/fme/ogcwktutil_fme.dll'),
'ole32.dll': None,
'pluginutils.dll': Path('FMEFlow/Server/fme/pluginutils.dll'),
'proj_fme.dll': Path('FMEFlow/Server/fme/proj_fme.dll'),
'projutil_fme.dll': Path('FMEFlow/Server/fme/projutil_fme.dll'),
'python311.dll': Path('FMEFlow/Server/fme/fmepython311/python311.dll'),
'rptree.dll': Path('FMEFlow/Server/fme/rptree.dll'),
'sqlite3_fme.dll': Path('FMEFlow/Server/fme/sqlite3_fme.dll'),
'stk.dll': Path('FMEFlow/Server/fme/stk.dll'),
'tbb.dll': Path('FMEFlow/Server/fme/tbb.dll'),
'tbbmalloc.dll': Path('FMEFlow/Server/fme/tbbmalloc.dll'),
'tcl_fme.dll': Path('FMEFlow/Server/fme/tcl_fme.dll'),
'threadlockmgr.dll': Path('FMEFlow/Server/fme/threadlockmgr.dll'),
'tiff_fme.dll': Path('FMEFlow/Server/fme/tiff_fme.dll'),
'unrar_fme.dll': Path('FMEFlow/Server/fme/unrar_fme.dll'),
'xerces-c_3_2_fme.dll': Path('FMEFlow/Server/fme/xerces-c_3_2_fme.dll'),
'zlib_fme.dll': Path('FMEFlow/Server/fme/zlib_fme.dll')}
And a few extra DLLs that are optionally loaded and might needed for all the features to be available.
'DWrite.dll': None,
'GDI32.dll': None,
'GTransTF.dll': None,
'NCSEcw.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/NCSEcw.dll'),
'OCI.dll': None,
'WININET.dll': None,
'dbgeng.dll': None,
'dedit.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/dedit.dll'),
'fmepythonpip.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/fmepythonpip.dll'),
'imagehlp.dll': None,
'meshsimplificationutil.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/meshsimplificationutil.dll'),
'opennurbs_fme.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/opennurbs_fme.dll'),
'pcache.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/pcache.dll'),
'solidvalidation.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/solidvalidation.dll'),
'straightskeleton.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/straightskeleton.dll'),
'trianglesurface.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/trianglesurface.dll'),
'trnapi.dll': WindowsPath('D:/Program Files/FMEFlow/Server/fme/trnapi.dll')
The DLLs with "None" written next to them are not in the FME Python search path, so they're expected to be in the Windows system path (most of these are standard Windows API DLLs, but some like OCI.dll are third-party).
At any rate, that's a lot of DLLs to carry around by hand just to get the basic features working, and I was certainly hoping there wouldn't be that many.
That's an impressive list, although I'm not very surprised. I have the impression that fmeobjects is just a relatively thin interface on top of the C++ libraries that make up the fundamental FME engine.
For what it's worth, here are my own current guidelines for developing Python with FME when the script is non-trivial:
- Develop the entire functionality in a proper IDE, e.g. Pycharm, without any references to fmeobjects whatsoever. Generalize everything into proper first-order Python classes without relying on the FME feature/object model. Write unit tests.
- Try to stick with the standard library as much as possible, or if necessary, only install minimalist packages for specific functionality. Try very hard to avoid monoliths such as Anaconda, Geopandas, etc. as they can be tricky to make work with FME due to dependencies.
- In your workspace (e.g. PythonCaller) import the above mentioned libraries and write the least amount of lines possible, only what's necessary to instantiate and interact with the business objects using the incoming feature.
The idea is to fully separate FME and the business logic code, making it easy to debug, extend, refactor and automate testing of the business objects without relying on FME.
That's an impressive list, although I'm not very surprised. I have the impression that fmeobjects is just a relatively thin interface on top of the C++ libraries that make up the fundamental FME engine.
For what it's worth, here are my own current guidelines for developing Python with FME when the script is non-trivial:
- Develop the entire functionality in a proper IDE, e.g. Pycharm, without any references to fmeobjects whatsoever. Generalize everything into proper first-order Python classes without relying on the FME feature/object model. Write unit tests.
- Try to stick with the standard library as much as possible, or if necessary, only install minimalist packages for specific functionality. Try very hard to avoid monoliths such as Anaconda, Geopandas, etc. as they can be tricky to make work with FME due to dependencies.
- In your workspace (e.g. PythonCaller) import the above mentioned libraries and write the least amount of lines possible, only what's necessary to instantiate and interact with the business objects using the incoming feature.
The idea is to fully separate FME and the business logic code, making it easy to debug, extend, refactor and automate testing of the business objects without relying on FME.
Developping functionnalities as importable, unit-tested, FME-independent modules is an interesting practice that I hadn't considered before and might be tempted to use for transformers that handle saner amounts of data (see my previous question regarding dataframes, since I'm currently handling too much data for pure Python to handle this with a reasonnable performance level). I like that it encourages making your transformer code portable. Still, I can't currently avoid pulling dependencies, whether FME-vendored ones or user-installed ones, and at the very least being able to use the same environment that FME would have access to at runtime would be very beneficial.
So I spent some time yesterday trying to do just that.
***
As it turns out, given the way FME loads Python interpreters, whether built-in or custom, FME can't make use of the environment constructed below and will fail to include the modules from either site-packages in its module search path, because the virtual environment is activated by python.exe, not python3.dll. This would only be useful for developping PythonCaller and PythonCreator (and possibly scripted user parameters) scripts outside of FME Workbench. The user-installed dependencies would still need to be installed in the native locations for both of these in order to run user scripts there.
Nevertheless, it turns out that making a standalone Python install that's compatible with the embedded version is possible and the result works raher well, so for good measure, here's how you can create one yourself for local developpment.
- Install Python 3.11 (preferably without pip) in a system location. Uncheck adding it to PATH and the environment, as well as py launcher. The locatino where you've installed is one I'm going to call `MYPYTHON_FME_RO` from now on.
- Copy everything from `FME2023/python` (minus the directories that start with "python3"), `FME2023/python/python311` and `FME2023\fmeobjects\python311` to `MYPYTHON_FME_RO/Lib/site-packages`. There shouldn't be anything in site-packages unless you asked the installer to install Pip, but if there's something in there, you can safely remove it. If you're copying the files from FME Server, these are instead located in `FMEFlow\Server\fme`.
- To make sure the DLLs from FME are on the DLL search path, add a `MYPYTHON_FME_RO/Lib/site-packages/sitecustomize.py` file that contains `import os;os.add_dll_directory(r"D:\Program Files\FME2023")` (replace that path with whatever FME Flow/FME Form directory you've been using copying things from so far, the one that contains fme.exe and all the DLLs I mentionned in my previous reply)
- A good idea is to also add a `MYPYTHON_FME_RO/Lib/EXTERNALLY-MANAGED` file (not in `site-packages`, this goes in `Lib` directly) to your new Python environment to prevent Pip from adding new packages to it. The contents should look like this:
; This is an INI file following the PEP 668 specification. It's pretty new.
pexternally-managed]
Error=This is a copy of the FME Python environment, please do not alter it.
- `MYPYTHON_FME_RO` is intended to be a read-only copy of the FME libs, so we're not going to be touching anything in there anymore. To avoid interfences, the user-installed libraries will be installed in a separate, virtual environment. Simply run `MYPYTHON_FME_RO\python.exe -mvenv --system-site-packages "MYPYTHON_RW"`, with `MYPYTHON_RW` being a new, distinct path from `MYPYTHON_FME_RO`.
- When your IDE of choice asks you what interpreter you would like to use, specify `MYPYTHON_RW/Scripts/python.exe`.
The DLL list I posted yesterday establishes that every non-system DLL needed by fmeobjects is in FMEFlow\Server\fme, so adding that to the DLL import path using sitecustomize instead of moving these by hand definitely seems like the most manageable option.
You can install any library you want in the `MYPYTHON_RW` virtual environment, and pip will resolve dependencies in accordance with the library versions supplied by FME. I wouldn't reccomend installing any upgraded versions of the included libraries, though, because while Pip will raise dependency warnings if such an upgrade were to break the requirements of another FME-supplied lib that's not being upgraded, it will still install the upgrade and break the other libraries. That is because Python will load the modules from `MYPYTHON_RW` in priority, including the ones being imported by modules located in `MYPYTHON_FME_RO` that expect to see the versions that are also in `MYPYTHON_FME_RO`.
Most crucially, though, is that you can run `import fmeobjects` from within this environment and it won't raise any errors. Pycharm and VSCode will both recognize and work with everything in this environment out of the box (something I haven't managed to do with Pycharm, despite the tutorial), with no additionnal configuration needed.
Disregard everything I said above regarding copying libraries and search path shenanigans and just use fme-packager config-env in any fresh Python venv to provide access to FME-specific libraries, including fmeobjects.
It took me that long to stumble upon it, but it’s official and a lot cleaner and easier to setup than what I was up to last year. It’s obviously not mentionned anywhere that’s not related to FME Transformer Designer, and has never been talked about once in the community forums, but it seems to be the one correct solution to this issue.
