• EN
Solved

FME Blob Storage in FME 2022.2

  • 29 February 2024
  • 4 replies
  • 118 views
osalex
Rank
Badge +4

Hi Community,

I have a workspace with some python in to create a container within a blob storage account (something the blob storage connector was not able to do). The problem i’m having is that as soon as i use the blob storage connector in my workspace the package is installed in C:\Users\%username%\AppData\Roaming\Safe Software\FME. This then seems to force FME to use this directory in my python caller. The problem is i’m using newer python packages where BlockBlobService no longer exists and is replaced by BlobServiceClient so they stop working despite being installed as per the instructions for installing python libraries. Does anyone have any suggestions?

 

icon

Best answer by debbiatsafe 7 March 2024, 02:17

View original

4 replies

debbiatsafe
Rank
Userlevel 3
Badge +17

Hello @osalex

You can try removing the AzureBlobStorage package directory from the PATH environment variable before importing BlobServiceClient.

If you are planning to use the AzureBlobStorageConnector transformer in the same workspace as your script creating the container, you should add the package directory back to PATH so the transformer runs as expected.

import fme
import fmeobjects
import sys, os

buildstring = fmeobjects.FME_BUILD_STRING[17:].lower().replace(" ","")
pkgPath = fr"{os.getenv('APPDATA')}\Safe Software\FME\Packages\{buildstring}\python\safe.azurestorage"
sys.path.remove(pkgPath)

from azure.storage.blob import BlobServiceClient

<your Script here>

# Add package path back to PATH
sys.path.insert(0,pkgPath)

 

Our development team also notes they are working on a rewrite for the Azure Storage package in which they are planning to make the package’s copy of Azure libraries private so it won't conflict with others. Once this package is rewritten, you should not need to remove the package directory from PATH. I hope this information helps.

danilo_fme
Rank
Userlevel 5
Badge +30

Hello @osalex

You can try removing the AzureBlobStorage package directory from the PATH environment variable before importing BlobServiceClient.

If you are planning to use the AzureBlobStorageConnector transformer in the same workspace as your script creating the container, you should add the package directory back to PATH so the transformer runs as expected.

import fme
import fmeobjects
import sys, os

buildstring = fmeobjects.FME_BUILD_STRING[17:].lower().replace(" ","")
pkgPath = fr"{os.getenv('APPDATA')}\Safe Software\FME\Packages\{buildstring}\python\safe.azurestorage"
sys.path.remove(pkgPath)

from azure.storage.blob import BlobServiceClient

<your Script here>

# Add package path back to PATH
sys.path.insert(0,pkgPath)

 

Our development team also notes they are working on a rewrite for the Azure Storage package in which they are planning to make the package’s copy of Azure libraries private so it won't conflict with others. Once this package is rewritten, you should not need to remove the package directory from PATH. I hope this information helps.

Greaaat!

rishi1804
Rank
Badge +4

Hi @danilo_fme I tried using your approach but the issue is once you append delete the path from sys.path and then again insert it back at the same position doesn’t work if there is AzureBlobStorageConnector  transformer in the same workbench. I don’t know why but FME tries to import BlockBlobService then. I printed the updated sys.path, the path was added correctly but don’t know it didn’t try to import from that location again. If I run the workbench in one go it fails but if I run step by step means one transformer after another then it works without any issue

debbiatsafe
Rank
Userlevel 3
Badge +17

Hello @rishi1804 

FME shares one Python interpreter in a workspace. Once you have changed sys.path and the newer Python library is imported, the reference to the non-FME package location remains in sys.modules even if you re-add the removed package path. You can either remove modules loaded with del sys.modules[“<moduleName>”]. Since Azure is a complex library with many individual modules ~300), it is best to do this programmatically.

import fme
import fmeobjects
import sys, os

buildstring = fmeobjects.FME_BUILD_STRING[17:].lower().replace(" ","")
pkgPath = fr"{os.getenv('APPDATA')}\Safe Software\FME\Packages\{buildstring}\python\safe.azurestorage"
sys.path.remove(pkgPath)

from azure.storage.blob import BlobServiceClient

class FeatureProcessor(object):
    def __init__(self):
        pass

    def has_support_for(self, support_type: int):
        return support_type == fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM

    def input(self, feature: fmeobjects.FMEFeature):
        self.pyoutput(feature)

    def close(self):
        pass

    def process_group(self):
        pass

#Find all modules starting with 'azure' and delete
az = [k for k in sys.modules if k.startswith('azure')]
for m in az:
    del sys.modules[m]        

sys.path.insert(0,pkgPath)

Alternatively, you can separate the portion that creates the container and the AzureBlobStorageConnector into separate workspaces and chain them together with a WorkspaceRunner. This should work since the changes to sys.path made in the workspace creating the container does not affect the workspace running the AzureBlobStorageConnector.

Partial runs of the workspace succeed because the Python interpreter is restarted for each run of the workspace.

 

Reply


Community Stats

28,833
Posts
109,885
Replies
36,639
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.