Solved

i would like to connect FME Desktop with Azure Data lake storage gen2

  • 7 April 2022
  • 5 replies
  • 76 views
gogopotter90
Rank
Badge +13

 

i would like to connect FME Desktop with Azure Data lake storage gen2,to let FME read the data from there .

 

my first question : will be possible by using only FME Desktop ?

i have read some articles that they are using FME Server ,i would like to know also , when we have decision to use FME Desktop or use FME Server ?

i have currently FME Desktop 2020

 

my second question : i have found this transformer : Azure Storage on FME hub

https://hub.safe.com/publishers/safe/packages/azurestorage

so this transformer where should i start my workspace , which steps should i follow to connect FME desktop with azure storage gen.2 ?

 

Thanks in advance .

FME Desktop 2020.2

icon

Best answer by keziaatsafe 14 April 2022, 18:11

View original

5 replies

keziaatsafe
Rank
Userlevel 1
Badge +6

Hi @gogopotter90​ ,

 

It is possible to only use FME Desktop to connect with Azure Storage. Using FME Server allows for more capabilities such as Azure Event Grid as a trigger and an external action in FME Server Automations, which allows you to integrate your Azure environment with complex automated workflows. Please see Connecting FME Server to Azure File Share and How to Connect to Microsoft Azure in FME for more details.

 

To get started with the Azure Storage package on FME hub, this Community article would be good to follow. As mentioned in the Read Me section of the Azure Storage package, you will first need to set up an Azure Storage account, then authorize access to your account.

 

Hope this helps!

 

gogopotter90
Rank
Badge +13

Hi @gogopotter90​ ,

 

It is possible to only use FME Desktop to connect with Azure Storage. Using FME Server allows for more capabilities such as Azure Event Grid as a trigger and an external action in FME Server Automations, which allows you to integrate your Azure environment with complex automated workflows. Please see Connecting FME Server to Azure File Share and How to Connect to Microsoft Azure in FME for more details.

 

To get started with the Azure Storage package on FME hub, this Community article would be good to follow. As mentioned in the Read Me section of the Azure Storage package, you will first need to set up an Azure Storage account, then authorize access to your account.

 

Hope this helps!

 

thanks for reply ,i have read articles and i have found that AzureBlobStorageConnector transformer ,it will be useful for my case .

but i would like to sure , if there are CSV files format in Data Lake Storage Gen2 ,so i can connect with FME Desktop to read this data ,right ?

and which format will you prefer to read from Data Lake Storage Gen2 ? they could offer different formats .

i have to design workflow to read the newest file there between lots of files ,wil be possible to create it with FME Desktop ?

keziaatsafe
Rank
Userlevel 1
Badge +6

thanks for reply ,i have read articles and i have found that AzureBlobStorageConnector transformer ,it will be useful for my case .

but i would like to sure , if there are CSV files format in Data Lake Storage Gen2 ,so i can connect with FME Desktop to read this data ,right ?

and which format will you prefer to read from Data Lake Storage Gen2 ? they could offer different formats .

i have to design workflow to read the newest file there between lots of files ,wil be possible to create it with FME Desktop ?

Hi @gogopotter90​ ,

 

There shouldn't be a problem with the CSV files format. FME’s AzureBlobStorageConnector allows you to access an Azure Data Lake Storage Gen2 directory to upload, download, delete files and folders, or list file/folder information with FME Desktop. Once you download the file using the AzureBlobStorageConnector, you can read it into your workspace using the FeatureReader transformer. I'm not too familiar with Data Lake Storage Gen2 so I can't give you my format preference, but you should be able to go with the format you like! This Blog may be helpful to check out too.

 

Yes, you should be able to work in some logic when designing your workflow to read in the newest file after it is downloaded. If there is a date attribute that may be easier.

S
Badge

Hello Friendly FME Community,

I have a question regarding connection to Azure Data Lake gen2 as well. Is it possible to connect to ADL gen2 using FME Desktop (my version is 2020.2) using service principle?

I couldn't find any complete tutorial on this.

I was thinking of some tricky way through API calls if that's possible. Do you guys have any experience with such connection and if yes could you please advise the direction?

Thank you in advance!

Regards!

gogopotter90
Rank
Badge +13

Hello Friendly FME Community,

I have a question regarding connection to Azure Data Lake gen2 as well. Is it possible to connect to ADL gen2 using FME Desktop (my version is 2020.2) using service principle?

I couldn't find any complete tutorial on this.

I was thinking of some tricky way through API calls if that's possible. Do you guys have any experience with such connection and if yes could you please advise the direction?

Thank you in advance!

Regards!

@siriosus it is possible .i did it before but I left the company since few months .so It will be hard to remember how it works .Could u please , create new question and sure u will find who could help u .

i am sorry , if I am not able to remember

Reply


Community Stats

28,706
Posts
109,463
Replies
36,550
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.