Question

Creating new CSV files to upload to Azure Blob Storage, via FME Server

  • 28 April 2020
  • 4 replies
  • 67 views
C
Badge

Hello,

I was hoping to be able to create an FME workspace to be uploaded to FME server, that would run an SQL script (via SQL Creator) and create a CSV file that would then be uploaded directly to Azure Blob storage.

I tried the 'AzureBlobStorageConnector' (using the 'File' upload option), but it appears that the file needs to be stored locally to use this option, which seems to be no good if I want to publish and automate this script via server. I also tried 'FeatureWriter', but couldn't see an option to write to a web service.

Any help would be greatly appreciated!

Thanks

Chris

4 replies

nielsgerrits
Rank
Userlevel 6
Badge +32

I did some work with Azure Blob Storage and I think you can't create files there like on a drive, you need to upload them, using the AzureBlobStorageConnector. This is not an FME limitation but how Blob storage works.

And it can work well. If you use a TempPathnameCreator, you can create a file, with a FeatureWriter to a temp directory, upload it using the Connector and it will get cleaned up when the workspace is run.

Attached 2019 sample workspace template. FeatureReader is just to show you can dynamically read the file you wrote to a temp directory. Replace the FeatureReader with the AzureBlobStorageConnector and you're good to go.

csv2azureblob.fmwt

C
Badge

I did some work with Azure Blob Storage and I think you can't create files there like on a drive, you need to upload them, using the AzureBlobStorageConnector. This is not an FME limitation but how Blob storage works.

And it can work well. If you use a TempPathnameCreator, you can create a file, with a FeatureWriter to a temp directory, upload it using the Connector and it will get cleaned up when the workspace is run.

Attached 2019 sample workspace template. FeatureReader is just to show you can dynamically read the file you wrote to a temp directory. Replace the FeatureReader with the AzureBlobStorageConnector and you're good to go.

csv2azureblob.fmwt

Thanks, the principal makes sense, however when I set the 'File to Upload' path in the AzureBlobStorageConnector using the same filepath up applied in the 'FeatureReader', it gets rejected. The 'fme_rejection_code' is 'INVALID_FILE' and 'fme_rejection_message' is 'Source file does not exist'. Any ideas how I can overcome this?

C
Badge

Thanks, the principal makes sense, however when I set the 'File to Upload' path in the AzureBlobStorageConnector using the same filepath up applied in the 'FeatureReader', it gets rejected. The 'fme_rejection_code' is 'INVALID_FILE' and 'fme_rejection_message' is 'Source file does not exist'. Any ideas how I can overcome this?

I managed to figure it out, and appears to be just a minor thing; I changed the 'File to Upload' path to , and it now seems to be working fine. Thank you so much for your help!

nielsgerrits
Rank
Userlevel 6
Badge +32

I managed to figure it out, and appears to be just a minor thing; I changed the 'File to Upload' path to , and it now seems to be working fine. Thank you so much for your help!

Great, thanks for your feedback :-)

Reply


Community Stats

28,707
Posts
109,479
Replies
36,552
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.