Solved

AzureBlobStorageConnector - Downloads files with 0 KB sometimes

  • 10 March 2021
  • 2 replies
  • 26 views

Have two automations happening where one downloads files off an Azure Blob storage. The other is actively listening to the folder where the first automation downloads files too to trigger a workspace. My problem is that for some reason AzureBlobStorageConnector sometimes downloads files as 0kb. Is that cause it's trying to download files that haven't fully uploaded to that Azure Blob Storage? I.e the same thing that happens when you download something off the internet and it goes to the download folder as a temp file - until it downloads fully.

 

Cheers

Dan

icon

Best answer by jlutherthomas 11 March 2021, 19:22

View original

2 replies

Badge +2

Hi @danielyen​ 

 

I think what you've suggested sounds plausible. It sounds like you have 2 separate automations, so there is a chance the second automation is starting as soon as the file arrives in Azure Blob Storage, before it's fully uploaded.

Are you able to combine the two automations into one, so that the AzureBlobStorageConnector only starts once it gets a feature from the first automation (that signals the upload has completed)?

Alternatively you could add a decellerator transformer before the AzureBlobStorageConnector (ABSC) or maybe use the List functionality of the ABSC to check the file size is about right before downloading.

 

Hi @danielyen​ 

 

I think what you've suggested sounds plausible. It sounds like you have 2 separate automations, so there is a chance the second automation is starting as soon as the file arrives in Azure Blob Storage, before it's fully uploaded.

Are you able to combine the two automations into one, so that the AzureBlobStorageConnector only starts once it gets a feature from the first automation (that signals the upload has completed)?

Alternatively you could add a decellerator transformer before the AzureBlobStorageConnector (ABSC) or maybe use the List functionality of the ABSC to check the file size is about right before downloading.

 

Hi @jlutherthomas Thanks for the reply. ​ 

 

The ABSC is setup in my workspace to not overwrite files. It looks like when I delete the 0KB file manually, it re-downloads as the complete file so perhaps there is a timing issue.

Anyway I've gone ahead with your second suggestion by adding more steps in to check for file size using the ABSC connector twice, one to list and add in some attributes like file size etc, used a filter and then used the _path given by the first ABSC to use as a path for the second ABSC. It seems to work fine in my testing at the moment.

 

Cheers for the help!

Dan

Reply