Hi @danielyen
I think what you've suggested sounds plausible. It sounds like you have 2 separate automations, so there is a chance the second automation is starting as soon as the file arrives in Azure Blob Storage, before it's fully uploaded.
Are you able to combine the two automations into one, so that the AzureBlobStorageConnector only starts once it gets a feature from the first automation (that signals the upload has completed)?
Alternatively you could add a decellerator transformer before the AzureBlobStorageConnector (ABSC) or maybe use the List functionality of the ABSC to check the file size is about right before downloading.
Hi @danielyen
I think what you've suggested sounds plausible. It sounds like you have 2 separate automations, so there is a chance the second automation is starting as soon as the file arrives in Azure Blob Storage, before it's fully uploaded.
Are you able to combine the two automations into one, so that the AzureBlobStorageConnector only starts once it gets a feature from the first automation (that signals the upload has completed)?
Alternatively you could add a decellerator transformer before the AzureBlobStorageConnector (ABSC) or maybe use the List functionality of the ABSC to check the file size is about right before downloading.
Hi @jlutherthomas Thanks for the reply.
The ABSC is setup in my workspace to not overwrite files. It looks like when I delete the 0KB file manually, it re-downloads as the complete file so perhaps there is a timing issue.
Anyway I've gone ahead with your second suggestion by adding more steps in to check for file size using the ABSC connector twice, one to list and add in some attributes like file size etc, used a filter and then used the _path given by the first ABSC to use as a path for the second ABSC. It seems to work fine in my testing at the moment.
Cheers for the help!
Dan
