I have an FMW that writes a CSV file and then uploads that CSV to Azure BLOB.
The CSV that's created on the server's disk is okay. It has 201 records as expected and opens up just fine.
That CSV--after FME uploads it to Azure BLOB via AzureBlobStorageConnector has about 25 less records--seemingly randomly removed. To rule out Azure BLOB itself as the problem, I used Microsoft Azure Storage Explorer to upload the CSV--when I do that, all records are in the CSV file when downloaded back down from Azure.
I have experimented w/ a folder upload (upload the file w/ a containing folder) vs. a file upload--no luck w/ that, same result.
What really bothers me about this is how it makes no sense. The CSV that gets uploaded should be the CSV that's on disk. It's a file and file-upload process. What could possibly be removing certain lines/records from the file?
Question
When Uploading CSV to Azure BLOB via AzureBlobStorageConnector, Some Records Missing in BLOB
+5- Contributor
+5- Contributor
I figured it out. It was a timing issue.
I was trying to use a Connection Run Time Order to run a CSV writer and then run AzureBlobStorageConnector (branched). Apparently, the CSV writer was still writing when the AzureBlobStorageConnector started.
I switched from using a CSV writer to a FeatureWriter. The FeatureWriter can write to CSV. And, because FeatureWriter is a transformer, the ETL can continue on from its output--no need to branch out and be concerned about order.
1 person upvoted this
Reply
Rich Text Editor, editor1
Editor toolbars
Press ALT 0 for help
Helpful Members This Week
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.