I have a bunch of large files to SFTP into the fme server. In order to not deplete the space on the local disk, I am moving them to S3 after they complete. How can I ensure the files are complete before using a directory watcher to move the files to s3? I am running my own fme server, not cloud version.
Question
Large dataset - many files. FTP to S3 bucket
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+25- Supporter
I think this depends on the ftpserver used. Some servers dont show the file until the complete file is uploaded, but some show incremental steps and the filesize increasing.
To be sure you can do a check using the
- Directory and file path reader
- Check the filesize (or date)
- Use a decelerator to wait 30 seconds
- Compare the filesize (or date) - if the filesize is similar, the file is completely uploaded, if not - then wait another 30 seconds.
+16- Contributor
@sigtill 's approach is nice and simple but it will tie up an FME Server Engine for as long as it takes the file to upload. If that's going to be an issue, then you could do something along the lines of what's discussed in this article:
https://knowledge.safe.com/articles/52791/directory-watch-publisher-with-idle-time-delay-adv.html
A variation on the above article and to sigtill's approach that I've used in the past is to use the FME Server REST API to dynamically create an FME Server schedule, instead of using the Decelerator. The schedule would simply call the same workspace again after a certain amount of time to check if the filesize has changed. The calling workspace is shutdown in between scheduled calls. Just don't forget to delete the schedule (using the REST API) after the final job is complete!