I'm finding that when certain events do not trigger wither a MODIFY or a CREATE trigger (FME Server 2020)
Does anyone know the specifics of how the s3 watch works?
s3 Watch - CREATE and MODIFY - how does FME Server know when something is new or modified?
Userlevel 1
+6
- Safer
- 148 replies
-
8 August 2022
Hi @virtualcitymatt ,
When an S3 Watch publication is started, we first fetch all of the objects in the S3 bucket to keep track of the initial state of the bucket. The AmazonS3.listObjects() API returns a maximum of 1000 objects so if a bucket has more than 1000 objects, then several requests will need to be made to get all the objects.
Events trigger based on changes to the entity Tag (ETag), which is a representation of file contents. No message is triggered if a change to an existing file does not update the ETag. If a file is overwritten and the ETag value changes, a MODIFY event is triggered (from Amazon S3 Bucket Triggers documentation).
Hope this helps!
-Kezia
Userlevel 4
+26
- Influencer
- 1640 replies
-
9 August 2022
Hi @virtualcitymatt ,
When an S3 Watch publication is started, we first fetch all of the objects in the S3 bucket to keep track of the initial state of the bucket. The AmazonS3.listObjects() API returns a maximum of 1000 objects so if a bucket has more than 1000 objects, then several requests will need to be made to get all the objects.
Events trigger based on changes to the entity Tag (ETag), which is a representation of file contents. No message is triggered if a change to an existing file does not update the ETag. If a file is overwritten and the ETag value changes, a MODIFY event is triggered (from Amazon S3 Bucket Triggers documentation).
Hope this helps!
-Kezia
OK great, thanks for this explanation - I thiiink what may have happened is that a few files with 0KB size were present in the folder when the service started. These files were deleted and later reuploaded (still with 0KB). I assume that the ETag was the same and so the polling did not pick up a change? could that be? The other possibility was that an ETag was never created.
Seems to be working now as expected
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.