Skip to main content
  • EN

Hi, is there a way to reference an s3 bucket using an fme server parameter so that we can read or write directly to/from the bucket? I know I can use the S3 uploader, but it would be really handy to be able to use the feature writer instead.

An S3 bucket isn't a format, it's just a file container, much like a directory on your PC, so you'll have to first write your data locally in the desired format either using a regular writer or a FeatureWriter before using the S3Uploader. The TempPathnameCreator can be helpful for creating such temporary files since the files created using this transformer automatically gets deleted when FME terminates.

You can user a regular TEXT type published parameter to specify the S3 bucket name used by the S3Uploader.

An S3 bucket isn't a format, it's just a file container, much like a directory on your PC, so you'll have to first write your data locally in the desired format either using a regular writer or a FeatureWriter before using the S3Uploader. The TempPathnameCreator can be helpful for creating such temporary files since the files created using this transformer automatically gets deleted when FME terminates.

You can user a regular TEXT type published parameter to specify the S3 bucket name used by the S3Uploader.

So here's my scenario. I have a file, called template.xlsx located in an s3 bucket. In order to update that file using FME cloud, I need to create the file inside an FME resource and reference it like $(FME_SHAREDRESOURCE_DATA)/template.xlsx, then use the s3 uploader to copy it to my s3 bucket. What I'm wondering, is there a way to create an FME parameter which would reference my s3 bucket which I already have connected as a resource in FME Cloud? Then I could simply use that parameter as the file location in any fme reader/writer.

 

thanks.

 

 

Hi. Unfortunately, at this point, AWS S3 buckets mounted as a Resource cannot be accessed from workspaces run from FME Server, unless that access is through an S3-related transformer, such as S3Uploader, S3Downloader, S3Deleter, or S3ObjectLister. The S3 resources mount is really just to enable backup workflows. There is an open request to enable access to the S3 resource mounts from within FME Workbench which I will add you to.

Hi. Unfortunately, at this point, AWS S3 buckets mounted as a Resource cannot be accessed from workspaces run from FME Server, unless that access is through an S3-related transformer, such as S3Uploader, S3Downloader, S3Deleter, or S3ObjectLister. The S3 resources mount is really just to enable backup workflows. There is an open request to enable access to the S3 resource mounts from within FME Workbench which I will add you to.

Thank-you!

 

 

Hi. Unfortunately, at this point, AWS S3 buckets mounted as a Resource cannot be accessed from workspaces run from FME Server, unless that access is through an S3-related transformer, such as S3Uploader, S3Downloader, S3Deleter, or S3ObjectLister. The S3 resources mount is really just to enable backup workflows. There is an open request to enable access to the S3 resource mounts from within FME Workbench which I will add you to.

Also feel free to up vote this idea:

 

https://knowledge.safe.com/idea/22137/ability-to-mount-s3-bucket-as-a-drive-from-within.html

 

 

Hi. Unfortunately, at this point, AWS S3 buckets mounted as a Resource cannot be accessed from workspaces run from FME Server, unless that access is through an S3-related transformer, such as S3Uploader, S3Downloader, S3Deleter, or S3ObjectLister. The S3 resources mount is really just to enable backup workflows. There is an open request to enable access to the S3 resource mounts from within FME Workbench which I will add you to.

Get the votes up people....we need this

 

 

@johnglick - If the file is already in s3 use the s3 downloader, put it in temp resources location - use a feature reader, manipulate and upload using s3 uploader.

you can't manipulate the file while it is in s3

Reply


Community Stats

30,366
Posts
115,373
Replies
38,143
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.