Skip to main content
Question

How do I use a CSV list to move files between S3 buckets?

T
Forum|alt.badge.img

Hi everyone, I hope my question will be clear to understand.

I have a workspace that generates a single list CSV of photos from multiple tables in a Postgres database, each with a UUID to identify it in the bucket. It looks like this:

 

This list is output to the (FME_SHAREDRESOURCE_DATA) directory on FME Server, which I then want to call it from and use to parse the photos from the S3 bucket and move them to another S3 bucket. I need to use the Cloud/Server resources as the total size of these photos can reach multiple GB’s and would cause too much latency on my office WAN.

How can I achieve this using the S3 transformers offered with FME to move objects between S3 buckets without downloading the photos locally? Or is this even possible in a single or multiple workspace(s)?

I would also need to perform this on video files as well.

Thanks in advance for the assistance.

Todd

4 replies

redgeographics
Celebrity
Forum|alt.badge.img+49

The S3 transformers that come with FME can be used, specifically the S3Downloader and S3Uploader but that'll make the file pass through your server, so data transfer in and out.

Alternatively, the Amazon S3 REST API offers tools to copy objects between buckets, presumably this does not involve the double data transfer.

T
Forum|alt.badge.img
  • tjpollard
  • Author
  • July 23, 2018
redgeographics wrote:

The S3 transformers that come with FME can be used, specifically the S3Downloader and S3Uploader but that'll make the file pass through your server, so data transfer in and out.

Alternatively, the Amazon S3 REST API offers tools to copy objects between buckets, presumably this does not involve the double data transfer.

Thank you @redgeographics for the quick response.

 

 

To follow up for a bit more clarification, if I use the S3 transformers and run the workspace on my FME server instance hosted by FME Cloud, I would guess I'm limited by the server resource allocations to the engines and memory, which I'm somewhat familiar with from review of various log files.

 

 

The other question is about the Amazon S3 REST API, is there a way to use that in a workspace?

 

 

G
Forum|alt.badge.img

@tjpollard

you are correct. If you use FME Cloud and the S3 transformers you need to make sure the FME Cloud instance has enough disk space to download those files before they are written to the 2nd S3 bucket. As @redgeographics already mentioned correctly, uploading the files to the 2nd bucket will be outbound data transfer which will be charged (0.13$ per GB, inbound data is free of charge).

 

If you choose this approach I would recommend using the Temporary Disk of the FME Cloud instance to make sure you can decrease the size of the disk again once you are done. Here is some info about the temp disk.

Here is a simple example.

In case you look into using the AWS S3 REST API you should check out the HTTPCaller. This is a REST Client that will allow you to interact with the REST API. It looks like you can not use the AWS Web Connections for authentication so you would need to look into Authenticating Requests and set up the correct Authorization Header for the REST requests.

I am not too familiar with the AWS S3 REST API, so I am not sure how hard it is to set up the correct calls for your transformation.

I hope this helps

T
Forum|alt.badge.img
  • tjpollard
  • Author
  • July 24, 2018
gerhardatsafe wrote:

@tjpollard

you are correct. If you use FME Cloud and the S3 transformers you need to make sure the FME Cloud instance has enough disk space to download those files before they are written to the 2nd S3 bucket. As @redgeographics already mentioned correctly, uploading the files to the 2nd bucket will be outbound data transfer which will be charged (0.13$ per GB, inbound data is free of charge).

 

If you choose this approach I would recommend using the Temporary Disk of the FME Cloud instance to make sure you can decrease the size of the disk again once you are done. Here is some info about the temp disk.

Here is a simple example.

In case you look into using the AWS S3 REST API you should check out the HTTPCaller. This is a REST Client that will allow you to interact with the REST API. It looks like you can not use the AWS Web Connections for authentication so you would need to look into Authenticating Requests and set up the correct Authorization Header for the REST requests.

I am not too familiar with the AWS S3 REST API, so I am not sure how hard it is to set up the correct calls for your transformation.

I hope this helps

@GerhardAtSafe, thanks for this information. I think I'm going to go the temp disk route and just increase the size of that disk to run these jobs at least, since our Cloud instance needed an upgrade anyway. But the other option of using the HTTPCaller and is one I will probably have to apply soon. Thanks again. Todd

 

Reply


Helpful Members This Week

Community Stats

31,139
Posts
118,062
Replies
38,758
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.