Question

AWS Region Detection in S3Uploader

K

Hi all,

We're attempting to use S3Uploader in FME 2015.1.1 and FME 2016.1 to upload data to a bucket in ap-southeast-2 (Sydney), but we're receiving an error from AWS that seems to indicate that FME is not uploading to ap-southeast-2.

“The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect;”

Googling this error results in many bug reports from other software using AWS where they have forgotten to specify an AWS region, and it's defaulted to a different region than where their bucket resides.

According to FME's documentation here, our AWS region should be automatically detected.

"As of build 2015.1.1, all AWS transformers support multiple regions. You don't have to do anything we automatically detect the region."

We've been unable to find any further information about how the automatic detection works, or how we're able to override it.

Is anyone aware of any workarounds for providing FME's AWS transformers with a hard-coded region?

Cheers,

Keith

3 replies

scyphers
Rank
Badge +5

Hi Keith,

I do not have 2016.1 to properly test but i have been able to repeat the 301 status error if i specify a correct/valid bucket name but specify a wrong region in the S3Uploader.  

The odd thing about S3 is that the DNS rules are 'different' in us-east-1 Virginia. I think even case sensitive bucket name spaces are honoured. Every other region came after and more stringent rules were applied from day 1.    

So my hacky workaround given that all AWS buckets are globally unique, If you provide the transformer with the name your ap-southeast-2 Sydney "located" bucket <uswest1-bucket> and point FME at us-east-1 Region you should be able to let AWS resolve the DNS on the S3_PUT and your files will land in Sydney bucket.

0684Q00000ArLfAQAV.png

The reason this works (for now) is that a bucket maybe does not support redirection thru REST API and under the hood the calls probably look something like putting something in Virginia us-east-1 

http://

bucketname
.s3.amazonaws.com  

And every other region  look something like. 

 http://s3-

region
.s3.amazonaws.com/
bucketname
 

So please try to Lie to your S3Uploader and see if you can upload your files to Sydney Region by naming you correct/valid bucket name but then using the wrong generic Virginia <us-east-1> Region.  

;)

K

Ta for the tip, Steven!

Unless I'm missing something, I'm not sure that's going to work for us though - we have no control over the region in S3 Uploader in FME 2015/16. All we can set is the bucket name and FME.

Actually according to this blog post from Safe our FME 2015/16 should already be trying to use the us-east-1 'trick'?

Our other hacky workaround we may fallback to for now is to drop down to Python Shutdown Script and handle uploads via Boto.

N

You can check this link too may be useful ( https://kodlogs.net/316/the-bucket-you-are-attempting-to-access-must-be-addressed-using-the-specified-endpoint )

Reply


Community Stats

28,692
Posts
109,433
Replies
36,544
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.