Skip to main content
Solved

Reading and then eventually writing to PostGIS database sitting on an AWS Serverless RDS?

Y

Hi All,

We're looking at setting up a PostGIS database on a PostGIS database sitting on an AWS Serverless RDS and are unclear whether our desktop FME will be able to successfully interact with it.

Our main concern is with some of our existing FME workspaces that we would want to read from the PostGIS database, process for 2-3 days then write the output back to the database. During this processing we'd expect the AWS Serverless RDS to switch itself off as all the processing is happening on my desktop, but will FME "wake-up" the Serverless RDS when it's time to write to the database?

 

Cheers

 

Rich

 

 

 

Best answer by david_r

In my experience FME is able to both read and write to RDS Postgresql, granted that all the security groups etc have been correctly configured.

Concerning interacting with the RDS instance itself, this is something that is out of FME's control, but you can e.g. use the AWS command line interface inside a SystemCaller to interact with the instance at different points in your process. In other words, you will have to manually implement the functionality to switch on or off the RDS instance.

View original
Did this help you find an answer to your question?

5 replies

david_r
Celebrity
  • david_rCelebrity
  • Best Answer
  • February 4, 2020

In my experience FME is able to both read and write to RDS Postgresql, granted that all the security groups etc have been correctly configured.

Concerning interacting with the RDS instance itself, this is something that is out of FME's control, but you can e.g. use the AWS command line interface inside a SystemCaller to interact with the instance at different points in your process. In other words, you will have to manually implement the functionality to switch on or off the RDS instance.

G
Forum|alt.badge.img

@yorkie As David pointed out FME as able to read & write from & to PostGIS databases.

 

 

When you say "AWS Serverless RDS" does that mean that you are using the Serverless option of the Amazon Aurora with PostgreSQL compatibility?

 

 

I am asking because the regular AWS RDS database with PostgreSQL/PostGIS engine type is not serverless. So you would be in charge of pausing and starting the RDS instance similar to what David described. Instead of using a SystemCaller to call the AWS CLI you could also use the AWS Simple Notification Service (SNS) and to notify an AWS Lambda function to stop & start your instance. This would probably give you more control and allow you to put safeguards in place that make sure your instance is indeed up & running and ready to be accessed by FME. Here are some resources that might be useful:

https://aws.amazon.com/blogs/database/implementing-db-instance-stop-and-start-in-amazon-rds/

 

 

https://docs.aws.amazon.com/lambda/latest/dg/with-sns.html
david_r
Celebrity
  • david_rCelebrity
  • February 4, 2020
gerhardatsafe wrote:

@yorkie As David pointed out FME as able to read & write from & to PostGIS databases.

 

 

When you say "AWS Serverless RDS" does that mean that you are using the Serverless option of the Amazon Aurora with PostgreSQL compatibility?

 

 

I am asking because the regular AWS RDS database with PostgreSQL/PostGIS engine type is not serverless. So you would be in charge of pausing and starting the RDS instance similar to what David described. Instead of using a SystemCaller to call the AWS CLI you could also use the AWS Simple Notification Service (SNS) and to notify an AWS Lambda function to stop & start your instance. This would probably give you more control and allow you to put safeguards in place that make sure your instance is indeed up & running and ready to be accessed by FME. Here are some resources that might be useful:

https://aws.amazon.com/blogs/database/implementing-db-instance-stop-and-start-in-amazon-rds/

 

 

https://docs.aws.amazon.com/lambda/latest/dg/with-sns.html

+1 for SNS + Lambda, a very powerful combination, although a step up in complexity from using the AWS CLI.

Y
  • yorkie
  • Author
  • February 4, 2020
gerhardatsafe wrote:

@yorkie As David pointed out FME as able to read & write from & to PostGIS databases.

 

 

When you say "AWS Serverless RDS" does that mean that you are using the Serverless option of the Amazon Aurora with PostgreSQL compatibility?

 

 

I am asking because the regular AWS RDS database with PostgreSQL/PostGIS engine type is not serverless. So you would be in charge of pausing and starting the RDS instance similar to what David described. Instead of using a SystemCaller to call the AWS CLI you could also use the AWS Simple Notification Service (SNS) and to notify an AWS Lambda function to stop & start your instance. This would probably give you more control and allow you to put safeguards in place that make sure your instance is indeed up & running and ready to be accessed by FME. Here are some resources that might be useful:

https://aws.amazon.com/blogs/database/implementing-db-instance-stop-and-start-in-amazon-rds/

 

 

https://docs.aws.amazon.com/lambda/latest/dg/with-sns.html

@gerhardatsafe @david_r

thanks for your response and the links. Yes, I mean Serverless option of the Amazon Aurora with PostgreSQL compatibility (sorry my terminology is poor). I'll forward your responses to our Developer and let you know how we get on!

Cheers,

Rich

 

 

G
Forum|alt.badge.img
yorkie wrote:

@gerhardatsafe @david_r

thanks for your response and the links. Yes, I mean Serverless option of the Amazon Aurora with PostgreSQL compatibility (sorry my terminology is poor). I'll forward your responses to our Developer and let you know how we get on!

Cheers,

Rich

 

 

@yorkie According to this doc I would assume your scenario works if the Automatic Pause and Resume for Aurora Serverless is configured correctly:

 

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.how-it-works.html

Reply


Helpful Members This Week

Community Stats

31,148
Posts
118,088
Replies
38,768
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.