Solved

How to set up incremental updates using FME Cloud?

  • 3 June 2014
  • 1 reply
  • 3 views
M
Our organization is exploring the use of FME cloud to push data out to Google Maps Engine(GME).  We currently have 2 FME Desktop 2014 licenses which we use to load data into GME.  We run change detection to incrementally update features in GME from our on site SQL Server 2012 database.  This is crucial as we have to pay for storage with GME and need to minimize update size to reduce costs.

 

 

I would like to publish our workspaces to FME Cloud but am trying to figure out the workflow for such a thing before investing in it.  Would our data need to be published as a shared resource in FME Cloud everytime we'd like the workspaces to run?  Is it possible to direct those workspaces to our database without punching a hole in the firewall?  Or is there a way to run change detection locally and then upload the new data to FME Cloud and have workspaces run only when things have been added, updated and/or deleted?
icon

Best answer by kathyross 4 June 2014, 02:37

View original

1 reply

K
Badge +2
If you reference the following site, FME Server is optimised for Local Source Data rather than FME Cloud:

 

http://fmecloud.com/public/benefits

 

 

That being said, there are options for FME Cloud to communicate with local data:

 

  • DMZ Source
If you have the resources, you could request IT to put a server on the DMZ (externally accessible, cannot access your network) and either fully or incrementally replicate your database to this location.

 

 

FME Cloud is able to access any external source and thus would be able to access your database via the DMZ server. IT could potentially place restrictions on IP addresses allowed to access the DMZ, assign logins, set read/write permissions, restrict access to ports, etc.

 

 

Cons of this approach is that it requires a DMZ server. It would also require data replication and a methodology to achieve such data replication.

 

 

  • Proxy Server or VPN
FME Cloud is able to communicate to local databases if there is a channel via which to access the data. Channels can be set up via proxy servers or VPN tunnels. The way that such would work in theory is that FME Cloud would access the source via the IP or URL of the proxy server as the database server name. 

 

 

Proxy Servers can be restricted to only allow certain IP addresses access through the tunnel. This is good as FME Cloud has a static IP address thus restricting access to any other person.I believe proxy servers can also restrict access to what the IP address has access to on your network. 

 

 

VPN may be another option. However VPN tunnels often have a timeout (for good reason!), and are not often easy to function with automated solutions.

 

 

Be aware that FME does not have proxy compatibility for all formats and transformers.

 

  • Amazon Solutions
There are other options available. For example, it may be worth looking into Amazon secure tunnels.

 

 

There is also the possibility to write to Amazon's S3 Cloud Storage via a local FME.

 

 

http://fmepedia.safe.com/articles/FAQ/Amazon-S3

 

 

However, in your case this would only seem to add another potentially unneeded tier of complexity.

 

 

The bottom line is that FME Cloud needs its source data accessible via the web and it is thus not as optimised for local data as FME Server is. (Yet!).

Reply


Community Stats

28,709
Posts
109,483
Replies
36,556
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.