Skip to main content
Archived

Delta source readers to speed up transformations

Related products:Integrations
  • May 3, 2021
  • 1 reply
  • 5 views
luisgarcia
kennyo
  • luisgarcia
    luisgarcia
  • kennyo
    kennyo

FME is an awesome ETL tool that is very efficient at transforming data. However, when moving large datasets on a schedule from on-site GIS databases over the WAN to cloud databases it has to reprocess and re-trasmit all of the data while very little of it has changed. For popular database sources (e.g. Postgres, Oracle, SQL Server) it would be optimal to mine database logs to only process the data deltas from the last time the workspace ran, instead of the entire dataset. This would greatly reduce the time required and data volumes to be transmitted.

1 reply

LizAtSafe
Safer
Forum|alt.badge.img+15
  • Safer
  • October 29, 2024
OpenArchived

Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings