Skip to main content
Archived

Delta source readers to speed up transformations

Related products:Integrations
  • May 3, 2021
  • 1 reply
  • 17 views

Forum|alt.badge.img

FME is an awesome ETL tool that is very efficient at transforming data. However, when moving large datasets on a schedule from on-site GIS databases over the WAN to cloud databases it has to reprocess and re-trasmit all of the data while very little of it has changed. For popular database sources (e.g. Postgres, Oracle, SQL Server) it would be optimal to mine database logs to only process the data deltas from the last time the workspace ran, instead of the entire dataset. This would greatly reduce the time required and data volumes to be transmitted.

1 reply

LizAtSafe
Safer
Forum|alt.badge.img+21
  • Safer
  • October 29, 2024
OpenArchived