Skip to main content
  • EN

Hi everyone,


I’m working on an FME workbench where I download a full dataset file geodatabase daily. I also have a CSV file that lists daily changes.

My goal is to remove files from the main dataset based on the CSV and add updated files from file geodatabase back into the main dataset.

I’ve tried using the FeatureMerger and FeatureJoiner transformers to identify changes, but they were not effective due to time constraints. Since this needs to be done daily, time efficiency is crucial.

Do you have any suggestions, Python scripts, or SQL queries, or any other ideas to speed up this process?

Thank you!

Have you looked into the ChangeDetector transformer?
What kind of runtimes have you been experiencing and hoping for?

Thank you liamfez for your reply.

The dataset has about 18 million records, so I do not want to run all of them daily to check the changed data. 
I just tested a small portion of the data, which takes about one hour!

I need as short as possible to be able to run daily.

I appreciate any assistance you can provide.

Thanks

​@marjan_nikoukar , you might need to explain in more detail what the overall goal is, and samples of your data and FME workflow.

If it is to apply changes back to an enterprise database, there are far better formats to process delta changes in than offline File Geodatabases.

Reply


Community Stats

30,141
Posts
114,407
Replies
37,782
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.