Hello there
We are using change detector in order to update a large data base with million if records
Is there a wày we could speed up the detection process
Thanks
No. By definition, you need to have the full set to detect changes. You could of course cluster changes on some hash function pi and paralise it, but the computing power would just increase and not solve this engineering problem.
Just add more memory so that FME runs smoothly would be my advise.
+2- Contributor
bozewolf wrote:
No. By definition, you need to have the full set to detect changes. You could of course cluster changes on some hash function pi and paralise it, but the computing power would just increase and not solve this engineering problem.
Just add more memory so that FME runs smoothly would be my advise.
And turn off feature caching.
Run > Enable Feature Caching (should be deselected).
+17- Contributor
Can we get some more details on this?
I have sped up change detection by writing data to a staging table in a database.
If the database is the destination, writing all records to a table and have the database calculate the delta can speed up the process quite a bit.
But all depends on what type of data you are using.
+6- Contributor
One trick is calculate a CRC for each record using all the fields and perform the change detection on just the CRC.
Reply
Helpful Members This Week
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.