Skip to main content
Question

Incremental (ongoing) migration

B
Forum|alt.badge.img
  • b4

Hi,

What is the simplest way of to create an ongoing process that will import all "recently modified" records.

"recent" means all rows in source where modified date > latest "Modified date" in destination.

  1. I have millions of rows in about 10 tables.
  2. Each row has a unique ID which can be used to overwrite the destination.
  3. I am planning to copy all fields and all rows (replica).
  4. Source database: MS SQL 2008B, Destination database: MongoDB.

Thank you,

2 replies

B
Forum|alt.badge.img
  • b4
  • Author
  • May 22, 2017
Idea 1:

 

  1. Create a private parameter for each table called "last_modified_date_[table_name]".
  2. Prepopulate this parameter by querying destination table
  3. Using this parameter in WHERE clause in source query.
Is this the simplest? I could use a hint on how to do each of these.

 

 

Thank you,

 

 

david_r
Celebrity
  • david_rCelebrity
  • May 23, 2017

I would try to do something like

  1. Creator
  2. FeatureReader or whatever is the most effective way to get the latest modification date of your MongoDB database. You could also use the Python MongoDB library inside a PythonCaller or PythonCreator to get the latest modification date.
  3. FeatureReader with a Where-clause based on the modification date from step 2

Reply


Helpful Members This Week

Community Stats

31,545
Posts
119,574
Replies
39,182
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.