Skip to main content
  • EN

Hi,

What is the simplest way of to create an ongoing process that will import all "recently modified" records.

"recent" means all rows in source where modified date > latest "Modified date" in destination.

  1. I have millions of rows in about 10 tables.
  2. Each row has a unique ID which can be used to overwrite the destination.
  3. I am planning to copy all fields and all rows (replica).
  4. Source database: MS SQL 2008B, Destination database: MongoDB.

Thank you,

Idea 1:

 

  1. Create a private parameter for each table called "last_modified_date__table_name]".
  2. Prepopulate this parameter by querying destination table
  3. Using this parameter in WHERE clause in source query.
Is this the simplest? I could use a hint on how to do each of these.

 

 

Thank you,

 

 

I would try to do something like

  1. Creator
  2. FeatureReader or whatever is the most effective way to get the latest modification date of your MongoDB database. You could also use the Python MongoDB library inside a PythonCaller or PythonCreator to get the latest modification date.
  3. FeatureReader with a Where-clause based on the modification date from step 2

Reply


Community Stats

30,092
Posts
114,247
Replies
37,720
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.