Is there any way you can set up a reader to read in a subset of the data? I am using a ODBC2 reader and I would like to only read data that is a month old (this table has 2 million records as it contains over a decade worth of data). I am currently using a tester transformer to remove the unwanted records, but the entire process takes over 6 hours.
Page 1 / 1
The ODBC Reader has a parameter labeled WHERE Clause. That can be used to select a subset of your data. That way the database will do the filtering.
Thank you @erik_jan! That worked perfectly. This dropped it down to 12 minutes!
you could also if its switching months and may have two months so can actually set the reader parameters to start at bottom and tell it how many to read. If it is always getting new records you could say read 1,000 or how many you normally get between months of the records. That way you would not have to change the where clause.