Is there any way you can set up a reader to read in a subset of the data? I am using a ODBC2 reader and I would like to only read data that is a month old (this table has 2 million records as it contains over a decade worth of data). I am currently using a tester transformer to remove the unwanted records, but the entire process takes over 6 hours.
Solved
Can a reader only read in a subset of the data?
Best answer by erik_jan
The ODBC Reader has a parameter labeled WHERE Clause. That can be used to select a subset of your data. That way the database will do the filtering.
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+23- Contributor
- Best Answer
The ODBC Reader has a parameter labeled WHERE Clause. That can be used to select a subset of your data. That way the database will do the filtering.
you could also if its switching months and may have two months so can actually set the reader parameters to start at bottom and tell it how many to read. If it is always getting new records you could say read 1,000 or how many you normally get between months of the records. That way you would not have to change the where clause.