Hi, is there a way to query a csv before it's read into a workspace, something akin to being able to run a sql script on the oracle reader? I have a csv with 2 million+ records but only want to read in 6000 or so with a specific attribute value. I know I could run a tester but would rather not read them into the workspace.
Question
CSV Reader pre-read processing
+3This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
I don't think you'll have much choice other than reading the whole file, unless you know the line numbers the need.
An alternative would be to import the CSV data into a proper database and do the querying there, but unless the queries are fairly complex that probably won't be any faster.
+21- Safer
Assuming you have a new (2018 for sure, maybe even 2017.1) FME, I'd strongly advise that using the latest CSV reader and going into an AttributeFilter followed by a Sampler will be about as fast as you could hope. In the AttributeFilter you'd put in the specific attr val you care about and in the sampler you'd do the first N. Using that sequence would allow the data to stay in the turbocharged FeatureTable mode. Once you have your 6000 features out of the Sampler, then do anything you need to do and we'll only make "old style" slower features for that part.