Hi, is there a way to query a csv before it's read into a workspace, something akin to being able to run a sql script on the oracle reader? I have a csv with 2 million+ records but only want to read in 6000 or so with a specific attribute value. I know I could run a tester but would rather not read them into the workspace.
Page 1 / 1
I don't think you'll have much choice other than reading the whole file, unless you know the line numbers the need.
An alternative would be to import the CSV data into a proper database and do the querying there, but unless the queries are fairly complex that probably won't be any faster.
Assuming you have a new (2018 for sure, maybe even 2017.1) FME, I'd strongly advise that using the latest CSV reader and going into an AttributeFilter followed by a Sampler will be about as fast as you could hope. In the AttributeFilter you'd put in the specific attr val you care about and in the sampler you'd do the first N. Using that sequence would allow the data to stay in the turbocharged FeatureTable mode. Once you have your 6000 features out of the Sampler, then do anything you need to do and we'll only make "old style" slower features for that part.