Hi, is there a way to query a csv before it's read into a workspace, something akin to being able to run a sql script on the oracle reader? I have a csv with 2 million+ records but only want to read in 6000 or so with a specific attribute value. I know I could run a tester but would rather not read them into the workspace.
Userlevel 4
- 8177 replies
-
11 September 2018
I don't think you'll have much choice other than reading the whole file, unless you know the line numbers the need.
An alternative would be to import the CSV data into a proper database and do the querying there, but unless the queries are fairly complex that probably won't be any faster.
Userlevel 4
+13
- Contributor
- 3725 replies
-
12 September 2018
Assuming you have a new (2018 for sure, maybe even 2017.1) FME, I'd strongly advise that using the latest CSV reader and going into an AttributeFilter followed by a Sampler will be about as fast as you could hope. In the AttributeFilter you'd put in the specific attr val you care about and in the sampler you'd do the first N. Using that sequence would allow the data to stay in the turbocharged FeatureTable mode. Once you have your 6000 features out of the Sampler, then do anything you need to do and we'll only make "old style" slower features for that part.
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.