Skip to main content
  • EN
  • Home
  • Forums
  • FME Form
  • Data
  • Filter option in CSV reader parameter in FME 2018.1 disappeared? How to filter at source?

Hi,

I'm working with a CSV of 7Millions + rows which takes ages to run,

As I am working and amending/ improving the workflow I'd like to use a sample of the data.

I want to only use rows where there is a AVL_ID field starting 101 or 110 or 111... data is looking like numbers (7 digits) e.g. 1011234

I'd like to only read and use the rows where the AVL_ID field are starting with 110.

1. 2016 workflow and dataset

I started my workflow in 2016, so opening my CSV reader parameters, I see a Filter option . I thought I could filter it at source but my data came with the fields in Numeric format so I can't use the Filter AVL_ID ^110* as described on this web page: - (under Schema Attributes > Filter by Field Value) http://docs.safe.com/fme/2016.0/html/FME_Desktop_Documentation/FME_ReadersWriters/csv/CSV_reader.htm

I tried AVL_ID >= 1100000 but it does not do anything.

see image below

 

2. 2019 workflow - FME 2018.1 CSV reader

If I add a new CSV working on it now on FME upgraded to 2018.1... I get AVL_ID as string ... BUT no filter options. see image below

Has this functionality being taken away ?

What's the alternative?

 

Thanks

 

:-)

I should add that I have tried re-pointing to the data with the 2016 CSV reader but it keeps the attributes in Numbers.

The new CSV reader reads the CSV file in bulk mode (much faster), the older CSV reader read the CSV file line by line (which allows filtering).

I guess the only option for filtering is using a Tester transformer (or other filtering transformer) after reading the data.

Reply


Community Stats

30,108
Posts
114,295
Replies
37,743
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.