Solved

max features to read on a folder

  • 30 November 2017
  • 4 replies
  • 3 views
barrett_h
Rank
Badge

Hi all,

I'm sure I'm missing something simple here but I can't see a way to limit the number of features that are read when the Reader is pointing to a folder of files eg. c:\\folder\\**\\*.jpg

As far as I can tell, the Max Features to Read options are only applied within a dataset, not across a number of datasets. I'm currently using a Sampler to throttle the number of features that are being processed at any one time but it still has to read every file into the workspace first. As the folder in question has thousands of image files, I would like an easy way to ensure that the process doesn't collapse!

Cheers,

Barrett

icon

Best answer by jlutherthomas 1 December 2017, 01:01

View original

4 replies

J
Badge +2

Hi @barrett_h

 

Can you use the Directory and File Pathnames reader first, to read in folders and then pass the first X through to a FeatureReader?

 

You set it to only read in folder names, and set the maximum number of features to read (Folders).

 

If you want to read only X number of features per folder, then in the FeatureReader you can also set the number of features to read, which it would read for each input folder coming from the PATH reader.

 

 

danilo_fme
Rank
Userlevel 4
Badge +30

Hi @barrett_h

 

Can you use the Directory and File Pathnames reader first, to read in folders and then pass the first X through to a FeatureReader?

 

You set it to only read in folder names, and set the maximum number of features to read (Folders).

 

If you want to read only X number of features per folder, then in the FeatureReader you can also set the number of features to read, which it would read for each input folder coming from the PATH reader.

 

 

Great suggestion @jlutherthomas

 

danilo_fme
Rank
Userlevel 4
Badge +30

Hi @barrett_h,

Just to illustrate that the @jlutherthomas did write you:

1) Reader - Directory and File Pathnames

2) Transformer Counter to generate id for each file your folder

3) Transformer Tester to filter by attribute counter =< max_feature ( published parameters )

4) Transformer FeatureReader to get files using the attribute path_windows.

Attached the Workspace.

Thanks, workspace-max-features-to-read.fmw

Danilo

barrett_h
Rank
Badge

Thank you both @jlutherthomas and @danilo_inovacao. It's not exactly the solution I had in mind but it works great none the less! I was intent on trying to ensure the workspace read only a set number of features but by using the Directory and File Pathnames reader it still reads all the features, but with little impact on the performance of the workspace. I ended up with the following set up (the Directory and File Pathnames reader has the Allowed Path Types set to FILE):

I found that the Max Features to Read parameter in the FeatureReader does not work as I want it to ie. it is counting within a dataset not across all the datasets.

Thanks very much for the suggestions.

Reply


Community Stats

28,700
Posts
109,451
Replies
36,548
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.