Question

List all files from folder(s), read in some files and process these in FME.

R
Badge

Hi!

 

Here's an overview of what I'd like to achieve:

  1. User selects a folder.
  2. FME reads in all the files plus those in sub folders.
  3. An attribute filter filters out those files required. The attributes to filter by will have already been set.
  4. Selected files are then read into workspace and processing continues.

(all files are either .csv, .xls or .xlsx)

 

Having read this post this user is trying to accomplish the same thing. But I am unable to set up the FeatureReader to read the contained data in a file.

 

Please can you let me know how to do this or is there a better method? Thanks.

8 replies

dustin
Rank
Userlevel 3
Badge +26

If I'm reading this correctly, there is no way test the contents of the file without reading them in first. You are on the right track by first reading the filenames using the Directory and File Pathnames reader, and then reading those in with the FeatureReader (exposing the fme_basename)

 

From there I would feed all the features into your AttributeFilter, and based on the results, use the FeatureMerger to get all the original features that match based on fme_basename. image

R
Badge

If I'm reading this correctly, there is no way test the contents of the file without reading them in first. You are on the right track by first reading the filenames using the Directory and File Pathnames reader, and then reading those in with the FeatureReader (exposing the fme_basename)

 

From there I would feed all the features into your AttributeFilter, and based on the results, use the FeatureMerger to get all the original features that match based on fme_basename. image

Thank you for the response.

 

I hope this better explains what I'm trying to achieve with the AttributeFilter.

The attributefilter is only checking the file names from the PATH input as these folders contain many files I don't need. I was planning to read in these filtered features using a the FeatureReader.

dustin
Rank
Userlevel 3
Badge +26

That simplifies things. The Directory and File Pathnames reader gives you several attributes you can filter by. I normally use path_rootname or path_filename. In your FeatureReader, you would point it to path_windows, which is the full path of the files.

R
Badge

If I'm reading this correctly, there is no way test the contents of the file without reading them in first. You are on the right track by first reading the filenames using the Directory and File Pathnames reader, and then reading those in with the FeatureReader (exposing the fme_basename)

 

From there I would feed all the features into your AttributeFilter, and based on the results, use the FeatureMerger to get all the original features that match based on fme_basename. image

The generic port from my FeatureReader only contains a column named <no schema>, how do you set up the FeatureReader to output values which your AttributreFilter is processing?

dustin
Rank
Userlevel 3
Badge +26

The generic port from my FeatureReader only contains a column named <no schema>, how do you set up the FeatureReader to output values which your AttributreFilter is processing?

You can expose attributes in the parameters of the FeatureReader.

image

R
Badge

The generic port from my FeatureReader only contains a column named <no schema>, how do you set up the FeatureReader to output values which your AttributreFilter is processing?

Would you be able to take a look at my attached workspace?

 

The workflow in the top bookmark is functioning for a single dataset. In the second workflow I'm attempting to use the same process to reveal the features of two datasets.

 

 

1 Attachment
dustin
Rank
Userlevel 3
Badge +26

The generic port from my FeatureReader only contains a column named <no schema>, how do you set up the FeatureReader to output values which your AttributreFilter is processing?

It's failing because it's trying to read a CSV file into a FeatureReader set to read an excel file. I would create a 3rd FeatureReader which is fed from the AttributeFilter CSV port. Then set that FeatureReader to CSV.

R

I'm attempting to do the same thing, having read the post I have managed to isolate the files i wish to process but can't work out the final stage as to ho to process them. any assistance appreciated.

Reply


Community Stats

28,670
Posts
109,366
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.