Skip to main content
Archived

Add subfolder hierarchy parameter to readers.

Related products:Integrations
  • August 29, 2016
  • 11 replies
  • 19 views
taojunabc
Participant
Forum|alt.badge.img+6

FME can be specified on the reader reads all subfolders with **.
eg. C:Data***.shp, C:Data***.csv

But sumetimes, I don't need to read all the files in subfolders.

If I have the following folder structure.

If i set CSV reader dataset to C:Data***.csv.
Case 1
I want to read csv files in the [a1],[a1],[b1],[b1], But I don't want to read files in the [other] folder.
I hope to add a [Max Hierarchies to Read] parameters, control the maximum number of the subfolders hierarchies. In this case, this parameter can be set to 3 .

Case 2
I just want to read csv files in the specified hierarchy folder. eg.[a1*.csv],[a2*.csv].
I hope to add a [Specific Hierarchy to Read] parameters, control the number of specific subfolders hierarchy. In this case, this parameter can be set to 2 .

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

11 replies

takashi
Celebrity
  • takashiCelebrity
  • August 29, 2016

As a workaround, a combination of the PATH reader and FeatureReader may be useful. Assuming that "C:\\Data" is set to the Source Dataset for the PATH reader:

Path Filter for Case 1

{a1,b1}\\{b,}\\*.csv or {a1,a1\\b,b1,b1\\b}\\*.csv

Path Filter for Case 2

{a1,a2}\\*.csv

 

It might be better if path filter expressions could be used in source dataset setting for all file/folder-based format readers.

Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
taojunabc
Participant
Forum|alt.badge.img+6
  • taojunabcParticipant
  • Author
  • Participant
  • August 31, 2016
@takashi

 

Thaks your comment. I tested {a1,b1}\\{b,}\\*.csv, {a1,a1\\b,b1,b1\\b}\\*.csv and {a1,a2}\\*.csv. But get the same results as follows. I tested in FME 2016.1.1.0

 

extra characters after close-brace

 

Program Terminating

 

Translation FAILED.

takashi
Celebrity
  • takashiCelebrity
  • August 31, 2016
Hi @taojunabc, my intention is to set a path filter expression to the 'Path Filter' parameter of the PATH reader, and then read matched CSV files with the FeatureReader. Did you configure your workspace like this?

'

Currently such a complex path filter expression cannot be used in other readers. I hope every reader for file/folder-based formats will accept any path filter expression through the Advanced Browser.
Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
takashi
Celebrity
  • takashiCelebrity
  • August 31, 2016
sorry, the Dataset setting for the FeatureReader in the previous screenshot was wrong. Updated the screenshot.

 

Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
taojunabc
Participant
Forum|alt.badge.img+6
  • taojunabcParticipant
  • Author
  • Participant
  • August 31, 2016
@takashi

 

Oh, sorry, I setted it to dataset.

 

I tested it on "path filter", It worked.

 

I found * parttens is allowed too. It is very convenience.

 

eg.{a1,b1}\\*\\*.csv, {*}\\{*}\\*.csv, *\\*\\*.csv

 

fmelizard
Safer
Forum|alt.badge.img+21
  • fmelizardSafer
  • Safer
  • September 2, 2016

I'm curious if the \\ is being interpretted as an "escape" character and goofing up the file expansion. @takashi / @taojunabc -- if you tried {a1,b1}/{b,}/*.csv, {a1,a1/b,b1,b1/b}/*.csv does it make any difference? FME internally is very happy to use / as the file separator on Windows machines and \\ has the sometimes sad side effect of escaping what follows. (I don't have quick access to FME on Windows right now so can't check myself...)

takashi
Celebrity
  • takashiCelebrity
  • September 2, 2016
Hi @daleatsafe, unfortunately slashes are also ineffective in the Dataset parameter setting for the CSV reader.

I tried adding a CSV reader with this setting (FME 2016.1.1, Windows 7).

'

However, the slashes have been changed to backslashes automatically and the comma within the path filter expression seems to be interpreted as a delimiter between paths.

'

Failed...

'

'
Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
takashi
Celebrity
  • takashiCelebrity
  • September 2, 2016
Interestingly, the reader feature type can be configured if you set the path filter through the Advanced Browser.

'

'

but the translation will fail.

'

path-filter-in-csv-reader-navigator-2_2.png
Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
fmelizard
Safer
Forum|alt.badge.img+21
  • fmelizardSafer
  • Safer
  • September 2, 2016

 

I was thinking that maybe the advanced browser might be a way to get this in there...but then I can see where we get defeated later on. I think though we might be able to fix that for this case. I've alerted the appropriate authorities ( @GraemeAtSafe in particular...)

 

takashi
Celebrity
  • takashiCelebrity
  • September 2, 2016
Thanks for your considerations. If every reader could handle path filter expressions like the PATH reader, it would be wonderful.

 

 

Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
taojunabc
Participant
Forum|alt.badge.img+6
  • taojunabcParticipant
  • Author
  • Participant
  • September 5, 2016
@daleatsafe,@takashi, Thanks for your comments.

 

As Takashi said, If every reader could handle path filter expressions like the PATH reader, it would be wonderful.

 

 

And I also think, If can privide [Max Hierarchies to Read] and [Specific Hierarchy to Read] parameters in advanced browser, May be more convenient to use.

 

 

Community Stats

32,219
Posts
122,352
Replies
39,914
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.