Skip to main content
Solved

Search Time optimization in FeatureReader Files

  • February 10, 2020
  • 2 replies
  • 15 views
clowncutie
Contributor
Forum|alt.badge.img+7

Hi! I have an Excel file with ID numbers of different train stops, 20 of them. I wanted to find all DWG files that are in the subfolder Z\\Modell\\.*03.dwg but it takes around 1 h to run, at this point is it faster just to find them manually and run the workspace. Is there any way to improve the search time?

 

The "root" folder is a drive, but all files are within a folder is called "4\\_CAD" so I limit the search to only the subfolders of my starting folder. Trying to make it faster my only use the subfolder I know contain the other folders and files.

I was thinking maybe change it to have 2 feature writers after eachother instead, one that only searches for the correct folder then look for the files within the found folders. Change allowed path type to only folders then only files.

 

Other suggestions?

Also I figured if I have 60 subfolders, for every feature read it will look in all 60 of them 20 times over, that's probably why it takes such a long time, is there any smarter way?

 

root_folder:

"R:\\5410\\10285959\\4_CAD"

One Example subfolder containing a DWG:

"R:\\5410\\10285959\\4_CAD\\04\\0480\\04800800\\04800800R009\\Z\\Modell"

Example TrainID from the Excel File provided, so its [ID]+\\Z\\Modell

"04800800R009"

 

My Current settings:

root is: "R:\\5410\\10285959\\4_CAD"

 

 

My current filter

*\\@Value(BussStopID)*\\Z\\Modell\\*03.dwg

Best answer by takashi

Hi @clowncutie, I guess that the three parts "04", "0480" and "04800800" in the directory path can be extracted from the ID. i.e.

"04" = @Substring(@Value(ID),0,2)
"0480" = @Substring(@Value(ID),0,4)
"04800800" = @Substring(@Value(ID),0,8)

If so, you can set this expression to the Dataset in the PATH reader, in order to directly specify the folder containing target dwg files,

$(root_folder)\@Substring(@Value(ID),0,2)\@Substring(@Value(ID),0,4)\@Substring(@Value(ID),0,8)\@Value(ID)\Z\Modell

and just set *03.dwg to the Path Filter.

View original
Did this help you find an answer to your question?

2 replies

takashi
Evangelist
  • takashiEvangelist
  • Best Answer
  • February 10, 2020

Hi @clowncutie, I guess that the three parts "04", "0480" and "04800800" in the directory path can be extracted from the ID. i.e.

"04" = @Substring(@Value(ID),0,2)
"0480" = @Substring(@Value(ID),0,4)
"04800800" = @Substring(@Value(ID),0,8)

If so, you can set this expression to the Dataset in the PATH reader, in order to directly specify the folder containing target dwg files,

$(root_folder)\@Substring(@Value(ID),0,2)\@Substring(@Value(ID),0,4)\@Substring(@Value(ID),0,8)\@Value(ID)\Z\Modell

and just set *03.dwg to the Path Filter.

clowncutie
Contributor
Forum|alt.badge.img+7
  • clowncutieContributor
  • Author
  • Contributor
  • February 10, 2020
takashi wrote:

Hi @clowncutie, I guess that the three parts "04", "0480" and "04800800" in the directory path can be extracted from the ID. i.e.

"04" = @Substring(@Value(ID),0,2)
"0480" = @Substring(@Value(ID),0,4)
"04800800" = @Substring(@Value(ID),0,8)

If so, you can set this expression to the Dataset in the PATH reader, in order to directly specify the folder containing target dwg files,

$(root_folder)\@Substring(@Value(ID),0,2)\@Substring(@Value(ID),0,4)\@Substring(@Value(ID),0,8)\@Value(ID)\Z\Modell

and just set *03.dwg to the Path Filter.

I went from this:

 

0684Q00000ArMLhQAN.png

to this:

 

0684Q00000ArMnrQAF.png

Not the prettiest solution but the first FeatureReader only find the all directories containing "Modell\Z", aroung 100 of them, then uses the FeatureMerger to match with the substring in the path_name with the ID from Excel, then moves on as usual but without recursing into the subfolders, so now it only looks for .dwg in the 20 found folders which reduced the runtime with like 95%. 

Reply


Helpful Members This Week

Community Stats

31,533
Posts
119,549
Replies
39,169
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.