Question

FeatureReader select multiple files with condition

  • 2 November 2022
  • 5 replies
  • 63 views
kalbert
Rank
Badge +9

I use a FeatureReader to read multiple shape files from a .zip.

The FeatureReader run in FME-Server. A published parameter is used as 'Dataset' (i.e. the FME server path to the .zip file). 'Feature Types to Read' is empty to read <All Feature Types> because I don't know the exact name of the provided shape files per monthly run.

The set up works pretty well. the Reader loops over all shapefiles in the zip and the records are provided through the <Generic> port as intended.

 

Now I have the following challenge:

One group of shapefiles in the .zip are point classes and the name starts with POINT_****.shp. Another group are lines and the name start with LINE_***.shp.

Question: Is there any option to read only one group with my FeatureReader setting? I tried to play with wildcards in the File/Filter field of the 'SelectMultiple...' control but with no success

 

My current mitigation is to use two FeatureReader. The first one is only reading schema and the result is used to filter the classes I'm looking for (Tester) and concatenate the resulting fme_feature_type names into an attribute (ListBuilder, ListConcatenator) which is used as 'Features to Read' attribute in the second FeatureReader.

 

image(FME Desktop and Server 2020.1.3)

5 replies

natalieatsafe
Rank
Badge +6

@kalbert​ Hi there, thanks for your question. If I understand your ask correctly, I believe you can point the FeatureReader to the .zip, and then use the "Feature Types to Read" parameter to filter out all of the files which contain "POINT_".

 

If you set up the FeatureReader for Format and Dataset first, and then click the ellipses next to Feature Types to Read, a new window will pop open displaying the contents of the .zip. Make sure to deselect the "Select All" option. Next, at the bottom left of this window is a Filter input box. If you enter "POINTS_" into the filter, that should result in the FeatureReader picking up just those points shapefiles. Now you can select the "Select All" option to grab just those "POINTS_" shapefiles.

 

Here are a couple of screenshots showing the process:

 

  • Set up the FeatureReader and click the ellipses next to "Feature Types to Read.

 

FeatureReader_FeaturesTypesToRead 

  • Now deselect "Select All" and enter your file name filter into the "Filter" text input, at bottom left.

 

SelectFeatureTypes 

Give this a try and let us know how it goes! Hopefully it will do the trick 🙂

kalbert
Rank
Badge +9

Hi natalie,

No . Your solution is the default way when the content of the zip is always the same. As I wrote I have a monthly run, where I don't know the exact content of the zip. Known is only the naming convention of the files so that I can distinguish point and line classes by name.

The process runs automated on FME server, so I can't let the Feature Reader read the zip first and then manually select the classes. That's why I tried to work with wild cards but with no success.

 

natalieatsafe
Rank
Badge +6

@kalbert​ My apologies, I forgot the FME Server context! I don't think the parameters of the FeatureReader can accept wildcards or other similar special characters like it. I'm not certain on this, but I've tried several different methods, and cannot get the wildcard to work. The only exception to this seems to be in the Dataset parameter, which can accept and use the wildcard.

 

When you feed a .zip into the FeatureReader Dataset parameter, it can be treated just like any other folder, where you can go a couple levels beyond the .zip and get right into the compressed folder contents. This is where you can use the wildcard to get at those POINTS_ shapefiles. I'll demonstrate.

 

I have a zipped file called "Data.zip." It contains two shapefiles and a CAD drawing file. One of my shapefiles starts with "Forward", and I'm interested in processing just that shapefile. My shapefile of interest is not stored in any subfolders within my .zip. My .zip is located at the following path:

 

D:\\MyDrive\\MyProject\\MyWorkingFolder\\Inputs\\Data.zip

 

I can get right into that .zip file and access just my "Forward" shapefile by inputting the following into the FeatureReader Dataset parameter:

 

D:\\MyDrive\\MyProject\\MyWorkingFolder\\Inputs\\Data.zip\\Data\\Forward*.shp

 

Adjust_DatasetParameterI ran the above FeatureReader and received only my expected shapefile as output.

 

As long as your shapefiles are not stored in subfolders within the .zip, you can repeat the name of the .zip as if it's an extracted folder, and then put in your POINTS_*.shp wildcard portion at the end.

 

If they are all in the same subfolder, you can just append the subfolder name to the path. If they're in separate subfolders, you'd likely have to use the "Select Multiple Folders / Files" option from the drop-down beside the Dataset parameter, and specify each file path with subfolders to grab those, or play with the Subfolders option therein.

 

I know you mentioned that you don't know the exact contents of the .zip, but if you do know the folder structure / path to your shapefiles within the .zip, then this method may work for your use case. You'd just need to adjust the published parameter that you're using for "Dataset" to manually add-on that extra path portion after the .zip, so that you can use the wildcard.

 

Please don't hesitate to ask for clarification on this if needed, and let me know if this works for you!

kalbert
Rank
Badge +9

@kalbert​ My apologies, I forgot the FME Server context! I don't think the parameters of the FeatureReader can accept wildcards or other similar special characters like it. I'm not certain on this, but I've tried several different methods, and cannot get the wildcard to work. The only exception to this seems to be in the Dataset parameter, which can accept and use the wildcard.

 

When you feed a .zip into the FeatureReader Dataset parameter, it can be treated just like any other folder, where you can go a couple levels beyond the .zip and get right into the compressed folder contents. This is where you can use the wildcard to get at those POINTS_ shapefiles. I'll demonstrate.

 

I have a zipped file called "Data.zip." It contains two shapefiles and a CAD drawing file. One of my shapefiles starts with "Forward", and I'm interested in processing just that shapefile. My shapefile of interest is not stored in any subfolders within my .zip. My .zip is located at the following path:

 

D:\\MyDrive\\MyProject\\MyWorkingFolder\\Inputs\\Data.zip

 

I can get right into that .zip file and access just my "Forward" shapefile by inputting the following into the FeatureReader Dataset parameter:

 

D:\\MyDrive\\MyProject\\MyWorkingFolder\\Inputs\\Data.zip\\Data\\Forward*.shp

 

Adjust_DatasetParameterI ran the above FeatureReader and received only my expected shapefile as output.

 

As long as your shapefiles are not stored in subfolders within the .zip, you can repeat the name of the .zip as if it's an extracted folder, and then put in your POINTS_*.shp wildcard portion at the end.

 

If they are all in the same subfolder, you can just append the subfolder name to the path. If they're in separate subfolders, you'd likely have to use the "Select Multiple Folders / Files" option from the drop-down beside the Dataset parameter, and specify each file path with subfolders to grab those, or play with the Subfolders option therein.

 

I know you mentioned that you don't know the exact contents of the .zip, but if you do know the folder structure / path to your shapefiles within the .zip, then this method may work for your use case. You'd just need to adjust the published parameter that you're using for "Dataset" to manually add-on that extra path portion after the .zip, so that you can use the wildcard.

 

Please don't hesitate to ask for clarification on this if needed, and let me know if this works for you!

@natalieatsafe​ thanks for feedback. I played with your approach and figured out that it works with the wildcard setting as below:

$(path_to_zip)/**/LINE_*.zip

Although the zip file has no subfolders I have to add the /**/ part in the Dataset string.

My $(path_to_zip) parameter is of type Filename(existing) (e.g. C:\\dev\\delivery\\month\\Myshapes.zip).

(Side note: I also tried it as parameter type Folder assuming a zip is treated as a folder, but this was failing).

Next step: Test the approach on FME Server and provide the result here for the community.

 

natalieatsafe
Rank
Badge +6

@kalbert​ Happy to hear it's looking like a possible solution! And nice use of the recursive wildcard, I myself overlook it too often as an option in paths. I look forward to seeing if this works on the FME Server side.

Reply


Community Stats

28,729
Posts
109,527
Replies
36,565
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.