Skip to main content
Solved

Restricting file types read when using a generic reader to read data from a ZIP file

  • November 19, 2021
  • 9 replies
  • 283 views

jackyd
Contributor
Forum|alt.badge.img+11

I have a bunch of ZIP files which we download the contain data in 3 formats, shape, csv and file gdb. The zips also contain numerous other files such as pdf and xml, which we don't need to read.

 

Now the tricky bit, I am using a PATH reader to check a directory to see what ZIPs are there and to only load the ones that have arrived in the last 24hrs. The load is triggered in a Sub workbench where the Path is passed to a FeatureReader set to Generic. I can add I file type filter to the path that is passed by adding [**\\*.shp], for example to the path of the zip found by the PATH reader and can read the ZIP 3 times with a different filter each time by building this string

""@Value(path_windows)[**\\*.shp]" "@Value(path_windows)[**\\*.csv]" "@Value(path_windows)[**\\*.shp]""

 

but can't work out how to add filters for csv and gdb to read the file once and look for shp, csv or gdb.

 

The above works, but I get error messages for the iterations that don't find the file type filtered for, which is messy.

 

I could hard code the file type for each file, but want to future proof for possible changes in the future

Best answer by ebygomm

I think this should work here

"@Value(path_windows)\**\*.{shp,csv,gdb}"

 

View original
Did this help you find an answer to your question?

9 replies

ebygomm
Influencer
Forum|alt.badge.img+38
  • Influencer
  • Best Answer
  • November 19, 2021

I think this should work here

"@Value(path_windows)\**\*.{shp,csv,gdb}"

 


jackyd
Contributor
Forum|alt.badge.img+11
  • Author
  • Contributor
  • November 22, 2021
ebygomm wrote:

I think this should work here

"@Value(path_windows)\**\*.{shp,csv,gdb}"

 

Thanks @ebygomm​ , I knew there was some way to specify more that one extension, though I found I had to add the square brackets as well

@Value(path_windows)[**\*.{shp,csv,gdb}]


felipeverdu
Enthusiast
Forum|alt.badge.img+10

This only works for me whenever I choose the input format as design (i.e Esri shapefile). if chosen “<Guess format name from extension>” it won’t work out


crystalatsafe
Safer
Forum|alt.badge.img+19

Hi @felipeverdu 

I may have gotten it to work on my end.


Could you share a screenshot of how you set up your reader? 
Also, which version of FME are you using? 

 

Crystal 


felipeverdu
Enthusiast
Forum|alt.badge.img+10
  • Enthusiast
  • May 23, 2024

Hi @crystalatsafe 

 

I do better than that so i upload my own workspace (fmwt) :)

I see that it works fine for most input formats but Json and OGC Geopackage.

Let me know how it should be done.

Br,

Felipe Verdú


crystalatsafe
Safer
Forum|alt.badge.img+19

Hi @felipeverdu 

 

I was able to reproduce the same issue with the generic format FeatureReader. I see that json and Geopackage files are being filtered through the rejected port.

I have filed a ticket for our team to take a closer look. [internal: FMEENGINE-82928]

 

I apologize for any inconvenience and I will make sure to keep you updated.

 

Crystal 


crystalatsafe
Safer
Forum|alt.badge.img+19

Hi @felipeverdu 

 

I’ve been looking into finding a workaround for this. I did find a way to filter those records through the generic port of the Generic Feature Reader. However, many files are missing from the final output when compared to the JSON Feature Reader you have set up.

It looks like the workaround you have found is the best option for your workspace but I wanted to share what I found just in case it helps. 

I was able to filter records through the generic port by creating a new attribute with conditional values where the value is set to the fme format short name.

 

Then in the parameters of the Generic Feature Reader I selected that newly created attribute as the input format.  

This was able to filter records through the generic port but doesn’t read in the files as well as the JSON Feature Reader does. 

There have been some ideas around adding the option to select multiple formats in the Feature Reader. If you would like, please feel free to add a comment to this idea: Limit or chose preference of which Feature Types to read in Generic FeatureReader as our team reviews these. 

Please let me know if you have any questions. 

Crystal


felipeverdu
Enthusiast
Forum|alt.badge.img+10
  • Enthusiast
  • June 24, 2024

Hi @crystalatsafe 

Nice worksaround!!!. I like it and it solved pretty much the challenges here. I found just one json file not be able to be read but I believe there is something else.

I have added a commet to the idea so it can be taken into account.


crystalatsafe
Safer
Forum|alt.badge.img+19

Hi @felipeverdu 

I’m happy to hear that the workaround worked well for you! 

A huge thanks to @jdh for proving this workaround in the community post: Can I change the default formats read by the Generic Reader? 


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings