Solved

how to read zipped shapefiles that are 3 folders deep

  • 15 January 2019
  • 7 replies
  • 24 views
tnarladni
Rank
Badge +9

I have a zip file that has some shapefiles I need to load. Only problem is that the shapefiles are buried 3 folders deep. Any advice?

icon

Best answer by hollyatsafe 16 January 2019, 01:30

View original

7 replies

H
Badge +2

Hi @tnarladni,

Please review the usage Notes here: https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Desktop/Workbench/archive_files_reading.htm

 

If the files you wish to open are in a folder nested within a top-level folder, include \\**\\ to navigate to the next folder. For example, C:\\Zipster\\*.zip\\**\\*.shp

itay
Rank
Badge +16

Thanks to @takashi this issue was solved with python sometime ago:

https://knowledge.safe.com/questions/4864/triple-zipped-xml.html

You just need to add the authentication part to the python script.

G

Hi @tnarladni,

Please review the usage Notes here: https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Desktop/Workbench/archive_files_reading.htm

 

If the files you wish to open are in a folder nested within a top-level folder, include \\**\\ to navigate to the next folder. For example, C:\\Zipster\\*.zip\\**\\*.shp

If I want to use a zipped shapefile in a specific subfolder, which is located in the internet, it doesn't work.

I tried it with fme 2018 with the following zipped shapefile:

http://www.bev.gv.at/pls/portal/docs/PAGE/BEV_PORTAL_CONTENT_ALLGEMEIN/0200_PRODUKTE/UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip

I want to read the shapefile in the subfolder VGD-Steiermark/Steiermark_gstgenau/*.shp, but I didn't make it.

I tried a lot of variants, but each failed:

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip/VGD-Steiermark/Steiermark_gstgenau/*.shp

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip/**/**/*.shp

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip/**/*.shp

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip\\VGD-Steiermark\\Steiermark_gstgenau\\*.shp

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip\\**\\**\\*.shp

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip\\**\\*.shp

 

Can someone help me?

H
Badge +2

If I want to use a zipped shapefile in a specific subfolder, which is located in the internet, it doesn't work.

I tried it with fme 2018 with the following zipped shapefile:

http://www.bev.gv.at/pls/portal/docs/PAGE/BEV_PORTAL_CONTENT_ALLGEMEIN/0200_PRODUKTE/UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip

I want to read the shapefile in the subfolder VGD-Steiermark/Steiermark_gstgenau/*.shp, but I didn't make it.

I tried a lot of variants, but each failed:

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip/VGD-Steiermark/Steiermark_gstgenau/*.shp

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip/**/**/*.shp

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip/**/*.shp

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip\\VGD-Steiermark\\Steiermark_gstgenau\\*.shp

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip\\**\\**\\*.shp

...UNENTGELTLICHE_PRODUKTE_DES_BEV/VGD-Steiermark.zip\\**\\*.shp

 

Can someone help me?

Hi @gtitz,

Are you attempting to access these shapefiles using the ESRI Shapefile Reader?

It seems I can't access the link you provided without a portal log in so this might be where the problem lies. Please could you try using a Creator >HTTP Caller>Feature Reader instead.

You will need to save the Response Body to a file in the HTTPCaller and have Create a file per feature=yes. This transformer also allows you to set the appropriate Authentication requirements.

Then in the Feature Reader you can set the dataset to read to the response_file_path attribute.

G

Hi @gtitz,

Are you attempting to access these shapefiles using the ESRI Shapefile Reader?

It seems I can't access the link you provided without a portal log in so this might be where the problem lies. Please could you try using a Creator >HTTP Caller>Feature Reader instead.

You will need to save the Response Body to a file in the HTTPCaller and have Create a file per feature=yes. This transformer also allows you to set the appropriate Authentication requirements.

Then in the Feature Reader you can set the dataset to read to the response_file_path attribute.

Hi @hollyatsafe,

yes, I read the zipped shapefile with the ESRI Shapefile Reader.

Did you try the first link? That should work. The other links are just shortened links, where I've cut away the first part for clarity.

You can also go to the following page and copy the URL of the highlighted link:

http://www.bev.gv.at/portal/page?_pageid=713,2601287&_dad=portal&_schema=PORTAL

I didn't try your workaround yet, because the zip file is accessible for everyone without login, so (as you said) it should work with the ESRI Shapefile Reader.

H
Badge +2

Hi @hollyatsafe,

yes, I read the zipped shapefile with the ESRI Shapefile Reader.

Did you try the first link? That should work. The other links are just shortened links, where I've cut away the first part for clarity.

You can also go to the following page and copy the URL of the highlighted link:

http://www.bev.gv.at/portal/page?_pageid=713,2601287&_dad=portal&_schema=PORTAL

I didn't try your workaround yet, because the zip file is accessible for everyone without login, so (as you said) it should work with the ESRI Shapefile Reader.

Hi @gtitz

My mistake - the first link you provided was taking me to Error:Document Not Found and when I tried to work back down the url I ended up at a portal log in page. Thanks for clearing that up for me.

Anyway I've done some further digging as I'm seeing the same problem and unfortunately it looks like specifying a subdirectory (/**) is not currently supported because the Reader cannot determine where the URL pattern for the dataset ends and the extension pattern starts.

Instead the workaround would be what I previously suggested using the HTTPCaller - you can then download the folder and in the Feature Reader you will be able to specify the subfolder/wildcard. I've attached a workspace using your url to demonstrate this.

https://s3.us-west-2.amazonaws.com/community-migration/QAComments/url-subdirectory-shp65939.fmw" alt="https://s3.us-west-2.amazonaws.com/community-migration/QAComments/url-subdirectory-shp65939.fmwhttps://s3.us-west-2.amazonaws.com/community-migration/QAComments/url-subdirectory-shp65939.fmw" target="_blank">URL_subdirectory_shp.fmw

G

Hi @gtitz

My mistake - the first link you provided was taking me to Error:Document Not Found and when I tried to work back down the url I ended up at a portal log in page. Thanks for clearing that up for me.

Anyway I've done some further digging as I'm seeing the same problem and unfortunately it looks like specifying a subdirectory (/**) is not currently supported because the Reader cannot determine where the URL pattern for the dataset ends and the extension pattern starts.

Instead the workaround would be what I previously suggested using the HTTPCaller - you can then download the folder and in the Feature Reader you will be able to specify the subfolder/wildcard. I've attached a workspace using your url to demonstrate this.

https://s3.us-west-2.amazonaws.com/community-migration/QAComments/url-subdirectory-shp65939.fmw" alt="https://s3.us-west-2.amazonaws.com/community-migration/QAComments/url-subdirectory-shp65939.fmwhttps://s3.us-west-2.amazonaws.com/community-migration/QAComments/url-subdirectory-shp65939.fmw" target="_blank">URL_subdirectory_shp.fmw

Maybe this can be fixed in a future version.

Thank you!! Your workaround works fine.

Reply


Community Stats

28,692
Posts
109,433
Replies
36,544
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.