Skip to main content
Solved

Having a problem using featurereader to read a downloaded zip file (using HTTPCaller). The zip file contains a spatial dataset.

M

Hey folks,

 

As per the question itself, I'm really struggling with (what I assumed) would be a straightforward task. (Straightforward in that I cannabalised an older workbench which works just fine and simply switched up the HTTPCaller url and featurereader parameters and thought it would work.)

 

The way (I assume) it works is that the HTTPCaller calls the zip file from an online repository, then the featurereader reads the contained shapefile and I can then go on to do other processing tasks. However, the workbench is failing at the featurereader stage. The logfile has an error message that says:

Unable to open shape file `C:\\Users\\[USER]\\AppData\\Local\\Temp\\FME_TempFileStore_161769518366_3180\\dsR_1_161769518367_3180\\DBCALegislatedLandandWaters.zip_161769518367_3180\\DBCALegislatedLandandWaters.shp' for read.

 

I have no idea what the issue is, I've purged temporary files, screwed around with the parameters of the feature reader but am unable to get beyond this point. Does anyone have any suggestions or encountered anything similar?

 

Happy to upload the workbench if required.

 

Cheers,

 

Marty

 

Best answer by hkingsbury

martyrocks wrote:

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

 

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

See below, you need to specify the path to read your shapefile like

@Value(zip_loc)/*.shp

Screenshot 2021-04-09 095943

View original
Did this help you find an answer to your question?

14 replies

nielsgerrits
VIP
Forum|alt.badge.img+54

Just to be sure, you downloaded all the files? Not only the .shp but also the .dbf and the .shx? I think those are the bare minimum you need to read the .shp.

M
  • martyrocks
  • Author
  • April 6, 2021
nielsgerrits wrote:

Just to be sure, you downloaded all the files? Not only the .shp but also the .dbf and the .shx? I think those are the bare minimum you need to read the .shp.

Good question. I'd have to say I'm not sure but I think so. The zip file (when I download it manually) contains all component shapefile parts, and I use the same url for the HTTP Caller, but I'm not sure how to check that within FME.

nielsgerrits
VIP
Forum|alt.badge.img+54
martyrocks wrote:

Good question. I'd have to say I'm not sure but I think so. The zip file (when I download it manually) contains all component shapefile parts, and I use the same url for the HTTP Caller, but I'm not sure how to check that within FME.

Do you unzip the file or read directly from zip?

hkingsbury
Celebrity
Forum|alt.badge.img+54
  • hkingsburyCelebrity
  • Celebrity
  • April 6, 2021

You can pass the url of the zip file straight into a SHP reader/featurereader and it will download it and treat it as if it was reading a zipped SHP directly off the machine

M
  • martyrocks
  • Author
  • April 7, 2021
hkingsbury wrote:

You can pass the url of the zip file straight into a SHP reader/featurereader and it will download it and treat it as if it was reading a zipped SHP directly off the machine

Hey thanks hkingsbury. Tried that, however within the featurereader, the output always goes to the 'rejected' section, no matter what I do.

M
  • martyrocks
  • Author
  • April 7, 2021
martyrocks wrote:

Good question. I'd have to say I'm not sure but I think so. The zip file (when I download it manually) contains all component shapefile parts, and I use the same url for the HTTP Caller, but I'm not sure how to check that within FME.

I read directly from the zip (using the response_file_path parameter) but in every case, no matter what settings I play with, the output always goes to the 'rejected' section of the FeatureReader.

Cheers Nielsgerrits.

kailinatsafe
Safer
Forum|alt.badge.img+21

Hello @martyrocks​ , would you be able to share your data? Here to help, Kailin

M
  • martyrocks
  • Author
  • April 8, 2021
kailinatsafe wrote:

Hello @martyrocks​ , would you be able to share your data? Here to help, Kailin

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

 

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

1 Attachments
hkingsbury
Celebrity
Forum|alt.badge.img+54
  • hkingsburyCelebrity
  • Celebrity
  • Best Answer
  • April 8, 2021
martyrocks wrote:

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

 

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

See below, you need to specify the path to read your shapefile like

@Value(zip_loc)/*.shp

Screenshot 2021-04-09 095943

M
  • martyrocks
  • Author
  • April 9, 2021
martyrocks wrote:

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

 

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

Hey thanks hkingsbury, that works a treat!

 

I'd love to be able to pull directly from the URL rather than download and then read, that seems to be the main stumbling block for me in this case!

 

Regardless, thanks very much for your help and input, its very much appreciated!

 

Marty

hkingsbury
Celebrity
Forum|alt.badge.img+54
  • hkingsburyCelebrity
  • Celebrity
  • April 9, 2021
martyrocks wrote:

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

 

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

I do note that WFS serivces are avaliable, however they were pretty slow and kept failing on me. You could just read this straight into the workbench using the WFS reader

https://catalogue.data.wa.gov.au/dataset/dbca-legislated-lands-and-waters

M
  • martyrocks
  • Author
  • April 13, 2021
martyrocks wrote:

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

 

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

Hey thanks again hkingsbury. May be a silly question - where does the 'zip_loc' parameter come from?

hkingsbury
Celebrity
Forum|alt.badge.img+54
  • hkingsburyCelebrity
  • Celebrity
  • April 13, 2021
martyrocks wrote:

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

 

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

@martyrocks​  you can add attributes to a creator by right clicking on the blue part of the output port and selecting "Add Attribute"

 

Screenshot 2021-04-14 074207

M
  • martyrocks
  • Author
  • April 14, 2021
martyrocks wrote:

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

 

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

Ah nice one. Thanks again hkingsbury!

Reply


Helpful Members This Week

Community Stats

31,344
Posts
118,874
Replies
38,936
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.