Having a problem using featurereader to read a downloaded zip file (using HTTPCaller). The zip file contains a spatial dataset. | Community

Hey folks,

As per the question itself, I'm really struggling with (what I assumed) would be a straightforward task. (Straightforward in that I cannabalised an older workbench which works just fine and simply switched up the HTTPCaller url and featurereader parameters and thought it would work.)

The way (I assume) it works is that the HTTPCaller calls the zip file from an online repository, then the featurereader reads the contained shapefile and I can then go on to do other processing tasks. However, the workbench is failing at the featurereader stage. The logfile has an error message that says:

Unable to open shape file `C:\\Users\\\USER]\\AppData\\Local\\Temp\\FME_TempFileStore_161769518366_3180\\dsR_1_161769518367_3180\\DBCALegislatedLandandWaters.zip_161769518367_3180\\DBCALegislatedLandandWaters.shp' for read.

I have no idea what the issue is, I've purged temporary files, screwed around with the parameters of the feature reader but am unable to get beyond this point. Does anyone have any suggestions or encountered anything similar?

Happy to upload the workbench if required.

Cheers,

Marty

Page 1 / 1

Just to be sure, you downloaded all the files? Not only the .shp but also the .dbf and the .shx? I think those are the bare minimum you need to read the .shp.

Just to be sure, you downloaded all the files? Not only the .shp but also the .dbf and the .shx? I think those are the bare minimum you need to read the .shp.

Good question. I'd have to say I'm not sure but I think so. The zip file (when I download it manually) contains all component shapefile parts, and I use the same url for the HTTP Caller, but I'm not sure how to check that within FME.

Good question. I'd have to say I'm not sure but I think so. The zip file (when I download it manually) contains all component shapefile parts, and I use the same url for the HTTP Caller, but I'm not sure how to check that within FME.

Do you unzip the file or read directly from zip?

You can pass the url of the zip file straight into a SHP reader/featurereader and it will download it and treat it as if it was reading a zipped SHP directly off the machine

You can pass the url of the zip file straight into a SHP reader/featurereader and it will download it and treat it as if it was reading a zipped SHP directly off the machine

Hey thanks hkingsbury. Tried that, however within the featurereader, the output always goes to the 'rejected' section, no matter what I do.

Good question. I'd have to say I'm not sure but I think so. The zip file (when I download it manually) contains all component shapefile parts, and I use the same url for the HTTP Caller, but I'm not sure how to check that within FME.

I read directly from the zip (using the response_file_path parameter) but in every case, no matter what settings I play with, the output always goes to the 'rejected' section of the FeatureReader.

Cheers Nielsgerrits.

Hello @martyrocks , would you be able to share your data? Here to help, Kailin

Hello @martyrocks , would you be able to share your data? Here to help, Kailin

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

See below, you need to specify the path to read your shapefile like

@Value(zip_loc)/*.shp

Screenshot 2021-04-09 095943

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

Hey thanks hkingsbury, that works a treat!

I'd love to be able to pull directly from the URL rather than download and then read, that seems to be the main stumbling block for me in this case!

Regardless, thanks very much for your help and input, its very much appreciated!

Marty

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

I do note that WFS serivces are avaliable, however they were pretty slow and kept failing on me. You could just read this straight into the workbench using the WFS reader

https://catalogue.data.wa.gov.au/dataset/dbca-legislated-lands-and-waters

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

Hey thanks again hkingsbury. May be a silly question - where does the 'zip_loc' parameter come from?

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

@martyrocks you can add attributes to a creator by right clicking on the blue part of the output port and selecting "Add Attribute"

Screenshot 2021-04-14 074207

Sure thing Kailin, and thanks again, file attached. And in case you also need it, the download link: https://maps.slip.wa.gov.au/datadownloads/SLIP_Public_Services/Property_and_Planning/DBCA_LegislatedLandsandWatersDBCA_011/DBCA_LegislatedLandsandWatersDBCA_011.zip

You need to create an account to download, but it's free. I've also factored that in as part of the workbench - using my credientials to authenticate.

Ah nice one. Thanks again hkingsbury!

Reply