I am trying to download a zip file containing some shapefiles from a website that requires logging into with a username/password. From what I've read so far, it appears that the HTTPCaller transformer is what I should use. I've pasted my settings below, but nothing seems to download when I run the workspace. Anyone else figured out a way to download shapefiles from a password protected site?
Hi @mlayman09,
I did have success to use the HttpCaller to do a download from web. Some configurations:
Is there Error message in Log file?
Thanks,
Danilo
Hi @mlayman09,
I did have success to use the HttpCaller to do a download from web. Some configurations:
Is there Error message in Log file?
Thanks,
Danilo
I didn't see any information about the transformer HttpCaller.
There is some trigger before your HttpCaller?
In my example I used the Creator to send trigger to HttpCaller.
I didn't see any information about the transformer HttpCaller.
There is some trigger before your HttpCaller?
In my example I used the Creator to send trigger to HttpCaller.
Great @mlayman09, your HttpCaller is working now!
Could you send the Workspace?
Great @mlayman09, your HttpCaller is working now!
Could you send the Workspace?
If you open this link request in your internet browser, its works?
If you open this link request in your internet browser, its works?
Hi @mlayman09, web browser moved to this web page when I pasted your download URL (.zip) to the address bar.
It means that you cannot access the download URL with the Basic Authentication method.
Firstly you will have to analyze the HTML source of the sign-in page to find what you should perform to sign in the site. If posting some form data was enough to sign in the site, possibly you could download the zip file with a chain of these three HTTPCallers.
- HTTPCaller (GET method, save cookie): Access the sign-in page to start the session.
HTTPCaller (POST method, save cookie): Post required form data to perform sign-in. You will have to analyze the HTML source of the sign-in page to know required URL and form data to be set to this HTTPCaller.- HTTPCaller (GET method, save cookie): Download the zip file.
Depending on the structure of the site, more HTTPCaller could be required, but basically you can consider the chain of HTTPCallers as a simulation of your manual operations on a web browser. Also, it may not be possible to achieve the goal depending on the site structure. Good luck.
Hi @mlayman09, web browser moved to this web page when I pasted your download URL (.zip) to the address bar.
It means that you cannot access the download URL with the Basic Authentication method.
Firstly you will have to analyze the HTML source of the sign-in page to find what you should perform to sign in the site. If posting some form data was enough to sign in the site, possibly you could download the zip file with a chain of these three HTTPCallers.
- HTTPCaller (GET method, save cookie): Access the sign-in page to start the session.
HTTPCaller (POST method, save cookie): Post required form data to perform sign-in. You will have to analyze the HTML source of the sign-in page to know required URL and form data to be set to this HTTPCaller.- HTTPCaller (GET method, save cookie): Download the zip file.
Depending on the structure of the site, more HTTPCaller could be required, but basically you can consider the chain of HTTPCallers as a simulation of your manual operations on a web browser. Also, it may not be possible to achieve the goal depending on the site structure. Good luck.
<form method="post" action= ... > ... </form>
<form method="post" action= ... > ... </form>
<form method="post" action= ... > ... </form>
https://derrick.quickbase.com/db/main?a=signin&what;=
https://derrick.quickbase.com/db/main?a=signin&what;=
https://derrick.quickbase.com/db/main?a=signin&what;=