I am trying to download a zip file containing some shapefiles from a website that requires logging into with a username/password. From what I've read so far, it appears that the HTTPCaller transformer is what I should use. I've pasted my settings below, but nothing seems to download when I run the workspace. Anyone else figured out a way to download shapefiles from a password protected site?
- Home
- Forums
- FME Form
- Transformers
- Download zip file from password protected website
Download zip file from password protected website
- April 11, 2018
- 19 replies
- 466 views
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
19 replies
- Evangelist
- 2069 replies
- April 11, 2018
Hi @mlayman09,
I did have success to use the HttpCaller to do a download from web. Some configurations:
Is there Error message in Log file?
Thanks,
Danilo
- Author
- 8 replies
- April 11, 2018
Hi @mlayman09,
I did have success to use the HttpCaller to do a download from web. Some configurations:
Is there Error message in Log file?
Thanks,
Danilo
- Evangelist
- 2069 replies
- April 11, 2018
- Author
- 8 replies
- April 11, 2018
- Evangelist
- 2069 replies
- April 11, 2018
I didn't see any information about the transformer HttpCaller.
There is some trigger before your HttpCaller?
In my example I used the Creator to send trigger to HttpCaller.
- Author
- 8 replies
- April 11, 2018
I didn't see any information about the transformer HttpCaller.
There is some trigger before your HttpCaller?
In my example I used the Creator to send trigger to HttpCaller.
- Evangelist
- 2069 replies
- April 11, 2018
Great @mlayman09, your HttpCaller is working now!
Could you send the Workspace?
- Author
- 8 replies
- April 11, 2018
Great @mlayman09, your HttpCaller is working now!
Could you send the Workspace?
- Evangelist
- 2069 replies
- April 11, 2018
If you open this link request in your internet browser, its works?
- Evangelist
- 2069 replies
- April 11, 2018
If you open this link request in your internet browser, its works?
- Author
- 8 replies
- April 11, 2018
- 7831 replies
- April 11, 2018
Hi @mlayman09, web browser moved to this web page when I pasted your download URL (.zip) to the address bar.
It means that you cannot access the download URL with the Basic Authentication method.
Firstly you will have to analyze the HTML source of the sign-in page to find what you should perform to sign in the site. If posting some form data was enough to sign in the site, possibly you could download the zip file with a chain of these three HTTPCallers.
- HTTPCaller (GET method, save cookie): Access the sign-in page to start the session.
- HTTPCaller (POST method, save cookie): Post required form data to perform sign-in. You will have to analyze the HTML source of the sign-in page to know required URL and form data to be set to this HTTPCaller.
- HTTPCaller (GET method, save cookie): Download the zip file.
Depending on the structure of the site, more HTTPCaller could be required, but basically you can consider the chain of HTTPCallers as a simulation of your manual operations on a web browser. Also, it may not be possible to achieve the goal depending on the site structure. Good luck.
- 7831 replies
- April 11, 2018
Hi @mlayman09, web browser moved to this web page when I pasted your download URL (.zip) to the address bar.
It means that you cannot access the download URL with the Basic Authentication method.
Firstly you will have to analyze the HTML source of the sign-in page to find what you should perform to sign in the site. If posting some form data was enough to sign in the site, possibly you could download the zip file with a chain of these three HTTPCallers.
- HTTPCaller (GET method, save cookie): Access the sign-in page to start the session.
HTTPCaller (POST method, save cookie): Post required form data to perform sign-in. You will have to analyze the HTML source of the sign-in page to know required URL and form data to be set to this HTTPCaller.- HTTPCaller (GET method, save cookie): Download the zip file.
Depending on the structure of the site, more HTTPCaller could be required, but basically you can consider the chain of HTTPCallers as a simulation of your manual operations on a web browser. Also, it may not be possible to achieve the goal depending on the site structure. Good luck.
1<form method="post" action= ... > ... </form>
- Author
- 8 replies
- April 12, 2018
1<form method="post" action= ... > ... </form>
1
- 7831 replies
- April 12, 2018
1<form method="post" action= ... > ... </form>
- Author
- 8 replies
- April 12, 2018
- 7831 replies
- April 12, 2018
1https://derrick.quickbase.com/db/main?a=signin&what;=
- Author
- 8 replies
- April 12, 2018
1https://derrick.quickbase.com/db/main?a=signin&what;=
- 7831 replies
- April 12, 2018
1https://derrick.quickbase.com/db/main?a=signin&what;=
I am trying to download a zip file containing some shapefiles from a website that requires logging into with a username/password. From what I've read so far, it appears that the HTTPCaller transformer is what I should use. I've pasted my settings below, but nothing seems to download when I run the workspace. Anyone else figured out a way to download shapefiles from a password protected site?
Hi @mlayman09,
I did have success to use the HttpCaller to do a download from web. Some configurations:
Is there Error message in Log file?
Thanks,
Danilo
Hi @mlayman09,
I did have success to use the HttpCaller to do a download from web. Some configurations:
Is there Error message in Log file?
Thanks,
Danilo
I didn't see any information about the transformer HttpCaller.
There is some trigger before your HttpCaller?
In my example I used the Creator to send trigger to HttpCaller.
I didn't see any information about the transformer HttpCaller.
There is some trigger before your HttpCaller?
In my example I used the Creator to send trigger to HttpCaller.
Great @mlayman09, your HttpCaller is working now!
Could you send the Workspace?
Great @mlayman09, your HttpCaller is working now!
Could you send the Workspace?
If you open this link request in your internet browser, its works?
If you open this link request in your internet browser, its works?
Hi @mlayman09, web browser moved to this web page when I pasted your download URL (.zip) to the address bar.
It means that you cannot access the download URL with the Basic Authentication method.
Firstly you will have to analyze the HTML source of the sign-in page to find what you should perform to sign in the site. If posting some form data was enough to sign in the site, possibly you could download the zip file with a chain of these three HTTPCallers.
- HTTPCaller (GET method, save cookie): Access the sign-in page to start the session.
- HTTPCaller (POST method, save cookie): Post required form data to perform sign-in. You will have to analyze the HTML source of the sign-in page to know required URL and form data to be set to this HTTPCaller.
- HTTPCaller (GET method, save cookie): Download the zip file.
Depending on the structure of the site, more HTTPCaller could be required, but basically you can consider the chain of HTTPCallers as a simulation of your manual operations on a web browser. Also, it may not be possible to achieve the goal depending on the site structure. Good luck.
Hi @mlayman09, web browser moved to this web page when I pasted your download URL (.zip) to the address bar.
It means that you cannot access the download URL with the Basic Authentication method.
Firstly you will have to analyze the HTML source of the sign-in page to find what you should perform to sign in the site. If posting some form data was enough to sign in the site, possibly you could download the zip file with a chain of these three HTTPCallers.
- HTTPCaller (GET method, save cookie): Access the sign-in page to start the session.
HTTPCaller (POST method, save cookie): Post required form data to perform sign-in. You will have to analyze the HTML source of the sign-in page to know required URL and form data to be set to this HTTPCaller.- HTTPCaller (GET method, save cookie): Download the zip file.
Depending on the structure of the site, more HTTPCaller could be required, but basically you can consider the chain of HTTPCallers as a simulation of your manual operations on a web browser. Also, it may not be possible to achieve the goal depending on the site structure. Good luck.
1<form method="post" action= ... > ... </form>
1<form method="post" action= ... > ... </form>
1
1<form method="post" action= ... > ... </form>
1https://derrick.quickbase.com/db/main?a=signin&what;=
1https://derrick.quickbase.com/db/main?a=signin&what;=
1https://derrick.quickbase.com/db/main?a=signin&what;=
Related Topics
Dividing Integers, Managing Number Typesicon
AuthoringIs it possible to create a " trapezoids" shaped line buffer?icon
TransformersIterativeSnipper - combine polylines and segment a continuous feature by a certain distanceicon
TransformersHi there, today I have a question about the SchemaMapper lookup table. The SchemaMapper filter allows me to define conditional clauses to perform attribute mappings based on specific conditions.icon
TransformersImprove the Form Example in Developer Information (in Run Workspace)
Helpful Members This Week
- redgeographics
18 votes
- ebygomm
15 votes
- hkingsbury
10 votes
- liamfez
9 votes
- david_r
9 votes
- geomancer
9 votes
- nielsgerrits
8 votes
- philippeb
8 votes
- takashi
8 votes
- alexbiz
6 votes
Recently Solved Questions
Community Stats
- 31,940
- Posts
- 121,312
- Replies
- 39,556
- Members
Latest FME
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Scanning file for viruses.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
OKThis file cannot be downloaded
Sorry, our virus scanner detected that this file isn't safe to download.
OKCookie policy
We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.
Cookie settings
We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.