Question

How do I download zip files with different number in filename every week from HTTPS site with authentication?

  • 20 February 2020
  • 4 replies
  • 6 views
J
Badge

Hi,

I'm trying to setup a script to download a set of zip files from a website that requires logging of username/password. After logging in, I need to select from 3 different zip files from the list. From what I've read so far, it looks HTTPCaller transformer with GET method is what I should use. The 3 zip files that I need to download each week is changing every week with a new # in a filename. How do I use HTTPCaller to select the zip files each week regardless of what number is in the filename?

4 replies

takashi
Rank
Userlevel 2
Badge +17

If the zip file URLs are provided in the HTML document of the download page, I think that you should be able to get the HTML with an HTTPCaller, parse it to retrieve the zip file URLs, and then download them with another HTTPCaller.

J
Badge

Hi Takashi,

 

The zip files URLs are not provided in the download page even when I select the zip file and I think this is where the problem is. After logging into the download page, I then have a choice to select each file or select all 3 zip files (see screen) to download which pops up on another screen asking me if I want to save the file.

I have tried the 3 steps HTTPCaller_Sign_In, HTTP_Post_Data and the HTTPCaller_Download_File but it kept failing at the HTTPCaller_Post_Data step.

 

takashi
Rank
Userlevel 2
Badge +17

Hi Takashi,

 

The zip files URLs are not provided in the download page even when I select the zip file and I think this is where the problem is. After logging into the download page, I then have a choice to select each file or select all 3 zip files (see screen) to download which pops up on another screen asking me if I want to save the file.

I have tried the 3 steps HTTPCaller_Sign_In, HTTP_Post_Data and the HTTPCaller_Download_File but it kept failing at the HTTPCaller_Post_Data step.

 

The error could be resolved hopefully if you set Yes to the Save Cookies parameter in those HTTPCallers. However, I'm not sure whether downloading is possible through the HTTPCaller, since it depends on how the web page is providing the zip files.

J
Badge

I did set Yes to the Save Cookies on 3 of the HTTPCallers and it's still failing.

Reply


Community Stats

28,723
Posts
109,511
Replies
36,559
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.