Skip to main content
Question

How do I download zip files with different number in filename every week from HTTPS site with authentication?

  • February 20, 2020
  • 4 replies
  • 47 views
J
Forum|alt.badge.img

Hi,

I'm trying to setup a script to download a set of zip files from a website that requires logging of username/password. After logging in, I need to select from 3 different zip files from the list. From what I've read so far, it looks HTTPCaller transformer with GET method is what I should use. The 3 zip files that I need to download each week is changing every week with a new # in a filename. How do I use HTTPCaller to select the zip files each week regardless of what number is in the filename?

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

4 replies

takashi
Celebrity
  • takashiCelebrity
  • 7843 replies
  • February 20, 2020

If the zip file URLs are provided in the HTML document of the download page, I think that you should be able to get the HTML with an HTTPCaller, parse it to retrieve the zip file URLs, and then download them with another HTTPCaller.

Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
J
Forum|alt.badge.img
  • jasminet
  • Author
  • 2 replies
  • February 20, 2020

Hi Takashi,

 

The zip files URLs are not provided in the download page even when I select the zip file and I think this is where the problem is. After logging into the download page, I then have a choice to select each file or select all 3 zip files (see screen) to download which pops up on another screen asking me if I want to save the file.

I have tried the 3 steps HTTPCaller_Sign_In, HTTP_Post_Data and the HTTPCaller_Download_File but it kept failing at the HTTPCaller_Post_Data step.

 

takashi
Celebrity
  • takashiCelebrity
  • 7843 replies
  • February 21, 2020

Hi Takashi,

 

The zip files URLs are not provided in the download page even when I select the zip file and I think this is where the problem is. After logging into the download page, I then have a choice to select each file or select all 3 zip files (see screen) to download which pops up on another screen asking me if I want to save the file.

I have tried the 3 steps HTTPCaller_Sign_In, HTTP_Post_Data and the HTTPCaller_Download_File but it kept failing at the HTTPCaller_Post_Data step.

 

The error could be resolved hopefully if you set Yes to the Save Cookies parameter in those HTTPCallers. However, I'm not sure whether downloading is possible through the HTTPCaller, since it depends on how the web page is providing the zip files.

Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
J
Forum|alt.badge.img
  • jasminet
  • Author
  • 2 replies
  • February 21, 2020

I did set Yes to the Save Cookies on 3 of the HTTPCallers and it's still failing.

Community Stats

32,102
Posts
121,896
Replies
39,749
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.