Question

download multiple zip files from url

  • 7 March 2019
  • 2 replies
  • 73 views
pat_uow
Rank
Badge

Hi all,

I'm trying to download all zip files from the URL below, keep a copy of the zip in one local folder, and a copy of the extracted csv in another local folder.

http://www.nemweb.com.au/REPORTS/CURRENT/Daily_Reports/

Using the workspace attached , I can successfully download and extract a single zip file and csv content.

I believe there may be a list of all zip files available from the URL hidden somewhere in the response, but how would I go about extracting all available zip files and storing the resulting csv ?

Thanks

2 replies

redgeographics
Rank
Userlevel 5
Badge +25

I've used a StringSearcher after the HTTPCaller with this regex to look for the (partial) urls:

/REPORTS/CURRENT/Daily_Reports/PUBLIC_DAILY_\d+_\d+.zip

 

0684Q00000ArKZbQAN.png

Set it to generate a list and you'll end up with a partial url for every zip file referenced on the page. From there you can explode the list and download them one by one.

arnovananrooij
Rank
Badge +4

You could use a HTTPCaller and a HTMLExtracter to get a list of all the Files. After this you can use the ZipExtracter to download the files I think.

 

Reply


Community Stats

28,694
Posts
109,434
Replies
36,544
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.