Skip to main content

Hi, I have successfully used APIs for simple HTTP GET requests.

However, at the moment I am trying to download a table (or the whole site) from https://st.hwmonline.com/DG/Secure/SitesList.aspx where monitor sensor information (name, id, address) is stored.

The problem is that when I call that URL (together with authentication headers) only the first 50 sites are shown. There are however thousands which I need access to. Entering the site via browser, there is a button 'show all sites' below the table which I need to somehow pass into my HTTP caller.

 

This is an excerpt of the source code that refers to said button. When trying to put "dropMessagesNumber=0" into the header nothing happens:

 

<select 

name="ctl00$ContentPlaceHolder1$siteListInfo$dropMessagesNumber" onchange="javascript:setTimeout('__doPostBack(\\'ctl00$ContentPlaceHolder1$siteListInfo$dropMessagesNumber\\',\\'\\')', 0)" id="dropMessagesNumber" style="width:200px;">

<option value="1">Show 50 Sites Per Page</option>

<option value="2">Show 100 Sites Per Page</option>

<option value="3">Show 200 Sites Per Page</option>

<option value="4">Show 300 Sites Per Page</option>

<option selected="selected" value="0">Show All Sites</option>

 

</select>

 

Any ideas how I can formulate my request or a hint that it is not at all possible are appreciated.

 

Cheers,

Marvin

I would use F12 in the browser or a program like Fiddler, to inspect the request/traffic when you click on the Next 50 etc, to understand what that request looks like, with the idea of replicating it in FME. This approach is very dependent on how the site is coded, but hopefully it provides a way for you to extract all the data.


Hi ctredinnick, thanks for your input!

I have inspected the site with the development tools active. I couldn't spot an obvious change in the headers parameters and I am a layman in terms of web coding so not too sure what I am looking for.

I will get in touch with the website's administrator and see if they can point me in the right direction.

 

Cheers!

 


Reply