Solved

Download a CSV file using HTTP caller

  • 30 March 2021
  • 6 replies
  • 80 views
pschout
Badge +1

Hi,

 

I am trying to download a csv with this url (https://waterinfo.rws.nl/api/Download/CSV?expertParameter=Waterhoogte___20Oppervlaktewater___20t.o.v.___20Normaal___20Amsterdams___20Peil___20in___20cm&locationSlug=Delfzijl(DLFZ)&timehorizon=-24,24) using FME.

The csv contains data about the water level in the past 24 hours and the expected water level in the next 24 hours for a specific location. I have tried with the http caller but that doensn't work. Is there a way to download the csv using FME or to read the data in another way with a FME reader/transformer? In the attachment I have included the csv that should be the result of the download. 

I hope someone can help me.

Peter

icon

Best answer by caracadrian 1 April 2021, 06:45

View original
1 Attachment

6 replies

caracadrian
Rank
Badge +20

How do you setup your HTTPCaller? What HTTP error do you get? Can you provide a log?

Do you use GET, do you use POST, are you seting the Query String Parameters right?

Have you checked the API documentation?

pschout
Badge +1

How do you setup your HTTPCaller? What HTTP error do you get? Can you provide a log?

Do you use GET, do you use POST, are you seting the Query String Parameters right?

Have you checked the API documentation?

See for the HTTP Caller setup the attached simple workbench. I think the Query string parameters are right. The recieved error is : HTTPCaller(HTTPFactory): Received HTTP response header: 'HTTP/1.1 500 Internal Server Error' from 'https://waterinfo.rws.nl/api/Download/CSV?expertParameter=Waterhoogte___20Oppervlaktewater___20t.o.v.___20Normaal___20Amsterdams___20Peil___20in___20cm&locationSlug=Delfzijl%28DLFZ%29&timehorizon=-24%2C24'

caracadrian
Rank
Badge +20

I looked into the website. I can't figure out where the API documentation is located.

But I could figure out how to get the data in JSON.

Use this link https://waterinfo.rws.nl/api/chart?mapType=waterhoogte&locationCode=Delfzijl(DLFZ)&values=-48,48 to get a JSON response then Fragment all the data out of it.

I recommend using JSONFragmenter instead of HTTPCaller as it has a nice GUI for creating the JSON Query.

pschout
Badge +1

I looked into the website. I can't figure out where the API documentation is located.

But I could figure out how to get the data in JSON.

Use this link https://waterinfo.rws.nl/api/chart?mapType=waterhoogte&locationCode=Delfzijl(DLFZ)&values=-48,48 to get a JSON response then Fragment all the data out of it.

I recommend using JSONFragmenter instead of HTTPCaller as it has a nice GUI for creating the JSON Query.

Thanks! This helps! This alternative via JSON works fine, I was focused on using the HttpCaller, but this method is perfectly usable. Thanks for the support!

 

debbiatsafe
Rank
Userlevel 3
Badge +17

Hi @pschout​ 

It sounds like you already have a great answer from @caracadrian​  that works even better for your workflow but still wanted to share a tip in case it helps other users in the future.

If you are seeing a difference in behaviour between a web browser and HTTPCaller, it may be helpful to try emulating the browser's request in the HTTPCaller. In this case, it appears that setting the request headers to be the same as the web browser allows the GET request to succeed.

HTTPCaller_ParamYou can usually find this information via the Network tab after selecting Inspect Element.

WebBrowser_Network

pschout
Badge +1

Hi @pschout​ 

It sounds like you already have a great answer from @caracadrian​  that works even better for your workflow but still wanted to share a tip in case it helps other users in the future.

If you are seeing a difference in behaviour between a web browser and HTTPCaller, it may be helpful to try emulating the browser's request in the HTTPCaller. In this case, it appears that setting the request headers to be the same as the web browser allows the GET request to succeed.

HTTPCaller_ParamYou can usually find this information via the Network tab after selecting Inspect Element.

WebBrowser_Network

Hi debbiatsafe (Safer),

I tried your suggestion with the HTTPCaller and the Header information and it works fine, thanks for the tip!

 

Reply


Community Stats

28,716
Posts
109,494
Replies
36,561
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.