Skip to main content
Solved

Download a CSV file using HTTP caller

pschout
Participant
Forum|alt.badge.img+2

Hi,

 

I am trying to download a csv with this url (https://waterinfo.rws.nl/api/Download/CSV?expertParameter=Waterhoogte___20Oppervlaktewater___20t.o.v.___20Normaal___20Amsterdams___20Peil___20in___20cm&locationSlug=Delfzijl(DLFZ)&timehorizon=-24,24) using FME.

The csv contains data about the water level in the past 24 hours and the expected water level in the next 24 hours for a specific location. I have tried with the http caller but that doensn't work. Is there a way to download the csv using FME or to read the data in another way with a FME reader/transformer? In the attachment I have included the csv that should be the result of the download. 

I hope someone can help me.

Peter

Best answer by caracadrian

I looked into the website. I can't figure out where the API documentation is located.

But I could figure out how to get the data in JSON.

Use this link https://waterinfo.rws.nl/api/chart?mapType=waterhoogte&locationCode=Delfzijl(DLFZ)&values=-48,48 to get a JSON response then Fragment all the data out of it.

I recommend using JSONFragmenter instead of HTTPCaller as it has a nice GUI for creating the JSON Query.

View original
1 Attachments
Did this help you find an answer to your question?

6 replies

caracadrian
Contributor
Forum|alt.badge.img+22
  • caracadrianContributor
  • Contributor
  • March 31, 2021

How do you setup your HTTPCaller? What HTTP error do you get? Can you provide a log?

Do you use GET, do you use POST, are you seting the Query String Parameters right?

Have you checked the API documentation?

pschout
Participant
Forum|alt.badge.img+2
  • pschoutParticipant
  • Author
  • Participant
  • March 31, 2021
caracadrian wrote:

How do you setup your HTTPCaller? What HTTP error do you get? Can you provide a log?

Do you use GET, do you use POST, are you seting the Query String Parameters right?

Have you checked the API documentation?

See for the HTTP Caller setup the attached simple workbench. I think the Query string parameters are right. The recieved error is : HTTPCaller(HTTPFactory): Received HTTP response header: 'HTTP/1.1 500 Internal Server Error' from 'https://waterinfo.rws.nl/api/Download/CSV?expertParameter=Waterhoogte___20Oppervlaktewater___20t.o.v.___20Normaal___20Amsterdams___20Peil___20in___20cm&locationSlug=Delfzijl%28DLFZ%29&timehorizon=-24%2C24'

caracadrian
Contributor
Forum|alt.badge.img+22
  • caracadrianContributor
  • Contributor
  • Best Answer
  • April 1, 2021

I looked into the website. I can't figure out where the API documentation is located.

But I could figure out how to get the data in JSON.

Use this link https://waterinfo.rws.nl/api/chart?mapType=waterhoogte&locationCode=Delfzijl(DLFZ)&values=-48,48 to get a JSON response then Fragment all the data out of it.

I recommend using JSONFragmenter instead of HTTPCaller as it has a nice GUI for creating the JSON Query.

pschout
Participant
Forum|alt.badge.img+2
  • pschoutParticipant
  • Author
  • Participant
  • April 1, 2021
caracadrian wrote:

I looked into the website. I can't figure out where the API documentation is located.

But I could figure out how to get the data in JSON.

Use this link https://waterinfo.rws.nl/api/chart?mapType=waterhoogte&locationCode=Delfzijl(DLFZ)&values=-48,48 to get a JSON response then Fragment all the data out of it.

I recommend using JSONFragmenter instead of HTTPCaller as it has a nice GUI for creating the JSON Query.

Thanks! This helps! This alternative via JSON works fine, I was focused on using the HttpCaller, but this method is perfectly usable. Thanks for the support!

 

debbiatsafe
Safer
Forum|alt.badge.img+20

Hi @pschout​ 

It sounds like you already have a great answer from @caracadrian​  that works even better for your workflow but still wanted to share a tip in case it helps other users in the future.

If you are seeing a difference in behaviour between a web browser and HTTPCaller, it may be helpful to try emulating the browser's request in the HTTPCaller. In this case, it appears that setting the request headers to be the same as the web browser allows the GET request to succeed.

HTTPCaller_ParamYou can usually find this information via the Network tab after selecting Inspect Element.

WebBrowser_Network

pschout
Participant
Forum|alt.badge.img+2
  • pschoutParticipant
  • Author
  • Participant
  • April 2, 2021
debbiatsafe wrote:

Hi @pschout​ 

It sounds like you already have a great answer from @caracadrian​  that works even better for your workflow but still wanted to share a tip in case it helps other users in the future.

If you are seeing a difference in behaviour between a web browser and HTTPCaller, it may be helpful to try emulating the browser's request in the HTTPCaller. In this case, it appears that setting the request headers to be the same as the web browser allows the GET request to succeed.

HTTPCaller_ParamYou can usually find this information via the Network tab after selecting Inspect Element.

WebBrowser_Network

Hi debbiatsafe (Safer),

I tried your suggestion with the HTTPCaller and the Header information and it works fine, thanks for the tip!

 

Reply


Helpful Members This Week

Community Stats

30,921
Posts
117,185
Replies
38,725
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.