Hello @juanmahere​ , thanks for providing the URL! If I try to make the same GET request in Postman, I get a pretty similar result. It looks like this website doesn't like people scraping. After a quick search, it looks like they have an API you can leverage (see Page 8, I think this is the most up to date doc, but it may be worth double checking). Would you mind trying to use the API? Best, Kailin.

Further to that, a 503 error indicates that the server is not able to handle the request, suggesting that there is something not 100% on the server. In this case it seems to be that they're blocking scraping. You could possibly get around it by including a different user agent or referer

Thanks @kailinatsafe​  and @hkingsbury​  for your tips.

I moved forward and from Postman, I could make the request using this structure (screenshot), but I cannot reproduce/translate into FME. Any ideas?

Looking at the above the only header you should need is the requestverificationtoken. This will go in the headers part of the httpcaller

I would hope you don't need the cookie header, if you do, thats going to need some additional calls and configuration

Hi @hkingsbury​ ,

Thanks for replying to this issue. Unfortunately, I'm sending that request you suggested, and I find no way to get it through.

Here is an example, passing the values as Published Params, and using "requestverificationtoken" on header:

Hmm, having a look at the site, its sitting behind cloudflare. The response i'm getting is that cloudflare has blocked it. I can get around it by setting a referer, cookie, user-agent and the token. However I imagine that both the token and cookie and time based and will expire eventually.

Basically its been setup to stop what you're doing in regards to scraping. If you need the data from it, your best bet is going to be to contact the company directly and see if they can supply it as a file, or have a developers portal you can sign up to

@hkingsbury​  many thanks for the investigation, as you suggested, it may take less pain if I directly contact the company. On the other hand, I was more interested in the FME HttpCall setup rather than the data, honestly.

Thanks everybody for your tips!

Juanma,