I'm trying to extract data from the website https://www.tax.service.gov.uk/check-council-tax-band/search. After submitting a postcode (e.g. LS26 0AG) it returns a coded URL with results, which I want to extract.
I've got the last part (extracting the results) working well but I can't figure out how to use HTTPCaller to submit a postcode and get the URL that is returned.
The form name is "postcode" but when I set it up and run I get a "HTTP/1.1 400 Bad Request" error.
Can anyone help please?
Looks like they're using a CSRF token to make life difficult for you: https://portswigger.net/web-security/csrf/tokens
You'll have to include the CSRF token when querying the form endpoint, here's an example POST payload when using the form to search for "E1 7AW":
csrfToken=355c9ad413278a74421550206b365c12adbc107d-1665049163161-7daf2c0e178f5292f96ad2da&postcode=E1+7AW&Search=
Looks like they're using a CSRF token to make life difficult for you: https://portswigger.net/web-security/csrf/tokens
You'll have to include the CSRF token when querying the form endpoint, here's an example POST payload when using the form to search for "E1 7AW":
csrfToken=355c9ad413278a74421550206b365c12adbc107d-1665049163161-7daf2c0e178f5292f96ad2da&postcode=E1+7AW&Search=
Thanks. Sorry for the newbie questions, but where do I put that in the HTTPCaller?
Thanks. Sorry for the newbie questions, but where do I put that in the HTTPCaller?
My recommendation is that you start by looking in the network traffic of the browser developer tools, so that you can see what the web page is doing. Having said that, I suspect that the CSRF token is there to make it very hard/impossible to accomplish what you're trying.