Hello, so we regularly download the whole catalogue of polygons for England, seen here:
Dataset Download - Use land and property data
I would very much want to automate this as its a tedious task to manually download them all one by one, so I’ve investigated the calls that my browser makes when calling this data and it first sends a very simple call to very simple URL’s containing the Zipped data, as seen here:
https://use-land-property-data.service.gov.uk/datasets/inspire/download/Amber_Valley_Borough_Council.zip
Which in turn takes you onward to a Amazon service that just gives you 60seconds to fetch the data, as seen here:
https://datapub-prd-s3-bucket.s3.amazonaws.com/inspire/Amber_Valley_Borough_Council.zip?response-content-type=application%2Fforce-download&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASNNJI2YGUWCWQTWF%2F20250522%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20250522T125638Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=c8a0384597f39f35e7be798cff4fcd1235d94cde0bbf5860e65b9eb8758339e2
I’ve tried to use the good old creator to HttpCaller to make a get call to the former URL which returns in a html code while making a call with the latter and longer Amazon URL one results in actual data (through a feature reader call), however that URL is generated on the fly and expires in 60 seconds.
This data is public so there should be some simple logic how to make a call that generates the tokens needed when needed and I’m not entirely sure how to do that, all input on how to best to get this done is appreciated :)
-Hlynur