Skip to main content
  • EN

Hello, so we regularly download the whole catalogue of polygons for England, seen here:
Dataset Download - Use land and property data

I would very much want to automate this as its a tedious task to manually download them all one by one, so I’ve investigated the calls that my browser makes when calling this data and it first sends a very simple call to very simple URL’s containing the Zipped data, as seen here:
https://use-land-property-data.service.gov.uk/datasets/inspire/download/Amber_Valley_Borough_Council.zip

Which in turn takes you onward to a Amazon service that just gives you 60seconds to fetch the data, as seen here:
https://datapub-prd-s3-bucket.s3.amazonaws.com/inspire/Amber_Valley_Borough_Council.zip?response-content-type=application%2Fforce-download&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASNNJI2YGUWCWQTWF%2F20250522%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20250522T125638Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=c8a0384597f39f35e7be798cff4fcd1235d94cde0bbf5860e65b9eb8758339e2

I’ve tried to use the good old creator to HttpCaller to make a get call to the former URL which returns in a html code while making a call with the latter and longer Amazon URL one results in actual data (through a feature reader call), however that URL is generated on the fly and expires in 60 seconds.

This data is public so there should be some simple logic how to make a call that generates the tokens needed when needed and I’m not entirely sure how to do that, all input on how to best to get this done is appreciated :)

 

-Hlynur

I think you want to use this instead of retro-engineer how the website create access to their s3 bucket : https://use-land-property-data.service.gov.uk/api-information

​@hlynur 

As mentioned by ​@alexbiz FME offers robust API support, enabling users to connect to and utilize various web services and APIs without writing code. 

FME uses transformers, readers, and writers to interact with APIs, including REST APIs, webhooks, and more.

It will be a good choice to get the data of your interest.

Happy FME:-)ing

Cheers

SRG

Hey ​@alexbiz and thanks for the quick answers, we have been told that this API you refer to doesn’t provide the Inspire properties, you have any experience with that?

Oh, that’s a shame then. I do have experience working with APIs, but not specifically with this one or its datasets. From what you're describing, it looks like you're trying to automate something that wasn’t really designed to be accessed programmatically (or worse, something that they might intentionally want to discourage).

In general, if you're planning to use FME to automate downloads, it's always better to rely on APIs that are explicitly designed for that kind of use. That way, your workspace will be much easier to maintain over time.

Since this service is labeled as “beta,” it might be worth reaching out to them directly. They may be able to clarify how you can programmatically access the datasets you need—whether through their API or via another method. In some cases, they can even provide special access or guidance for bulk downloads.

Anyone else wants to give this a go, there has to be some logic in that call that can be adjusted

Reply


Community Stats

31,814
Posts
120,830
Replies
39,443
Members

Latest FME

Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.