Skip to main content
Question

Inspire directs you to Amazon - Httpcaller get data calls


hlynur
Contributor
Forum|alt.badge.img+7

Hello, so we regularly download the whole catalogue of polygons for England, seen here:
Dataset Download - Use land and property data

I would very much want to automate this as its a tedious task to manually download them all one by one, so I’ve investigated the calls that my browser makes when calling this data and it first sends a very simple call to very simple URL’s containing the Zipped data, as seen here:
https://use-land-property-data.service.gov.uk/datasets/inspire/download/Amber_Valley_Borough_Council.zip

Which in turn takes you onward to a Amazon service that just gives you 60seconds to fetch the data, as seen here:
https://datapub-prd-s3-bucket.s3.amazonaws.com/inspire/Amber_Valley_Borough_Council.zip?response-content-type=application%2Fforce-download&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASNNJI2YGUWCWQTWF%2F20250522%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20250522T125638Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=c8a0384597f39f35e7be798cff4fcd1235d94cde0bbf5860e65b9eb8758339e2

I’ve tried to use the good old creator to HttpCaller to make a get call to the former URL which returns in a html code while making a call with the latter and longer Amazon URL one results in actual data (through a feature reader call), however that URL is generated on the fly and expires in 60 seconds.

This data is public so there should be some simple logic how to make a call that generates the tokens needed when needed and I’m not entirely sure how to do that, all input on how to best to get this done is appreciated :)

 

-Hlynur

5 replies

alexbiz
Enthusiast
Forum|alt.badge.img+12
  • Enthusiast
  • May 22, 2025

I think you want to use this instead of retro-engineer how the website create access to their s3 bucket : https://use-land-property-data.service.gov.uk/api-information


raghavendrans
Enthusiast
Forum|alt.badge.img+14

@hlynur 

As mentioned by ​@alexbiz FME offers robust API support, enabling users to connect to and utilize various web services and APIs without writing code. 

FME uses transformers, readers, and writers to interact with APIs, including REST APIs, webhooks, and more.

It will be a good choice to get the data of your interest.

Happy FME:-)ing

Cheers

SRG


hlynur
Contributor
Forum|alt.badge.img+7
  • Author
  • Contributor
  • May 22, 2025

Hey ​@alexbiz and thanks for the quick answers, we have been told that this API you refer to doesn’t provide the Inspire properties, you have any experience with that?


alexbiz
Enthusiast
Forum|alt.badge.img+12
  • Enthusiast
  • May 22, 2025

Oh, that’s a shame then. I do have experience working with APIs, but not specifically with this one or its datasets. From what you're describing, it looks like you're trying to automate something that wasn’t really designed to be accessed programmatically (or worse, something that they might intentionally want to discourage).

In general, if you're planning to use FME to automate downloads, it's always better to rely on APIs that are explicitly designed for that kind of use. That way, your workspace will be much easier to maintain over time.

Since this service is labeled as “beta,” it might be worth reaching out to them directly. They may be able to clarify how you can programmatically access the datasets you need—whether through their API or via another method. In some cases, they can even provide special access or guidance for bulk downloads.


hlynur
Contributor
Forum|alt.badge.img+7
  • Author
  • Contributor
  • May 23, 2025

Anyone else wants to give this a go, there has to be some logic in that call that can be adjusted


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings