Skip to main content
  • EN

Hi all, this is my first run with both FME and HTTPS API endpoints. The data I'm requesting comes back to me as paginated JSON in sets of 25 and the API uses offset=[Integer] to allow me to cycle through the data.

 

This means that when I want to pull almost any data in my day-to-day I need to hit the same endpoint multiple times with offsets 0, 25, 50 etc... until I've got all the data. The JSON comes back with a "has_next_page" boolean so I was hoping to use that to do the following:

 

1: establish a variable called 'offset' = 0 and an attribute called 'endpoint' = =UrlString] which references 'offset' at the end, run the HTTPCaller

2: when the JSON comes back, take the "has_next_page" value and turn it into an attribute

3: run a test to determine if it's true

4: if true, offset+25, run HTTPCaller again

5: repeat until 'has_next_page' returns false

 

what's the best way to go about this?

If there's no way of establishing the number of requests required up front, then you can do this by looping in a custom transformer

If there's no way of establishing the number of requests required up front, then you can do this by looping in a custom transformer

there's one instance where I can (kind of) get the total number of values up front, one of my queries gives me a 'total' value - it's not exactly up front because I have to run the HTTPCaller once to get the information but I figure that's quite easy to work around. Can you briefly outline what you might do in this case? I'm not familiar with custom transformers but will look into it for my other queries, thanks!

there's one instance where I can (kind of) get the total number of values up front, one of my queries gives me a 'total' value - it's not exactly up front because I have to run the HTTPCaller once to get the information but I figure that's quite easy to work around. Can you briefly outline what you might do in this case? I'm not familiar with custom transformers but will look into it for my other queries, thanks!

If you can make a single http call to get the total, you can then use this number to calculate how many calls you need to make, by dividing by 25 and rounding up.

Then use a cloner to create a feature for each call, and use the copy number multiplied by 25 to get the offset to put into the http call

Capture

Reply


Community Stats

30,162
Posts
114,512
Replies
37,820
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.