Solved

API multiple records with limit

  • 23 October 2018
  • 4 replies
  • 32 views
A
Badge

Hello

I am trying to download multiple records using HTTPCaller from a API endpoint. I need to get all records updated in last 7 days. I plan to schedule to run this on daily interval.I have used a query to get the max date and passing it to get recent data.

The issue is that the API only allows 1000 records at a time. One has to set an offset start at 0,1000,2000 etc to get additional records. I dont know the number of records updated until i query all of them 1000 at a time.

What is the best way to loop the Httpcaller and automate it so when i run it all records that were updated in last 7 days let's say it is 1800 are downloaded.

Unfortunately I cannot share the API details.

icon

Best answer by jdh 23 October 2018, 19:31

View original

4 replies

jdh
Rank
Badge +22

Does your api return a cursor, or nextPage, or even the total number of results, when you make the first query (offset =0)?

A
Badge

No, it only shows a result "page":{"offset":0,"limit":1000,"hasmore":true}

I am thinking maybe i somehow parse out hasmore = true and then keep running the caller until is false. I just wasnt sure how to add a loop.

jdh
Rank
Badge +22

Here is the outline of a custom transformer that loops through the api results. The httpCaller should have the _offset attribute in the url, rather than have a number hardcoded.

 

 

Note that regardless of the results of hasMore, the current result set is processed.

fetchloop.fmx

A
Badge

Here is the outline of a custom transformer that loops through the api results. The httpCaller should have the _offset attribute in the url, rather than have a number hardcoded.

 

 

Note that regardless of the results of hasMore, the current result set is processed.

fetchloop.fmx

 

Thank you, this is exactly what i needed.

Reply


Community Stats

28,650
Posts
109,263
Replies
36,523
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.