Skip to main content
  • EN

Hello

I am trying to download multiple records using HTTPCaller from a API endpoint. I need to get all records updated in last 7 days. I plan to schedule to run this on daily interval.I have used a query to get the max date and passing it to get recent data.

The issue is that the API only allows 1000 records at a time. One has to set an offset start at 0,1000,2000 etc to get additional records. I dont know the number of records updated until i query all of them 1000 at a time.

What is the best way to loop the Httpcaller and automate it so when i run it all records that were updated in last 7 days let's say it is 1800 are downloaded.

Unfortunately I cannot share the API details.

Does your api return a cursor, or nextPage, or even the total number of results, when you make the first query (offset =0)?

No, it only shows a result "page":{"offset":0,"limit":1000,"hasmore":true}

I am thinking maybe i somehow parse out hasmore = true and then keep running the caller until is false. I just wasnt sure how to add a loop.

Here is the outline of a custom transformer that loops through the api results. The httpCaller should have the _offset attribute in the url, rather than have a number hardcoded.

 

 

Note that regardless of the results of hasMore, the current result set is processed.

fetchloop.fmx

Here is the outline of a custom transformer that loops through the api results. The httpCaller should have the _offset attribute in the url, rather than have a number hardcoded.

 

 

Note that regardless of the results of hasMore, the current result set is processed.

fetchloop.fmx

 

Thank you, this is exactly what i needed.

Reply


Community Stats

30,351
Posts
115,343
Replies
38,086
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.