Skip to main content
Solved

API multiple records with limit

  • October 23, 2018
  • 4 replies
  • 266 views
ayraju
Contributor
Forum|alt.badge.img+1

Hello

I am trying to download multiple records using HTTPCaller from a API endpoint. I need to get all records updated in last 7 days. I plan to schedule to run this on daily interval.I have used a query to get the max date and passing it to get recent data.

The issue is that the API only allows 1000 records at a time. One has to set an offset start at 0,1000,2000 etc to get additional records. I dont know the number of records updated until i query all of them 1000 at a time.

What is the best way to loop the Httpcaller and automate it so when i run it all records that were updated in last 7 days let's say it is 1800 are downloaded.

Unfortunately I cannot share the API details.

Best answer by jdh

Here is the outline of a custom transformer that loops through the api results. The httpCaller should have the _offset attribute in the url, rather than have a number hardcoded.

 

 

Note that regardless of the results of hasMore, the current result set is processed.

fetchloop.fmx

danilo_fme
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

4 replies

jdh
Contributor
Forum|alt.badge.img+38
  • jdhContributor
  • Contributor
  • October 23, 2018

Does your api return a cursor, or nextPage, or even the total number of results, when you make the first query (offset =0)?

danilo_fme
ayraju
Contributor
Forum|alt.badge.img+1
  • ayrajuContributor
  • Author
  • Contributor
  • October 23, 2018

No, it only shows a result "page":{"offset":0,"limit":1000,"hasmore":true}

I am thinking maybe i somehow parse out hasmore = true and then keep running the caller until is false. I just wasnt sure how to add a loop.

danilo_fme
jdh
Contributor
Forum|alt.badge.img+38
  • jdhContributor
  • Contributor
  • Best Answer
  • October 23, 2018

Here is the outline of a custom transformer that loops through the api results. The httpCaller should have the _offset attribute in the url, rather than have a number hardcoded.

 

 

Note that regardless of the results of hasMore, the current result set is processed.

fetchloop.fmx

danilo_fme
ayraju
Contributor
Forum|alt.badge.img+1
  • ayrajuContributor
  • Author
  • Contributor
  • October 23, 2018

Here is the outline of a custom transformer that loops through the api results. The httpCaller should have the _offset attribute in the url, rather than have a number hardcoded.

 

 

Note that regardless of the results of hasMore, the current result set is processed.

fetchloop.fmx

 

Thank you, this is exactly what i needed.
danilo_fme

Community Stats

32,128
Posts
121,966
Replies
39,779
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.