Skip to main content
Solved

httpcaller limitation of items per page

L

Hi community,

I would need your help to make my FME code smarter and more dynamic.

 

I am retrieving data via API. The problem is that I can only retrieve 100 items per page. I have 6 pages so I duplicate 6 httpcaller to be able to recover all of my data.

cap1Wouldn't there be a more dynamic way to retrieve this data without having to duplicate the httpcaller, maybe by looping but I don't know if it's possible on FME.

In the first call, I have the information about the number of pages.

callThanks for your help

Best answer by ebygomm

I would personally avoid looping if you know from the first call how many calls you need to make in total.

So httpcaller to retrieve first 100 items, grab the total page count from this output, use this value to clone a number of features (total_pages-1), and then a second httpcaller using the copy number to structure the call with which page/start number of records you want to retrieve (so in the scenario above, you would send 5 features to the second http caller.

View original
Did this help you find an answer to your question?

8 replies

redgeographics
Celebrity
Forum|alt.badge.img+49

You could use looping inside a custom transformer. The particular API you're using has a total_pages and next_page attribut, so you could use that to feed the HTTPCaller that's inside the loop.

 

Can you share the URL you're trying to access?

ebygomm
Influencer
Forum|alt.badge.img+32
  • ebygommInfluencer
  • Influencer
  • Best Answer
  • June 8, 2023

I would personally avoid looping if you know from the first call how many calls you need to make in total.

So httpcaller to retrieve first 100 items, grab the total page count from this output, use this value to clone a number of features (total_pages-1), and then a second httpcaller using the copy number to structure the call with which page/start number of records you want to retrieve (so in the scenario above, you would send 5 features to the second http caller.

debbiatsafe
geomancer
2 people upvoted this
  • debbiatsafe
    debbiatsafe
  • geomancer
    geomancer
L
  • lribot
  • Author
  • June 9, 2023
redgeographics wrote:

You could use looping inside a custom transformer. The particular API you're using has a total_pages and next_page attribut, so you could use that to feed the HTTPCaller that's inside the loop.

 

Can you share the URL you're trying to access?

Yes it's :

https://hub-digital.daxium-air.com/spie-act-infra/groups?per_page=100&page=0

L
  • lribot
  • Author
  • June 9, 2023
ebygomm wrote:

I would personally avoid looping if you know from the first call how many calls you need to make in total.

So httpcaller to retrieve first 100 items, grab the total page count from this output, use this value to clone a number of features (total_pages-1), and then a second httpcaller using the copy number to structure the call with which page/start number of records you want to retrieve (so in the scenario above, you would send 5 features to the second http caller.

thanks ebygomm, I will try it.

L
  • lribot
  • Author
  • June 9, 2023
lribot wrote:

thanks ebygomm, I will try it.

Perfect it's works!

redgeographics
Celebrity
Forum|alt.badge.img+49
ebygomm wrote:

I would personally avoid looping if you know from the first call how many calls you need to make in total.

So httpcaller to retrieve first 100 items, grab the total page count from this output, use this value to clone a number of features (total_pages-1), and then a second httpcaller using the copy number to structure the call with which page/start number of records you want to retrieve (so in the scenario above, you would send 5 features to the second http caller.

That's actually a very elegant way of solving it (provided you can tell from the first call how many there will be in total).

david_r
Evangelist
  • david_rEvangelist
  • June 9, 2023
redgeographics wrote:

That's actually a very elegant way of solving it (provided you can tell from the first call how many there will be in total).

Agree, it's a very elegant solution. Unfortunately not all APIs will let you know the total number of pages up front, since there is often a performance hit associated with calculating the number.

sworkman
Contributor
Forum|alt.badge.img+4
  • sworkmanContributor
  • Contributor
  • March 5, 2024

Hey all, total newby and stumbling through. I have an httpcaller which returns the first 100 which I am assuming is page 1. I am struggling to find where you are seeing total page count and how to set up each caller to return a different page. I am connecting via a Microsoft Graph connection. Any chance you could tell me how to set up which page it is calling? Thanks!

 

 

Reply


Helpful Members This Week

Community Stats

31,043
Posts
117,673
Replies
38,823
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.