• EN
Solved

httpcaller limitation of items per page

  • 8 June 2023
  • 8 replies
  • 40 views
L

Hi community,

I would need your help to make my FME code smarter and more dynamic.

 

I am retrieving data via API. The problem is that I can only retrieve 100 items per page. I have 6 pages so I duplicate 6 httpcaller to be able to recover all of my data.

cap1Wouldn't there be a more dynamic way to retrieve this data without having to duplicate the httpcaller, maybe by looping but I don't know if it's possible on FME.

In the first call, I have the information about the number of pages.

callThanks for your help

icon

Best answer by ebygomm 8 June 2023, 18:32

View original

8 replies

redgeographics
Rank
Userlevel 5
Badge +26

You could use looping inside a custom transformer. The particular API you're using has a total_pages and next_page attribut, so you could use that to feed the HTTPCaller that's inside the loop.

 

Can you share the URL you're trying to access?

ebygomm
Rank
Userlevel 1
Badge +21

I would personally avoid looping if you know from the first call how many calls you need to make in total.

So httpcaller to retrieve first 100 items, grab the total page count from this output, use this value to clone a number of features (total_pages-1), and then a second httpcaller using the copy number to structure the call with which page/start number of records you want to retrieve (so in the scenario above, you would send 5 features to the second http caller.

L

You could use looping inside a custom transformer. The particular API you're using has a total_pages and next_page attribut, so you could use that to feed the HTTPCaller that's inside the loop.

 

Can you share the URL you're trying to access?

Yes it's :

https://hub-digital.daxium-air.com/spie-act-infra/groups?per_page=100&page=0

L

I would personally avoid looping if you know from the first call how many calls you need to make in total.

So httpcaller to retrieve first 100 items, grab the total page count from this output, use this value to clone a number of features (total_pages-1), and then a second httpcaller using the copy number to structure the call with which page/start number of records you want to retrieve (so in the scenario above, you would send 5 features to the second http caller.

thanks ebygomm, I will try it.

L

thanks ebygomm, I will try it.

Perfect it's works!

redgeographics
Rank
Userlevel 5
Badge +26

I would personally avoid looping if you know from the first call how many calls you need to make in total.

So httpcaller to retrieve first 100 items, grab the total page count from this output, use this value to clone a number of features (total_pages-1), and then a second httpcaller using the copy number to structure the call with which page/start number of records you want to retrieve (so in the scenario above, you would send 5 features to the second http caller.

That's actually a very elegant way of solving it (provided you can tell from the first call how many there will be in total).

david_r
Rank
Userlevel 5

That's actually a very elegant way of solving it (provided you can tell from the first call how many there will be in total).

Agree, it's a very elegant solution. Unfortunately not all APIs will let you know the total number of pages up front, since there is often a performance hit associated with calculating the number.

sworkman
Rank
Badge +1

Hey all, total newby and stumbling through. I have an httpcaller which returns the first 100 which I am assuming is page 1. I am struggling to find where you are seeing total page count and how to set up each caller to return a different page. I am connecting via a Microsoft Graph connection. Any chance you could tell me how to set up which page it is calling? Thanks!

 

 

Reply


Community Stats

28,823
Posts
109,858
Replies
36,638
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.