I've got a HTTPCaller in a loop that retrieves data page by page (I use offset and limit). I have a tester that tests whether 'has_last_page' is true. This all seems to works ok but the translation does not stop - it just keeps repeating the last URL even though all records have been returned.
Best answer by nick824
View original
Maybe the sample workspace posted by @bruceharold here can give you some clues: https://community.safe.com/s/question/0D54Q000080hPDg/resolving-pagination-with-httpcallerloop
ok thanks, I'll take a look
+26
As an alternative, if you know how many records per page and how many total records there are you can use a cloner to create the required number of features and calculate their offset. This let's you avoid loops, however, if the total number of records isn't present then this isn't an option.
Thanks for the advice. Unfortunately I don't know the number of total records. I just have 'has_next_page' and 'total_rows'. I know that there are 25 rows/page but the 'total_rows' only reports a maximum of 501 so does not necessarily reflect actual.
Hi all, I tried cloner option and worked very well if I can get records per page etc. When the only data available is 'has_next_page' I eventually found that by moving the JSON extractor and JSON fragmenter stages out of the loop and placing them just before the writer it works like a treat.
+26
Hi all, I tried cloner option and worked very well if I can get records per page etc. When the only data available is 'has_next_page' I eventually found that by moving the JSON extractor and JSON fragmenter stages out of the loop and placing them just before the writer it works like a treat.
Nice, good result!
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
