I've got a HTTPCaller in a loop that retrieves data page by page (I use offset and limit). I have a tester that tests whether 'has_last_page' is true. This all seems to works ok but the translation does not stop - it just keeps repeating the last URL even though all records have been returned.
Solved
Hi all, I am fairly new to FME but have a problem with an httpcaller in a loop. I cannot seem to stop the translation when all records returned. Any help would be appreciated.
Best answer by nick824
Hi all, I tried cloner option and worked very well if I can get records per page etc. When the only data available is 'has_next_page' I eventually found that by moving the JSON extractor and JSON fragmenter stages out of the loop and placing them just before the writer it works like a treat.
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
- 8392 replies
Maybe the sample workspace posted by @bruceharold here can give you some clues: https://community.safe.com/s/question/0D54Q000080hPDg/resolving-pagination-with-httpcallerloop
+47- Celebrity
- 2000 replies
As an alternative, if you know how many records per page and how many total records there are you can use a cloner to create the required number of features and calculate their offset. This let's you avoid loops, however, if the total number of records isn't present then this isn't an option.
Thanks for the advice. Unfortunately I don't know the number of total records. I just have 'has_next_page' and 'total_rows'. I know that there are 25 rows/page but the 'total_rows' only reports a maximum of 501 so does not necessarily reflect actual.
Hi all, I tried cloner option and worked very well if I can get records per page etc. When the only data available is 'has_next_page' I eventually found that by moving the JSON extractor and JSON fragmenter stages out of the loop and placing them just before the writer it works like a treat.
+47- Celebrity
- 2000 replies
Hi all, I tried cloner option and worked very well if I can get records per page etc. When the only data available is 'has_next_page' I eventually found that by moving the JSON extractor and JSON fragmenter stages out of the loop and placing them just before the writer it works like a treat.
Nice, good result!