Question

Pagination - Fetching Additional Responses

  • 12 December 2017
  • 3 replies
  • 6 views

Badge

Hello: I am trying to piece my way through building a workspace to process surveymonkey responses. I'm using an HTTPCaller to get the results. However, I have roughly under a thousand responses and I have a max limit of 100 responses per page. I take the JSON results, validate then and use a Flattener to get the attributes I need. But I have to go grab the next 100 results, then next and per the SurveyMonkey documentation, I need to fetch 100 results with the resource url retutned in the links.next field (ex.

https://api.surveymonkey.net/v3/surveys/{survey_id}/responses/bulk?page=2&per;_page=100). The issue I have is I need guidance on what's next. I added another HTTPCaller and added the above to it thinking it would get the next results but it's not responding correctly. I know I have to loop and the max will be the number of responses / 100. I could use a good push in the right direction if anyone has done something similar. Thanks in Advance


3 replies

Badge +16

Hi

Hopefully you can unpick this; I can't share the actual workspace but in the attachment is a doc showing the pattern using a looping custom transformer, which is in the doc somewhere....

main.zip

Userlevel 6
Badge +31

Did something similar fetching WFS results using responsepaging before it was implemented in the reader. I don't know the api but you could use the Cloner transformer after the initial request to generate requests to fetch the results. No Workbench on this device so no sample but it should be fairly easy.

 

- Get total results. HTTPCaller / JsonFlattener?

 

- Calculate total number of pages you need to request. AttributeCreator or ExpressionEvaluator. Total results / 100.

 

- Use this as input for the Cloner to generate features.

 

- Create startindex and number of features per clone.

 

- Do all the requests with a second HTTPCaller.

Badge

Thank you! I will take a look. As I started to write my question, I had an idea of what I needed to do (smile). Perhaps I just had to jot it down for me to actually think through it a bit :)

Reply