I have an HTTPCaller that fetches all the source rows in one call however the downstream transformers make the caller run again for each row. How to tell the transformers they already have all the data and to loop only within?
Userlevel 4
- 8170 replies
-
21 October 2022
Here's a good starting point: https://community.safe.com/s/article/Getting-Started-with-API-Pagination
+20
- Contributor
- 563 replies
-
21 October 2022
I assume your HTTPCaller is inside a Custom transformer. Otherwise, you wouldn't be able to create loops.
So, basically, you are telling it to run for each row.
After getting the first response, extract the page number, number of total elements or whatever information the API sends you for total number of features in the dataset you are querying. Use a Tester to send the response back through the HTTPCaller only if you haven't already fetched the entire dataset. Do some calculations to determine if you already have all the data, increment the offset/page number as needed. Stop, via the Tester, when you think it is done.
From the image you attached I think HTTPCaller only runs once but you fragment the JSON response into to many pieces.
+8
- Contributor
- 67 replies
-
24 October 2022
After a weekend and restarting my computer I'm not seeing the same behavior. Today the caller runs only once. Unfortunately I didn't keep the previous log so I can't verify I actually saw what I described in opening.
Other strange things were happening sometimes too, for instance sometimes the workspace said it ended with an error, but that was the only error in the log, all individual steps seemed to be okay. Concomitant with this, Inspect Cache would do nothing. No error, just no response. I tried purge cache and run again. Maybe it work on the next run and maybe not. Closing FME and re-opening the workbench also may or may not let it work again. So far today everything seems to be okay. I guess the rule of thumb is: "have you tried unplugging it and plugging it back in again?"
(And no this isn't a custom transformer. The workspace is complicated though. )
So for now: close, can't reproduce.
+20
- Contributor
- 563 replies
-
25 October 2022
After a weekend and restarting my computer I'm not seeing the same behavior. Today the caller runs only once. Unfortunately I didn't keep the previous log so I can't verify I actually saw what I described in opening.
Other strange things were happening sometimes too, for instance sometimes the workspace said it ended with an error, but that was the only error in the log, all individual steps seemed to be okay. Concomitant with this, Inspect Cache would do nothing. No error, just no response. I tried purge cache and run again. Maybe it work on the next run and maybe not. Closing FME and re-opening the workbench also may or may not let it work again. So far today everything seems to be okay. I guess the rule of thumb is: "have you tried unplugging it and plugging it back in again?"
(And no this isn't a custom transformer. The workspace is complicated though. )
So for now: close, can't reproduce.
For what you just described there might be some limiting factors on the API/server side.
Set Maximum Number of Concurrent HTTP Requests to 1 and limit the rate of requests per second.
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.