+3Hey everyone,
I have an API that hold historical data. I would need to retrieve all the data for the past year. I am using a HTTP Caller to retrieve data.
The API only allows for 30 days for one location to be retrieved at once.
I have 36 locations, and about 15 months worth of data to retrieve.
I already have workbench that retrieve one month for one location worth of data.
How can I create a loop that would go through all these locations and date list automatically?
Best answer by nielsgerrits
For each location:
Attached sample workspace demonstrating this.
Â
+60Multiple ways to solve this:
Â
As a custom transformer has its limitations and is more difficult to build, pre creating the requests is the preferred route.
This is a good starting point: https://community.safe.com/s/article/Getting-Started-with-API-Pagination
Â
+3This is a good starting point: https://community.safe.com/s/article/Getting-Started-with-API-Pagination
Â
Hi David,
Thank you, but I have seen those already and unfortunately they are not helping in my case as far as i could see.
I can only pass one location id and one date range each time. I cant see a way for the loop to input new variable on each iteration.
+3Multiple ways to solve this:
Â
As a custom transformer has its limitations and is more difficult to build, pre creating the requests is the preferred route.
Hello there,
the HTTPCaller can only accept one location ID and one date range.
I came up with a similar idea to yours before reading your post, please see below:
I was going to create a separate HTTPCaller for each location and then find a way to loop through the dates.
Could you clarify on on using the @dateTimeDiff() can help me here? How Would that help with the iteration? Or I need to create 432 features? There are 36 locations and each has at least 12 months worth of data.
Hi David,
Thank you, but I have seen those already and unfortunately they are not helping in my case as far as i could see.
I can only pass one location id and one date range each time. I cant see a way for the loop to input new variable on each iteration.
If we can assume that you know exactly how many http calls to make (36 * 15), then you could e.g. use two Cloners in series, the first one to make 36 clones, then the next one to make 15 clones of those again. For example:
It's then simply a matter of mapping location_no (0..35) and period_no (0..14) to whatever your API expects and pass it into a single HTTPCaller.
+60Hello there,
the HTTPCaller can only accept one location ID and one date range.
I came up with a similar idea to yours before reading your post, please see below:
I was going to create a separate HTTPCaller for each location and then find a way to loop through the dates.
Could you clarify on on using the @dateTimeDiff() can help me here? How Would that help with the iteration? Or I need to create 432 features? There are 36 locations and each has at least 12 months worth of data.
I think you need only one HTTPCaller, and generate features for all possible combinations. Then you can ask:
For location A, return data for day 1 to 30.
For location A, return data for day 31 to 61.
For location B, return data for day 1 to 30.
For location B, return data for day 31 to 61.
etc.
Â
You can use attribute input in both the Request URL (and body if POST), so you can use one HTTPCaller for all locations combined with all date ranges. You just need to prepare all the "questions" you want to ask, and feed those in the tool that asks, the HTTPCaller.
+3Sorry guys, but am I missing something about the cloners?
How would the cloners know which date range or location to produce based on the HTTPCaller above?
Have I created the caller wrong? I did consider passing the location ids and date ranges as attribute under value, but was not sure how to do that, is that what the cloners are suppose to do?
Sorry guys, but am I missing something about the cloners?
How would the cloners know which date range or location to produce based on the HTTPCaller above?
Have I created the caller wrong? I did consider passing the location ids and date ranges as attribute under value, but was not sure how to do that, is that what the cloners are suppose to do?
The following phrase was collapsed under my answer: It's then simply a matter of mapping location_no (0..35) and period_no (0..14) to whatever your API expects and pass it into a single HTTPCaller.
You can e.g. use a couple of AttributeValueMappers to e.g. map location_no 0 to 30215, etc.
+60Sorry guys, but am I missing something about the cloners?
How would the cloners know which date range or location to produce based on the HTTPCaller above?
Have I created the caller wrong? I did consider passing the location ids and date ranges as attribute under value, but was not sure how to do that, is that what the cloners are suppose to do?
For each location:
+60For each location:
Attached sample workspace demonstrating this.
Â
+3Thank you both, please allow me some time to go over the materials you provided, and get back to you.
+3For each location:
Hi,
Thank you so much, I understand now. One more thing. I need to send the date to the API as datetime, at which stage should i convert the date to datetime?
For each location:
REST APIs normally don't have a notion of a datetime data type, they usually expect a string that can be reliably interpreted as a datetime, e.g. ISO 8601 or similar. So my suggestion is to use the DateTimeConverter before the HTTPCaller, but only if necessary.
+3For each location:
Thank you so much both. I believe I have what I need now.
+60For each location:
Nice :)
