Hello,
I'm looking for a method that will allow me to extract a summary from a JSON, but due to the size of each JSON, fragmenting the data first would be extremely time consuming.
For a bit of background, I want to extract daily data from the WebTRIS traffic counter API, with the aim of creating a summary of average traffic speed for the sites in question. The API allows for the download of multiple report styles, however only the daily report allows for the extraction of average speed. The report provides data at 15 minute intervals for each counter for the periods of time that the counter is active. If I wanted to extract the average traffic speed over a year, assuming the counter was active for the whole year, this might mean summarising 35,040 records, per site.
I've looked at using the JSON fragmentor, but due to the potential size of the data that would require summarising, this would create an excessive number of records per site. What I would ideally like to do is to be able to take the average speed attribute contained within the nested JSON, treat it like a list, and summarise the list without exploding it first.
I've thought about trying to reduce the number of records I get from the API, but as I need an annual summary and due to the limitations of the API itself, it is not very flexible, with only the site ID and date range allowing for any control over what records are downloaded.
If anyone has any tips on how I might do this, or perhaps an alternate method I might use to be able to extract an average speed summary for a site, that would be much appreciated.