Question

List in httpfetcher url

  • 7 August 2015
  • 2 replies
  • 4 views

I'm setting up a workflow to download data from the police.uk API and eventually I want to set it up to run every few months or so to get updated data.

 

I'm using the street-crimes method to feed in a lng/lat string and get all crimes inside the polygon the coords generate. https://data.police.uk/docs/method/crime-street/ 

 

This initial bit is working well so far.

 

 

The method outputs a single months data and what I want to do is get the httpfetcher transformer to send multiple requests to get several months worth of data.  I don't mind it rerunning this each time as the police may update older crimes so getting updates would be fine.  

 

 

My thought was that I could build a list with each month I want to request data for and feed that into the target url httpfetcher.  I don't really know if that would work and I haven't really worked with lists yet so I'm not really sure how to create one.  

 

 

Any thoughts on this approach and any tips on creating the list of months initially?

2 replies

Userlevel 4
Hi

 

 

By looking at the API it seems that it wants a parameter like 2015-08 to specify the month. One way to solve it using FME is to create a feature (e.g. Creator or Cloner) for each month and then do a separate call. It is possible to calculate incrementing month and year numbers using e.g. the ModuloCounter, but is is (IMNSHO) a bit long-winded. 

 

 

Here is a PythonCaller script that will return an incrementing series of Year-Month values for each feature that passes through it:

 

 

---

 

import fme

 

import fmeobjects

 

 

class IncrementMonthYear(object):

 

    def __init__(self):

 

        self.month = int(FME_MacroValues['START_MONTH'])

 

        self.year = int(FME_MacroValues['START_YEAR'])

 

        self.count = 0

 

        

 

    def input(self,feature):

 

        if self.count > 0:

 

            self.month += 1

 

        if self.month > 12:

 

            self.year += 1

 

            self.month = 1

 

        self.count += 1

 

        feature.setAttribute('year_month', '%04d-%02d' % (self.year, self.month))

 

        self.pyoutput(feature)

 

--- It assumes you have defined two published parameters START_YEAR and START_MONTH that says when to start.

 

 

You can then expose the attribute "year_month" and use it in your HttpCaller parameter.

 

 

David
Thanks, I'll have a look at that, not done anything with Python before so will see if I can work it out!

Reply