Skip to main content
I'm setting up a workflow to download data from the police.uk API and eventually I want to set it up to run every few months or so to get updated data.

 

I'm using the street-crimes method to feed in a lng/lat string and get all crimes inside the polygon the coords generate. https://data.police.uk/docs/method/crime-street/ 

 

This initial bit is working well so far.

 

 

The method outputs a single months data and what I want to do is get the httpfetcher transformer to send multiple requests to get several months worth of data.  I don't mind it rerunning this each time as the police may update older crimes so getting updates would be fine.  

 

 

My thought was that I could build a list with each month I want to request data for and feed that into the target url httpfetcher.  I don't really know if that would work and I haven't really worked with lists yet so I'm not really sure how to create one.  

 

 

Any thoughts on this approach and any tips on creating the list of months initially?
Hi

 

 

By looking at the API it seems that it wants a parameter like 2015-08 to specify the month. One way to solve it using FME is to create a feature (e.g. Creator or Cloner) for each month and then do a separate call. It is possible to calculate incrementing month and year numbers using e.g. the ModuloCounter, but is is (IMNSHO) a bit long-winded. 

 

 

Here is a PythonCaller script that will return an incrementing series of Year-Month values for each feature that passes through it:

 

 

---

 

import fme

 

import fmeobjects

 

 

class IncrementMonthYear(object):

 

    def __init__(self):

 

        self.month = int(FME_MacroValuesi'START_MONTH'])

 

        self.year = int(FME_MacroValuesa'START_YEAR'])

 

        self.count = 0

 

        

 

    def input(self,feature):

 

        if self.count > 0:

 

            self.month += 1

 

        if self.month > 12:

 

            self.year += 1

 

            self.month = 1

 

        self.count += 1

 

        feature.setAttribute('year_month', '%04d-%02d' % (self.year, self.month))

 

        self.pyoutput(feature)

 

--- It assumes you have defined two published parameters START_YEAR and START_MONTH that says when to start.

 

 

You can then expose the attribute "year_month" and use it in your HttpCaller parameter.

 

 

David
Thanks, I'll have a look at that, not done anything with Python before so will see if I can work it out!

Reply