Skip to main content

It's not up there with previous oddball ideas (sundials, QR codes) but randomly scheduled jobs certainly isn't something everyone will need. On the other hand, perhaps you might want to do data QA at random intervals, or sample a live feed at unscheduled times (what is the weather/traffic load at this particular point?)

 

I thought about how I might achieve this, and here are the methods I used and the results I got.

 

FME Server Scheduling

Here's what the scheduling dialog looks like in FME Server:

 

RandomSchedules1 

As you can see there are no options to randomize time, which is sensible really. If FME had to pick random times then the user would have to enter the probability of an event (say, 1:10,000) and FME would have to continuously test for that, taking up valuable resources while doing so.

 

Similarly the CRON command doesn't have the option, and the solutions I saw online involved more of a command-line CRON process.

 

So, I considered two viable FME processes:

 

  1. Have one workspace run and pick a random delay. That delay is passed to a Decelerator transformer. After the delay is done an FMEServerJobSubmitter runs the actual job.
  2. Have one workspace pick a random datetime, and set up an actual schedule to match by using the REST API.

 

The first option I don't like because it means you have a workspace running and taking up an engine, to do nothing but wait for the Decelerator. Not attractive. So I went for the REST API option...

 

My Random Haiku Tweeter

First of all I needed to find a task that could use randomized scheduling. I came up with an FME haiku poem generator that tweeted the result to my FMEEvangelist account. So as not to bore anyone, I'll tweet one haiku per week; but to avoid being predictable, I'll tweet at a random time/date.

 

Here's my haiku-generating workspace:

 

RandomSchedules2 

There's not much to see because I'm using a Python script that I found online. So... start, generate the haiku, and tweet it online:

 

RandomSchedules3Ignore the random h3 tag, which was part of a test for styling.

 

I now set up a schedule in Server, to run at a fixed date, and triggered it to make sure it works. So I have my workspace and I have my schedule, now to set up randomizing that schedule...

 

Scheduling with the API

The REST API lets me create a schedule, but it also lets me replace one too:

 

RandomSchedules4 

Replacement is good, because I don't want to create a new schedule every week and jam up the system with unused schedules.

 

So having found that API call, I set about using it in a workspace:

 

RandomSchedules5 

The workspace basically takes the current time, adds a random number of days (using the DateTimeCalculator to automatically take care of overflows), and then sets a random time of day. Notice use of the FME Hub transformer, DateTimeExploder. The HTTPCaller then sends the request to FME Server. Here's the key part of the request body:

 

RandomSchedules6When I run that workspace, it automatically updates the HaikuTweeter schedule with a new datetime, somewhere in the coming week. But how do I automate this workspace? With another schedule of course!

 

RandomSchedules7 

So I have two schedules. One of them (HaikuScheduler) runs at a set time every Sunday afternoon. It runs the randomization workspace that updates the second schedule (HaikuTweeter) through the REST API.

 

You can see that I'm scheduled to tweet a haiku on 5th November at 04:23, server time (I think). It's set to run every 30 days, but it won't get that far, because it will be replaced next Sunday.

 

So that's how you can set up a randomized schedule on FME Server. If you are interested, feel free to check out my workspaces, which are attached below - and feel free to suggest any words or improvements to my random haiku generator.

 

Issues

The big issue around scheduling is knowing what timezone to use. Build a "Basic" schedule and you use the Server timezone. Build a "Repeat on Interval" schedule and you use your local timezone. The API doesn't make clear which of these it is using, so we'll have to wait and see. We have a couple of requests for the developers to improve this design, and I think that would help a lot.

 

The other issue I have is password changes! It seems to be more and more common that systems are set up to expire passwords on a monthly or quarterly basis. It's great for security, but dreadful for automation. When my FME Server password expires, I suspect my REST API calls will start to fail until I update the workspace with a new connection. I could, of course, set up an automation to watch for failures and send me an email alert... but what happens if my email password expires at the same time?!

 

I'm not sure how to get around that one, but it has spurred me to investigate what might happen.

 

Thanks

Many thanks to my colleagues who provided assistance and suggestions, particularly Laura, Holly, Jen, Dean, and Rylan.

Be the first to reply!

Reply