Hello FME'ers,

Here's a question-of-the-week post…

This one wasn't posted as a "question", but was instead a comment from @sigtill on the tutorial for FME Server automations:

Q) Can you create a Key within a workspace - expose it - and then use it in automations? For instance an AttributeCreator and the attributes myattribute = myValue - and then use the "myattribute" within the Automations?

A) Yes you can! Below are the steps to pass attributes to a Server automation...

Let's cover an automation in brief.

An Automation in FME Server is a tool that helps carry out tasks automatically (yes, I know Automation = Automatically! Who knew?)

So we can carry out a task at a set time, or as triggered by some other action. The task doesn't need to be just running a Workspace - there are many different tasks you can do - but it usually is a workspace because that's where we have the most processing tools.

Here's a sample automation:

We watch for new files in Dropbox. When one appears we run a workspace. If the workspace succeeds, we copy the output to an FTP site. If the workspace fails, we send an email. All simple enough.

In FME2019 all we could tell was whether the workspace passed or failed, but some additional details were also passed as an Event Key...

So in FME Workbench, the main unit of information (other than geometry) is an attribute. The equivalent unit in Server is a "key". There are several fixed keys that are available, called Event Keys:

Those are the Event Keys for the workspace when it failed. You could use this information in the email that you send (for example):

But there was a barrier preventing keys passing between workspace and Server. So the question is, how - in FME 2020 - can we send information from a workspace attribute to a Server key?

The answer to that question is the FME Server Automations Writer. The Automations Writer is a new format specially designed to pass information to Server.

For example, my workspace might look like this:

My workspace (called myWorkspace) has an Automations writer, with a single feature type called MyFeatureType. It has an attribute called myAttribute, with the value myValue.

In FME Server, I've set up this layout:

The important thing is that the Run Workspace object has extra keys now:

Oh look! There is a key called route.MyFeatureType.myAttribute. I can now use that key in my Automation. In fact, I set up the Filter action to test for the value of myAttribute, and route it to log different messages on that basis.

So it’s very simple, but there are a few "gotchas" to be aware of...

It took me a while to figure this out and the reason was this: when you select the workspace in Run Workspace, it doesn't immediately show the Feature Type on the "canvas" or the related event keys!

See? I picked myWorkspace and yet there are no keys! The trick is to click the Apply button to apply the workspace, and then the feature types and attribute Event Keys appear.

The second gotcha is that none of the attribute Event Keys appear on the passed/failed ports. They only appear on the Feature Type output port:

So for a while I was looking in the wrong place. But it makes sense really because I could have multiple feature types, and you wouldn't want all of the keys for all of the feature types, on the same port. Additionally, if the workspace failed you wouldn't get event keys anyway.

One final thing to consider, each feature emerging from the workspace sends a separate message through your automation. On one hand, that's great. You can set up batch processing (for example) where each feature represents a dataset to process; because this is Server you automagically get parallel processing!

But I only want a single message; so I add a Sampler transformer to pass only a single feature.

Oh, one final thing; it's not specific to attributes/keys, but I find it hard to remember: the Log tool in an Automation doesn’t write to the workspace log! It should be obvious really, since it's a separate action and you don't need a workspace in an Automation anyway. So, where can I find the logs? Here:

Now that I've written it down, maybe I'll remember where to look!

So that's how to pass attribute values from a workspace to a Server automation event key. It's actually pretty straightforward.

If it helps, I made a brief movie here.

I know I often go on about how "this FME is the best ever", but it really feels that the whole FME platform has taken a quantum leap forward in terms of enterprise-level setup. With this, enhancements to the automations tool as a whole, dynamic engines, app enhancements, and a lot more; there really is a lot to see in 2020.

At the very least, go view Don and Jen's recorded webinar, to see these tools in action.

This was a bumper week for notable questions!

• Merge Mode and Feature Types
  • This question from @pascalmortz was close to being the question-of-the-week, but lost out to Server keys. In brief, it's an interesting look at the interaction of feature types, source data, Merge Filters, and Feature-Types-To-Read. As I like to say, a Merge Filter is like a highway toll booth. Set it to allow <All> traffic and anything can pass through. Feature-Types-To-Read plays a similar role, but at a higher level. It's like the law says "tractors aren’t allowed on a highway"; so even if the tollbooth says <All>, tractors won't pass because they're not permitted traffic! So don't permit layer x and layer x won't enter the workspace, even if the merge filter allows it.
• ShortestPathFinder returns different routes for same From-To locations
  • The ShortestPathFinder is a favourite transformer; a geographic/spatial problem, it’s a member of the interesting NP problem set; it has lots of different parameters (I'm a control freak); it has great performance; and though I know how the algorithms work, there's still an element of mystery!
  • Anyway, @afod wondered how it returns different routes for the same start/end points. The answer is that the transformer starts with a random route and tries to improve it. If you have enough iterations and verifications, then you always get the same. Otherwise you get routes that are close, but not necessarily identical.
• Create Google Maps Traffic equivalent
  • It sounds like @stevemossfeldt has a very interesting project. How do you take a series of GPS tracks along the same road, and merge them together to get a single route? It's a new question so there aren't many answers yet. I think this one is going to produce some very interesting suggestions.
• No Module Named FME Objects in 2020
  • User @koen asks why their Python script throws an error in FME2020. As @virtualcitymatt notes, the answer is the script uses Python 2.7, and FME no longer installs Python 2.7 by default. Instead, there is a choice in the installation wizard where you must specifically pick Python 2.7 to be installed. That’s because bundling libraries that include potential security holes is a red flag for us! So while the immediate problem is solved, I really recommend upgrading your scripts to a newer Python, if possible.
• Folder Existence Checker in Python
  • It's an interesting requirement from @gylona, but what I really wanted to concentrate on was the answer from @ebygomm, because it's something that catches me out. When you place a PythonCaller, the default script includes a template for a Function (processFeature) and a Class (FeatureProcessor). Normally you use one of these and delete the other, which was issue #1 here. But also the Class or Function to Process parameter is set to FeatureProcessor by default. i.e. it defaults to the Class, not the Function. So if you use the Function (and keep it named as processFeature) then you need to change this parameter. Experienced Pythonistas won't get confused, but newbies like me miss that all of the time!

OK, stay safe folks, and have a great weekend.