Skip to main content
  • EN

I too ran into the issue raised by @jurgen​  in https://community.safe.com/s/question/0D54Q000084ATxeSAG/how-can-i-merge-an-automation-flow-when-using-dynamic-workspaces - that my workspace that uses an automation writer, can't be funnelled into a merge-action. In the response, @daraghatsafe​ said that an enhancement request is on the way - but I needed to solve the issue before then.

 

 

Screenshot 2020-09-21 132313

This connection isn't possible right now

 

Problem: We need some way of knowing that all the workspaces had run successfully, before we'd run the final step. In my case, I didn't want to update the "I've imported all the data up to and including this date"-field, until all the imports were done.

 

Idea: What we need, is to build a state store, where we can keep track of the completed workspaces for one specific run of one specific automation. We could build this in f.ex. any database, but then we run into issues with having to create schemas/tables for each automation. Instead, I realized that the state store I wanted to work with already existed - the job log on FME Server! If we can ask the FME Server API to provide all successful jobs, we should be able to filter out the rest - surely?

 

Solution: Enter the MergeBox. It takes four parameters:

  • FME Server webconnection (for the API interaction with the FME Server)
  • The name of the automation
  • The name of the first workspace of the automation
  • The names of the workspaces that all have to be finished, before the MergeBox should give the go-ahead signal.

 

It then asks the FME Server API for all successful automation-jobs. Then we filter out those that match the name of the automation and sorting them by date. We then filter out (with the sorter) the most recent run for each workspace in the job log. Thereafter, using the variable setter / retriever, we extract the finish time for the first workspace in the automation. This step is to give us a baseline time that we can filter against. Since the MergeBox is run inside the same automation, we can expect all the runs we're looking for to happen after the first workspace is run. If, for example, one of them ran yesterday but hasn't run today yet - we can filter it out.

 

Having extracted the start time for this run of the automation, we check all the workspaces we've got (i.e. the latest run of any workspace with started by the correct automation name) against that date, keeping only those who've started after the start time.

Thereafter, we join them against an exploded list from the comma-separated workspace name parameter. By exploding this list, we also know how many matches we should have. Each workspace in the comma-separated list parameter, should have a match, if the MergeBox should give the green light.

 

After joining all the filtered out candidates, against the exploded list, we re-aggregate and check that we've got as many matches as we expect. If we do, the automations-writer outputs a dummy feature that serves as the green light. In this case, we don't actually require anything from the input - we simply need to know everything's run successfully.

 

TL;DR:

 

In an automation with an automation writer, the "merge messages"-functionality doesn't work. The MergeBox introduces a general solution to this. It works by checking for when red, green, purple and black workspaces all have successfully processed in this run of the automation. If so, the MergeBox releases a feature to let the light blue workspace run.

Screenshot 2020-09-21 131931

 

I hope this helps someone who's run into the same problems as me - also, is there a better way to filter out the current run of the automation? I tried to look for an ID, but it seems to be automation-level and not this specific run of it. If you've got automations that are triggered by something external and you can have two running in parallel, this solution is no help to you, unless it can be based on some automation-run-ID.

Hi @fhilding​ I wanted to say thank you so much for sharing your solution for this merging trouble in Automations! It's a great example of the power of what you can build with a workspace and some creativity. We'll certainly still keep working on building this functionality into the Merge action, but this is a great way to get up and running without having to wait for that change.

We really appreciate you taking the time to post this to the FME Community. I think there are a good number of other FME'ers out there who will benefit from this. I think I may borrow this for some workflows I'm building now too. :)

Hi @fhilding​ I wanted to say thank you so much for sharing your solution for this merging trouble in Automations! It's a great example of the power of what you can build with a workspace and some creativity. We'll certainly still keep working on building this functionality into the Merge action, but this is a great way to get up and running without having to wait for that change.

We really appreciate you taking the time to post this to the FME Community. I think there are a good number of other FME'ers out there who will benefit from this. I think I may borrow this for some workflows I'm building now too. :)

@lauraatsafe​  - do you have any progress update on this, or is it still a future implementation candidate?

@lauraatsafe​  - do you have any progress update on this, or is it still a future implementation candidate?

Hi @fhilding​ Yes! We do have an update on this! In FME 2021.1, we introduced an object called a Split-Merge Block to Automations. This will allow for merging workspaces that contain automations writers. There is some information about how they work in this article: https://community.safe.com/s/article/job-orchestration-with-automations under the section title "7. Add a Merge Action"

@lauraatsafe​  - do you have any progress update on this, or is it still a future implementation candidate?

Fine work @lauraatsafe​ and crew :)

Reply


Community Stats

30,099
Posts
114,258
Replies
37,729
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.