I too ran into the issue raised by @jurgen in https://community.safe.com/s/question/0D54Q000084ATxeSAG/how-can-i-merge-an-automation-flow-when-using-dynamic-workspaces - that my workspace that uses an automation writer, can't be funnelled into a merge-action. In the response, @daraghatsafe said that an enhancement request is on the way - but I needed to solve the issue before then.
This connection isn't possible right now
Problem: We need some way of knowing that all the workspaces had run successfully, before we'd run the final step. In my case, I didn't want to update the "I've imported all the data up to and including this date"-field, until all the imports were done.
Idea: What we need, is to build a state store, where we can keep track of the completed workspaces for one specific run of one specific automation. We could build this in f.ex. any database, but then we run into issues with having to create schemas/tables for each automation. Instead, I realized that the state store I wanted to work with already existed - the job log on FME Server! If we can ask the FME Server API to provide all successful jobs, we should be able to filter out the rest - surely?
Solution: Enter the MergeBox. It takes four parameters:
- FME Server webconnection (for the API interaction with the FME Server)
- The name of the automation
- The name of the first workspace of the automation
- The names of the workspaces that all have to be finished, before the MergeBox should give the go-ahead signal.
It then asks the FME Server API for all successful automation-jobs. Then we filter out those that match the name of the automation and sorting them by date. We then filter out (with the sorter) the most recent run for each workspace in the job log. Thereafter, using the variable setter / retriever, we extract the finish time for the first workspace in the automation. This step is to give us a baseline time that we can filter against. Since the MergeBox is run inside the same automation, we can expect all the runs we're looking for to happen after the first workspace is run. If, for example, one of them ran yesterday but hasn't run today yet - we can filter it out.
Having extracted the start time for this run of the automation, we check all the workspaces we've got (i.e. the latest run of any workspace with started by the correct automation name) against that date, keeping only those who've started after the start time.
Thereafter, we join them against an exploded list from the comma-separated workspace name parameter. By exploding this list, we also know how many matches we should have. Each workspace in the comma-separated list parameter, should have a match, if the MergeBox should give the green light.
After joining all the filtered out candidates, against the exploded list, we re-aggregate and check that we've got as many matches as we expect. If we do, the automations-writer outputs a dummy feature that serves as the green light. In this case, we don't actually require anything from the input - we simply need to know everything's run successfully.
TL;DR:
In an automation with an automation writer, the "merge messages"-functionality doesn't work. The MergeBox introduces a general solution to this. It works by checking for when red, green, purple and black workspaces all have successfully processed in this run of the automation. If so, the MergeBox releases a feature to let the light blue workspace run.
I hope this helps someone who's run into the same problems as me - also, is there a better way to filter out the current run of the automation? I tried to look for an ID, but it seems to be automation-level and not this specific run of it. If you've got automations that are triggered by something external and you can have two running in parallel, this solution is no help to you, unless it can be based on some automation-run-ID.
Best answer by lauraatsafe
View original
