If automation 2 is dependant on automation 1 is there a reason for not just making this 1 automation?

Yes there is.

For clarity's sake, I'm importing data in auto1, and doing post-processing in auto2. There may be multiple automations run in parallel in auto1, but I just wanted to make my question clearer.

And no, auto2 is not dependent on what happens in auto1, it just cannot run before all auto1's are done running. Doing so will corrupt data.

Right, I think I understand. I guess the problem comes from submitting the auto1 in parallel because there's no way to get the result of the jobs. They just start.

I guess one not great method is to add in a 'pre' step for auto2 which essentially just checks the job table using the REST API? Every 5 seconds or so...

Once a job with workspacex.fmw is no longer found with a status of RUNNING or QUEUED then you can trigger the other jobs.

There are probably better ways but that's how I'd do it - feels pretty hacky though

Alas, I cannot afford to have a running job waiting for a certain condition to occur, as I have only 2 engines on the system, and that would tie up 50% of my resources doing basically nothing. Step 1 is the important one, and needs all the ressources they can get. I really just need a simple way to coordinate steps 1 and 2.

Surely once all of the jobs from automation 1 are submitted then you can trigger automation 2? The first job in automation 2 should not take any resources away from automation 1 as all of the jobs from automation 1 should be finished except the last one?

Or is it the case that automation 2 gets triggered randomly but should not start until all jobs from automation 1 are complete?

Alas, the parallel processes are truly parallel, they are triggered by a single notification trigger in the automation system, so they run in an arbitrary order.

I'm instead looking into having a semphore hold a protected count variable, having each automation decrease the count until zero, and have a post process workspace run in every automation, checking for a count of zero before triggering step 2.

But the whole "semaphore/protected count" is not built-in, so I'll need to build this from the ground up.

And hard-coding the initial count somewhere along the way don't sit well with me, anyway.

Have you had a look here (note the split merge block)?

https://community.safe.com/s/article/job-orchestration-with-automations

There are some pretty good tips here - I imagine something like this could work?

It's not just workspaces, it's several automations, I need to handle. Each automation contains a chain of workspaces, that implement some business logic, that I'm reusing for several datasets. I do not want to break open those automations, as this would defeat the purpose of the encapsulation.

And again, they're completely independent wrt. each other, and triggered by a single notification.

Well, after much hard thinking, I ended up with a fairly simple solution. Unfortunately it's partly hardcoded, so it's not optimal, but it'll work for now.

I used the database to build a simple semaphore-mechanism with a protected count.

My inital workspace creates the semaphore and sets its expected count (hardcoded). It then issues a topic, that a number of automation (data import) react upon, and they all finish off by calling a workspace, that decreses the semaphore count by one, and issues the next topic (for post-processing) if the count is now zero.

The semaphore is implemented as a table and two stored procedures in my database.

My workflow is then as such:

FMW (create semaphore,issue topic 1) - Automations react to topic 1, do their thing, and call FMW(decrease semaphore count,issue topic 2 if count is now 0) - Automations react to topic 2, do their thing.