FME Flow 2024.1 Build 24612 - distributed setup

I’m raising with Safe but if anyone has any bright ideas before Vancouver wakes up :-)

Recently upgraded our pre-production environment to the above version. Noticed an issue however with some jobs getting stuck in a queued state and never moving on.

The jobs are triggered by automation that runs every 10 minutes and assigns the jobs to a specific queue which in turn has two engines available. Sometimes the job gets submitted and runs successfully, sometimes the job gets put in a queue where it remains forever. 

e.g. the job triggered at 11:20 was processed successfully, the job triggered at 11:30 ended up in the queue, the job triggered at 11:40 was processed as expected

The automation log (running in debug mode) doesn’t show any errors or warnings, the queued jobs show as submitted but with no other information

Other perhaps relevant information, the jobs should also expire once in the queue beyond a certain length of time and that is not happening, I also cannot delete these jobs from the queue (I can delete other jobs in the queue in this environment). The job numbers aren’t sequential, the job number for the job sent to the queue at 11:30  is 40 numbers higher than the job processed successfully at 11:40

The job queues themselvers show 0 running, x queued but with a red icon which shows at Waiting for Engines on hover, however all engines are showing as idle with no jobs running.

Any suggestions, log files to dig into etc.?