It does require a free engine, but an occasional check to see if there's more than x jobs of any user in the queue and if so, moving them to another engine / assigning them lower priority would do the trick I suppose.
It does require a free engine, but an occasional check to see if there's more than x jobs of any user in the queue and if so, moving them to another engine / assigning them lower priority would do the trick I suppose.
Well this sounds exactly the kind of thing I'd like to do! Is there a way though to move already submitted jobs to a different queue? I couldn't figure that out. Once a job is submitted and queued it seems like it can't be changed. Perhaps I'm missing something here. I suppose I could always cancel and then resubmit them with lower priority.
But there are topics set up to handle fails which would make that a little messy
Well this sounds exactly the kind of thing I'd like to do! Is there a way though to move already submitted jobs to a different queue? I couldn't figure that out. Once a job is submitted and queued it seems like it can't be changed. Perhaps I'm missing something here. I suppose I could always cancel and then resubmit them with lower priority.
But there are topics set up to handle fails which would make that a little messy
Hmmm... I was under the impression that the API offers a way to assign tags to queued jobs, but it doesn't appear to do that.
I was thinking about conterra's Dynamic Engine Management solution, but that just adds extra engines to queues if there's more than X jobs in a monitored queue, so that's easier that influencing specific jobs.
Well this sounds exactly the kind of thing I'd like to do! Is there a way though to move already submitted jobs to a different queue? I couldn't figure that out. Once a job is submitted and queued it seems like it can't be changed. Perhaps I'm missing something here. I suppose I could always cancel and then resubmit them with lower priority.
But there are topics set up to handle fails which would make that a little messy
Yeah exactly this - I'm considering temporarily redirecting all new requests to a higher priority queue with a higher priority using the "tm_tag" directive. This will, hopefully, priority new requests over the backlog.
Would this not be a good case for Job Routing and Engine Assignment rules?
For one of our clients, we dump 100s of jobs onto server overnight, and occasionally 10s of jobs during the day
We have the following rule:
Which basically means, at any point, these jobs will run on machine 02 using standard engines. If that queue gets above 10, all dynamic engines across all hosts can pull jobs from that queue
Would this not be a good case for Job Routing and Engine Assignment rules?
For one of our clients, we dump 100s of jobs onto server overnight, and occasionally 10s of jobs during the day
We have the following rule:
Which basically means, at any point, these jobs will run on machine 02 using standard engines. If that queue gets above 10, all dynamic engines across all hosts can pull jobs from that queue
Yeah actually maybe this would help. We're still stuck on FME2022 though. But it's nice to see it's easy to use the queue length allow for adding more engines.
I don't really want to add more engines though, just prioritize newer jobs following a glut submission. Well I mean I'd love to add more engines but just limited by the server size.
Perhaps I just need to rethink the way I'm currently handling the queues in the first place. It seems there are not options to be a lot smarter about it.
Well this sounds exactly the kind of thing I'd like to do! Is there a way though to move already submitted jobs to a different queue? I couldn't figure that out. Once a job is submitted and queued it seems like it can't be changed. Perhaps I'm missing something here. I suppose I could always cancel and then resubmit them with lower priority.
But there are topics set up to handle fails which would make that a little messy
I'm suspecting you'll have to a) read the queued job information (notably the published parameter values), b) cancel the job and then c) resubmit it with a queue specifier. It shouldn't be too hard using the REST API, I believe step a) returns more or less the format needed for step c).
Well this sounds exactly the kind of thing I'd like to do! Is there a way though to move already submitted jobs to a different queue? I couldn't figure that out. Once a job is submitted and queued it seems like it can't be changed. Perhaps I'm missing something here. I suppose I could always cancel and then resubmit them with lower priority.
But there are topics set up to handle fails which would make that a little messy
Yep, this was my thinking. I would need to first temporarily disable the subscriptions as to not send out failure notifications.
