• EN
Question

Anyone have any ideas or tips on how to move submitted jobs to a new queue?

  • 5 July 2023
  • 8 replies
  • 11 views
virtualcitymatt
Rank
Userlevel 5
Badge +28

We (rarely) get a glut of job submissions to some of our FME Server queues (Running on FME Cloud).

 

These glut of jobs usually come from someone who has automated a bunch of submissions (we're totally fine with that). We want these jobs to be processed, but at the same time we still want to be able to handle other user requests in a timely manner.

 

I created an idea over here which would help me: https://community.safe.com/s/bridea/a0rDm000001Sf7kIAC/option-to-move-submitted-queued-jobs-to-new-queue

It outlines the scenario in more details.

 

Just looking for tips or idea on how to work around this.

 

The service in question is essentially just a clip and snip but depending on what the user wants the request can take anywhere between 10 seconds and an hour or so to process.

8 replies

redgeographics
Rank
Userlevel 5
Badge +26

It does require a free engine, but an occasional check to see if there's more than x jobs of any user in the queue and if so, moving them to another engine / assigning them lower priority would do the trick I suppose.

 

virtualcitymatt
Rank
Userlevel 5
Badge +28

It does require a free engine, but an occasional check to see if there's more than x jobs of any user in the queue and if so, moving them to another engine / assigning them lower priority would do the trick I suppose.

 

Well this sounds exactly the kind of thing I'd like to do! Is there a way though to move already submitted jobs to a different queue? I couldn't figure that out. Once a job is submitted and queued it seems like it can't be changed. Perhaps I'm missing something here. I suppose I could always cancel and then resubmit them with lower priority.

But there are topics set up to handle fails which would make that a little messy

redgeographics
Rank
Userlevel 5
Badge +26

Well this sounds exactly the kind of thing I'd like to do! Is there a way though to move already submitted jobs to a different queue? I couldn't figure that out. Once a job is submitted and queued it seems like it can't be changed. Perhaps I'm missing something here. I suppose I could always cancel and then resubmit them with lower priority.

But there are topics set up to handle fails which would make that a little messy

Hmmm... I was under the impression that the API offers a way to assign tags to queued jobs, but it doesn't appear to do that.

 

I was thinking about conterra's Dynamic Engine Management solution, but that just adds extra engines to queues if there's more than X jobs in a monitored queue, so that's easier that influencing specific jobs.

virtualcitymatt
Rank
Userlevel 5
Badge +28

Well this sounds exactly the kind of thing I'd like to do! Is there a way though to move already submitted jobs to a different queue? I couldn't figure that out. Once a job is submitted and queued it seems like it can't be changed. Perhaps I'm missing something here. I suppose I could always cancel and then resubmit them with lower priority.

But there are topics set up to handle fails which would make that a little messy

Yeah exactly this - I'm considering temporarily redirecting all new requests to a higher priority queue with a higher priority using the "tm_tag" directive. This will, hopefully, priority new requests over the backlog.

hkingsbury
Rank
Userlevel 5
Badge +29

Would this not be a good case for Job Routing and Engine Assignment rules?

 

For one of our clients, we dump 100s of jobs onto server overnight, and occasionally 10s of jobs during the day

 

We have the following rule:image 

Which basically means, at any point, these jobs will run on machine 02 using standard engines. If that queue gets above 10, all dynamic engines across all hosts can pull jobs from that queue

Check out the Abley FME Firesides on Youtube: https://www.youtube.com/@ableyltd
virtualcitymatt
Rank
Userlevel 5
Badge +28

Would this not be a good case for Job Routing and Engine Assignment rules?

 

For one of our clients, we dump 100s of jobs onto server overnight, and occasionally 10s of jobs during the day

 

We have the following rule:image 

Which basically means, at any point, these jobs will run on machine 02 using standard engines. If that queue gets above 10, all dynamic engines across all hosts can pull jobs from that queue

Yeah actually maybe this would help. We're still stuck on FME2022 though. But it's nice to see it's easy to use the queue length allow for adding more engines.

I don't really want to add more engines though, just prioritize newer jobs following a glut submission. Well I mean I'd love to add more engines but just limited by the server size.

 

Perhaps I just need to rethink the way I'm currently handling the queues in the first place. It seems there are not options to be a lot smarter about it.

david_r
Rank
Userlevel 5

Well this sounds exactly the kind of thing I'd like to do! Is there a way though to move already submitted jobs to a different queue? I couldn't figure that out. Once a job is submitted and queued it seems like it can't be changed. Perhaps I'm missing something here. I suppose I could always cancel and then resubmit them with lower priority.

But there are topics set up to handle fails which would make that a little messy

I'm suspecting you'll have to a) read the queued job information (notably the published parameter values), b) cancel the job and then c) resubmit it with a queue specifier. It shouldn't be too hard using the REST API, I believe step a) returns more or less the format needed for step c).

virtualcitymatt
Rank
Userlevel 5
Badge +28

Well this sounds exactly the kind of thing I'd like to do! Is there a way though to move already submitted jobs to a different queue? I couldn't figure that out. Once a job is submitted and queued it seems like it can't be changed. Perhaps I'm missing something here. I suppose I could always cancel and then resubmit them with lower priority.

But there are topics set up to handle fails which would make that a little messy

Yep, this was my thinking. I would need to first temporarily disable the subscriptions as to not send out failure notifications.

Reply


Community Stats

28,866
Posts
109,982
Replies
36,671
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.