Solved

Best Practices for Job Queue configuration

  • 8 September 2017
  • 2 replies
  • 6 views

Badge +8

We are looking to utilize Job Queues. Does anyone have any best practices?

  • We have 6 engines to utilize.
  • 2 folders with various real-time operation jobs that run every 3-10 minutes.
  • 1 folder for nightly ETL jobs
  • 1 folder for on-demand jobs that support a web API
  • Multiple folders for other on-demand/weekly to monthly scheduled jobs

Thanks in advance!

icon

Best answer by rylanatsafe 8 September 2017, 22:48

View original

2 replies

Badge +11

Great question! I see three main categories to consider...

Real-Time: At least one or two dedicated FME Engines. The big conditional is how long the jobs take to process. A frequency of "3-10 minutes" might warrant only one FME Engine, but if the translation time is equal or greater than the frequency... you might need more dedicated resources.

 

On-Demand: At least one dedicated FME Engine – no one likes to wait! Adjust as needed per frequency of requests.

 

Scheduled (Daily/Nightly Jobs): These could probably pull from the larger "Default" pool of FME Engines. Though it depends if you have any mission critical schedules that absolutely need to process as soon as they trigger – those might be better treated as "On-Demand".

 

If you reserve FME Engines for "Real-Time" and "On-Demand" – consider to also let them pull from the larger "Default" pool. In times of high demand the workload might be better spread out.

It's also worth investigating into the FME Server Job Directives.

Hope this helps! I'm sure there will be some other great replies from the FME Community.

Badge +22

We have three queues for jobs defined as light (<1 min), default and intensive (20min+).

 

the light queue has access to all engines, the default queue to all the engines but one, and the intensive to a specific engine.

 

This insures that there is at least one engine available for the web API interaction, though we also set the tm_priority of those jobs very high, so they can be squeezed in between the more intensive jobs on the other engines.

Reply