We are looking to utilize Job Queues. Does anyone have any best practices?
Thanks in advance!
Great question! I see three main categories to consider...
Real-Time: At least one or two dedicated FME Engines. The big conditional is how long the jobs take to process. A frequency of "3-10 minutes" might warrant only one FME Engine, but if the translation time is equal or greater than the frequency... you might need more dedicated resources.
On-Demand: At least one dedicated FME Engine – no one likes to wait! Adjust as needed per frequency of requests.
Scheduled (Daily/Nightly Jobs): These could probably pull from the larger "Default" pool of FME Engines. Though it depends if you have any mission critical schedules that absolutely need to process as soon as they trigger – those might be better treated as "On-Demand".
If you reserve FME Engines for "Real-Time" and "On-Demand" – consider to also let them pull from the larger "Default" pool. In times of high demand the workload might be better spread out.
It's also worth investigating into the FME Server Job Directives.
Hope this helps! I'm sure there will be some other great replies from the FME Community.
We have three queues for jobs defined as light (<1 min), default and intensive (20min+).
the light queue has access to all engines, the default queue to all the engines but one, and the intensive to a specific engine.
This insures that there is at least one engine available for the web API interaction, though we also set the tm_priority of those jobs very high, so they can be squeezed in between the more intensive jobs on the other engines.