I had an engine queue pileup where job where just getting added to the queue and never ran. Since there were no failures none of my automated notifications were going out. So I completely missed this.
Is there a way I can set up hourly polling to check the number of jobs in the queue? Did a quick search for doing this with Rest API but nothing jumped out at me. Thanks for any suggestions you may have.
Have a look at the /transformations/jobs/queued endpoint
This will give you a json response with the number of queued jobs
{
"offset": -1,
"limit": -1,
"totalCount": 0,
"items": l]
}Then using something like uptimerobot, set a Keyword Not Exists trigger to find "totalCount": 0.
Then have it check every minute, and configure it so that if it can't find the keyword for 5 minutes it sends a notification.
Ideally you could parse the json and say "if count > X for Y minutes send email"
Have a look at the /transformations/jobs/queued endpoint
This will give you a json response with the number of queued jobs
{
"offset": -1,
"limit": -1,
"totalCount": 0,
"items": l]
}Then using something like uptimerobot, set a Keyword Not Exists trigger to find "totalCount": 0.
Then have it check every minute, and configure it so that if it can't find the keyword for 5 minutes it sends a notification.Ideally you could parse the json and say "if count > X for Y minutes send email"
Great tip @hkingsbury - love the use here of UptimeRobot here.
Thanks, guys - that was tremendously helpful. I have a working solution deployed already,
Not experience with UptimeRobot. - My instinct was to build a workbench with an HTTPCaller... well that could south if you plan on running that on FME Server and the jobs gets shoved in the same backed up queue you're trying to monitor.
In my case, I may actually be okay because my jam happened in a dedicated queue on a remote engine-only machine... so processes on the main core machine were still processing. The workspace grabs the count of queued job and if >x, reads the workbench names and throws them into an email. Alternatives that come to mind given my arsenal or tools are:
Either way, I love to have this working and for the first time I'm getting live insights into the status of the queue without needing to hop on the web console. Thanks again!
Thanks, guys - that was tremendously helpful. I have a working solution deployed already,
Not experience with UptimeRobot. - My instinct was to build a workbench with an HTTPCaller... well that could south if you plan on running that on FME Server and the jobs gets shoved in the same backed up queue you're trying to monitor.
In my case, I may actually be okay because my jam happened in a dedicated queue on a remote engine-only machine... so processes on the main core machine were still processing. The workspace grabs the count of queued job and if >x, reads the workbench names and throws them into an email. Alternatives that come to mind given my arsenal or tools are:
Either way, I love to have this working and for the first time I'm getting live insights into the status of the queue without needing to hop on the web console. Thanks again!
We've also explored the idea of have the dev server monitor the production server and vice-versa
We've also explored the idea of have the dev server monitor the production server and vice-versa
Clever idea there
@hkingsbury - that's how I tested it for me. But I am not sure if leaving that way violates licensing terms. Just glad to have found another way to monitor this.
@hkingsbury - that's how I tested it for me. But I am not sure if leaving that way violates licensing terms. Just glad to have found another way to monitor this.
Whats your concern about the licensing terms?
Whats your concern about the licensing terms?
Oh, I don't know.... using anything that's running in Test/Dev consuming a Test license as if it were Production, always has been worried about getting into hot water with the licensing police. Maybe someone from Safe could chime in here... wondering what's allowed.
Whats your concern about the licensing terms?
Just to clarify any confusion, "Software under a trial or evaluation license cannot be used for any production purposes and must be used solely for the purposes of evaluations, demonstrations, proof of concepts, prototypes, and self-learning" You can find some more information about our licensing here: https://www.safe.com/legal/fme-sla/
Hope this helps!
@danminneyatsaf - and just to clarify from my end. I'm fully licensed here. What I was calling 'Test' wasn't an EVAL license but what Safe may refer to as "Staging". Basically a permanent test environment. Monitoring your Prod environment with a process in Test seems innocuous if you're licensed for both types of environment. But that's where my doubts came in.
@danminneyatsaf - and just to clarify from my end. I'm fully licensed here. What I was calling 'Test' wasn't an EVAL license but what Safe may refer to as "Staging". Basically a permanent test environment. Monitoring your Prod environment with a process in Test seems innocuous if you're licensed for both types of environment. But that's where my doubts came in.
@agelfert, replying with my official Safe account this time.
I can say that monitoring a Production environment with a process in Test environment fits under the umbrella of testing/staging, so I wouldn't worry about this! Using it for that purpose is all good with us 😄