Question

FME SERVER Job Time out


Badge
Hi All,

 

Are there possibilities to specify a finite timeout for a job? I need to stop a job that is running and is never ended without using admin interface?

 

 

Thanks an advance;

 

Farfar

17 replies

Userlevel 4
Hi,

 

 

have you looked at this article (http://fmepedia.safe.com/articles/Error_Unexpected_Behavior/FME-Server-Jobs-not-Processing)?

 

 

David
Badge
Hi David,

 

Yes i looked this article before. But it's not resolved my problem. I explain my problem :

 

If i run my job with coorect input shape file, the job is executed in 3 minutes. But if i use uncorrect input file, this job is executed in 3 hours and sometimes it doesn't ends.  My idea is to stop this job if it is running more than 5 minutes for example.  

 

Thanks in advance.

 

Farfar
Badge

Hi Farfar,

Do you have any answer to your question?

I am facing the same issue and can't find any answer. Thanks

.JP

Badge +10

An interesting challenge... I do recall that there is an option somewhere in FME Server that allows jobs to timeout, BUT, I have a feeling that is related to the queue. I.e. If a job sits in the queue waiting to be processed for greater that x minutes/hours before being processed, it may no longer be viable so ignore it.

Obviously that won't help you.

So by way of a consolation prize please find attached a workspace that should do what you need. You'll need to add the first section and the last into your process, right up front and at the very end of any existing transformers. This workflow then simple terminates the job if it goes on for more than 5 minutes; a value that you can of course alter.

Hope it helps, but just beware, with the super quick testing I have done, the time difference will be actually quite a bit less than the FME Session duration, so you'll need to do some experimentation with this if you use it... or just build on the principle.

Download: transformer-sequence-to-terminate-long-run-jobs.fmw

cc. @jpvo

Userlevel 4
Badge +25

The other solution I can think of is that you have an FME workspace that runs all the time (or is run on a schedule every minute). It uses the HTTPCaller to access the FME Server REST api to get a list of current jobs. That will include the start time of each job. You calculate which have been running for more than five minutes and then kill those off using the HTTPCaller again with another REST call.

The two commands would (I think) be:

<servername>/fmerest/v2/transformations/jobs/running

<servername>/fmerest/v2/transformations/commands/remove/running

You can access the API docs at: <servername>/fmerest/v2/apidoc

Badge +10

The other solution I can think of is that you have an FME workspace that runs all the time (or is run on a schedule every minute). It uses the HTTPCaller to access the FME Server REST api to get a list of current jobs. That will include the start time of each job. You calculate which have been running for more than five minutes and then kill those off using the HTTPCaller again with another REST call.

The two commands would (I think) be:

<servername>/fmerest/v2/transformations/jobs/running

<servername>/fmerest/v2/transformations/commands/remove/running

You can access the API docs at: <servername>/fmerest/v2/apidoc

Good idea Mark and smart to use the REST API. Presumably though this would only work for setups with more than 1 engine?

Userlevel 2
Badge +12

I do not know what kind of Job you are referring to, but could it be done inside the workspace?

You can use the TimeStamper to get a starting time and if you are using a loop inside the workspace you can compare the current time against the starting time. If the difference is larger than the specified timeout you can exit the loop and therefor the job.

Userlevel 4
Badge +25

Good idea Mark and smart to use the REST API. Presumably though this would only work for setups with more than 1 engine?

Good point - it would use up one of your engines. That hadn't occurred to me (doh!)

Badge

The other solution I can think of is that you have an FME workspace that runs all the time (or is run on a schedule every minute). It uses the HTTPCaller to access the FME Server REST api to get a list of current jobs. That will include the start time of each job. You calculate which have been running for more than five minutes and then kill those off using the HTTPCaller again with another REST call.

The two commands would (I think) be:

<servername>/fmerest/v2/transformations/jobs/running

<servername>/fmerest/v2/transformations/commands/remove/running

You can access the API docs at: <servername>/fmerest/v2/apidoc

I like this but my engines are pretty much busy, I will have to set this scheduled "timeout" up to a high priority to make sure it doesn't get stuck in the queue. Problem comes when two engines get stuck with a never ending job.

BTW the origin of it all is that some dwg won't be read properly and start runing for ever and fill the disk in with GB of logs.

Badge

this job is executed in 3 hours and sometimes it doesn't ends

Can you submit this idea with an option (reconnect) in FMEJOBSUBMITTER ? FmeJobSubmitter can lost the connection with the server. Maybe the transformer resends a notification to know what is the status of the job.

Badge

The other solution I can think of is that you have an FME workspace that runs all the time (or is run on a schedule every minute). It uses the HTTPCaller to access the FME Server REST api to get a list of current jobs. That will include the start time of each job. You calculate which have been running for more than five minutes and then kill those off using the HTTPCaller again with another REST call.

The two commands would (I think) be:

<servername>/fmerest/v2/transformations/jobs/running

<servername>/fmerest/v2/transformations/commands/remove/running

You can access the API docs at: <servername>/fmerest/v2/apidoc

this job is executed in 3 hours and sometimes it doesn't ends

Can you submit this idea with an option (reconnect) in FMEJOBSUBMITTER ? FmeJobSubmitter can lost the connection with the server. Maybe the transformer can resent a notification to know what is the status of the job.

Userlevel 3
Badge +13

I can definitely see the value in this setting. In fact, we've seen this requested by a number of users this year. It would be really great if one of the community members (@farfar ?) turned this into an Idea: https://knowledge.safe.com/content/idea/post.html

Then the rest of the community can vote it up. More votes equals more likely to get done. Thanks for getting involved!

Badge

Would be a nice feature to add!

 

@aaronkoning: This idea was already posted by @FMELizard apparently, so +1 for me :)

An interesting challenge... I do recall that there is an option somewhere in FME Server that allows jobs to timeout, BUT, I have a feeling that is related to the queue. I.e. If a job sits in the queue waiting to be processed for greater that x minutes/hours before being processed, it may no longer be viable so ignore it.

Obviously that won't help you.

So by way of a consolation prize please find attached a workspace that should do what you need. You'll need to add the first section and the last into your process, right up front and at the very end of any existing transformers. This workflow then simple terminates the job if it goes on for more than 5 minutes; a value that you can of course alter.

Hope it helps, but just beware, with the super quick testing I have done, the time difference will be actually quite a bit less than the FME Session duration, so you'll need to do some experimentation with this if you use it... or just build on the principle.

Download: transformer-sequence-to-terminate-long-run-jobs.fmw

cc. @jpvo

Such a routine is however only able to determine any time excession at the end, after the run is already done, and the time has been exceeded. it will not terminate the run right at the moment when the set time is just consumed.

Userlevel 3
Badge +13

I can definitely see the value in this setting. In fact, we've seen this requested by a number of users this year. It would be really great if one of the community members (@farfar ?) turned this into an Idea: https://knowledge.safe.com/content/idea/post.html

Then the rest of the community can vote it up. More votes equals more likely to get done. Thanks for getting involved!

Here is the idea that you can vote for:

 

https://knowledge.safe.com/content/idea/18984/configure-time-limits-for-a-job-and-alertcancel-if.html

Hi all I made a template https://hub.safe.com/templates/cancelinqueue that can scan the FME server queue and cancel jobs if they are to long in the queue are to many of the same. Maybe useful for others.

Badge +11

@farfar - This is now possible in FME Server 2017.0 and Newer with the Running Job Expiry Time parameter.

Reply