Skip to main content
Question

FME SERVER Job Time out


farfar
Contributor
Forum|alt.badge.img+11
Hi All,

 

Are there possibilities to specify a finite timeout for a job? I need to stop a job that is running and is never ended without using admin interface?

 

 

Thanks an advance;

 

Farfar

17 replies

david_r
Celebrity
  • April 30, 2014

farfar
Contributor
Forum|alt.badge.img+11
  • Author
  • Contributor
  • April 30, 2014
Hi David,

 

Yes i looked this article before. But it's not resolved my problem. I explain my problem :

 

If i run my job with coorect input shape file, the job is executed in 3 minutes. But if i use uncorrect input file, this job is executed in 3 hours and sometimes it doesn't ends.  My idea is to stop this job if it is running more than 5 minutes for example.  

 

Thanks in advance.

 

Farfar

jpvo
Participant
Forum|alt.badge.img+1
  • Participant
  • December 16, 2015

Hi Farfar,

Do you have any answer to your question?

I am facing the same issue and can't find any answer. Thanks

.JP


davideagle
Contributor
Forum|alt.badge.img+21
  • Contributor
  • December 16, 2015

An interesting challenge... I do recall that there is an option somewhere in FME Server that allows jobs to timeout, BUT, I have a feeling that is related to the queue. I.e. If a job sits in the queue waiting to be processed for greater that x minutes/hours before being processed, it may no longer be viable so ignore it.

Obviously that won't help you.

So by way of a consolation prize please find attached a workspace that should do what you need. You'll need to add the first section and the last into your process, right up front and at the very end of any existing transformers. This workflow then simple terminates the job if it goes on for more than 5 minutes; a value that you can of course alter.

Hope it helps, but just beware, with the super quick testing I have done, the time difference will be actually quite a bit less than the FME Session duration, so you'll need to do some experimentation with this if you use it... or just build on the principle.

Download: transformer-sequence-to-terminate-long-run-jobs.fmw

cc. @jpvo


mark2atsafe
Safer
Forum|alt.badge.img+45
  • Safer
  • December 16, 2015

The other solution I can think of is that you have an FME workspace that runs all the time (or is run on a schedule every minute). It uses the HTTPCaller to access the FME Server REST api to get a list of current jobs. That will include the start time of each job. You calculate which have been running for more than five minutes and then kill those off using the HTTPCaller again with another REST call.

The two commands would (I think) be:

<servername>/fmerest/v2/transformations/jobs/running

<servername>/fmerest/v2/transformations/commands/remove/running

You can access the API docs at: <servername>/fmerest/v2/apidoc


davideagle
Contributor
Forum|alt.badge.img+21
  • Contributor
  • December 16, 2015
mark2atsafe wrote:

The other solution I can think of is that you have an FME workspace that runs all the time (or is run on a schedule every minute). It uses the HTTPCaller to access the FME Server REST api to get a list of current jobs. That will include the start time of each job. You calculate which have been running for more than five minutes and then kill those off using the HTTPCaller again with another REST call.

The two commands would (I think) be:

<servername>/fmerest/v2/transformations/jobs/running

<servername>/fmerest/v2/transformations/commands/remove/running

You can access the API docs at: <servername>/fmerest/v2/apidoc

Good idea Mark and smart to use the REST API. Presumably though this would only work for setups with more than 1 engine?


erik_jan
Contributor
Forum|alt.badge.img+18
  • Contributor
  • December 16, 2015

I do not know what kind of Job you are referring to, but could it be done inside the workspace?

You can use the TimeStamper to get a starting time and if you are using a loop inside the workspace you can compare the current time against the starting time. If the difference is larger than the specified timeout you can exit the loop and therefor the job.


mark2atsafe
Safer
Forum|alt.badge.img+45
  • Safer
  • December 17, 2015
davideagle wrote:

Good idea Mark and smart to use the REST API. Presumably though this would only work for setups with more than 1 engine?

Good point - it would use up one of your engines. That hadn't occurred to me (doh!)


jpvo
Participant
Forum|alt.badge.img+1
  • Participant
  • December 17, 2015
mark2atsafe wrote:

The other solution I can think of is that you have an FME workspace that runs all the time (or is run on a schedule every minute). It uses the HTTPCaller to access the FME Server REST api to get a list of current jobs. That will include the start time of each job. You calculate which have been running for more than five minutes and then kill those off using the HTTPCaller again with another REST call.

The two commands would (I think) be:

<servername>/fmerest/v2/transformations/jobs/running

<servername>/fmerest/v2/transformations/commands/remove/running

You can access the API docs at: <servername>/fmerest/v2/apidoc

I like this but my engines are pretty much busy, I will have to set this scheduled "timeout" up to a high priority to make sure it doesn't get stuck in the queue. Problem comes when two engines get stuck with a never ending job.

BTW the origin of it all is that some dwg won't be read properly and start runing for ever and fill the disk in with GB of logs.


Forum|alt.badge.img+5
  • December 17, 2015

this job is executed in 3 hours and sometimes it doesn't ends

Can you submit this idea with an option (reconnect) in FMEJOBSUBMITTER ? FmeJobSubmitter can lost the connection with the server. Maybe the transformer resends a notification to know what is the status of the job.


Forum|alt.badge.img+5
  • December 17, 2015
mark2atsafe wrote:

The other solution I can think of is that you have an FME workspace that runs all the time (or is run on a schedule every minute). It uses the HTTPCaller to access the FME Server REST api to get a list of current jobs. That will include the start time of each job. You calculate which have been running for more than five minutes and then kill those off using the HTTPCaller again with another REST call.

The two commands would (I think) be:

<servername>/fmerest/v2/transformations/jobs/running

<servername>/fmerest/v2/transformations/commands/remove/running

You can access the API docs at: <servername>/fmerest/v2/apidoc

this job is executed in 3 hours and sometimes it doesn't ends

Can you submit this idea with an option (reconnect) in FMEJOBSUBMITTER ? FmeJobSubmitter can lost the connection with the server. Maybe the transformer can resent a notification to know what is the status of the job.


fmelizard
Safer
Forum|alt.badge.img+19
  • Safer
  • December 28, 2015

I can definitely see the value in this setting. In fact, we've seen this requested by a number of users this year. It would be really great if one of the community members (@farfar ?) turned this into an Idea: https://knowledge.safe.com/content/idea/post.html

Then the rest of the community can vote it up. More votes equals more likely to get done. Thanks for getting involved!


geosander
Forum|alt.badge.img+7
  • January 11, 2016

Would be a nice feature to add!

 

@aaronkoning: This idea was already posted by @FMELizard apparently, so +1 for me :)

frantsch
Contributor
Forum|alt.badge.img+4
  • Contributor
  • February 4, 2016
davideagle wrote:

An interesting challenge... I do recall that there is an option somewhere in FME Server that allows jobs to timeout, BUT, I have a feeling that is related to the queue. I.e. If a job sits in the queue waiting to be processed for greater that x minutes/hours before being processed, it may no longer be viable so ignore it.

Obviously that won't help you.

So by way of a consolation prize please find attached a workspace that should do what you need. You'll need to add the first section and the last into your process, right up front and at the very end of any existing transformers. This workflow then simple terminates the job if it goes on for more than 5 minutes; a value that you can of course alter.

Hope it helps, but just beware, with the super quick testing I have done, the time difference will be actually quite a bit less than the FME Session duration, so you'll need to do some experimentation with this if you use it... or just build on the principle.

Download: transformer-sequence-to-terminate-long-run-jobs.fmw

cc. @jpvo

Such a routine is however only able to determine any time excession at the end, after the run is already done, and the time has been exceeded. it will not terminate the run right at the moment when the set time is just consumed.


fmelizard
Safer
Forum|alt.badge.img+19
  • Safer
  • March 27, 2016
fmelizard wrote:

I can definitely see the value in this setting. In fact, we've seen this requested by a number of users this year. It would be really great if one of the community members (@farfar ?) turned this into an Idea: https://knowledge.safe.com/content/idea/post.html

Then the rest of the community can vote it up. More votes equals more likely to get done. Thanks for getting involved!

Here is the idea that you can vote for:

 

https://knowledge.safe.com/content/idea/18984/configure-time-limits-for-a-job-and-alertcancel-if.html

Forum|alt.badge.img+3
  • June 19, 2016

Hi all I made a template https://hub.safe.com/templates/cancelinqueue that can scan the FME server queue and cancel jobs if they are to long in the queue are to many of the same. Maybe useful for others.


rylanatsafe
Safer
Forum|alt.badge.img+13
  • Safer
  • September 21, 2017

@farfar - This is now possible in FME Server 2017.0 and Newer with the Running Job Expiry Time parameter.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings