Solved

FME Problem with licenses when using FME Server Job Submitter

  • 31 March 2019
  • 8 replies
  • 60 views
W
Badge

Hello,

 

We have a server on which we have 4 engines installed. FME Version is 2016.

In our setup on FME Server we have main workspace which is responsible for some preprocessing and then it submits child workspace using ServerJobSubmitter.

 

In some cases this child workspace also has ServerJobSubmitter because it processes huge data set and if we didn't split it into several runs then we woud get out of memory exception or have other problems.

 

It looks quite nice but in some cases (probably when few jobs are executed also on other engines) in that child workspace we have an error:

FME license system failure: The maximum number of concurrent FME instances specified in the license has been reached (-508)

 

 

What can be the reasson for that?

ServerJobSubmitter is set to wait for jobs to complete

icon

Best answer by hollyatsafe 3 April 2019, 01:27

View original

8 replies

david_r
Rank
Userlevel 4

Have you verified in the logs that both the master and the child workspace actually runs on the same engine?

A common issue is if the FMEServerJobSubmitter references a different host name than used by the master workspace, FME Server will spin up the job on a separate engine, even if both host names are physically referencing the same server.

Typical example:

Master workspace runs on my.fmeserver.local, but the FMEServerJobSubmitter connects to localhost. The child workspace will then be forced to run on a separate engine.

I would also recommend that you consider upgrading FME Server, there has been a lot of improvements since 2016 in this regard.

W
Badge

Have you verified in the logs that both the master and the child workspace actually runs on the same engine?

A common issue is if the FMEServerJobSubmitter references a different host name than used by the master workspace, FME Server will spin up the job on a separate engine, even if both host names are physically referencing the same server.

Typical example:

Master workspace runs on my.fmeserver.local, but the FMEServerJobSubmitter connects to localhost. The child workspace will then be forced to run on a separate engine.

I would also recommend that you consider upgrading FME Server, there has been a lot of improvements since 2016 in this regard.

Hello,

 

thank you, Yes I verified it and for all jobs it is the same engine.

 

 

Do you know what is the limit of concurent fme processes and the same server? Is it per engine or per server?
david_r
Rank
Userlevel 4

Hello,

 

thank you, Yes I verified it and for all jobs it is the same engine.

 

 

Do you know what is the limit of concurent fme processes and the same server? Is it per engine or per server?

One engine = one job. If you let a child workspace run on the same workspace as the master workspace, the master workspace will be paused ("put aside") as long as the child workspace runs. The master workspace will continue once the child workspace has termina

W
Badge

One engine = one job. If you let a child workspace run on the same workspace as the master workspace, the master workspace will be paused ("put aside") as long as the child workspace runs. The master workspace will continue once the child workspace has termina

Ok, but then I get

 

 

FME license system failure: The maximum number of concurrent FME instances specified in the license has been reached (-508)

 

 

So I guess that even if I have ServerJobSubmitter is set to wait for jobs to complete

new FME.Exe processes so questions is how many FME.exe processes you can have on one FME engine?

 

 

david_r
Rank
Userlevel 4

Ok, but then I get

 

 

FME license system failure: The maximum number of concurrent FME instances specified in the license has been reached (-508)

 

 

So I guess that even if I have ServerJobSubmitter is set to wait for jobs to complete

new FME.Exe processes so questions is how many FME.exe processes you can have on one FME engine?

 

 

There should (normally) be no instances of fme.exe running on FME Server, only fmeengine.exe, which, confusingly, presents itself as "FME EXE" in the Task Manager.

I suppose you do not have any WorkspaceRunners?

You may also want to contact Safe support for this.

W
Badge

There should (normally) be no instances of fme.exe running on FME Server, only fmeengine.exe, which, confusingly, presents itself as "FME EXE" in the Task Manager.

I suppose you do not have any WorkspaceRunners?

You may also want to contact Safe support for this.

Thanks for hints,

 

I only use ServerJobSubmitters. And in my case it looks like this:

 

 

I sent message to safe support. We will see :)

C
Badge

Hi @witos,

There can be a maximum of 7 child processes kicked off by a job ( for a total of 8 processes). However, in FME 2016 it didn't always do the right thing - have you checked out this article - https://knowledge.safe.com/articles/55369/fme-server-troublshooting-fmeserverjobsubmitter.html which highlights the differences in 2016 and 2017+.

Cheers,

Cat

H
Badge +2

Hi @witos,

Thank you for contacting Safe Software Support. Now that we have understood the issue I wanted to also update this post on the Forum so that the answer is available publicly for anyone else that may run into this issue.

The reason you are running into this error is because the FME licensing agreement only allows a maximum of eight fme.exe processes to run in parallel.

In FME Server 2016 there are three ways I know of that could cause you to hit this error:

  1. Running on a single engine your parent workspace contains 9 layers of workspaces with fmeserverjobsubmitters in each. Note only one fme.exe process is launched per workspace no matter how many JobSubmitters it contains.
  2. Across multiple engines you have workspaces that launch multiple layers of child jobs which combined add up to more than 8.
  3. If you cancelled a job containing a JobSubmitter the fme.exe process failed to shut down so is taking up part of the total count even though it isn't being used - this was a known issue in 2016.

This is because in 2016 the behaviour mirrored that of FME Desktop and so was monitoring these fme.exe processes on a per machine basis. If you have multiple engines the workaround would be to deploy these on different machines.

In FME Server 2017.1 and newer this has been rectified and is now on a per engine basis. This can be observed in Windows Task Manager as it now launches additional fmeengine.exe processes instead of fme.exe processes to avoid these being counted like for desktop engines.

There were a number of problems with FMEServerJobSubmitter in 2016 which Cat highlighted. However I have also updated the Maximum Concurrent Processes article that covers this error to include the fme.exe per machine limitation that is on 2017 and older.

Reply


Community Stats

28,670
Posts
109,365
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.