Skip to main content
Solved

FME Problem with licenses when using FME Server Job Submitter

W
Forum|alt.badge.img

Hello,

 

We have a server on which we have 4 engines installed. FME Version is 2016.

In our setup on FME Server we have main workspace which is responsible for some preprocessing and then it submits child workspace using ServerJobSubmitter.

 

In some cases this child workspace also has ServerJobSubmitter because it processes huge data set and if we didn't split it into several runs then we woud get out of memory exception or have other problems.

 

It looks quite nice but in some cases (probably when few jobs are executed also on other engines) in that child workspace we have an error:

FME license system failure: The maximum number of concurrent FME instances specified in the license has been reached (-508)

 

 

What can be the reasson for that?

ServerJobSubmitter is set to wait for jobs to complete

Best answer by hollyatsafe

Hi @witos,

Thank you for contacting Safe Software Support. Now that we have understood the issue I wanted to also update this post on the Forum so that the answer is available publicly for anyone else that may run into this issue.

The reason you are running into this error is because the FME licensing agreement only allows a maximum of eight fme.exe processes to run in parallel.

In FME Server 2016 there are three ways I know of that could cause you to hit this error:

  1. Running on a single engine your parent workspace contains 9 layers of workspaces with fmeserverjobsubmitters in each. Note only one fme.exe process is launched per workspace no matter how many JobSubmitters it contains.
  2. Across multiple engines you have workspaces that launch multiple layers of child jobs which combined add up to more than 8.
  3. If you cancelled a job containing a JobSubmitter the fme.exe process failed to shut down so is taking up part of the total count even though it isn't being used - this was a known issue in 2016.

This is because in 2016 the behaviour mirrored that of FME Desktop and so was monitoring these fme.exe processes on a per machine basis. If you have multiple engines the workaround would be to deploy these on different machines.

In FME Server 2017.1 and newer this has been rectified and is now on a per engine basis. This can be observed in Windows Task Manager as it now launches additional fmeengine.exe processes instead of fme.exe processes to avoid these being counted like for desktop engines.

There were a number of problems with FMEServerJobSubmitter in 2016 which Cat highlighted. However I have also updated the Maximum Concurrent Processes article that covers this error to include the fme.exe per machine limitation that is on 2017 and older.

View original
    Did this help you find an answer to your question?

    8 replies

    david_r
    Evangelist
    • david_rEvangelist
    • April 1, 2019

    Have you verified in the logs that both the master and the child workspace actually runs on the same engine?

    A common issue is if the FMEServerJobSubmitter references a different host name than used by the master workspace, FME Server will spin up the job on a separate engine, even if both host names are physically referencing the same server.

    Typical example:

    Master workspace runs on my.fmeserver.local, but the FMEServerJobSubmitter connects to localhost. The child workspace will then be forced to run on a separate engine.

    I would also recommend that you consider upgrading FME Server, there has been a lot of improvements since 2016 in this regard.

    W
    Forum|alt.badge.img
    • witos
    • Author
    • April 1, 2019
    david_r wrote:

    Have you verified in the logs that both the master and the child workspace actually runs on the same engine?

    A common issue is if the FMEServerJobSubmitter references a different host name than used by the master workspace, FME Server will spin up the job on a separate engine, even if both host names are physically referencing the same server.

    Typical example:

    Master workspace runs on my.fmeserver.local, but the FMEServerJobSubmitter connects to localhost. The child workspace will then be forced to run on a separate engine.

    I would also recommend that you consider upgrading FME Server, there has been a lot of improvements since 2016 in this regard.

    Hello,

     

    thank you, Yes I verified it and for all jobs it is the same engine.

     

     

    Do you know what is the limit of concurent fme processes and the same server? Is it per engine or per server?
    david_r
    Evangelist
    • david_rEvangelist
    • April 1, 2019
    witos wrote:

    Hello,

     

    thank you, Yes I verified it and for all jobs it is the same engine.

     

     

    Do you know what is the limit of concurent fme processes and the same server? Is it per engine or per server?

    One engine = one job. If you let a child workspace run on the same workspace as the master workspace, the master workspace will be paused ("put aside") as long as the child workspace runs. The master workspace will continue once the child workspace has termina

    W
    Forum|alt.badge.img
    • witos
    • Author
    • April 1, 2019
    david_r wrote:

    One engine = one job. If you let a child workspace run on the same workspace as the master workspace, the master workspace will be paused ("put aside") as long as the child workspace runs. The master workspace will continue once the child workspace has termina

    Ok, but then I get

     

     

    FME license system failure: The maximum number of concurrent FME instances specified in the license has been reached (-508)

     

     

    So I guess that even if I have ServerJobSubmitter is set to wait for jobs to complete

    new FME.Exe processes so questions is how many FME.exe processes you can have on one FME engine?

     

     

    david_r
    Evangelist
    • david_rEvangelist
    • April 1, 2019
    witos wrote:

    Ok, but then I get

     

     

    FME license system failure: The maximum number of concurrent FME instances specified in the license has been reached (-508)

     

     

    So I guess that even if I have ServerJobSubmitter is set to wait for jobs to complete

    new FME.Exe processes so questions is how many FME.exe processes you can have on one FME engine?

     

     

    There should (normally) be no instances of fme.exe running on FME Server, only fmeengine.exe, which, confusingly, presents itself as "FME EXE" in the Task Manager.

    I suppose you do not have any WorkspaceRunners?

    You may also want to contact Safe support for this.

    W
    Forum|alt.badge.img
    • witos
    • Author
    • April 1, 2019
    david_r wrote:

    There should (normally) be no instances of fme.exe running on FME Server, only fmeengine.exe, which, confusingly, presents itself as "FME EXE" in the Task Manager.

    I suppose you do not have any WorkspaceRunners?

    You may also want to contact Safe support for this.

    Thanks for hints,

     

    I only use ServerJobSubmitters. And in my case it looks like this:

     

     

    I sent message to safe support. We will see :)

    C
    Forum|alt.badge.img
    • catatsafe
    • April 2, 2019

    Hi @witos,

    There can be a maximum of 7 child processes kicked off by a job ( for a total of 8 processes). However, in FME 2016 it didn't always do the right thing - have you checked out this article - https://knowledge.safe.com/articles/55369/fme-server-troublshooting-fmeserverjobsubmitter.html which highlights the differences in 2016 and 2017+.

    Cheers,

    Cat

    H
    Forum|alt.badge.img+2
    • hollyatsafe
    • Best Answer
    • April 2, 2019

    Hi @witos,

    Thank you for contacting Safe Software Support. Now that we have understood the issue I wanted to also update this post on the Forum so that the answer is available publicly for anyone else that may run into this issue.

    The reason you are running into this error is because the FME licensing agreement only allows a maximum of eight fme.exe processes to run in parallel.

    In FME Server 2016 there are three ways I know of that could cause you to hit this error:

    1. Running on a single engine your parent workspace contains 9 layers of workspaces with fmeserverjobsubmitters in each. Note only one fme.exe process is launched per workspace no matter how many JobSubmitters it contains.
    2. Across multiple engines you have workspaces that launch multiple layers of child jobs which combined add up to more than 8.
    3. If you cancelled a job containing a JobSubmitter the fme.exe process failed to shut down so is taking up part of the total count even though it isn't being used - this was a known issue in 2016.

    This is because in 2016 the behaviour mirrored that of FME Desktop and so was monitoring these fme.exe processes on a per machine basis. If you have multiple engines the workaround would be to deploy these on different machines.

    In FME Server 2017.1 and newer this has been rectified and is now on a per engine basis. This can be observed in Windows Task Manager as it now launches additional fmeengine.exe processes instead of fme.exe processes to avoid these being counted like for desktop engines.

    There were a number of problems with FMEServerJobSubmitter in 2016 which Cat highlighted. However I have also updated the Maximum Concurrent Processes article that covers this error to include the fme.exe per machine limitation that is on 2017 and older.

    Reply


    Helpful Members This Week

    Community Stats

    30,937
    Posts
    117,248
    Replies
    38,740
    Members

    Latest FME

    Terms & ConditionsCookie settings

    Cookie policy

    We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

     
    Cookie settings
    A product of
    Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.