Skip to main content
Question

How to split jobs to multiple cores in FME server?

  • June 18, 2019
  • 3 replies
  • 28 views
G
Forum|alt.badge.img

Hi all,

 

I have a list of urls (50 in number). For each url there are 10s of thousends requests to be made.

12 engines in my fme server.

What I want to do use 5 of fme engines to run the jobs in parallel - so 10 jobs each engine.

How can I do that?

At the moment I have:

xls reader -> sampler2 (first 10 urls) -> FMEServerJobSubmitter1 (submit in sequence, wait to finish) , sampler2 (connected to sampler1,not_sampled port (first 10) -> FMEServerJobSubmitter2 and so on.

 

But it just runs the 10 jobs on one engine in FMEServerJobSubmitter1 one after another.

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

3 replies

redgeographics
Celebrity
Forum|alt.badge.img+59
  • redgeographicsCelebrity
  • Celebrity
  • 3699 replies
  • June 18, 2019

If you set the FMEServerJobSubmitter to submit in parallel and don't wait to finish it will simply submit all of the jobs and let the queue sort it out. You can limit the number of engines to be used by setting up a job queue (in the Engines and Licensing section of the UI, you'll need to have access to that of course)

FME rocks! \m/
G
Forum|alt.badge.img
  • garnet
  • Author
  • 25 replies
  • June 19, 2019

If you set the FMEServerJobSubmitter to submit in parallel and don't wait to finish it will simply submit all of the jobs and let the queue sort it out. You can limit the number of engines to be used by setting up a job queue (in the Engines and Licensing section of the UI, you'll need to have access to that of course)

Hi, thanks.

Can you elaborate more on setting the engines count. I want to use 5 out of 12 engines for this 50 jobs. Is there a setting like this for a job to tell it 'use only X engines'?

redgeographics
Celebrity
Forum|alt.badge.img+59
  • redgeographicsCelebrity
  • Celebrity
  • 3699 replies
  • June 19, 2019

Hi, thanks.

Can you elaborate more on setting the engines count. I want to use 5 out of 12 engines for this 50 jobs. Is there a setting like this for a job to tell it 'use only X engines'?

No, you'd have to add 5 specific engines to a job queue and limit that to the repository that holds your workspace.

I like the idea though, maybe you should make a new entry for it on the Ideas page.

FME rocks! \m/

Community Stats

32,062
Posts
121,727
Replies
39,689
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.