Skip to main content
Solved

FME Serve Custom Engine Properties

  • November 25, 2021
  • 2 replies
  • 52 views
hkingsbury
Celebrity
Forum|alt.badge.img+55

Hi All

 

I've been having a look at the new engine assignment rules on recent version of FME Server. When creating a new Engine Assignment Rule you have the ability to create a custom engine property to use to assign engines.

image 

However I see no way of subsequently assigning that newely created property to an engine to use in the engine routing. In the Queue Control documentation it suggests that you can add custom properties to Engines, but only using Kubernetes

 

https://docs.safe.com/fme/html/FME_Server_Documentation/WebUI/Queue-Control.htm (towards the bottom of the page under 'To add a queue to a new rule')

 

Is there something i'm missing? i've had a dig through the config files (thinking it might have to be a manual assignment similar to older versions of FME Server) also had a dig through and could find anything apparent in the REST API (side note - noted theres a new v4 API in the works)

 

This could be really powerful if you were able to individually tag engines with custom properties....

Best answer by hollyatsafe

Hi @hkingsbury​ , 

 

The ability to add custom properties was implemented specifically for containerized installations (Docker/K8S) because in these deployment types FME Engines are regularly created and destroyed and because of that, the Engine name is continuously changing, so in order for users to be able to make use of Queue Control, they needed to be able to provide Engines with a constant value.

 

That being said, it is possible to add custom properties to a standard deployment by making a change to the processMonitorConfigEngine.txt:

 

1. Run a text editor as administrator and open processMonitorConfigEngine.txt located in <FMEInstallDir>\Server\

2. Scroll down to the bottom of the file to the section # ENGINE process configuration template and find the line starting with TEMPLATE_START_ENGINE (~line 270)

3. At the end of the line add the parameter '-ENGINE_PROP <value>' where <value> is the custom property name. If you’d like to add multiple properties, repeat this with a space to separate them. For example:

TEMPLATE_START_ENGINE=“C:/Program Files/FMEServer/Server/fme/FMEEngine.exe” REGISTER_FME_ENGINE localhost 7070 DEFAULT “C:/Program Files/FMEServer/Server/fmeEngineConfig.txt” -ENGINE_HOST_NAME HC-HUMMINGBIRD -ENGINE_PROP Prop1 -ENGINE_PROP Prop2

4. Save the file and restart the FME Server Engine Service, in the Web UI go to the Engines page and you’ll now see your new properties assigned.

Screen Shot 2021-11-30 at 5.50.35 AMIf you do this, you should keep in mind:

1. This property will be applied to all FME Engines running on that host, you cannot assign custom properties to a single Engine unless you distribute it so it has its own configuration file.

2. This approach is not documented because it is not the intended use case, as such it is possible it will not work in future versions of FME.

 

I’d love to learn a little bit more about what you’d like to use custom properties for, do you have a use case in mind? 

I’d highly recommend you post an idea, sharing your use case, this way we can monitor if this is something a lot of users would find helpful and consider making improvements to the product to be able to fully support custom properties on standard deployments in future.

View original
Did this help you find an answer to your question?

2 replies

H
Forum|alt.badge.img+2
  • hollyatsafe
  • Best Answer
  • November 30, 2021

Hi @hkingsbury​ , 

 

The ability to add custom properties was implemented specifically for containerized installations (Docker/K8S) because in these deployment types FME Engines are regularly created and destroyed and because of that, the Engine name is continuously changing, so in order for users to be able to make use of Queue Control, they needed to be able to provide Engines with a constant value.

 

That being said, it is possible to add custom properties to a standard deployment by making a change to the processMonitorConfigEngine.txt:

 

1. Run a text editor as administrator and open processMonitorConfigEngine.txt located in <FMEInstallDir>\Server\

2. Scroll down to the bottom of the file to the section # ENGINE process configuration template and find the line starting with TEMPLATE_START_ENGINE (~line 270)

3. At the end of the line add the parameter '-ENGINE_PROP <value>' where <value> is the custom property name. If you’d like to add multiple properties, repeat this with a space to separate them. For example:

TEMPLATE_START_ENGINE=“C:/Program Files/FMEServer/Server/fme/FMEEngine.exe” REGISTER_FME_ENGINE localhost 7070 DEFAULT “C:/Program Files/FMEServer/Server/fmeEngineConfig.txt” -ENGINE_HOST_NAME HC-HUMMINGBIRD -ENGINE_PROP Prop1 -ENGINE_PROP Prop2

4. Save the file and restart the FME Server Engine Service, in the Web UI go to the Engines page and you’ll now see your new properties assigned.

Screen Shot 2021-11-30 at 5.50.35 AMIf you do this, you should keep in mind:

1. This property will be applied to all FME Engines running on that host, you cannot assign custom properties to a single Engine unless you distribute it so it has its own configuration file.

2. This approach is not documented because it is not the intended use case, as such it is possible it will not work in future versions of FME.

 

I’d love to learn a little bit more about what you’d like to use custom properties for, do you have a use case in mind? 

I’d highly recommend you post an idea, sharing your use case, this way we can monitor if this is something a lot of users would find helpful and consider making improvements to the product to be able to fully support custom properties on standard deployments in future.

hkingsbury
Celebrity
Forum|alt.badge.img+55
  • hkingsburyCelebrity
  • Author
  • Celebrity
  • November 30, 2021

Thanks for the response @hollyatsafe​ 

 

I'll definitely post an idea around that, I think it could be a really powerful feature for FME Server

 

Our use case is a bit complex, but i'll try to distill it as simply as possible....

 

The production environment comprises of a failover environment, on each of the servers there are 2 standard engines (4 in total) and a number of dynamic engines (exact number undecided at this stage).

There are a range of processes that run in this environment that for simplicity can be categorized into three groups:

  1. Real time on demand - these run when a user click go, and then the user waits for a response (usually through an external web based application)
  2. On demand - lower priority jobs that are requested but a user is not expecting a prompt response
  3. Scheduled jobs - Generally run overnight, but some schedules will kick off jobs that a re classified as #2

 

What we're wanting to achieve is maximising the usage of the standard engines before falling over to dynamic engines. We may also look at dynamically scallign the number of dynamic engines we have available depending on exactly what process are required to be run

To assist with that I was aiming to have all the dynamic engines available to process jobs in #1 so when the standard engines are fully utilized, jobs begin to roll over onto those.

Currently #2 jobs are locked to a queue where one engine is generally assigned, this engine is configured to be redeployed to run #1 jobs when they begin to queue. If we begin to see queue length, or time in queue increasing for #2 then I want to be able to assign some of the dynamic engines to this queue to begin this processing.

 

Whilst i could achieve this by specifying manual engine names in the engine assignment rules, if we begin to scale up/down dynamic engines as well, keeping track of those names is going to become a bit harder. What I was envisioning when I saw those custom properties was that I could assign properties to engines such as "Overflow" or "Primary" and that these could then be assigned to individual engines on an as need basis.

 

Separate, but also related, is then out of work hours, have all standard engines available for #1, #2 and #3 to run on - Be handy to have time dependent engine assignment rules.

 

So in short, I could achieve what i'm wanting to do by setting rules using engine names, it would be handy to be able to seperate categorise engines into 'groups'

Reply


Helpful Members This Week

Community Stats

31,428
Posts
119,163
Replies
39,050
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.