Question

No FME server engine available


Issue: 

FME Server is licensed, but under Engine management, no engines are available. 

Refering to this article: Known Issue: FME Server Engines Missing Failed to Connect to Job Queue (safe.com) I've tried to delete the log from queue folder, FME can regenerate the log, but still didn't solve the issue. 

Queue log information:

[6732] 25 Nov 12:14:18.692 # Creating Server TCP listening socket 127.0.0.1:6379: bind: No error

FMEserver log:

Thu-25-Nov-2021 01:53:57.238 PM   INFORM   fmeenginemgrnodeclient   402911 : Connecting to Job Queue localhost node host localhost port 6379.
Thu-25-Nov-2021 01:53:57.304 PM   INFORM   fmeenginemgrnodeclient   402911 : Connecting to Job Queue localhost node host localhost port 6379.
Thu-25-Nov-2021 01:53:57.387 PM   ERROR    fmeenginemgrnodeclient   402902 : Failed to connect to Job Queue. Please ensure Job Queue is started.
Thu-25-Nov-2021 01:53:57.388 PM   ERROR    fmeenginemgrnodeclient   Could not get a resource from the pool
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at COM.safe.fmeserver.JobRouterConfig.loadAllTags(JobRouterConfig.java:163)
at COM.safe.fmeserver.JobRouterConfig.refresh(JobRouterConfig.java:194)
at COM.safe.fmeserver.JobRouterConfig.init(JobRouterConfig.java:142)
at COM.safe.fmeserver.FMEServerJobRouter.getJobRouterConfig(FMEServerJobRouter.java:296)
at COM.safe.fmeserver.FMEServerJobRouter.checkActiveQueueNodeAlive(FMEServerJobRouter.java:144)
at COM.safe.fmeserver.jobs.EngineManagerNodeOps.checkActiveQueueNodeAlive(EngineManagerNodeOps.java:112)
at COM.safe.fmeserver.jobs.EngineManagerNodeClient.executeLeaderOp(EngineManagerNodeClient.java:90)
at COM.safe.fmeserver.database.NodeClient.run(NodeClient.java:110)
at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisDataException: ERR invalid password
at redis.clients.jedis.Protocol.processError(Protocol.java:127)
at redis.clients.jedis.Protocol.process(Protocol.java:161)
at redis.clients.jedis.Protocol.read(Protocol.java:215)
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340)
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239)
at redis.clients.jedis.BinaryJedis.auth(BinaryJedis.java:2139)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:108)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 10 more

I did notice there's another application running on the same server that also got redis-server running. Not sure if this will cause the error. 


11 replies

Userlevel 4
Badge +26

Another redis server could be part of the problem I suppose. You should check which port the other service is running on if you can. It shouldn't make a difference if the port is different I wouldn't think.

Can you see if the FME Server queue process is running in the services? do you get any errors when you try to restart fme server at all?

Here's the list of ports which FME Server uses: http://docs.safe.com/fme/html/FME_Server_Documentation/ReferenceManual/FME-Server-Ports.htm

You should also check that these are open but the probably are

Something kind of odd is this line here:

"Caused by: redis.clients.jedis.exceptions.JedisDataException: ERR invalid password"

It's hard to tell if this is important of just some random message

And are you 100% sure that the user running the FME Server service has access to write to the log file?

 

Another redis server could be part of the problem I suppose. You should check which port the other service is running on if you can. It shouldn't make a difference if the port is different I wouldn't think. 

Can you see if the FME Server queue process is running in the services? do you get any errors when you try to restart fme server at all?

Here's the list of ports which FME Server uses: http://docs.safe.com/fme/html/FME_Server_Documentation/ReferenceManual/FME-Server-Ports.htm

You should also check that these are open but the probably are

Something kind of odd is this line here:

"Caused by: redis.clients.jedis.exceptions.JedisDataException: ERR invalid password"

It's hard to tell if this is important of just some random message

And are you 100% sure that the user running the FME Server service has access to write to the log file?

 

Thanks Matt, the other service is using the same port as FME's redis. So I managed to change the redis port from 6379 to 6380. How do I config FME listening to the new port instead of the default 6379? Thank you.

queue log:

 29 Nov 09:04:01.098 * Redis 3.0.504 (00000000/0) 64 bit, standalone mode, port 6380, pid 17436 ready to start.
[17436] 29 Nov 09:04:01.098 # Server started, Redis version 3.0.504
[17436] 29 Nov 09:04:01.098 * The server is now ready to accept connections on port 6380

fmeserver log:

Continued from C:\ProgramData\Safe Software\FME Server\resources\logs\core\old\2021-11-29\fmeserver_2021-11-29_092540.log
Opening log file using character encoding UTF-8
Mon-29-Nov-2021 09:25:40.619 AM   INFORM   RequestHandler-Thread   402911 : Connecting to Job Queue localhost node host localhost port 6379.
Mon-29-Nov-2021 09:25:40.638 AM   ERROR    RequestHandler-Thread   402902 : Failed to connect to Job Queue. Please ensure Job Queue is started.
Mon-29-Nov-2021 09:25:40.638 AM   ERROR    RequestHandler-Thread   Could not get a resource from the pool
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at COM.safe.fmeserver.JobRouterConfig.loadAllTags(JobRouterConfig.java:163)
at COM.safe.fmeserver.JobRouterConfig.refresh(JobRouterConfig.java:194)
at COM.safe.fmeserver.JobRouterConfig.init(JobRouterConfig.java:142)
at COM.safe.fmeserver.FMEServerJobRouter.getJobRouterConfig(FMEServerJobRouter.java:296)
at COM.safe.fmeserver.FMEServerJobRouter.getJobTagMetrics(FMEServerJobRouter.java:682)
at COM.safe.fmeserver.FMEEngineRuleEvaluator.getQueueMetrics(FMEEngineRuleEvaluator.java:554)
at COM.safe.fmeserver.FMEEngineRuleEvaluator.lambda$getQueueMetricsCached$1(FMEEngineRuleEvaluator.java:533)
at COM.safe.fmeserver.database.ManagerBase.execute(ManagerBase.java:83)
at COM.safe.fmeserver.database.ManagerBase.execute(ManagerBase.java:61)
at COM.safe.fmeserver.FMEEngineRuleEvaluator.getQueueMetricsCached(FMEEngineRuleEvaluator.java:533)
at COM.safe.fmeserver.FMEEngineRuleEvaluator.getQueuesFromEngineAssignRules(FMEEngineRuleEvaluator.java:376)
at COM.safe.fmeserver.FMEServerJobRouter.registerEngine(FMEServerJobRouter.java:459)
at COM.safe.fmeserver.TransformationManager.registerFMEEngine(TransformationManager.java:1919)
at COM.safe.fmeserver.TransformationRequestsDispatcher.registerFMEEngine(TransformationRequestsDispatcher.java:841)
at COM.safe.fmeserver.FMEServer.handleFMEEngineRegistrationRequest(FMEServer.java:363)
at COM.safe.fmeserver.RequestHandler.handleServerSocketChannelConnectRequest(RequestHandler.java:163)
at COM.safe.fmeserver.RequestHandlerBase.run(RequestHandlerBase.java:882)
at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisDataException: ERR invalid password
at redis.clients.jedis.Protocol.processError(Protocol.java:127)
at redis.clients.jedis.Protocol.process(Protocol.java:161)
at redis.clients.jedis.Protocol.read(Protocol.java:215)
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340)
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239)
at redis.clients.jedis.BinaryJedis.auth(BinaryJedis.java:2139)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:108)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 19 more

 

Userlevel 4
Badge +26

Thanks Matt, the other service is using the same port as FME's redis. So I managed to change the redis port from 6379 to 6380. How do I config FME listening to the new port instead of the default 6379? Thank you.

queue log:

 29 Nov 09:04:01.098 * Redis 3.0.504 (00000000/0) 64 bit, standalone mode, port 6380, pid 17436 ready to start.
[17436] 29 Nov 09:04:01.098 # Server started, Redis version 3.0.504
[17436] 29 Nov 09:04:01.098 * The server is now ready to accept connections on port 6380

fmeserver log:

Continued from C:\ProgramData\Safe Software\FME Server\resources\logs\core\old\2021-11-29\fmeserver_2021-11-29_092540.log
Opening log file using character encoding UTF-8
Mon-29-Nov-2021 09:25:40.619 AM   INFORM   RequestHandler-Thread   402911 : Connecting to Job Queue localhost node host localhost port 6379.
Mon-29-Nov-2021 09:25:40.638 AM   ERROR    RequestHandler-Thread   402902 : Failed to connect to Job Queue. Please ensure Job Queue is started.
Mon-29-Nov-2021 09:25:40.638 AM   ERROR    RequestHandler-Thread   Could not get a resource from the pool
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at COM.safe.fmeserver.JobRouterConfig.loadAllTags(JobRouterConfig.java:163)
at COM.safe.fmeserver.JobRouterConfig.refresh(JobRouterConfig.java:194)
at COM.safe.fmeserver.JobRouterConfig.init(JobRouterConfig.java:142)
at COM.safe.fmeserver.FMEServerJobRouter.getJobRouterConfig(FMEServerJobRouter.java:296)
at COM.safe.fmeserver.FMEServerJobRouter.getJobTagMetrics(FMEServerJobRouter.java:682)
at COM.safe.fmeserver.FMEEngineRuleEvaluator.getQueueMetrics(FMEEngineRuleEvaluator.java:554)
at COM.safe.fmeserver.FMEEngineRuleEvaluator.lambda$getQueueMetricsCached$1(FMEEngineRuleEvaluator.java:533)
at COM.safe.fmeserver.database.ManagerBase.execute(ManagerBase.java:83)
at COM.safe.fmeserver.database.ManagerBase.execute(ManagerBase.java:61)
at COM.safe.fmeserver.FMEEngineRuleEvaluator.getQueueMetricsCached(FMEEngineRuleEvaluator.java:533)
at COM.safe.fmeserver.FMEEngineRuleEvaluator.getQueuesFromEngineAssignRules(FMEEngineRuleEvaluator.java:376)
at COM.safe.fmeserver.FMEServerJobRouter.registerEngine(FMEServerJobRouter.java:459)
at COM.safe.fmeserver.TransformationManager.registerFMEEngine(TransformationManager.java:1919)
at COM.safe.fmeserver.TransformationRequestsDispatcher.registerFMEEngine(TransformationRequestsDispatcher.java:841)
at COM.safe.fmeserver.FMEServer.handleFMEEngineRegistrationRequest(FMEServer.java:363)
at COM.safe.fmeserver.RequestHandler.handleServerSocketChannelConnectRequest(RequestHandler.java:163)
at COM.safe.fmeserver.RequestHandlerBase.run(RequestHandlerBase.java:882)
at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisDataException: ERR invalid password
at redis.clients.jedis.Protocol.processError(Protocol.java:127)
at redis.clients.jedis.Protocol.process(Protocol.java:161)
at redis.clients.jedis.Protocol.read(Protocol.java:215)
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340)
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239)
at redis.clients.jedis.BinaryJedis.auth(BinaryJedis.java:2139)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:108)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 19 more

 

I think this should be configurable in one of the FME Server configuration file, but perhaps @hkingsbury​ knows where to look he's pretty expert in FME Server.

Userlevel 5
Badge +29

Have a look at what @jlutherthomas​  posted here around some more general redis debugging steps

https://community.safe.com/s/question/0D54Q000080hYGm/no-fme-server-engines-available-in-20181-failed-to-connect-to-job-queue

 

Its too late here (10pm) for me to start diving into some FME Server config files so will have a look in the morning

Have a look at what @jlutherthomas​  posted here around some more general redis debugging steps

https://community.safe.com/s/question/0D54Q000080hYGm/no-fme-server-engines-available-in-20181-failed-to-connect-to-job-queue

 

Its too late here (10pm) for me to start diving into some FME Server config files so will have a look in the morning

Thank you. I've checked this post before. We had another application using redis with the same default port. I changed the FME redis server port to a new one, but can't find any information about how to config the FME server to use the new port defined.

Userlevel 5
Badge +29

Thank you. I've checked this post before. We had another application using redis with the same default port. I changed the FME redis server port to a new one, but can't find any information about how to config the FME server to use the new port defined. 

Looking through the config files there seems to be two locations with references to that port (i'm looking at 2020.1)

 

line 181 in fmeCommonConfig.txt

QUEUE_PORT=6379

line 245 in processMonitorConfigCore.txt

STOP_SYSTEM_Queue="C:/Program Files/FMEServer/Utilities/redis/redis-cli.exe" -p 6379 -a sozPpbLfgdI9WJoPejNMpSxGw shutdown

I cant see any other places this may be referenced... of course there might be others!

 

As always, take a back up of those two config files before you make any changes :)

Thank you. I've checked this post before. We had another application using redis with the same default port. I changed the FME redis server port to a new one, but can't find any information about how to config the FME server to use the new port defined. 

Interesting, I changed those two config's ports. The fmeserver log still indicates it's looking for the default 6379 port. Testing on 2021.2. 

Tue-30-Nov-2021 09:11:03.118 AM   INFORM   fmeenginemgrnodeclient   402911 : Connecting to Job Queue localhost node host localhost port 6379.
Tue-30-Nov-2021 09:11:03.123 AM   ERROR    fmeenginemgrnodeclient   Could not load all queues due to connection issue to job router. Please try again.
COM.safe.fmeserver.api.FMEServerException: Could not load all queues due to connection issue to job router. Please try again.
at COM.safe.fmeserver.JobRouterConfig.loadAllTags(JobRouterConfig.java:188)
at COM.safe.fmeserver.JobRouterConfig.refresh(JobRouterConfig.java:194)
at COM.safe.fmeserver.JobRouterConfig.init(JobRouterConfig.java:142)
at COM.safe.fmeserver.FMEServerJobRouter.getJobRouterConfig(FMEServerJobRouter.java:296)
at COM.safe.fmeserver.FMEServerJobRouter.getCurrentJobRouterConnectInfo(FMEServerJobRouter.java:127)
at COM.safe.fmeserver.jobs.EngineManagerNodeOps.engineManagerNodeKeepAlive(EngineManagerNodeOps.java:344)
at COM.safe.fmeserver.jobs.EngineManagerNodeClient.executeLeaderOp(EngineManagerNodeClient.java:96)
at COM.safe.fmeserver.database.NodeClient.run(NodeClient.java:110)
at java.lang.Thread.run(Thread.java:748)

 

Userlevel 5
Badge +29

Thank you. I've checked this post before. We had another application using redis with the same default port. I changed the FME redis server port to a new one, but can't find any information about how to config the FME server to use the new port defined.

Hmmm, might be worth touching base with support, they'll be able to put you in touch with someone on the server team who will know a lot more than I do!

Badge +11

@brandonguo

 

Please submit a support case on this one.

I've not known of anyone who's modified the Redis port with FME Server. Perhaps we (Safe), might consider changing this port to the 70xx range to align with our other FME Server related ports.

 

Just in case, please ensure you are stopping FME Server completely, that means to say, check the Task Manager to ensure there are no FME Server processes running in the Task Manager. Then startup. If the error remains, it is possible we have something buried deep in code. But hopefully not! I can give this a test once the case is submitted. Cheers.

 

@brandonguo

 

Please submit a support case on this one.

I've not known of anyone who's modified the Redis port with FME Server. Perhaps we (Safe), might consider changing this port to the 70xx range to align with our other FME Server related ports.

 

Just in case, please ensure you are stopping FME Server completely, that means to say, check the Task Manager to ensure there are no FME Server processes running in the Task Manager. Then startup. If the error remains, it is possible we have something buried deep in code. But hopefully not! I can give this a test once the case is submitted. Cheers.

 

Thanks, Steve. Case submitted.

Badge +11

Hi everyone, I've just learned that it is not possible with this version (or previous) to update the FME Server redis port from 6379. An issue will be filed. You'll need to move FME Server to another system, or Remove the Redis installation, or update the existing Redis installation to use another port.

Reply