@alexpillonel FME does not yet support Rabbit AMQP and therefore doesn't support RabbitMQ. I have added your post to the product change request we have to support RabbitMQ. Perhaps consider posting your request to the Ideas page to see if there are other users interested in RabbitMQ support
I finally
had some success with my JMS RabbitMQ configuration on FME Server. Here the
results of my research :
RabbitMQ
For
information, I send message to the RabbitMQ server via an Exchange (ex-fme-flight-info) binded to a queue (qu-cep-flight-info)
Configuration
on the server where FME Server is installed:
- I copied
rabbitmq-jms client and its dependencies in the FME Server folder :
"../Server/lib/jms" - I configured a JNDI bindings
file with the RabbitMQ configuration : C:\jndi\.bindings
Parts of the .binding file with the important configuration (where I
broke my head ;-)) :
#Factory parameters
ConnectionFactory/ClassName=javax.jms.QueueConnectionFactory
ConnectionFactory/FactoryName=com.rabbitmq.jms.admin.RMQObjectFactory
…
…
#Destination queue
SendFlightQueue/ClassName=javax.jms.Queue
SendFlightQueue/FactoryName=com.rabbitmq.jms.admin.RMQObjectFactory
…
…
SendFlightQueue/RefAddr/2/Content=ex-fme-flight-info
SendFlightQueue/RefAddr/2/Type=amqpExchangeName
SendFlightQueue/RefAddr/2/Encoding=String
…
…
FME Server
- I created a
"Topic" : IODA_OUT_FlightCurrent - I created a
"Subscription" with the protocol JMS and defined the
"Destination" name with the name defined in the jndi binding
files :
ProtocolJMSProvider Typecom.sun.jndi.fscontext.RefFSContextFactoryProvider URLfile:/c:/jndi/Connection FactoryConnectionFactoryDestinationSendFlightQueue
- I created a FME Workspace
with a "FMEServerNotifier" transformer configured to send data
to the Topic previously created :
Unconverted
Try
I tried to send data
with the "JMSSender" transformer, but without success. FME engine
return the following error :
javax.naming.NoInitialContextException: Cannot instantiate class: com.sun.jndi.fscontext.RefFSContextFactory tRoot exception is java.lang.ClassNotFoundException: com.sun.jndi.fscontext.RefFSContextFactory]
Failed to connect to JMS provider with url "file:/c:/jndi/" and initial context "com.sun.jndi.fscontext.RefFSContextFactory"
I hope it will help
I finally
had some success with my JMS RabbitMQ configuration on FME Server. Here the
results of my research :
RabbitMQ
For
information, I send message to the RabbitMQ server via an Exchange (ex-fme-flight-info) binded to a queue (qu-cep-flight-info)
Configuration
on the server where FME Server is installed:
- I copied
rabbitmq-jms client and its dependencies in the FME Server folder :
"../Server/lib/jms" - I configured a JNDI bindings
file with the RabbitMQ configuration : C:\jndi\.bindings
Parts of the .binding file with the important configuration (where I
broke my head ;-)) :
#Factory parameters
ConnectionFactory/ClassName=javax.jms.QueueConnectionFactory
ConnectionFactory/FactoryName=com.rabbitmq.jms.admin.RMQObjectFactory
…
…
#Destination queue
SendFlightQueue/ClassName=javax.jms.Queue
SendFlightQueue/FactoryName=com.rabbitmq.jms.admin.RMQObjectFactory
…
…
SendFlightQueue/RefAddr/2/Content=ex-fme-flight-info
SendFlightQueue/RefAddr/2/Type=amqpExchangeName
SendFlightQueue/RefAddr/2/Encoding=String
…
…
FME Server
- I created a
"Topic" : IODA_OUT_FlightCurrent - I created a
"Subscription" with the protocol JMS and defined the
"Destination" name with the name defined in the jndi binding
files :
ProtocolJMSProvider Typecom.sun.jndi.fscontext.RefFSContextFactoryProvider URLfile:/c:/jndi/Connection FactoryConnectionFactoryDestinationSendFlightQueue
- I created a FME Workspace
with a "FMEServerNotifier" transformer configured to send data
to the Topic previously created :
Unconverted
Try
I tried to send data
with the "JMSSender" transformer, but without success. FME engine
return the following error :
javax.naming.NoInitialContextException: Cannot instantiate class: com.sun.jndi.fscontext.RefFSContextFactory tRoot exception is java.lang.ClassNotFoundException: com.sun.jndi.fscontext.RefFSContextFactory]
Failed to connect to JMS provider with url "file:/c:/jndi/" and initial context "com.sun.jndi.fscontext.RefFSContextFactory"
I hope it will help
Thanks for posting this information. Yes, I hope by having it here some other users will benefit from the knowledge.
Nice work, Alex! Your answer would have saved me a bunch of time--I must have missed it by a day or two. Just some additional details, I was able to get both the JMSSender and JMSReceiver working in a local Workspace as well as get FME Server Pub/Sub working. I've posted the .bindings file that works for us.
For the local workspace, you need to make sure that the Jar containing com.sun.jndi.fscontext.RefFSContextFactory is somewhere in your class path.. I used fscontext-4.5-b25.jar from Maven here: https://mvnrepository.com/artifact/com.sun.messaging.mq/fscontext/4.5-b25. As a quick hack, I put this Jar as well as the RabbitMQ JMS jars into my default FME Java Plugins directory: C:\Users\<USERNAME>\Documents\FME\Plugins\Java.
Configuring log4j on top of the slf4j was useful for debugging--the RabbitMQ client code does output some decent traces. I put a screenshot of all the jars I used to get this working below.
Bindings File (in c:\RabbitMQBindings\.bindings for our config)
ConnectionFactory/ClassName=javax.jms.ConnectionFactory
ConnectionFactory/FactoryName=com.rabbitmq.jms.admin.RMQObjectFactory
#We are using a virtual host
ConnectionFactory/RefAddr/0/Content=fmetest
ConnectionFactory/RefAddr/0/Type=virtualHost
ConnectionFactory/RefAddr/0/Encoding=String
ConnectionFactory/RefAddr/1/Content=127.0.0.1
ConnectionFactory/RefAddr/1/Type=host
ConnectionFactory/RefAddr/1/Encoding=String
# JMS Queue where we have a RabbitMQ queue called "JMS2" boundto an exchange called "jms.durable.queues:.
# I named this very confusinginly, but FME will connect to this using the "JMS"
# destination. See screenshot below.
JMS/ClassName=javax.jms.Queue
JMS/FactoryName=com.rabbitmq.jms.admin.RMQObjectFactory
JMS/RefAddr/0/Content=jms/Queue
JMS/RefAddr/0/Type=name
JMS/RefAddr/0/Encoding=String
JMS/RefAddr/1/Content=true
JMS/RefAddr/1/Type=amqp
JMS/RefAddr/1/Encoding=String
JMS/RefAddr/2/Content=JMS
JMS/RefAddr/2/Type=amqpRoutingKey
JMS/RefAddr/2/Encoding=String
JMS/RefAddr/3/Content=com.rabbitmq.jms.admin.RMQDestination
JMS/RefAddr/3/Type=type
JMS/RefAddr/3/Encoding=String
JMS/RefAddr/4/Content=jms.durable.queues
JMS/RefAddr/4/Type=amqpExchangeName
JMS/RefAddr/4/Encoding=String
JMS/RefAddr/5/Content=JMS2
JMS/RefAddr/5/Type=destinationName
JMS/RefAddr/5/Encoding=String
JMS/RefAddr/6/Content=JMS2
JMS/RefAddr/6/Type=amqpQueueName
JMS/RefAddr/6/Encoding=String
FME Workspace Configuration
Thanks for posting this!
I tried it out and had some success.
I found that to get the JMSReceiver-transformer to actually present a jms_content I had to set the message header "JMSType" to "TextMessage"
Example using python (pika) to post message with the header:
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
msg_properties = pika.spec.BasicProperties(headers={'JMSType': 'TextMessage'})
channel.basic_publish(exchange='jms.durable.queues',
routing_key='JMS',
properties=msg_properties,
body='Hello World!')
I could not find a way to set this header using the JMSSender-transformer though so I don't know how to "eat my own dogfood" yet.
/Peter
For the record, I was able to configure JMSSender and JMSReceiver to work in FME Desktop 2019.2 recently.
However, somewhere in the FME Server 2018.1 cycle the support for RabbitMQ and the JMSPublisher stopped working. We are hoping to resolve this in a later release of FME Server. Stay tuned.
Let us know - via an idea - if you are using or looking to use FME with RabbitMQ. Thanks.
For the record, I was able to configure JMSSender and JMSReceiver to work in FME Desktop 2019.2 recently.
However, somewhere in the FME Server 2018.1 cycle the support for RabbitMQ and the JMSPublisher stopped working. We are hoping to resolve this in a later release of FME Server. Stay tuned.
Let us know - via an idea - if you are using or looking to use FME with RabbitMQ. Thanks.
I have found I can trigger topics in FME Server 2019 by setting the JMSType=TextMessage. If this isn't set in the header of the message, FME Server will not pass the message and thus won't trigger a related topic.
Here's an example of this in the RabbitMQ Web UI
@alexpillonel FME does not yet support Rabbit AMQP and therefore doesn't support RabbitMQ. I have added your post to the product change request we have to support RabbitMQ. Perhaps consider posting your request to the Ideas page to see if there are other users interested in RabbitMQ support
Just a quick update. I recently had a Case related to this subject and it raised questions if FME was able to support RabbitMQ or not.
Back in 2016, this statement was true. However, some users were still able to configure FME and have success. More recently, I have been doing some testing and discovered that for a few releases of FME Server (2018.1, 2019.0, 2019.1) RabbitMQ messages were not triggering topics in FME Server. This was because of a change made.
For FME Server 2019.2 we resolved this. It still requires manual configuration but it is possible to get it to work. You'll need to provide the necessary RabbitMQ client libraries and be able to make sense of the .bindings file.
In summary, I have been able to configure FME Server to support RabbitMQ message types JMSType=TextMessage. I've also been able to use the JMSSender and JMSReciever in FME Desktop as well. This was all made possible with the help of @david_r when he discovered that things were not working in FME Server 2019.0/1 as they were before.
We have made changes to FME Server 2019.2 and even if JMSType is not set to TextMessage a notification topic will be triggered like it used to be in 2018.0 and older. Hope this helps.
If you have questions on this, please do reach out to us at support - add SteveatSafe to the message. I'm working on a KB article to share the configuration (similar to above) that was successful for me.
Just a quick update. I recently had a Case related to this subject and it raised questions if FME was able to support RabbitMQ or not.
Back in 2016, this statement was true. However, some users were still able to configure FME and have success. More recently, I have been doing some testing and discovered that for a few releases of FME Server (2018.1, 2019.0, 2019.1) RabbitMQ messages were not triggering topics in FME Server. This was because of a change made.
For FME Server 2019.2 we resolved this. It still requires manual configuration but it is possible to get it to work. You'll need to provide the necessary RabbitMQ client libraries and be able to make sense of the .bindings file.
In summary, I have been able to configure FME Server to support RabbitMQ message types JMSType=TextMessage. I've also been able to use the JMSSender and JMSReciever in FME Desktop as well. This was all made possible with the help of @david_r when he discovered that things were not working in FME Server 2019.0/1 as they were before.
We have made changes to FME Server 2019.2 and even if JMSType is not set to TextMessage a notification topic will be triggered like it used to be in 2018.0 and older. Hope this helps.
If you have questions on this, please do reach out to us at support - add SteveatSafe to the message. I'm working on a KB article to share the configuration (similar to above) that was successful for me.
I'm working on a KB article to share the configurationExcellent!
