Skip to main content
  • EN

Using a RabbitMQConnector in Stream mode to continuously listen for change events. It is set to not Auto Acknowledge messages as we do not want that to happen until confirmation it is written to the database. Was thinking to use a FeatureWriter as it chains outputs after writing. Doing this with Transaction Type of None and it inserts/updates the features in real-time fine. But the output from the FeatureWriter seems to always be blocking. We ideally want to use that output to acknowledge the message in RabbitMQ.

 

Would try a SQLExecutor but also need to write the shape to an ESRI geodatabase. Worst case can probably use a PythonCaller using arcpy. But would be nice to have a persistent connection with an actual writer.

 

Any thoughts on how to do something like this while keeping a data steam going?

 

FME Ack Messages Issue

Think we have a decent workaround but still would love any additional ideas that might be out there.

 

Our application databases such as this one are SQL Server so wasn't thinking about the fact that ESRI uses SQL Server Spatial geometries (I'm more used thinking Oracle and separate ST libs) so we can just use a query. So using a SQLExecutor to write each feature and the Initiator output flows through without blocking. Yay!

update DBO.TASK
SET 
    ...,
    SHAPE = geometry::Point(@Value(LON), @Value(LAT), 4269)  
WHERE TASK_ID = @Value(TASK_ID)

 

FYI, still not crazy about using the SQLExecutor as we need to check for nulls, dates, etc. and do some string manipulation for the insert/update query. Doable, but extra items to setup and maintain, especially frustrating when the FeaureWriter is everything we need minus the blocking.

 

So another alternative I am throwing around is to use a cloner to duplicate the record, set its fme_db_operation to something bogus like IGNORE (this seems to do nothing in the FeatureWriter), then in the FeatureWriter set the Group By to the copy number attribute with Mode to Process When Group Changes. Then drop the clone after the writer.

 

So using the Group By mode to not block with this workaround. Thoughts on if this is a bad idea? I am going to put in a feature request to add a non-blocking option to the FeatureWriter too.

 

Clone Hack To Not Block

@ryan​ we have a series of articles and a webinar on working with streaming data & FME:

These might give you some ideas on how to process you stream data.

@ryan​ we have a series of articles and a webinar on working with streaming data & FME:

These might give you some ideas on how to process you stream data.

Hi @Mark Stoakes​ . Thanks for sharing those articles. Unfortunately I still do not see a great solution. The article links "Writing to Databases When Running in Stream Mode" which talks about how to write with small transactions, but that doesn't stop FeatureWriter from blocking. And it does mention the option for groupby which is where my clone hack mentioned above comes in I guess.

 

The article also talks about "Automations" which I really want to use, but there is no RabbitMQ trigger event there. Look forward to the day when it is 🙂. There are ways to route Rabbit to JMS but I believe that requires additional components I didn't really want to get into.

Reply


Community Stats

30,189
Posts
114,589
Replies
37,854
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.