Non-Blocking FeatureWriter Output For Real-Time integrations?

Think we have a decent workaround but still would love any additional ideas that might be out there.

Our application databases such as this one are SQL Server so wasn't thinking about the fact that ESRI uses SQL Server Spatial geometries (I'm more used thinking Oracle and separate ST libs) so we can just use a query. So using a SQLExecutor to write each feature and the Initiator output flows through without blocking. Yay!

update DBO.TASK
SET 
    ...,
    SHAPE = geometry::Point(@Value(LON), @Value(LAT), 4269)  
WHERE TASK_ID = @Value(TASK_ID)

FYI, still not crazy about using the SQLExecutor as we need to check for nulls, dates, etc. and do some string manipulation for the insert/update query. Doable, but extra items to setup and maintain, especially frustrating when the FeaureWriter is everything we need minus the blocking.

So another alternative I am throwing around is to use a cloner to duplicate the record, set its fme_db_operation to something bogus like IGNORE (this seems to do nothing in the FeatureWriter), then in the FeatureWriter set the Group By to the copy number attribute with Mode to Process When Group Changes. Then drop the clone after the writer.

So using the Group By mode to not block with this workaround. Thoughts on if this is a bad idea? I am going to put in a feature request to add a non-blocking option to the FeatureWriter too.

@ryan​ we have a series of articles and a webinar on working with streaming data & FME:

• Article series: FME and Stream Processing
• Webinar recording: Introduction to Data Stream Processing

These might give you some ideas on how to process you stream data.

Hi @Mark Stoakes​ . Thanks for sharing those articles. Unfortunately I still do not see a great solution. The article links "Writing to Databases When Running in Stream Mode" which talks about how to write with small transactions, but that doesn't stop FeatureWriter from blocking. And it does mention the option for groupby which is where my clone hack mentioned above comes in I guess.

The article also talks about "Automations" which I really want to use, but there is no RabbitMQ trigger event there. Look forward to the day when it is :). There are ways to route Rabbit to JMS but I believe that requires additional components I didn't really want to get into.