Question

How to get notified about MQTT-connectors disconnecting?


We sometimes have a problem with our MQTT-connector that is disconnecting (always running on our FME Server), but it only shows in the log as a warning.

 

21318 2021-5-17 11:08:21 | MQTTConnector: Unexpectedly disconnected from broker. Will attempt to reconnect automatically... Error: Connection Refused: unacceptable protocol version.

 

Restarting it works just fine, but it didn't try to reconnect, or at least no log message about it. It was haulted like that for 10 days until I manually stopped & started it …

 

When this happenes, the job seems to be running like normal, and you manually has to go into the log (when running) to see that the job has stopped receiving data from the MQTT-provider.

 

I had to implement a separate test on a different FME Server job to check whether sensordata came in or not, but the optimal would be to get a notification (eg. by email) if the connection is not reestablished within a few seconds.

 

Looking forward to suggestions & future updates.

 

Kenneth.

 

Edit:

We are planning to move this FME Server job to 'Streams'. Will this help with our problem, and if so, how?

 

Edit2:

When the job is running, it's possible to go in and check what's happening. When the job is stopped (manually, as it is set to always run), it is sometimes not possible to see the log – Error: The log file for job ID "4" does not exist. Other times it shows the log.

 

Kind of strange …


2 replies

Hi @kenneth_oslo​ , did you solve the problem? I'm in the same situation right now.

Hi @Jacob Million​. Our project has now ended, but no, we didn't find any other workaround except having two different scheduled jobs. One checking the log every 20 mins, and the other hourly (could indeed have been more often) getting data from the data providers (brokers) API. But in fact, this proved to be quite a good solution after all, as the broker sometimes had lag or other issues on their side.

The "log checking job" was simply checking if any activity last 30 mins, and if not, sending email warning. The job fetching data from the API, was comparing the received data with the data stored in our database from the MQTT Stream. If the number of features with mismatching data was 5 or more, it sent an email warning. Note that if you have a decent amount of sensor activity, it should be higher than 0, because of some ms latency, so you would then expect a few features showing mismatching data (API call vs MQTT). The number of features with mismatching data accepted, should be tweaked until it seems right. When receiving email warnings, we had to manually check what could be wrong. Sometimes the MQTT Stream just needed a restart, and sometimes our data supplier had to fix a problem on their side (broker). In general, these things didn't happen more than perhaps 7-8 times total during a 12 month period after we setup the MQTT job as a Stream on FME Server, except sometimes during holidays when sensor activity was abnormally low during night time (false warnings).

Oh, one more thing: Depending on the QoS (Quality of Service) set up on the MQTT Stream (and by the broker), you might want to regularly run API calls anyways, and use the fetched data to update the features in your database that's has failed to update via MQTT.

GL!

Reply