Chirpstack v3 networkserver stops handling most (but not all) traffic when switched over to new MQTT broker

Question

We have this Chirpstack v3 stack, installed a few years ago using the official containers and docker-compose.yml file. Among the components, there is an Eclipse Mosquitto broker for a MQTT backend.

(There are currently no plans for migration to a v4 stack)

When we placed this stack in production, we failed to realize that the MQTT broker was running with absoulutely no authentication, no passwords, nada. This stack grew and now has 2 networkserver instances (for regions US915 and AU915), and more than 40 active gateway bridges running in RPi4 installed across the country, all connected to this unauthenticated MQTT broker. I am now tasked with adding authentication to the MQTT broker.

The strategy we decided is to spin up a new MQTT broker (another Mosquitto instance), with the proper authentication in place. This new broker is configured as a bidirectional bridge with the old broker on the topics application/# and gateway#, so that all traffic published on either broker is forwarded to the other. After this, we planned to switch over each of the components of the Chirpstack v3 stack, one at a time, so that, at every moment, we had a fully functional system, with as little downtime as possible.

With every component, the entire switchover procedure was to change the server directive from "tcp://{OLDBROKER}:1883" to "ssl://{NEWBROKER}:8883", add username=... and password=... as required, and restart the component. No other directive of the component was supposed to be changed.

We started with the chirpstack-application-server component. The switchover was completed, with apparently no issues.

When we tried next to do the same to one of our network servers (the AU915 one), we realized that the amount of messages published by the application server to the topics under application/# dropped to almost (but not exactly) zero. Instead of several application packets published per second, there were only one every minute or so. Obviously there is a huge packet loss somewhere. The only way to fix this so far is reverting the change and switching the network server back to the old broker.

We have already checked the following:

The new broker uses the mosquitto dynauth plugin. The ACLs for both the application server and the network server appear to be OK, and such an incompatibility would be inconsistent with the observation that a few packets manage to be published, just not all the expected amount.
The Google AI first suggested that there might be a false loop detection caused by the MQTT bridge configuration. Changing this as suggested had no effect.
Then the Google AI suggested flushing the redis cache and immediately restarting the network server with the new broker. This also had no effect, and I think it additionally caused a few cached OTAA sessions to be lost.
The packets seem to reach the network server in both cases (old and new broker), but when switching over to the new broker, something causes the network server to ignore most of the traffic. I enabled log_level=5 for debug, and I see the SQL queries to Postgresql, but no further clue of why most of the gateway traffic is ignored.

Orne Brocaar · Answer

What might be an easier approach is configuring two listeners:

One for accepting external connections (e.g. publicly available over the network)
Once which is only accessible for localhost connections

E.g. you could use something like this (make updates where needed with regards to TLS / passwords etc... this config works with client-certificates as documented here https://www.chirpstack.io/docs/guides/mosquitto-tls-configuration.html, this config demonstrates how to setup two listeners)

per_listener_settings true

listener 1883 127.0.0.1
allow_anonymous true

listener 8883
cafile /etc/mosquitto/certs/ca.pem
certfile /etc/mosquitto/certs/server.pem
keyfile /etc/mosquitto/certs/server-key.pem
allow_anonymous false
require_certificate true
use_identity_as_username true
acl_file /etc/mosquitto/acl

Chirpstack v3 networkserver stops handling most (but not all) traffic when switched over to new MQTT broker

1 Answers