Chirpstack-mqtt-forwarder publishes at QoS 0 — any way to enable QoS 1 for offline buffering?

Question

Hi,

I'm running a Dragino gateway (Armbian, armv7l) with chirpstack-mqtt-forwarder 4.4.0 and a local Mosquitto 2.0.21 bridging to a remote ChirpStack v4 server over WireGuard.

Architecture:

SX1302 concentrator
↓ Semtech UDP :1700
chirpstack-mqtt-forwarder 4.4.0
↓ MQTT (localhost:1883)
Mosquitto 2.0.21 (local, bridge mode)
↓ bridge over WireGuard
Mosquitto 2.0.22 (VPS) → ChirpStack v4.18

Problem: during internet outages, roughly 33-62% of uplinks are lost and never delivered after reconnection, despite a bridge configuration built for buffering.

Bridge config:

queue_qos0_messages true
max_queued_messages 100000
cleansession false
local_cleansession false
topic eu868/# out 1
topic eu868/# in 1
max_inflight_messages 1
restart_timeout 5 30

What I verified with logs during outages:

Frames are correctly received by the local broker: Received PUBLISH from (d0, q0, r0, m0, 'eu868/gateway/.../event/up', ...)
Note the q0 — the forwarder publishes at QoS 0
After reconnection, only a fraction of the queued frames appear as "Sending PUBLISH to local...-bridge"

What I already tested without success:

Upgraded Mosquitto 2.0.11 → 2.0.21 (no measurable change over a 10h outage test)
Explicit client_id in the forwarder config
Direct connection (forwarder → remote broker, bypassing local Mosquitto) — worse, no buffering at all, only "MQTT error: Network unreachable" repeated every second with no queued replay
AT+CFM=0 on the end device (this one did help: 59% → 33% loss)

My question: since queue_qos0_messages is documented as non-standard and QoS 0 has no delivery guarantee, is there any way to make chirpstack-mqtt-forwarder publish at QoS 1? I could not find a qos option in the configuration reference.

If not, is there a recommended approach for reliable store-and-forward on the gateway during long outages? My use case is water metering where gaps create visible artifacts in consumption graphs.

Thanks a lot for any pointers.

Orne Brocaar · Accepted Answer

Please see the ChirpStack MQTT Forwarder configuration documentation:
https://www.chirpstack.io/docs/chirpstack-mqtt-forwarder/configuration.html

There is a qos option under the [mqtt] section:

  # Quality of service level
  #
  # 0: at most once
  # 1: at least once
  # 2: exactly once
  #
  # Note: an increase of this value will decrease the performance.
  # For more information: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels
  qos=0

HAMANI Saddek · Answer

Thanks to @brocaar for the earlier reply on the support thread about qos — this is a follow-up with a more precise root cause and reproducible data.

Environment
ChirpStack 4.18.0 (Docker), MQTT gateway backend (rumqttc). chirpstack-mqtt-forwarder 4.4.0, Dragino LPS8v2, Mosquitto bridge over WAN (RTT ~25-55ms).

Problem

After a gateway reconnects following an outage, the Mosquitto bridge correctly replays all buffered uplinks (zero drops at broker level, confirmed via $SYS/broker/messages/stored). ChirpStack then processes several uplinks from the same device concurrently, and a frame with an older FCnt processed after a newer one is rejected:

Get device-session: FCntUp already updated (dev_addr: XXX)

Root cause

In chirpstack/src/gateway/backend/mqtt.rs, each uplink on event/up is dispatched via tokio::spawn(uplink::deduplicate_uplink(...)) without awaiting completion — no per-DevAddr lock serializes FCnt validation. Likely related to the device-session lock removed in 4.15.0 ("Remove FCnt increment check from get_for_phypayload").

Mitigations tested (same device, ~8-9.5h outages, loss cross-checked via event log + docker logs | grep -c "FCntUp already updated" + downstream platform, all in agreement):

Mitigation Loss
Baseline 59-69%
max_inflight_messages 1 (both bridge ends) 23-48% (variable)

Bridge QoS 2 (~110ms inter-message spacing, ~2x RTT) 53.6% — no improvement
"Disable frame-counter validation" (Device Profile) Confirmed ineffective — rejections still logged

QoS 2 doubling the arrival spacing had no measurable effect, suggesting the bottleneck is internal processing time (Redis/Postgres in deduplicate_uplink) under burst load rather than MQTT delivery timing — rumqttc appears to ack messages independent of task completion.

Question

Is per-device serialization out of scope by design (deadlock tradeoff per 4.15.0 changelog), or would a per-DevAddr lock/queue in deduplicate_uplink be in scope for a PR? Happy to test a patch against this reproduction.

Chirpstack-mqtt-forwarder publishes at QoS 0 — any way to enable QoS 1 for offline buffering?

2 Answers