Hello,
I am seeing retransmissions from nodes and achlowledgement downlink only seen at latest uplink. So, I assume node is not receiving achnowledgemt and retransmitting same frame. I also noticed that such retransmissions happen only where gateway is using mqtt. Gateway with UDP forwarder are fine.
image869×211 14.2 KB
image889×577 41 KB
When I checked mosquitto logs on server side I see GW is trying to open new connection (same IP but different port) every 2-3 seconds, so mqtt is dropping previous connection and opening new one. it is most probably due to this reconnections that some downlinks are lost.
1758378584: eu868/gateway/0016c001ff1e958f/state/conn
1758378584: Sending CONNACK to 0016c001ff1e958f (0, 0)
1758378584: Received SUBSCRIBE from 0016c001ff1e958f
1758378584: eu868/gateway/0016c001ff1e958f/command/+ (QoS 0)
1758378584: 0016c001ff1e958f 0 eu868/gateway/0016c001ff1e958f/command/+
1758378584: Sending SUBACK to 0016c001ff1e958f
1758378584: Received PUBLISH from 0016c001ff1e958f (d0, q0, r1, m0, 'eu868/gateway/0016c001ff1e958f/state/conn', ... (20 bytes))
1758378585: Client 0016c001ff1e958f already connected, closing old connection.
1758378585: New client connected from 37.75.133.70:36802 as 0016c001ff1e958f (p5, c0, k30, u'apnisiGW').
1758378585: eu868/gateway/0016c001ff1e958f/state/conn
1758378585: Sending CONNACK to 0016c001ff1e958f (0, 0)
1758378585: Received SUBSCRIBE from 0016c001ff1e958f
1758378585: eu868/gateway/0016c001ff1e958f/command/+ (QoS 0)
1758378585: 0016c001ff1e958f 0 eu868/gateway/0016c001ff1e958f/command/+
1758378585: Sending SUBACK to 0016c001ff1e958f
1758378585: Received PUBLISH from 0016c001ff1e958f (d0, q0, r1, m0, 'eu868/gateway/0016c001ff1e958f/state/conn', ... (20 bytes))
1758378586: Client 0016c001ff1e958f already connected, closing old connection.
1758378586: New client connected from 37.75.133.70:36812 as 0016c001ff1e958f (p5, c0, k30, u'apnisiGW').
1758378586: eu868/gateway/0016c001ff1e958f/state/conn
1758378586: Sending CONNACK to 0016c001ff1e958f (0, 0)
1758378586: Received SUBSCRIBE from 0016c001ff1e958f
1758378586: eu868/gateway/0016c001ff1e958f/command/+ (QoS 0)
1758378586: 0016c001ff1e958f 0 eu868/gateway/0016c001ff1e958f/command/+
1758378586: Sending SUBACK to 0016c001ff1e958f
1758378586: Received PUBLISH from 0016c001ff1e958f (d0, q0, r1, m0, 'eu868/gateway/0016c001ff1e958f/state/conn', ... (20 bytes))
1758378587: Client 0016c001ff1e958f already connected, closing old connection.
1758378587: New client connected from 37.75.133.70:54504 as 0016c001ff1e958f (p5, c0, k30, u'apnisiGW').
1758378587: eu868/gateway/0016c001ff1e958f/state/conn
1758378587: Sending CONNACK to 0016c001ff1e958f (0, 0)
1758378587: Received SUBSCRIBE from 0016c001ff1e958f
1758378587: eu868/gateway/0016c001ff1e958f/command/+ (QoS 0)
1758378587: 0016c001ff1e958f 0 eu868/gateway/0016c001ff1e958f/command/+
1758378587: Sending SUBACK to 0016c001ff1e958f
1758378587: Received PUBLISH from 0016c001ff1e958f (d0, q0, r1, m0, 'eu868/gateway/0016c001ff1e958f/state/conn', ... (20 bytes))
1758378588: Client 0016c001ff1e958f already connected, closing old connection.
1758378588: New client connected from 37.75.133.70:54510 as 0016c001ff1e958f (p5, c0, k30, u'apnisiGW').
1758378588: eu868/gateway/0016c001ff1e958f/state/conn
1758378588: Sending CONNACK to 0016c001ff1e958f (0, 0)
1758378589: Received SUBSCRIBE from 0016c001ff1e958f
1758378589: eu868/gateway/0016c001ff1e958f/command/+ (QoS 0)
1758378589: 0016c001ff1e958f 0 eu868/gateway/0016c001ff1e958f/command/+
1758378589: Sending SUBACK to 0016c001ff1e958f
1758378589: Received PUBLISH from 0016c001ff1e958f (d0, q0, r1, m0, 'eu868/gateway/0016c001ff1e958f/state/conn', ... (20 bytes))
From journalctl -u mosquitto I see another issue, but this one looks originating from server side as MQTT service is restarting from time to time (but intervals are higher here, several hours). So this is probably secondary issue, not the main root cause:
Sep 17 06:57:39 ApnisiChirpstack01 mosquitto[896829]: warning: can't get client address: Connection reset by peer
Sep 18 00:00:02 ApnisiChirpstack01 systemd[1]: Reloading mosquitto.service - Mosquitto MQTT Broker...
Sep 18 00:00:02 ApnisiChirpstack01 systemd[1]: Reloaded mosquitto.service - Mosquitto MQTT Broker.
Sep 19 00:00:02 ApnisiChirpstack01 systemd[1]: Reloading mosquitto.service - Mosquitto MQTT Broker...
Sep 19 00:00:02 ApnisiChirpstack01 systemd[1]: Reloaded mosquitto.service - Mosquitto MQTT Broker.
Sep 20 00:00:02 ApnisiChirpstack01 systemd[1]: Reloading mosquitto.service - Mosquitto MQTT Broker...
Sep 20 00:00:02 ApnisiChirpstack01 systemd[1]: Reloaded mosquitto.service - Mosquitto MQTT Broker.
Sep 20 02:10:56 ApnisiChirpstack01 mosquitto[896829]: warning: can't get client address: Connection reset by peer
Sep 20 18:08:33 ApnisiChirpstack01 systemd[1]: Stopping mosquitto.service - Mosquitto MQTT Broker...
Sep 20 18:08:33 ApnisiChirpstack01 systemd[1]: mosquitto.service: Deactivated successfully.
Sep 20 18:08:33 ApnisiChirpstack01 systemd[1]: Stopped mosquitto.service - Mosquitto MQTT Broker.
Sep 20 18:08:33 ApnisiChirpstack01 systemd[1]: mosquitto.service: Consumed 25min 39.302s CPU time.
Sep 20 18:08:33 ApnisiChirpstack01 systemd[1]: Starting mosquitto.service - Mosquitto MQTT Broker...
Sep 20 18:08:33 ApnisiChirpstack01 mosquitto[1124203]: 1758377313: Loading config file /etc/mosquitto/conf.d/default.conf
Sep 20 18:08:33 ApnisiChirpstack01 systemd[1]: Started mosquitto.service - Mosquitto MQTT Broker.
On gateway side, system logs I see downlink is received, but there are MQTT errors as well “closed by peer”. Seems from Server side each gateway uplink is treated as different client (even though it is coming from same IP) and server is thus dropping previous connection
Sat Sep 20 19:56:29 2025 user.info chirpstack-mqtt-forwarder[2345]: Received uplink frame, uplink_id: 2228320446
Sat Sep 20 19:56:29 2025 user.info chirpstack-mqtt-forwarder[2345]: Sending uplink event, uplink_id: 2228320446, topic: eu868/gateway/0016c001ff1a09b4/event/up
Sat Sep 20 19:56:30 2025 user.info chirpstack-mqtt-forwarder[2345]: Received message, topic: eu868/gateway/0016c001ff1a09b4/command/down, qos: AtMostOnce
Sat Sep 20 19:56:30 2025 user.info chirpstack-mqtt-forwarder[2345]: Received downlink command, downlink_id: 858586403, topic: eu868/gateway/0016c001ff1a09b4/command/down
Sat Sep 20 19:56:30 2025 user.info chirpstack-mqtt-forwarder[2345]: Sending downlink frame, downlink_id: 858586403
Sat Sep 20 19:56:30 2025 user.info chirpstack-gateway-mesh[2311]: Downlink command received - [downlink_id: 858586403 - [freq: 867100000, power: 16, mod: [LORA - sf: 7, bw: 125000], timing: [DELAY: 1], [freq: 869525000, power: 29, mod: [LORA - sf: 12, bw: 125000], timing: [DELAY: 2]]
Sat Sep 20 19:56:30 2025 user.info chirpstack-gateway-mesh[2311]: Proxying LoRaWAN downlink, downlink: [downlink_id: 858586403 - [freq: 867100000, power: 16, mod: [LORA - sf: 7, bw: 125000], timing: [DELAY: 1], [freq: 869525000, power: 29, mod: [LORA - sf: 12, bw: 125000], timing: [DELAY: 2]]
Sat Sep 20 19:56:30 2025 user.info chirpstack-gateway-mesh[2311]: Sending downlink frame - [downlink_id: 858586403 - [freq: 867100000, power: 16, mod: [LORA - sf: 7, bw: 125000], timing: [DELAY: 1], [freq: 869525000, power: 29, mod: [LORA - sf: 12, bw: 125000], timing: [DELAY: 2]]
Sat Sep 20 19:56:30 2025 user.info chirpstack-concentratord-sx1302[1739]: Enqueueing timestamped packet, downlink_id: 858586403, counter_us: 1412380719, current_counter_us: 1411841166
Sat Sep 20 19:56:30 2025 user.info chirpstack-concentratord-sx1302[1739]: Item tracked, band: [label: L, freq_min: 865000000, freq_max: 868000000, dc_max: 1.00%], freq: 867100000, tx_power_eirp: 16, start_counter_us: 1412380719, end_counter_us: 1412413719, duration: 33ms
Sat Sep 20 19:56:30 2025 user.info chirpstack-mqtt-forwarder[2345]: Received ack, items: ["OK", "IGNORED"], downlink_id: 858586403
Sat Sep 20 19:56:30 2025 user.info chirpstack-mqtt-forwarder[2345]: Sending ack event, downlink_id: 858586403, topic: eu868/gateway/0016c001ff1a09b4/event/ack
Sat Sep 20 19:56:30 2025 user.info chirpstack-concentratord-sx1302[1739]: Scheduled packet for TX, downlink_id: 858586403, count_us: 1412380719, freq: 867100000, bw: 125000, mod: LoRa, dr: SF7
Sat Sep 20 19:56:30 2025 user.info chirpstack-mqtt-forwarder[2334]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:30 2025 user.info chirpstack-mqtt-forwarder[2334]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:30 2025 user.err chirpstack-mqtt-forwarder[2345]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:32 2025 user.info chirpstack-mqtt-forwarder[2345]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:32 2025 user.info chirpstack-mqtt-forwarder[2345]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:32 2025 user.err chirpstack-mqtt-forwarder[2334]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:33 2025 user.info chirpstack-mqtt-forwarder[2334]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:33 2025 user.info chirpstack-mqtt-forwarder[2334]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:33 2025 user.err chirpstack-mqtt-forwarder[2345]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:34 2025 user.info chirpstack-mqtt-forwarder[2345]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:34 2025 user.info chirpstack-mqtt-forwarder[2345]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:34 2025 user.err chirpstack-mqtt-forwarder[2334]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:35 2025 user.info chirpstack-mqtt-forwarder[2334]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:35 2025 user.info chirpstack-mqtt-forwarder[2334]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:35 2025 user.err chirpstack-mqtt-forwarder[2345]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:36 2025 user.info chirpstack-mqtt-forwarder[2345]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:36 2025 user.info chirpstack-mqtt-forwarder[2345]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:36 2025 user.err chirpstack-mqtt-forwarder[2334]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:37 2025 user.info chirpstack-mqtt-forwarder[2334]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:37 2025 user.info chirpstack-mqtt-forwarder[2334]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:37 2025 user.err chirpstack-mqtt-forwarder[2345]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:39 2025 user.info chirpstack-mqtt-forwarder[2345]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:39 2025 user.info chirpstack-mqtt-forwarder[2345]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:39 2025 user.err chirpstack-mqtt-forwarder[2334]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:40 2025 user.info chirpstack-mqtt-forwarder[2334]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:40 2025 user.info chirpstack-mqtt-forwarder[2334]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:40 2025 user.err chirpstack-mqtt-forwarder[2345]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:41 2025 user.info chirpstack-mqtt-forwarder[2345]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:41 2025 user.info chirpstack-mqtt-forwarder[2345]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:41 2025 user.err chirpstack-mqtt-forwarder[2334]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:42 2025 user.info chirpstack-mqtt-forwarder[2334]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:42 2025 user.info chirpstack-mqtt-forwarder[2334]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:42 2025 user.err chirpstack-mqtt-forwarder[2345]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:43 2025 user.err chirpstack-mqtt-forwarder[2334]: MQTT error, error: Mqtt state: Io error: Custom { kind: ConnectionAborted, error: "connection closed by peer" }
Sat Sep 20 19:56:43 2025 user.info chirpstack-mqtt-forwarder[2345]: Subscribing to command topic, topic: eu868/gateway/0016c001ff1a09b4/command/+
Sat Sep 20 19:56:43 2025 user.info chirpstack-mqtt-forwarder[2345]: Sending conn state, topic: eu868/gateway/0016c001ff1a09b4/state/conn
Sat Sep 20 19:56:44 2025 user.info chirpstack-concentratord-sx1302[1739]: Duty-cyle stats: [label: M, freq_min: 868000000, freq_max: 868600000, dc_max: 1.00%] - current_dc: 0.00%
Sat Sep 20 19:56:44 2025 user.info chirpstack-concentratord-sx1302[1739]: Duty-cyle stats: [label: L, freq_min: 865000000, freq_max: 868000000, dc_max: 1.00%] - current_dc: 0.00%
Sat Sep 20 19:56:44 2025 user.info chirpstack-concentratord-sx1302[1739]: Publishing stats event, rx_received: 1, rx_received_ok: 1, tx_received: 1, tx_emitted: 1
Unfortunately on GW Luci GUI, I did not find any configuration options for MQTT exept of QoS and Clean session. Currently QoS is 0 and Clean session is unticked. Can anyone suggest why I am seeing this behavior and how to fix it?
Server:
chirpstack 4.14.0
MQTT version 5.0/3.1.1/3.1
Gateway:
OS 4.8.2 Base
