No device uplink data published to MQTT (gateways communicate fine)

Viewed 63

About 10/5 my standard chirpstack installation stopped getting gateway messages and stopped sending application messages to MQTT. MQTT is working OK with chirpstack-gateway-bridge. gateway messages show up on MQTT. No application messages. I don't think I changed anything that would cause that. Debug logs are not helpful. No errors and no messages being sent to MQTT. Now running Chirpstack 4.15.0. No change. Any help on troubleshooting this problem? Read all the docs as far as I know. It was working perfectly previously.

Kevin

2 Answers

Good questions. Activity in the gateway LoRaWAN frames tab also stopped before 10/6. No application/xxx/device/xxx/event/up on MQTT. Lots of gateway messages on MQTT however. No messages on the Events tab of the devices. As I mentioned, logs have not been helpful. Without log messages I'm lost as to what has gone wrong. See below, no application messages at all. My own gateway is shown as connected and receiving events. Chirpstack doesn't see anything.

192.168.2.223
gateway
2f8e59802eb7f497
715095fb124cab5d
477e6973c1d6ad62
b827ebfffe721c8a
state
event
stats = �Ŵ� �dp��E@i:;�Q�_@ (�b827ebfffe721c8a
up = 4�����+���P��ߪ�Q���Ԓݲ0HY(�j�������!mȋd�2r
�|-"���� �� (*> b827ebfffe721c8a� �Ĵ�����0���������=33A@jx��T�
b73e349f5744cc96
7dd4d42118809f86

Oct 13 12:10:52 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:10:52.027497Z INFO gRPC{uri=/api.FuotaService/ListDeployments}: chirpstack::api: Finished processing request status="200" latency=19.344054ms
Oct 13 12:10:52 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:10:52.029817Z INFO gRPC{uri=/api.MulticastGroupService/List}: chirpstack::api: Finished processing request status="200" latency=22.14889ms
Oct 13 12:10:52 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:10:52.033440Z INFO gRPC{uri=/api.FuotaService/ListDeployments}: chirpstack::api: Finished processing request status="200" latency=21.572172ms
Oct 13 12:10:52 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:10:52.096038Z INFO gRPC{uri=/api.MulticastGroupService/List}: chirpstack::api: Finished processing request status="200" latency=14.000373ms
Oct 13 12:10:52 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:10:52.097092Z INFO gRPC{uri=/api.MulticastGroupService/List}: chirpstack::api: Finished processing request status="200" latency=13.342927ms
Oct 13 12:10:52 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:10:52.097935Z INFO gRPC{uri=/api.FuotaService/ListDeployments}: chirpstack::api: Finished processing request status="200" latency=14.839329ms
Oct 13 12:10:52 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:10:52.101485Z INFO gRPC{uri=/api.FuotaService/ListDeployments}: chirpstack::api: Finished processing request status="200" latency=15.682243ms
Oct 13 12:10:52 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:10:52.103409Z INFO gRPC{uri=/api.FuotaService/ListDeployments}: chirpstack::api: Finished processing request status="200" latency=16.054949ms
Oct 13 12:10:52 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:10:52.105713Z INFO gRPC{uri=/api.MulticastGroupService/List}: chirpstack::api: Finished processing request status="200" latency=19.11395ms
Oct 13 12:10:52 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:10:52.157121Z INFO gRPC{uri=/api.GatewayService/List}: chirpstack::api: Finished processing request status="200" latency=10.138302ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.266244Z INFO gRPC{uri=/api.InternalService/Login}: chirpstack::api: Finished processing request status="200" latency=3.78978ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.335388Z INFO gRPC{uri=/api.InternalService/Profile}: chirpstack::api: Finished processing request status="200" latency=6.970759ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.414577Z INFO gRPC{uri=/api.TenantService/List}: chirpstack::api: Finished processing request status="200" latency=14.791099ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.469612Z INFO gRPC{uri=/api.InternalService/GetVersion}: chirpstack::api: Finished processing request status="200" latency=2.48921ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.493281Z INFO gRPC{uri=/api.InternalService/GetVersion}: chirpstack::api: Finished processing request status="200" latency=2.716293ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.542725Z INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=3.613999ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.569023Z INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=5.188734ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.580530Z INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=6.60701ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.591545Z INFO gRPC{uri=/api.InternalService/GetVersion}: chirpstack::api: Finished processing request status="200" latency=5.722065ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.676301Z INFO gRPC{uri=/api.InternalService/GetGatewaysSummary}: chirpstack::api: Finished processing request status="200" latency=23.708622ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.686781Z INFO gRPC{uri=/api.GatewayService/List}: chirpstack::api: Finished processing request status="200" latency=17.206924ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.689458Z INFO gRPC{uri=/api.InternalService/GetDevicesSummary}: chirpstack::api: Finished processing request status="200" latency=27.933191ms
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.799461Z INFO gRPC{uri=/assets/markers-soft@2x-5qZIy0Cz.png}: chirpstack::api: Finished processing request status="200" latency=665.154µs
Oct 13 12:12:48 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:48.807135Z INFO gRPC{uri=/assets/fa-solid-900-CTAAxXor.woff2}: chirpstack::api: Finished processing request status="200" latency=667.654µs
Oct 13 12:12:49 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:49.194753Z INFO gRPC{uri=/assets/fa-regular-400-BjRzuEpd.woff2}: chirpstack::api: Finished processing request status="200" latency=531.873µs
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.189397Z INFO gRPC{uri=/api.InternalService/GetVersion}: chirpstack::api: Finished processing request status="200" latency=5.748471ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.196135Z INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=13.40152ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.202668Z INFO gRPC{uri=/api.ApplicationService/List}: chirpstack::api: Finished processing request status="200" latency=16.214635ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.205743Z INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=7.917735ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.215231Z INFO gRPC{uri=/api.GatewayService/List}: chirpstack::api: Finished processing request status="200" latency=29.58277ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.284581Z INFO gRPC{uri=/api.MulticastGroupService/List}: chirpstack::api: Finished processing request status="200" latency=15.789533ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.286493Z INFO gRPC{uri=/api.MulticastGroupService/List}: chirpstack::api: Finished processing request status="200" latency=16.095261ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.288112Z INFO gRPC{uri=/api.FuotaService/ListDeployments}: chirpstack::api: Finished processing request status="200" latency=18.443951ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.288081Z INFO gRPC{uri=/api.FuotaService/ListDeployments}: chirpstack::api: Finished processing request status="200" latency=17.231612ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.293444Z INFO gRPC{uri=/api.FuotaService/ListDeployments}: chirpstack::api: Finished processing request status="200" latency=16.935154ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.351935Z INFO gRPC{uri=/api.MulticastGroupService/List}: chirpstack::api: Finished processing request status="200" latency=10.95986ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.357483Z INFO gRPC{uri=/api.MulticastGroupService/List}: chirpstack::api: Finished processing request status="200" latency=13.816935ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.362032Z INFO gRPC{uri=/api.FuotaService/ListDeployments}: chirpstack::api: Finished processing request status="200" latency=16.944113ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.364623Z INFO gRPC{uri=/api.MulticastGroupService/List}: chirpstack::api: Finished processing request status="200" latency=16.956613ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.364775Z INFO gRPC{uri=/api.FuotaService/ListDeployments}: chirpstack::api: Finished processing request status="200" latency=15.701356ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.367503Z INFO gRPC{uri=/api.MulticastGroupService/List}: chirpstack::api: Finished processing request status="200" latency=16.846665ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.426167Z INFO gRPC{uri=/api.FuotaService/ListDeployments}: chirpstack::api: Finished processing request status="200" latency=13.595217ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.431153Z INFO gRPC{uri=/api.GatewayService/List}: chirpstack::api: Finished processing request status="200" latency=16.219948ms
Oct 13 12:12:51 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:51.434387Z INFO gRPC{uri=/api.GatewayService/List}: chirpstack::api: Finished processing request status="200" latency=17.743641ms
Oct 13 12:12:53 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:53.215600Z INFO gRPC{uri=/api.InternalService/GetVersion}: chirpstack::api: Finished processing request status="200" latency=5.353056ms
Oct 13 12:12:53 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:53.216788Z INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=7.259039ms
Oct 13 12:12:53 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:53.224449Z INFO gRPC{uri=/api.GatewayService/Get}: chirpstack::api: Finished processing request status="200" latency=13.69006ms
Oct 13 12:12:53 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:53.312953Z INFO gRPC{uri=/api.GatewayService/GetMetrics}: chirpstack::api: Finished processing request status="200" latency=11.030902ms
Oct 13 12:12:53 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:53.315602Z INFO gRPC{uri=/api.GatewayService/GetDutyCycleMetrics}: chirpstack::api: Finished processing request status="200" latency=13.073083ms
Oct 13 12:12:54 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:54.772537Z INFO gRPC{uri=/api.InternalService/GetVersion}: chirpstack::api: Finished processing request status="200" latency=3.499468ms
Oct 13 12:12:54 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:54.775392Z INFO gRPC{uri=/api.TenantService/Get}: chirpstack::api: Finished processing request status="200" latency=6.95128ms
Oct 13 12:12:54 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:54.780685Z INFO gRPC{uri=/api.GatewayService/Get}: chirpstack::api: Finished processing request status="200" latency=8.81768ms
Oct 13 12:12:54 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:54.781038Z INFO gRPC{uri=/api.InternalService/StreamGatewayFrames}: chirpstack::api: Finished processing request status="200" latency=10.254186ms
Oct 13 12:12:55 cytron-lora-gw chirpstack[548663]: 2025-10-13T16:12:55.100888Z INFO gRPC{uri=/api.InternalService/StreamGatewayFrames}: chirpstack::api: Finished processing request status="200" latency=2.84447ms

Chirpstack seems to be running but nothing gets in or out.

Restart Chirpstack with "debug" enabled. Nothing that helps me.

Oct 13 12:22:33 cytron-lora-gw systemd[1]: Stopping chirpstack.service - ChirpStack open-source LoRaWAN Network Server...
Oct 13 12:22:33 cytron-lora-gw systemd[1]: chirpstack.service: Deactivated successfully.
Oct 13 12:22:33 cytron-lora-gw systemd[1]: Stopped chirpstack.service - ChirpStack open-source LoRaWAN Network Server.
Oct 13 12:22:33 cytron-lora-gw systemd[1]: chirpstack.service: Consumed 1.089s CPU time.
Oct 13 12:22:33 cytron-lora-gw systemd[1]: Started chirpstack.service - ChirpStack open-source LoRaWAN Network Server.
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.584012Z INFO chirpstack::cmd::root: Starting ChirpStack LoRaWAN Network Server version="4.15.0" docs="https://www.chirpstack.io/"
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.584149Z INFO chirpstack::storage::postgres: Setting up PostgreSQL connection pool
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.585258Z INFO chirpstack::storage: Applying schema migrations
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.682188Z INFO chirpstack::storage: Setting up Redis client
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.682745Z INFO chirpstack::region: Setting up regions
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.682933Z INFO setup{common_name=US915 region_id=us915_1}: chirpstack::region: Configuring region
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.683319Z INFO chirpstack::backend::joinserver: Setting up Join Server clients
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.683379Z INFO chirpstack::backend::roaming: Setting up roaming clients
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.683398Z INFO chirpstack::adr: Setting up adr algorithms
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.683468Z INFO chirpstack::integration: Setting up global integrations
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.683489Z INFO chirpstack::integration::redis: Initializing Redis integration
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.683508Z INFO chirpstack::integration::mqtt: Initializing MQTT integration
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.738409Z INFO chirpstack::integration::mqtt: Connecting to MQTT broker server_uri=tcp://127.0.0.1:1883/ client_id=99a7cdadc9fd596c clean_session=false
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.738621Z INFO chirpstack::gateway::backend: Setting up gateway backends for the different regions
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.738648Z INFO chirpstack::gateway::backend: Setting up gateway backend for region region_id=us915_1 region_common_name=US915
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.739708Z INFO chirpstack::integration::mqtt: Starting MQTT event loop
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.742604Z INFO chirpstack::integration::mqtt: Subscribing to command topic command_topic=application/+/device/+/command/+
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.770696Z INFO chirpstack::gateway::backend::mqtt: Connecting to MQTT broker region_id=us915_1 server_uri=tcp://localhost:1883 clean_session=false client_id=b8b850333b32d2c2
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.770942Z INFO chirpstack::gateway::backend::mqtt: Starting MQTT event loop
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.770947Z INFO chirpstack::downlink: Setting up Class-B/C scheduler loop
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.771070Z INFO chirpstack::downlink: Setting up multicast scheduler loop
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.771097Z INFO chirpstack::applayer::fuota: Setting up FUOTA scheduler loop
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.771271Z INFO chirpstack::api: Setting up API interface bind=0.0.0.0:8080
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.775322Z INFO chirpstack::gateway::backend::mqtt: Subscribing to gateway event topic region_id=us915_1 event_topic=$share/chirpstack/us915_1/gateway/+/event/+
Oct 13 12:22:33 cytron-lora-gw chirpstack[725884]: 2025-10-13T16:22:33.834777Z INFO chirpstack::api::backend: Backend interfaces API interface is disabled

Kevin

If you do not see any gateway messages in the logs (and in the ChirpStack web-interface), but gateway messages are being published to MQTT, then most likely there is a mismatch between the topic-prefix the gateway sends its data to and the MQTT topic-prefix ChirpStack subscribes.

From your logs, ChirpStack subscribes to:

us915_1/gateway/+/event/+

Your gateway might not be forwarding to the us915_1 prefix. Therefore ChirpStack does not receive any data, hence the logs are not very helpful since there is no data to show.

Either update your ChirpStack configuration or update your gateway configuration (ChirpStack MQTT Forwarder) to use the correct region configuration. Make also sure that the configured channels on the gateway map to the region-configuration you are forwarding to. E.g. us915_1 prefix maps by default to US915 channels 8-15.