Azure Managed Redis Error

Question

Hello,

We are trying to upgrade chirpstack from v4.10.1 to the latest version (v4.15.0).

At the same time, we are migrating from Azure Cache for Redis (Redis v6.0) to Azure Managed Redis (Redis v 7.4) since the latest chirpstack version is incompatible with Redis v6.0 and Azure Cache for Redis is being discontinued and does not support a later redis version that chirpstack needs.

With the new Redis we noticed that we are missing a lot of uplinks (roughly 75%). In the logs we can find the following errors since the redis switch:

ERROR chirpstack::uplink::stats: Handle gateway stats error error=Save gateway stats: An error was signalled by the server - Moved: 9254 :8500

ERROR chirpstack::uplink: Deduplication error error=Deduplication put and get lock: An error was signalled by the server - Moved: 3825 :8500

ERROR chirpstack::api::internal: Reading frame-log returned error: XREAD frame stream

At the same time, we can see some device uplinks and LoRa frames in chirpstack, so it's not entirely broken.

Is there some configuration we are missing? All other services did not need any configuration change (apart from host:ip & credentials) and work fine with the new redis instance.

Our chirpstack config for redis looks like this:

[redis]
servers=$REDIS__SERVERS
cluster=false
key_prefix="$REDIS__KEY_PREFIX"
max_open_connections=100
min_idle_connections=10

With the following env vars:

REDIS__SERVERS=["rediss://:redacted@redacted:10000"]
REDIS__KEY_PREFIX=csv4

For reference, this is how we provision the redis instance through terraform:

resource "azurerm_managed_redis" "chirpstack_redis" {
  name                = "redacted"
  resource_group_name = "redacted"
  location            = "redacted"
  sku_name            = "Balanced_B0"

  default_database {
    client_protocol = "Encrypted"
    access_keys_authentication_enabled = true
    eviction_policy = "VolatileLRU"
    clustering_policy = "OSSCluster"
  }
}

Any help in getting Chirpstack to run with Azure Managed Redis is appreciated.

Edit:

I also tried to enable cluster mode, but then chirpstack is not able to connect at all:

ERROR chirpstack::uplink::stats: Handle gateway stats error error=Save gateway stats: No connections found- ClusterConnectionNotFound

Pascal Ludwig · Accepted Answer

Turns out, chirpstack's redis client library does not seem to support Redis Clustering (properly?).

As per https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/cache-best-practices-enterprise-tiers#clustering-on-enterprise

The OSS clustering policy implements the same Redis Cluster API as open-source Redis. The Redis Cluster API allows the Redis client to connect directly to each Redis node, minimizing latency and optimizing network throughput. As a result, near-linear scalability is obtained when scaling out the cluster with more nodes. The OSS clustering policy generally provides the best latency and throughput performance, but requires your client library to support Redis Clustering. OSS clustering policy also can't be used with the RediSearch module.

The Enterprise clustering policy is a simpler configuration that utilizes a single endpoint for all client connections. Using the Enterprise clustering policy routes all requests to a single Redis node that is then used as a proxy, internally routing requests to the correct node in the cluster. The advantage of this approach is that Redis client libraries don’t need to support Redis Clustering to take advantage of multiple nodes. The downside is that the single node proxy can be a bottleneck, in either compute utilization or network throughput. The Enterprise clustering policy is the only one that can be used with the RediSearch module.

I switched redis to clustering_policy = "EnterpriseCluster" and with that everything seems to work fine.

I would think the cluster=true is for this case, but as I mentioned in the OP, that didn't work for me with OSSCluster. Maybe I'm missing something else?

For now I'll live without OSS, though it would be great to be able to switch to that in the future.

Azure Managed Redis Error

1 Answers