> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.itential.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.itential.com/_mcp/server. # Administer Redis > Routine administration and maintenance procedures for the Redis Sentinel cluster backing Itential Platform, including health checks, failover, memory management, and TLS certificate rotation. Redis serves as the message queue backing store for Itential Platform and stores authentication tokens. This guide covers the routine maintenance tasks required to keep a Redis Sentinel cluster healthy, performant, and secure. For configuration properties and how Platform connects to standalone or HA Redis topologies, see [Configure Redis and monitor health](/itential-platform/configure/redis/configure-and-monitor-health). For detailed metrics, PromQL expressions, and alert thresholds, see [Redis metrics reference](/itential-platform/monitor/metrics-reference/redis) and [Redis Sentinel metrics reference](/itential-platform/monitor/metrics-reference/redis-sentinel). ## Before you begin The commands in this guide assume the following [environment variables](/itential-platform/configure/environment-variables-properties-reference) are set: ```bash REDIS_NODES=" " REDIS_PORT=6379 SENTINEL_PORT=26379 SENTINEL_MASTER= SSH_KEY_PATH=~/.ssh/.pem REDIS_SSH_USER= ``` Retrieve `REDIS_ADMIN_PASS` and `REDIS_SENTINEL_ADMIN_PASS` from your secrets manager at the start of each maintenance session and hold them only in shell variables for the session's duration. The examples below use HashiCorp Vault syntax. Adjust the retrieval commands to match your secrets manager's API: ```bash source vault-env.sh REDIS_ADMIN_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \ $VAULT_ADDR/ \ | jq -r '.data.data.redisAdmin') REDIS_SENTINEL_ADMIN_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \ $VAULT_ADDR/ \ | jq -r '.data.data.redisSentinelAdmin') ``` Passwords must never be stored in scripts, configuration files, or shell history. Retrieve credentials from your secrets manager at runtime and let them leave scope when the session ends. The current primary is managed by Sentinel and can change at any time after a failover. Always resolve it at runtime rather than hardcoding a hostname: ```bash REDIS_PRIMARY=$(redis-cli \ -h -p $SENTINEL_PORT \ --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \ SENTINEL get-master-addr-by-name $SENTINEL_MASTER | head -1) ``` TLS configuration varies by deployment. When TLS is enabled, append `--tls --cacert ` to every `redis-cli` command in this guide. ## Check health and replication status Redis is a single-threaded, in-memory store. In a Sentinel-managed deployment, one node holds the primary role and accepts all writes while the others replicate asynchronously. If the primary becomes unreachable, Sentinel promotes a replica, but only if quorum is healthy and replication lag is acceptable at the time. Regular health checks confirm that all nodes are reachable, replication is current, and Platform has a functioning data store. ### PING all nodes ```bash for node in $REDIS_NODES; do result=$(redis-cli -h $node -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning PING 2>/dev/null) echo "$node -> $result" done ``` All nodes should respond `PONG`. Investigate any node that doesn't respond immediately; it may have crashed, lost network connectivity, or exhausted its client connection limit before Sentinel has triggered a failover. ### Check replication status ```bash redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning INFO replication ``` | Field | Expected value | | ------------------------------------------- | --------------------------------------------- | | `role` | `master` on the primary | | `connected_slaves` | Equal to the number of expected replicas | | `master_sync_in_progress` | `0` (`1` means a replica is actively syncing) | | `slave_repl_offset` vs `master_repl_offset` | Small delta indicates low lag | | `master_link_status` | `up` on all replicas | Flag any replica with `master_link_status:down` or a large offset delta; a lagging replica can't safely become primary in a failover. ### Identify the current primary ```bash redis-cli -h -p $SENTINEL_PORT \ --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \ SENTINEL get-master-addr-by-name $SENTINEL_MASTER ``` The response is two lines: the hostname on the first line and the port on the second. Resolve the primary before any write operation or maintenance task rather than relying on a cached value. ## Manage the Sentinel cluster Redis Sentinel monitors Redis nodes, detects failures, and coordinates failovers. A majority of Sentinel processes (quorum) must agree the primary is unreachable before a failover triggers. If too few Sentinel instances are running, or they disagree on the primary, automatic failover doesn't work. ### Check quorum ```bash for node in $REDIS_NODES; do echo "=== $node: quorum check ===" redis-cli -h $node -p $SENTINEL_PORT \ --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \ SENTINEL ckquorum $SENTINEL_MASTER 2>/dev/null done ``` Expect `OK N usable Sentinels. Quorum and failover authorization can be reached` on every node. In a three-node Sentinel cluster, quorum is two; if only one Sentinel is reachable, the cluster is effectively unprotected. ### List Sentinel members ```bash redis-cli -h -p $SENTINEL_PORT \ --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \ SENTINEL sentinels $SENTINEL_MASTER ``` Confirm the number of entries matches the expected Sentinel count and each shows `flags: sentinel` with no `disconnected` flag. Sentinel's view of membership can drift after a node replacement or network partition. ### Interpret the failover event timeline When reviewing Sentinel logs after an unexpected failover, events appear in a predictable order: | Event | Meaning | | --------------------- | ---------------------------------------------------- | | `+sdown` | One Sentinel marked the primary subjectively down | | `+odown` | Quorum agreed the primary is objectively down | | `+failover-triggered` | A Sentinel was elected to lead the failover | | `+elected-leader` | The leading Sentinel is confirmed | | `+promoted-slave` | A replica was promoted to primary | | `-odown` | The former primary was demoted; failover is complete | A `+sdown` never followed by `+odown` means quorum wasn't reached and no failover occurred, indicating a split-brain or Sentinel availability problem. ## Manage Redis and Sentinel services Redis and Redis Sentinel run as separate systemd services managed independently. Restarting the primary triggers a Sentinel failover; stopping a majority of Sentinel instances prevents any future failover. Always restart replicas before the primary. ### Check status on all nodes ```bash for node in $REDIS_NODES; do echo "=== $node: redis ===" ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node 'sudo systemctl status redis' 2>/dev/null echo "=== $node: redis-sentinel ===" ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node 'sudo systemctl status redis-sentinel' 2>/dev/null done ``` ### Start, stop, or restart Redis on a single node Restarting a replica is low-risk. Restarting the primary triggers a Sentinel failover, which briefly interrupts writes for Platform. If you need to restart the primary, trigger a controlled failover first (`SENTINEL failover $SENTINEL_MASTER`) so the node you restart is already a replica. ```bash # Replace with: start, stop, or restart ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ 'sudo systemctl redis' ``` After a start or restart, verify the node responds and replication re-establishes: ```bash redis-cli -h -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning PING redis-cli -h -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \ INFO replication | grep -E "role|master_link_status|master_sync_in_progress" ``` ### Start, stop, or restart Sentinel on a single node Stopping Sentinel on one node reduces quorum. Stopping it on a majority of nodes disables automatic failover entirely until Sentinel is restored. ```bash ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ 'sudo systemctl redis-sentinel' ``` After a start or restart, verify quorum is restored: ```bash redis-cli -h -p $SENTINEL_PORT \ --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \ SENTINEL ckquorum $SENTINEL_MASTER ``` ### Review configuration files The Redis configuration file is at `/etc/redis/redis.conf` on each node; the Sentinel configuration file is at `/etc/redis/sentinel.conf`. Sentinel also writes back to its file at runtime when a failover occurs, updating the known primary address, so review it after a failover to confirm it reflects current topology. ```bash ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ 'sudo cat /etc/redis/redis.conf' ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ 'sudo cat /etc/redis/sentinel.conf' ``` | File | Key parameters | | --------------- | ----------------------------------------------------------------------------------------------------------- | | `redis.conf` | `maxmemory`, `maxmemory-policy`, `maxclients`, `aclfile`, `save`, `appendonly`, `bind` | | `sentinel.conf` | `sentinel monitor` (primary, port, quorum), `sentinel down-after-milliseconds`, `sentinel failover-timeout` | ### Enable services to start on boot ```bash ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ 'sudo systemctl enable redis redis-sentinel' ``` A node that reboots without this enabled stays offline until someone manually starts it, silently reducing quorum. ## Manage memory and eviction Redis stores all data in memory. When usage approaches the configured limit, Redis evicts keys according to its eviction policy. If eviction can't keep up with writes, new commands are rejected, which can stall queue workers or make job status unavailable. ### Check memory usage and eviction policy ```bash for node in $REDIS_NODES; do echo "=== $node: memory ===" redis-cli -h $node -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \ INFO memory 2>/dev/null | grep -E \ "used_memory_human|used_memory_peak_human|maxmemory_human|maxmemory_policy|mem_fragmentation_ratio|rss_overhead_ratio" done ``` | Metric | Concern | | ------------------------------------- | ---------------------------------------------------------------- | | `used_memory` approaching `maxmemory` | Eviction or OOM risk is imminent | | `maxmemory_policy: noeviction` | Redis rejects writes rather than evicting keys when full | | `mem_fragmentation_ratio` above 1.5 | High fragmentation; consider `MEMORY PURGE` or a planned restart | | `mem_fragmentation_ratio` below 1.0 | Redis is using swap; severe performance impact | ### Check eviction and connection rejection stats ```bash redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \ INFO stats | grep -E "evicted_keys|rejected_connections|total_commands_processed" ``` `evicted_keys` above 0 means Redis is discarding message queue data to stay within `maxmemory`; review key TTLs and the eviction policy. `rejected_connections` above 0 means `maxclients` was hit, a critical condition that causes client errors and can trigger a Sentinel failover if the primary can no longer serve health checks. ### Check client connection counts ```bash for node in $REDIS_NODES; do echo "=== $node: clients ===" redis-cli -h $node -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \ INFO clients 2>/dev/null | grep -E "connected_clients|blocked_clients|maxclients" redis-cli -h $node -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \ CONFIG GET maxclients 2>/dev/null done ``` Flag any node where `connected_clients` is within 20% of `maxclients`. `blocked_clients` above 0 indicates clients waiting on `BLPOP`, `BRPOP`, or `WAIT`; a small number is normal for queue workers, but a growing count indicates stalled consumers. ## Manage keys ### Check key count and average TTL ```bash redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning INFO keyspace ``` A database with many keys and an average TTL of zero contains keys that never expire on their own, the first candidates for cleanup if memory pressure is growing. ### Scan and inspect keys Never use `KEYS` in production; it blocks Redis while it iterates the entire keyspace. Use `SCAN` instead. ```bash # Scan by pattern redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \ --scan --pattern '' | head -20 # Inspect a specific key redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning TYPE redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning TTL ``` A TTL of `-1` means the key has no expiry and persists indefinitely; `-2` means the key doesn't exist. ## Check persistence health Redis can persist to disk using RDB snapshots, an append-only file (AOF), or both. A silent persistence failure means Redis appears healthy but isn't writing to disk; if the process restarts, queued jobs representing in-flight Platform work are lost. ### Check RDB snapshot status ```bash redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \ INFO persistence | grep -E "rdb_enabled|rdb_last_bgsave_status|rdb_last_bgsave_time_sec|rdb_last_save_time|loading" ``` Expect `rdb_last_bgsave_status: ok` and `loading: 0`. A failed snapshot doesn't stop Redis from serving requests, so check this routinely rather than waiting for a restart to reveal a stale on-disk copy. ### Check AOF status ```bash redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \ INFO persistence | grep -E "aof_enabled|aof_last_write_status|aof_last_rewrite_time_sec" ``` An AOF write failure is more serious than an RDB failure; it can indicate Redis is falling behind on durability, not just snapshotting. If both report errors simultaneously, check the Redis log for a `MISCONF` error, which indicates a configuration conflict between the two mechanisms. ## Manage users and credentials Redis uses an access control list (ACL) to manage permissions. This deployment uses the following ACL users: | Username | Purpose | | ------------- | ----------------------------------------------------------- | | `admin` | Full-access account used for all maintenance operations | | `itential` | Application service account used by Platform | | `replication` | Used by replicas to authenticate with the primary | | `prometheus` | Read-only metrics scraping account | | `sentinel` | Used by Sentinel processes to authenticate with Redis nodes | ### List ACL users ```bash redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning ACL LIST ``` Flag any expected user that's missing or shows `off`; a disabled ACL user causes the dependent service to fail authentication in a way that may not be obvious from application logs. ### Rotate a user password High-risk. Because Platform reads its Redis password from your secrets manager at startup, rotating a credential requires coordinating a secrets manager update with a Platform restart. Update your secrets manager first, then update Redis. Updating Redis first creates a window where the two are out of sync. #### Update the secret in your secrets manager Generate a new password and store it before making any change to Redis. #### Retrieve the new password The example below uses HashiCorp Vault syntax; adjust the retrieval command to match your secrets manager's API: ```bash source vault-env.sh NEW_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \ $VAULT_ADDR/ \ | jq -r '.data.data.') ``` #### Apply the new password to Redis ```bash redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \ ACL SETUSER >"$NEW_PASS" ``` #### Persist the ACL change ```bash redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning ACL SAVE ``` #### Verify the new password authenticates ```bash redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \ --user -a "$NEW_PASS" --no-auth-warning PING ``` #### Restart dependent services Restart Platform if the rotated user is `itential`, or restart Sentinel on all nodes if the rotated user is `sentinel`. ## Inspect logs Redis and Sentinel write separate log files. The Redis log records server-level events: memory pressure, persistence failures, replication changes, and client connection errors. The Sentinel log records the complete history of every failover, quorum decision, and topology change. | Log file | Default path | Configuration key | | ------------ | ----------------------------- | ---------------------------- | | Redis server | `/var/log/redis/redis.log` | `logfile` in `redis.conf` | | Sentinel | `/var/log/redis/sentinel.log` | `logfile` in `sentinel.conf` | Confirm these paths against your deployment; they can be customized. ### Search for key patterns ```bash # Redis log ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ \ 'sudo grep -hE "OOM|out of memory|MISCONF|LOADING|MASTERDOWN|CLUSTERDOWN|Connection refused|ERR max number" \ /var/log/redis/redis.log | tail -30' # Sentinel log ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ \ 'sudo grep -hE "\+failover|\+odown|\+sdown|-odown|\+elected|\+promoted|NOAUTH|WRONGPASS|Disconnecting" \ /var/log/redis/sentinel.log | tail -30' ``` | Pattern | Meaning | | ----------------------------------- | -------------------------------------------------------------- | | `OOM` / `out of memory` | Redis hit `maxmemory` and can't evict under the current policy | | `ERR max number of clients reached` | `maxclients` exhausted; clients were rejected | | `MISCONF` | RDB/AOF configuration conflict; persistence is broken | | `MASTERDOWN` | A replica lost its connection to the primary | | `NOAUTH` / `WRONGPASS` | Credential mismatch between Sentinel and Redis | ### Search for a specific pattern ```bash # Redis log for node in $REDIS_NODES; do echo "=== $node ===" ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node \ "sudo grep -iE '' /var/log/redis/redis.log | tail -50" 2>/dev/null done # Sentinel log for node in $REDIS_NODES; do echo "=== $node ===" ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node \ "sudo grep -iE '' /var/log/redis/sentinel.log | tail -50" 2>/dev/null done ``` ### View recent log entries on a single node ```bash # Redis log ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ \ 'sudo tail -100 /var/log/redis/redis.log' # Sentinel log ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ \ 'sudo tail -100 /var/log/redis/sentinel.log' ``` ## Rotate TLS certificates TLS certificates have a fixed expiry date. When a node's certificate expires, clients and Sentinel processes can no longer verify its identity and refuse to connect, breaking replication and preventing new Platform connections. Unlike services that degrade gradually, an expired Redis certificate causes an immediate hard failure. Redis and Sentinel maintain separate TLS configurations; both must be updated when certificates change. If the CA certificate changes, update every `redis-cli` command in this guide to reference the new CA file path. TLS certificate files are referenced in `/etc/redis/redis.conf` on each node: | Config key | Typical path | Contents | | ------------------ | -------------------------- | ---------------------------- | | `tls-cert-file` | `/etc/ssl/redis/redis.crt` | Server certificate | | `tls-key-file` | `/etc/ssl/redis/redis.key` | Server private key | | `tls-ca-cert-file` | `/etc/ssl/redis/ca.crt` | Certificate authority bundle | #### Check current certificate expiry Plan the rotation at least two weeks before the earliest expiry. ```bash for node in $REDIS_NODES; do ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node \ 'sudo openssl x509 -in /etc/ssl/redis/redis.crt -noout -dates 2>/dev/null' done ``` #### Verify the new certificate locally before deploying Run these checks on the machine where the new certificate was generated: ```bash openssl x509 -in new-redis.crt -noout -text | grep -E "Subject:|Not Before:|Not After :" openssl verify -CAfile new-ca.crt new-redis.crt # Confirm the certificate and key are a matching pair (hashes must be identical) openssl x509 -noout -modulus -in new-redis.crt | openssl md5 openssl rsa -noout -modulus -in new-redis.key | openssl md5 ``` Don't proceed if `openssl verify` returns an error or the modulus hashes don't match. #### Copy new certificate files to each node ```bash for node in $REDIS_NODES; do scp -i $SSH_KEY_PATH new-redis.crt new-redis.key new-ca.crt \ $REDIS_SSH_USER@$node:/tmp/ ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node ' sudo cp /tmp/new-redis.crt /etc/ssl/redis/redis.crt sudo cp /tmp/new-redis.key /etc/ssl/redis/redis.key sudo cp /tmp/new-ca.crt /etc/ssl/redis/ca.crt sudo chown redis:redis /etc/ssl/redis/redis.crt /etc/ssl/redis/redis.key /etc/ssl/redis/ca.crt sudo chmod 400 /etc/ssl/redis/redis.key sudo chmod 444 /etc/ssl/redis/redis.crt /etc/ssl/redis/ca.crt ' done ``` Adjust ownership if your Redis process doesn't run as `redis:redis`. Back up the previous files first (for example `redis.crt.bak`) so a rollback is a one-command restore. #### Roll out the change: replicas first, primary last For each replica: ```bash ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ 'sudo systemctl restart redis' redis-cli -h -p $REDIS_PORT --tls --cacert /etc/ssl/redis/ca.crt \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning PING ``` Don't restart the next node until this one responds `PONG`. For the primary, trigger a controlled failover first, then restart what is now a replica: ```bash redis-cli -h -p $SENTINEL_PORT --tls --cacert /etc/ssl/redis/ca.crt \ --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \ SENTINEL failover $SENTINEL_MASTER ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ 'sudo systemctl restart redis' ``` #### Restart Sentinel on each node Restart one at a time; avoid restarting a majority simultaneously, which removes quorum. ```bash ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ 'sudo systemctl restart redis-sentinel' redis-cli -h -p $SENTINEL_PORT --tls --cacert /etc/ssl/redis/ca.crt \ --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning SENTINEL ckquorum $SENTINEL_MASTER ``` #### Verify end-to-end connectivity ```bash redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT --tls --cacert /etc/ssl/redis/ca.crt \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \ INFO replication | grep -E "role|connected_slaves|master_link_status" ``` Then update the `--tls --cacert` path in any local scripts that reference the old CA. ### Rollback If a new certificate causes connection failures, restore the backed-up files and restart both services: ```bash ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@ ' sudo cp /etc/ssl/redis/redis.crt.bak /etc/ssl/redis/redis.crt sudo cp /etc/ssl/redis/redis.key.bak /etc/ssl/redis/redis.key sudo cp /etc/ssl/redis/ca.crt.bak /etc/ssl/redis/ca.crt sudo systemctl restart redis sudo systemctl restart redis-sentinel ' ``` Verify connectivity after rollback: ```bash redis-cli -h -p $REDIS_PORT \ --tls --cacert /etc/ssl/redis/ca.crt.bak \ --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning PING ```