Platform On-PremAdminister

Administer Redis

Redis serves as the message queue backing store for Itential Platform and stores authentication tokens. This guide covers the routine maintenance tasks required to keep a Redis Sentinel cluster healthy, performant, and secure.

For configuration properties and how Platform connects to standalone or HA Redis topologies, see Configure Redis and monitor health. For detailed metrics, PromQL expressions, and alert thresholds, see Redis metrics reference and Redis Sentinel metrics reference.

Before you begin

The commands in this guide assume the following environment variables are set:

$REDIS_NODES="<node1> <node2> <node3>"
$REDIS_PORT=6379
$SENTINEL_PORT=26379
$SENTINEL_MASTER=<sentinel-master-name>
$SSH_KEY_PATH=~/.ssh/<your-key>.pem
$REDIS_SSH_USER=<ssh-user>

Retrieve REDIS_ADMIN_PASS and REDIS_SENTINEL_ADMIN_PASS from your secrets manager at the start of each maintenance session and hold them only in shell variables for the session’s duration.

The examples below use HashiCorp Vault syntax. Adjust the retrieval commands to match your secrets manager’s API:

$source vault-env.sh
$
$REDIS_ADMIN_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
>  $VAULT_ADDR/<platform-credentials-path> \
>  | jq -r '.data.data.redisAdmin')
$
$REDIS_SENTINEL_ADMIN_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
>  $VAULT_ADDR/<platform-credentials-path> \
>  | jq -r '.data.data.redisSentinelAdmin')

Passwords must never be stored in scripts, configuration files, or shell history. Retrieve credentials from your secrets manager at runtime and let them leave scope when the session ends.

The current primary is managed by Sentinel and can change at any time after a failover. Always resolve it at runtime rather than hardcoding a hostname:

$REDIS_PRIMARY=$(redis-cli \
>  -h <any-sentinel-node> -p $SENTINEL_PORT \
>  --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \
>  SENTINEL get-master-addr-by-name $SENTINEL_MASTER | head -1)

TLS configuration varies by deployment. When TLS is enabled, append --tls --cacert <ca-cert-path> to every redis-cli command in this guide.

Check health and replication status

Redis is a single-threaded, in-memory store. In a Sentinel-managed deployment, one node holds the primary role and accepts all writes while the others replicate asynchronously. If the primary becomes unreachable, Sentinel promotes a replica, but only if quorum is healthy and replication lag is acceptable at the time. Regular health checks confirm that all nodes are reachable, replication is current, and Platform has a functioning data store.

PING all nodes

$for node in $REDIS_NODES; do
$  result=$(redis-cli -h $node -p $REDIS_PORT \
>    --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning PING 2>/dev/null)
$  echo "$node -> $result"
$done

All nodes should respond PONG. Investigate any node that doesn’t respond immediately; it may have crashed, lost network connectivity, or exhausted its client connection limit before Sentinel has triggered a failover.

Check replication status

$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning INFO replication
FieldExpected value
rolemaster on the primary
connected_slavesEqual to the number of expected replicas
master_sync_in_progress0 (1 means a replica is actively syncing)
slave_repl_offset vs master_repl_offsetSmall delta indicates low lag
master_link_statusup on all replicas

Flag any replica with master_link_status:down or a large offset delta; a lagging replica can’t safely become primary in a failover.

Identify the current primary

$redis-cli -h <any-sentinel-node> -p $SENTINEL_PORT \
>  --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \
>  SENTINEL get-master-addr-by-name $SENTINEL_MASTER

The response is two lines: the hostname on the first line and the port on the second.

Resolve the primary before any write operation or maintenance task rather than relying on a cached value.

Manage the Sentinel cluster

Redis Sentinel monitors Redis nodes, detects failures, and coordinates failovers. A majority of Sentinel processes (quorum) must agree the primary is unreachable before a failover triggers. If too few Sentinel instances are running, or they disagree on the primary, automatic failover doesn’t work.

Check quorum

$for node in $REDIS_NODES; do
$  echo "=== $node: quorum check ==="
$  redis-cli -h $node -p $SENTINEL_PORT \
>    --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \
>    SENTINEL ckquorum $SENTINEL_MASTER 2>/dev/null
$done

Expect OK N usable Sentinels. Quorum and failover authorization can be reached on every node. In a three-node Sentinel cluster, quorum is two; if only one Sentinel is reachable, the cluster is effectively unprotected.

List Sentinel members

$redis-cli -h <any-sentinel-node> -p $SENTINEL_PORT \
>  --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \
>  SENTINEL sentinels $SENTINEL_MASTER

Confirm the number of entries matches the expected Sentinel count and each shows flags: sentinel with no disconnected flag. Sentinel’s view of membership can drift after a node replacement or network partition.

Interpret the failover event timeline

When reviewing Sentinel logs after an unexpected failover, events appear in a predictable order:

EventMeaning
+sdownOne Sentinel marked the primary subjectively down
+odownQuorum agreed the primary is objectively down
+failover-triggeredA Sentinel was elected to lead the failover
+elected-leaderThe leading Sentinel is confirmed
+promoted-slaveA replica was promoted to primary
-odownThe former primary was demoted; failover is complete

A +sdown never followed by +odown means quorum wasn’t reached and no failover occurred, indicating a split-brain or Sentinel availability problem.

Manage Redis and Sentinel services

Redis and Redis Sentinel run as separate systemd services managed independently. Restarting the primary triggers a Sentinel failover; stopping a majority of Sentinel instances prevents any future failover. Always restart replicas before the primary.

Check status on all nodes

$for node in $REDIS_NODES; do
$  echo "=== $node: redis ==="
$  ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node 'sudo systemctl status redis' 2>/dev/null
$  echo "=== $node: redis-sentinel ==="
$  ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node 'sudo systemctl status redis-sentinel' 2>/dev/null
$done

Start, stop, or restart Redis on a single node

Restarting a replica is low-risk. Restarting the primary triggers a Sentinel failover, which briefly interrupts writes for Platform. If you need to restart the primary, trigger a controlled failover first (SENTINEL failover $SENTINEL_MASTER) so the node you restart is already a replica.

$# Replace <action> with: start, stop, or restart
$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<target-node> 'sudo systemctl <action> redis'

After a start or restart, verify the node responds and replication re-establishes:

$redis-cli -h <target-node> -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning PING
$
$redis-cli -h <target-node> -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \
>  INFO replication | grep -E "role|master_link_status|master_sync_in_progress"

Start, stop, or restart Sentinel on a single node

Stopping Sentinel on one node reduces quorum. Stopping it on a majority of nodes disables automatic failover entirely until Sentinel is restored.

$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<target-node> 'sudo systemctl <action> redis-sentinel'

After a start or restart, verify quorum is restored:

$redis-cli -h <target-node> -p $SENTINEL_PORT \
>  --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \
>  SENTINEL ckquorum $SENTINEL_MASTER

Review configuration files

The Redis configuration file is at /etc/redis/redis.conf on each node; the Sentinel configuration file is at /etc/redis/sentinel.conf. Sentinel also writes back to its file at runtime when a failover occurs, updating the known primary address, so review it after a failover to confirm it reflects current topology.

$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<target-node> 'sudo cat /etc/redis/redis.conf'
$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<target-node> 'sudo cat /etc/redis/sentinel.conf'
FileKey parameters
redis.confmaxmemory, maxmemory-policy, maxclients, aclfile, save, appendonly, bind
sentinel.confsentinel monitor (primary, port, quorum), sentinel down-after-milliseconds, sentinel failover-timeout

Enable services to start on boot

$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<target-node> 'sudo systemctl enable redis redis-sentinel'

A node that reboots without this enabled stays offline until someone manually starts it, silently reducing quorum.

Manage memory and eviction

Redis stores all data in memory. When usage approaches the configured limit, Redis evicts keys according to its eviction policy. If eviction can’t keep up with writes, new commands are rejected, which can stall queue workers or make job status unavailable.

Check memory usage and eviction policy

$for node in $REDIS_NODES; do
$  echo "=== $node: memory ==="
$  redis-cli -h $node -p $REDIS_PORT \
>    --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \
>    INFO memory 2>/dev/null | grep -E \
>    "used_memory_human|used_memory_peak_human|maxmemory_human|maxmemory_policy|mem_fragmentation_ratio|rss_overhead_ratio"
$done
MetricConcern
used_memory approaching maxmemoryEviction or OOM risk is imminent
maxmemory_policy: noevictionRedis rejects writes rather than evicting keys when full
mem_fragmentation_ratio above 1.5High fragmentation; consider MEMORY PURGE or a planned restart
mem_fragmentation_ratio below 1.0Redis is using swap; severe performance impact

Check eviction and connection rejection stats

$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \
>  INFO stats | grep -E "evicted_keys|rejected_connections|total_commands_processed"

evicted_keys above 0 means Redis is discarding message queue data to stay within maxmemory; review key TTLs and the eviction policy. rejected_connections above 0 means maxclients was hit, a critical condition that causes client errors and can trigger a Sentinel failover if the primary can no longer serve health checks.

Check client connection counts

$for node in $REDIS_NODES; do
$  echo "=== $node: clients ==="
$  redis-cli -h $node -p $REDIS_PORT \
>    --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \
>    INFO clients 2>/dev/null | grep -E "connected_clients|blocked_clients|maxclients"
$
$  redis-cli -h $node -p $REDIS_PORT \
>    --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \
>    CONFIG GET maxclients 2>/dev/null
$done

Flag any node where connected_clients is within 20% of maxclients. blocked_clients above 0 indicates clients waiting on BLPOP, BRPOP, or WAIT; a small number is normal for queue workers, but a growing count indicates stalled consumers.

Manage keys

Check key count and average TTL

$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning INFO keyspace

A database with many keys and an average TTL of zero contains keys that never expire on their own, the first candidates for cleanup if memory pressure is growing.

Scan and inspect keys

Never use KEYS in production; it blocks Redis while it iterates the entire keyspace. Use SCAN instead.

$# Scan by pattern
$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \
>  --scan --pattern '<pattern>' | head -20
$
$# Inspect a specific key
$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning TYPE <key-name>
$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning TTL <key-name>

A TTL of -1 means the key has no expiry and persists indefinitely; -2 means the key doesn’t exist.

Check persistence health

Redis can persist to disk using RDB snapshots, an append-only file (AOF), or both. A silent persistence failure means Redis appears healthy but isn’t writing to disk; if the process restarts, queued jobs representing in-flight Platform work are lost.

Check RDB snapshot status

$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \
>  INFO persistence | grep -E "rdb_enabled|rdb_last_bgsave_status|rdb_last_bgsave_time_sec|rdb_last_save_time|loading"

Expect rdb_last_bgsave_status: ok and loading: 0. A failed snapshot doesn’t stop Redis from serving requests, so check this routinely rather than waiting for a restart to reveal a stale on-disk copy.

Check AOF status

$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \
>  INFO persistence | grep -E "aof_enabled|aof_last_write_status|aof_last_rewrite_time_sec"

An AOF write failure is more serious than an RDB failure; it can indicate Redis is falling behind on durability, not just snapshotting. If both report errors simultaneously, check the Redis log for a MISCONF error, which indicates a configuration conflict between the two mechanisms.

Manage users and credentials

Redis uses an access control list (ACL) to manage permissions. This deployment uses the following ACL users:

UsernamePurpose
adminFull-access account used for all maintenance operations
itentialApplication service account used by Platform
replicationUsed by replicas to authenticate with the primary
prometheusRead-only metrics scraping account
sentinelUsed by Sentinel processes to authenticate with Redis nodes

List ACL users

$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning ACL LIST

Flag any expected user that’s missing or shows off; a disabled ACL user causes the dependent service to fail authentication in a way that may not be obvious from application logs.

Rotate a user password

High-risk. Because Platform reads its Redis password from your secrets manager at startup, rotating a credential requires coordinating a secrets manager update with a Platform restart. Update your secrets manager first, then update Redis. Updating Redis first creates a window where the two are out of sync.

1

Update the secret in your secrets manager

Generate a new password and store it before making any change to Redis.

2

Retrieve the new password

The example below uses HashiCorp Vault syntax; adjust the retrieval command to match your secrets manager’s API:

$source vault-env.sh
$NEW_PASS=$(curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
>  $VAULT_ADDR/<platform-credentials-path> \
>  | jq -r '.data.data.<password-key>')
3

Apply the new password to Redis

$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \
>  ACL SETUSER <username> >"$NEW_PASS"
4

Persist the ACL change

$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning ACL SAVE
5

Verify the new password authenticates

$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT \
>  --user <username> -a "$NEW_PASS" --no-auth-warning PING
6

Restart dependent services

Restart Platform if the rotated user is itential, or restart Sentinel on all nodes if the rotated user is sentinel.

Inspect logs

Redis and Sentinel write separate log files. The Redis log records server-level events: memory pressure, persistence failures, replication changes, and client connection errors. The Sentinel log records the complete history of every failover, quorum decision, and topology change.

Log fileDefault pathConfiguration key
Redis server/var/log/redis/redis.loglogfile in redis.conf
Sentinel/var/log/redis/sentinel.loglogfile in sentinel.conf

Confirm these paths against your deployment; they can be customized.

Search for key patterns

$# Redis log
$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<target-node> \
>  'sudo grep -hE "OOM|out of memory|MISCONF|LOADING|MASTERDOWN|CLUSTERDOWN|Connection refused|ERR max number" \
>   /var/log/redis/redis.log | tail -30'
$
$# Sentinel log
$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<target-node> \
>  'sudo grep -hE "\+failover|\+odown|\+sdown|-odown|\+elected|\+promoted|NOAUTH|WRONGPASS|Disconnecting" \
>   /var/log/redis/sentinel.log | tail -30'
PatternMeaning
OOM / out of memoryRedis hit maxmemory and can’t evict under the current policy
ERR max number of clients reachedmaxclients exhausted; clients were rejected
MISCONFRDB/AOF configuration conflict; persistence is broken
MASTERDOWNA replica lost its connection to the primary
NOAUTH / WRONGPASSCredential mismatch between Sentinel and Redis

Search for a specific pattern

$# Redis log
$for node in $REDIS_NODES; do
$  echo "=== $node ==="
$  ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node \
>    "sudo grep -iE '<your-pattern>' /var/log/redis/redis.log | tail -50" 2>/dev/null
$done
$
$# Sentinel log
$for node in $REDIS_NODES; do
$  echo "=== $node ==="
$  ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node \
>    "sudo grep -iE '<your-pattern>' /var/log/redis/sentinel.log | tail -50" 2>/dev/null
$done

View recent log entries on a single node

$# Redis log
$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<target-node> \
>  'sudo tail -100 /var/log/redis/redis.log'
$
$# Sentinel log
$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<target-node> \
>  'sudo tail -100 /var/log/redis/sentinel.log'

Rotate TLS certificates

TLS certificates have a fixed expiry date. When a node’s certificate expires, clients and Sentinel processes can no longer verify its identity and refuse to connect, breaking replication and preventing new Platform connections. Unlike services that degrade gradually, an expired Redis certificate causes an immediate hard failure.

Redis and Sentinel maintain separate TLS configurations; both must be updated when certificates change. If the CA certificate changes, update every redis-cli command in this guide to reference the new CA file path.

TLS certificate files are referenced in /etc/redis/redis.conf on each node:

Config keyTypical pathContents
tls-cert-file/etc/ssl/redis/redis.crtServer certificate
tls-key-file/etc/ssl/redis/redis.keyServer private key
tls-ca-cert-file/etc/ssl/redis/ca.crtCertificate authority bundle
1

Check current certificate expiry

Plan the rotation at least two weeks before the earliest expiry.

$for node in $REDIS_NODES; do
$  ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node \
>    'sudo openssl x509 -in /etc/ssl/redis/redis.crt -noout -dates 2>/dev/null'
$done
2

Verify the new certificate locally before deploying

Run these checks on the machine where the new certificate was generated:

$openssl x509 -in new-redis.crt -noout -text | grep -E "Subject:|Not Before:|Not After :"
$openssl verify -CAfile new-ca.crt new-redis.crt
$
$# Confirm the certificate and key are a matching pair (hashes must be identical)
$openssl x509 -noout -modulus -in new-redis.crt | openssl md5
$openssl rsa  -noout -modulus -in new-redis.key | openssl md5

Don’t proceed if openssl verify returns an error or the modulus hashes don’t match.

3

Copy new certificate files to each node

$for node in $REDIS_NODES; do
$  scp -i $SSH_KEY_PATH new-redis.crt new-redis.key new-ca.crt \
>    $REDIS_SSH_USER@$node:/tmp/
$
$  ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@$node '
>    sudo cp /tmp/new-redis.crt /etc/ssl/redis/redis.crt
>    sudo cp /tmp/new-redis.key /etc/ssl/redis/redis.key
>    sudo cp /tmp/new-ca.crt    /etc/ssl/redis/ca.crt
>    sudo chown redis:redis /etc/ssl/redis/redis.crt /etc/ssl/redis/redis.key /etc/ssl/redis/ca.crt
>    sudo chmod 400 /etc/ssl/redis/redis.key
>    sudo chmod 444 /etc/ssl/redis/redis.crt /etc/ssl/redis/ca.crt
>  '
$done

Adjust ownership if your Redis process doesn’t run as redis:redis. Back up the previous files first (for example redis.crt.bak) so a rollback is a one-command restore.

4

Roll out the change: replicas first, primary last

For each replica:

$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<replica-node> 'sudo systemctl restart redis'
$redis-cli -h <replica-node> -p $REDIS_PORT --tls --cacert /etc/ssl/redis/ca.crt \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning PING

Don’t restart the next node until this one responds PONG. For the primary, trigger a controlled failover first, then restart what is now a replica:

$redis-cli -h <any-sentinel-node> -p $SENTINEL_PORT --tls --cacert /etc/ssl/redis/ca.crt \
>  --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning \
>  SENTINEL failover $SENTINEL_MASTER
$
$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<former-primary> 'sudo systemctl restart redis'
5

Restart Sentinel on each node

Restart one at a time; avoid restarting a majority simultaneously, which removes quorum.

$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<target-node> 'sudo systemctl restart redis-sentinel'
$redis-cli -h <target-node> -p $SENTINEL_PORT --tls --cacert /etc/ssl/redis/ca.crt \
>  --user admin -a "$REDIS_SENTINEL_ADMIN_PASS" --no-auth-warning SENTINEL ckquorum $SENTINEL_MASTER
6

Verify end-to-end connectivity

$redis-cli -h $REDIS_PRIMARY -p $REDIS_PORT --tls --cacert /etc/ssl/redis/ca.crt \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning \
>  INFO replication | grep -E "role|connected_slaves|master_link_status"

Then update the --tls --cacert path in any local scripts that reference the old CA.

Rollback

If a new certificate causes connection failures, restore the backed-up files and restart both services:

$ssh -i $SSH_KEY_PATH $REDIS_SSH_USER@<target-node> '
>  sudo cp /etc/ssl/redis/redis.crt.bak /etc/ssl/redis/redis.crt
>  sudo cp /etc/ssl/redis/redis.key.bak /etc/ssl/redis/redis.key
>  sudo cp /etc/ssl/redis/ca.crt.bak    /etc/ssl/redis/ca.crt
>  sudo systemctl restart redis
>  sudo systemctl restart redis-sentinel
>'

Verify connectivity after rollback:

$redis-cli -h <target-node> -p $REDIS_PORT \
>  --tls --cacert /etc/ssl/redis/ca.crt.bak \
>  --user admin -a "$REDIS_ADMIN_PASS" --no-auth-warning PING