Redis metrics reference

This page expands on the Redis alert thresholds in Exporters with a per-metric description, the conditions that should trigger an alert, and how to troubleshoot each one. Use this page when you’re building alert rules from scratch or investigating an alert that already fired.

Redis handles session management for Platform and stores the active job data the Itential backend uses to coordinate workflow execution. Every metric on this page comes from redis_exporter running against a Redis data instance (primary or replica). See Redis Sentinel metrics reference for the metrics that cover the Sentinel failover layer instead.

Critical system metrics

Uptime

redis_uptime_in_seconds tracks how long the Redis process has been running since its last start. An uptime of zero means the instance has crashed or been stopped, so this is the most critical Redis alert: it represents complete data unavailability for BullMQ job processing.

ThresholdConditionDuration
Criticalredis_uptime_in_seconds == 030 sec

If this fires, check the Redis log (typically /var/log/redis/redis-server.log) and look for an out-of-memory (OOM) kill in the system log (dmesg | grep -i kill). Confirm the host hasn’t run out of memory or disk space and check for a configuration error that’s preventing startup before restarting the service.

CPU usage

This is the percentage of CPU time consumed by the Redis process and any child processes it spawns for point-in-time snapshot (RDB) or Append Only File (AOF) persistence, measured over a 5-minute window. Elevated CPU usage suggests Redis is under heavy load from too many operations, an expensive command such as KEYS *, or a background persistence operation competing for CPU.

ThresholdConditionDuration
WarningCPU usage > 70%5 min
CriticalCPU usage > 85%2 min

Memory usage

redis_memory_used_bytes is the total memory Redis has allocated for data, overhead, and buffers. Redis is an in-memory database, so exhausting available memory forces it to either evict keys, which loses data, or reject new writes, which fails the application.

ThresholdConditionDuration
Warningredis_memory_used_bytes > 80% of redis_config_maxmemory5 min
Criticalredis_memory_used_bytes > 95% of redis_config_maxmemory1 min

Connected replicas (primary perspective)

redis_connected_slaves is the number of replicas currently connected to a primary. Replicas provide the redundancy Sentinel needs to perform a failover, so a drop in this count means that redundancy is degraded or gone.

ThresholdConditionDuration
Warningredis_connected_slaves < expected replica count1 min
Criticalredis_connected_slaves == 0 and role == "master"30 sec

If a replica is missing, check its Redis log for connection errors, verify network connectivity and firewall rules between the primary and replica, confirm the replica process is running (systemctl status redis-server), and check the Sentinel logs for a failover event that may have changed the topology. Also verify the replication password (CONFIG GET masterauth) matches on both sides.

redis_master_link_up reports 1 when a replica is connected to its primary and 0 when it isn’t. A broken link means the replica is serving stale data and isn’t safe to promote during a failover.

ThresholdConditionDuration
Criticalredis_master_link_up == 0 and role == "slave"30 sec

Check master_link_down_since_seconds in the replica’s log, confirm the primary is reachable (redis-cli -h <primary-host> PING), and verify authentication credentials and the configured primary host and port on the replica.

Active clients

redis_connected_clients counts client connections, excluding replica connections. Each connection consumes memory and a file descriptor, so a high or rapidly growing count usually points to a connection leak or an application that isn’t pooling connections.

ThresholdConditionDuration
Warningredis_connected_clients > 80005 min
Criticalredis_connected_clients > 90% of redis_config_maxclients2 min

Use CLIENT LIST to see which clients are connected and from where, check whether a recent deployment introduced a connection leak, and confirm the timeout setting is closing idle connections as expected.

Memory usage and fragmentation

Memory fragmentation ratio

redis_mem_fragmentation_ratio compares the physical memory Redis is using (resident set size, or RSS) to the memory it has logically allocated. A ratio above 1.0 means Redis is holding onto physical memory that isn’t storing data, which wastes resources and can contribute to an out-of-memory condition even when logical memory usage looks fine.

ThresholdConditionDuration
Warningratio > 1.510 min
Criticalratio > 2.05 min
Warningratio < 1.0 (under-allocation, Redis may be using swap)10 min

A restart during a maintenance window defragments Redis by reloading from AOF or RDB into a clean allocation. This is usually caused by workloads that repeatedly create and delete keys of varying sizes.

Fragmented memory

redis_allocator_frag_bytes is the absolute amount of wasted memory (RSS minus allocated memory), in bytes. It quantifies exactly how much physical RAM the fragmentation ratio is wasting.

ThresholdConditionDuration
Warning> 512 MB10 min
Critical> 2 GB5 min

Replication and AOF persistence

AOF health

This composite condition checks for a delayed fsync, which happens when Redis can’t sync writes to disk fast enough. A delayed fsync means write operations are sitting in memory without being persisted, so it’s a direct data loss risk.

ThresholdConditionDuration
Criticalredis_aof_enabled == 1 and redis_aof_delayed_fsync > 030 sec

Check disk I/O with iostat -x 1 for saturation, review other processes competing for I/O (iotop), and confirm the disk isn’t failing. Changing appendfsync from always to everysec trades one second of potential data loss for meaningfully better write performance. In cloud environments, increasing provisioned input/output operations per second (IOPS) is often the fastest fix.

Delayed AOF writes

redis_aof_delayed_fsync counts pending fsync operations waiting to reach disk. Any value above zero is a data loss emergency, since it means Redis has queued writes that haven’t been persisted yet.

ThresholdConditionDuration
Criticalredis_aof_delayed_fsync > 010 sec

Treat this the same as AOF health above, plus check for available disk space (df -h) and filesystem errors (dmesg | grep -i error). If this occurs during a rewrite, aof-rewrite-incremental-fsync yes spreads the I/O load out instead of bursting it.

AOF rewrite duration

redis_aof_last_rewrite_duration_sec is how long the most recent AOF rewrite took. A rewrite forks the Redis process, which briefly pauses it and causes a CPU spike, so a long rewrite can also cause replication lag.

ThresholdConditionDuration
Warning> 300 secevaluated after completion

Monitor the trend over time to see whether rewrite duration is growing with the dataset. Tune auto-aof-rewrite-percentage and auto-aof-rewrite-min-size so rewrites land during low-traffic periods, and confirm there’s enough free memory for the fork, since the child process temporarily doubles memory usage.

Pending AOF data

redis_aof_pending_bio_fsync is the number of background I/O operations waiting to sync AOF data to disk. It’s a real-time, leading indicator of persistence backlog, ahead of an actual delayed fsync.

ThresholdConditionDuration
Warning> 102 min
Critical> 501 min

Replica lag (time)

This is how many seconds behind the primary a replica is, based on replication stream timing. High lag means promoting that replica during a failover would lose data.

ThresholdConditionDuration
Warningredis_connected_slave_lag_seconds > 52 min
Criticalredis_connected_slave_lag_seconds > 101 min

Identify the lagging replica with redis-cli INFO replication on the primary, check the replica’s CPU, memory, and disk usage for resource constraints, and verify network bandwidth between primary and replica with a tool like iperf. A lag above 10 seconds also risks min-replicas-to-write blocking writes on the primary, which is the safety mechanism working as intended, not a separate failure.

Replica lag (bytes)

This measures the byte offset difference between the primary’s replication position and a replica’s position. It quantifies exactly how far behind the replica is and, unlike time-based lag, tells you directly how close you are to exceeding the replication backlog.

ThresholdConditionDuration
Warning(redis_master_repl_offset - redis_connected_slave_offset_bytes) > 10 MB5 min
Critical(redis_master_repl_offset - redis_connected_slave_offset_bytes) > 100 MB2 min

If byte lag approaches repl-backlog-size, increase it immediately (CONFIG SET repl-backlog-size 256mb) to avoid forcing a full resync.

AOF file growth

This tracks the rate at which the AOF file is growing, which is useful for capacity planning and for spotting unexpected write spikes.

ThresholdConditionDuration
WarningAOF growth rate > 100 MB/hour30 min
InfoAOF size / AOF base size > 1.8 (approaching rewrite threshold)5 min

Confirm the growth rate matches expected job processing volume, and review whether job data can be compressed or job retention tightened so completed and failed jobs are cleaned up more aggressively.

Keys and cache performance

Key evictions

redis_evicted_keys_total counts keys Redis has forcibly removed under memory pressure. In a queue-backed deployment where noeviction is the expected policy, any eviction is data loss: a job or its result has disappeared.

ThresholdConditionDuration
Warningeviction rate > 0/5 min5 min
Criticaleviction rate > 100/5 min2 min

Investigate why maxmemory was exceeded, check for a spike in job creation or unusually large job payloads, and confirm BullMQ’s built-in cleanup is removing completed and failed jobs as expected.

ACL violations (security)

These four metrics track access control list (ACL) violations and should be zero in a correctly configured deployment. Any nonzero value is either a misconfigured credential or an unauthorized access attempt, so alert on them at zero tolerance rather than waiting for a rate threshold.

Authentication failures

redis_acl_access_denied_auth_total counts failed authentication attempts.

ThresholdConditionDuration
Criticalany increase over 5 minimmediate

Command failures

redis_acl_access_denied_cmd_total counts commands rejected because the authenticated user lacks permission to run them.

ThresholdConditionDuration
Criticalincrease > 5 over 5 min1 min

Key access failures

redis_acl_access_denied_key_total counts attempts to read or write keys outside a user’s allowed key pattern.

ThresholdConditionDuration
Criticalincrease > 5 over 5 min1 min

Channel failures

redis_acl_access_denied_channel_total counts denied pub/sub channel access. BullMQ uses pub/sub extensively for job events and worker coordination, so denials here break event-driven communication between workers.

ThresholdConditionDuration
Criticalany increase over 5 min1 min

For any of the four, use ACL GETUSER <username> to see the user’s allowed command, key, and channel patterns, and adjust with ACL SETUSER once you’ve confirmed the application legitimately needs the access it was denied.

Command performance

Command throughput

redis_commands_processed_total is the total number of commands Redis processes per second, across all command types.

ThresholdConditionDuration
Info> 50,000 commands/sec10 min
Warning> 100,000 commands/sec5 min

Compare against your baseline to spot an unusual spike, and use SLOWLOG GET to find expensive commands consuming resources.

Command latency

Average command latency is total execution time divided by total commands processed.

ThresholdConditionDuration
Warningaverage latency > 1 ms5 min
Criticalaverage latency > 10 ms2 min

Use SLOWLOG GET to find slow operations, and check for blocking commands like KEYS, SORT without LIMIT, or operations against very large sets and lists.

Network I/O

Network throughput

This is the combined rate of redis_net_input_bytes_total and redis_net_output_bytes_total, in bytes per second.

ThresholdConditionDuration
Infocombined rate > 500 MB/sec10 min
Warningcombined rate > 900 MB/sec (approaching 1 Gbps)5 min

Investigate whether large keys are being retrieved, whether replication traffic has grown, and whether compressing job payloads at the application layer would reduce transfer volume.

Composite alerts

These conditions combine several of the metrics above into a single higher-level signal, which is useful for a top-level dashboard panel or a single page-worthy alert.

Primary write safety fires when the primary no longer meets its own write-safety guarantee:

redis_connected_slaves{role="master"} < 1
OR redis_connected_slave_lag_seconds > 10

Overall health check fires on any condition that represents an outright Redis failure:

redis_uptime_in_seconds == 0
OR redis_memory_used_bytes > redis_config_maxmemory * 0.95
OR redis_aof_delayed_fsync > 0
OR redis_connected_slaves{role="master"} == 0
OR redis_master_link_up{role="slave"} == 0

Performance degradation fires on any condition that indicates Redis is still up but struggling:

CPU usage > 80%
OR redis_mem_fragmentation_ratio > 1.5
OR average command latency > 5 ms
OR hit rate < 70%

Redis Sentinel

Redis Sentinel requires monitoring in addition to the data instances above. At minimum, monitor Sentinel’s systemd uptime, CPU usage, memory usage, and disk usage the same way you would any other host. See Redis Sentinel metrics reference for the Sentinel-specific metrics covering quorum, failover readiness, and topology.