Use exporters
An exporter is a small process that runs alongside a monitored component and translates that component’s internal data into Prometheus metrics format. Prometheus scrapes each exporter on its configured interval. Each exporter listens on a specific port.
The following exporters are deployed in the Itential monitoring stack.
node_exporter
node_exporter monitors the Linux operating system on each host, collecting system-level metrics directly from the kernel and reading from the /proc and /sys filesystems to expose them in Prometheus format. It runs as a long-lived daemon on every Linux server that’s part of the Itential deployment.
Port
node_exporter listens on port 9100.
Collected metrics
- CPU usage per core and per mode (user, system, idle, I/O wait)
- Physical memory usage, available memory, and swap utilization
- Disk space usage and I/O throughput per device
- Network interface throughput and error rates
- Filesystem usage per mount point
- System load averages
- Open file descriptor counts
Impact
Most Platform, Gateway, MongoDB, and Redis problems show up as a change in the host’s own CPU, memory, or disk usage. For example, if MongoDB replication lag shows up around the same time the disk is struggling to keep up with read and write requests, that points to the disk as the underlying cause. If a Platform performance problem shows up alongside high memory usage, that points to Platform needing more memory than the host has available. node_exporter gives you that host-level data, so you can make these connections yourself. Without it, the metrics stack can tell you that MongoDB is slow, but not whether the underlying cause is the disk, the CPU, or memory.
Standard alert thresholds for Itential deployments
See Platform metrics reference and Gateway metrics reference for PromQL expressions and per-component (Platform, and Gateway runner/server/client) threshold breakdowns.
process_exporter
node_exporter reports totals for the whole host. process_exporter monitors individual processes and process groups on each host, breaking those totals down so you can see how much CPU or memory a specific process, such as mongod, is using.
Port
process_exporter listens on port 9256.
Collected metrics
- CPU time consumed per named process or process group
- Resident and virtual memory usage per process
- Open file descriptor count per process
- Thread counts per process
- Read and write I/O statistics per process
Impact
Host-level CPU at 90% tells you the machine is under pressure. process_exporter tells you that the mongod process is consuming 85% of that CPU. This distinction matters for triage. For Itential deployments, process_exporter is most useful for tracking the mongod, redis-server, and itential-platform process resource footprints independently on shared hosts.
Standard alert thresholds
mongodb_exporter
mongodb_exporter monitors MongoDB replica set health, performance, and storage.
It connects to MongoDB using a dedicated monitoring user and runs the same diagnostic commands a database administrator would run manually: serverStatus, replSetGetStatus, dbStats, collStats, and oplog queries. The results are converted to Prometheus metrics and exposed for scraping.
Port
mongodb_exporter listens on port 9216.
Collected metrics
- Replica set member health and state (
PRIMARY,SECONDARY,RECOVERING,DOWN) - Replication lag between primary and each secondary, measured in seconds
- Oplog window size (the time range of operations in the replication log)
- Active, available, and rejected connection counts
- Operation counters: inserts, updates, queries, deletes
- WiredTiger storage engine cache utilization
- Database and collection storage sizes
- Disk read and write latency per device
- Lock contention statistics
- Index usage
Impact
MongoDB is Platform’s primary datastore. Every job, task, workflow definition, adapter configuration, and audit record lives there, so a degraded MongoDB instance affects everything Platform does. Replication lag matters most. If a secondary falls too far behind and a failover happens during that lag, writes that hadn’t replicated yet become inaccessible until the replica catches up.
Watch the oplog window metric closely. The oplog is a fixed-size buffer of write operations on the primary. If a secondary stays offline long enough for new writes to overwrite the part of the oplog it needs to resume from, it must fully resync, which takes hours on large datasets.
Standard alert thresholds
See MongoDB metrics reference for PromQL expressions and troubleshooting guidance for each metric above.
redis_exporter (data instances)
redis_exporter connects to each Redis data instance and runs INFO commands to collect health, memory, replication, and queue activity data.
Redis handles session management for Platform and stores the active job data the Itential backend uses to coordinate workflow execution, so Redis health has a direct and immediate effect on workflow processing.
Port
redis_exporter listens on port 9121.
Collected metrics
- Memory usage and configured maximum (
maxmemory) - Memory fragmentation ratio (the gap between allocated memory and what the OS reports)
- Connected client count
- Connected replica count and primary link status
- Key eviction counts and rates
- Append Only File (AOF) persistence health, including delayed fsync events
- Replication lag in seconds and bytes
- Access control list (ACL) access denial counts (authentication failures, blocked commands, key access violations)
- Command throughput and average command latency
- Network I/O throughput
Impact
Redis is configured with a memory limit. When that limit is reached, Redis must evict keys to make room for new ones. When keys are evicted, the Itential backend loses job data, which causes jobs to fail or become permanently stuck. The redis_exporter memory usage alert fires before eviction begins, giving you time to act.
The replication metrics are equally important. If a Redis replica loses its connection to the primary (redis_master_link_up == 0), that replica is serving stale data. If a Sentinel-managed cluster loses all replicas, it loses the candidates needed to perform an automatic failover. The exporter surfaces both conditions as alertable signals.
ACL violation metrics are a security signal. Any authentication failure or blocked command in production typically indicates either a misconfigured application credential or an unauthorized access attempt. These alerts are configured at zero tolerance.
Standard alert thresholds
See Redis metrics reference for a per-metric description and troubleshooting guidance for each entry above.
redis_exporter (Sentinel)
A separate instance of redis_exporter monitors Redis Sentinel cluster health and failover readiness by watching the Redis Sentinel processes. Redis Sentinel is the high-availability layer that watches the Redis data instances and automatically promotes a replica to primary if the primary becomes unavailable. Monitoring Sentinel separately ensures that the failover mechanism itself is healthy, not just the data instances it protects.
Port
redis_exporter listens on port 9122.
Collected metrics
- Number of active Sentinel instances and whether quorum is achievable
- TILT mode status on each Sentinel
- Master status as seen by each Sentinel
- Current master address, used to detect that a failover has occurred
- Healthy and total replica counts per monitored master
- Failover configuration (
down_after_milliseconds,failover_timeout,parallel_syncs) - ACL authentication failure and command denial rates
- Network throughput and command processing rate per Sentinel instance
Impact
A Redis cluster can appear healthy in terms of data availability while its high-availability mechanism is silently broken. Quorum loss means Sentinel can’t agree on a new primary during a failover. TILT mode means Sentinel has stopped making failover decisions entirely, typically due to clock skew, not enough CPU time, or scheduling delays. Neither condition is visible by monitoring only the Redis data instances.
TILT mode is a subtle failure; Sentinel continues to respond to queries, the data instances continue to serve traffic, but the cluster has effectively lost its self-healing capability. An alert on redis_sentinel_tilt > 0 catches this condition before it matters, while the cluster is still operational.
Standard alert thresholds
See Redis Sentinel metrics reference for a per-metric description and troubleshooting guidance for each entry above.
Platform job-metrics-exporter
Platform job-metrics-exporter is a lightweight Go daemon that reads job and task data from MongoDB and publishes it as Prometheus metrics. While other exporters track infrastructure health, this one tracks Platform’s own operational health such as how many jobs are running or complete, and whether the backlog is growing or draining.
Collected metrics
- Count of Platform jobs grouped by status (running, complete, failed, canceled)
- Count of Platform tasks grouped by status
- Job and task backlog trends over time
- Itential backend queue depth and throughput (jobs waiting to be dispatched to Gateway)
Impact
Infrastructure metrics tell you Platform is running. Job metrics tell you it’s actually working. A Platform node can look healthy on CPU, memory, and connectivity while a backlog of stuck jobs builds up behind the scenes, a condition only the job metrics reveal.
These metrics also help you plan maintenance windows. Check the running job count and drain rate before a shutdown to estimate when the queue will empty, then confirm recovery afterward by watching throughput return to baseline.
Backend queue depth is a leading indicator for Gateway capacity: a sustained increase alongside healthy Gateway resource usage means the Gateway fleet needs more capacity. Queue depth at zero combined with jobs stuck in a running state points to a connectivity problem between Platform and Redis instead.