Gateway metrics reference

This page covers CPU, memory, and disk metrics for the three node types in an Itential Gateway deployment: runners, servers, and clients. See Choose a deployment architecture for what each node type does.

Runners

Runner nodes execute automation jobs. Resource pressure on a runner shows up directly as job execution delays, timeouts, or failures.

CPU usage

High CPU on a runner can indicate a heavy automation workload, an inefficient script or playbook, too many jobs running concurrently for the runner’s capacity, or resource contention between jobs. Sustained high CPU causes job execution delays, timeout failures, and a growing job queue.

PromQL:

# System-level
100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle",instance=~".*runner.*"}[5m])) * 100)
# Container-level
sum by (container, pod, namespace) (rate(container_cpu_usage_seconds_total{pod=~".*runner.*"}[5m])) * 100

Identify resource-intensive jobs, review job concurrency and distribution across runners, and check for inefficient automation code before adding more runners to spread the load.

Memory usage

Runners load automation logic and inventory data and keep job state in memory while a job runs. High usage can point to a large automation payload, a memory leak in custom script code, or too many jobs running concurrently for the runner’s memory allocation. Exhaustion causes out-of-memory (OOM) kills, runner restarts, and job failures mid-execution, which for a network automation job can mean losing track of a change that’s already in flight.

PromQL:

# System-level
100 * (1 - ((node_memory_MemAvailable_bytes{instance=~".*runner.*"} or
(node_memory_MemFree_bytes{instance=~".*runner.*"} +
node_memory_Cached_bytes{instance=~".*runner.*"} +
node_memory_Buffers_bytes{instance=~".*runner.*"})) /
node_memory_MemTotal_bytes{instance=~".*runner.*"}))
# Container-level
sum by (container, pod, namespace) (container_memory_usage_bytes{pod=~".*runner.*"}) /
sum by (container, pod, namespace) (container_spec_memory_limit_bytes{pod=~".*runner.*"}) * 100

Identify memory-intensive jobs, review job concurrency limits per runner, and check for a memory leak in custom automation code before increasing runner memory allocation or adding more runners.

Disk usage

Runners write job logs, temporary files, downloaded artifacts, and cache data to disk. Running out of space fails new jobs immediately, stops log writes (losing the troubleshooting data you’d need to explain the failure), and can trigger container eviction in Kubernetes.

PromQL:

# System-level
100 - ((node_filesystem_avail_bytes{instance=~".*runner.*",mountpoint=~"/data.*|/var.*"} /
node_filesystem_size_bytes{instance=~".*runner.*",mountpoint=~"/data.*|/var.*"}) * 100)
# Container-level
sum by (pod, namespace) (kubelet_volume_stats_used_bytes{persistentvolumeclaim=~".*runner.*"}) /
sum by (pod, namespace) (kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~".*runner.*"}) * 100

Identify what’s consuming disk space (logs, temp files, cache), implement or fix log rotation, and clean up old job artifacts before increasing disk allocation.

Servers

Gateway servers handle job orchestration, API requests, runner registration, state coordination, and client connections. Server resource pressure affects the entire control plane, not a single job.

CPU usage

High CPU can come from heavy API traffic, a large number of concurrent jobs, an inefficient job scheduling pattern, or connection management overhead. Sustained high CPU delays API responses and job submissions, and can cause runner registration failures and web UI slowdowns.

PromQL:

# System-level
100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle",instance=~".*server.*"}[5m])) * 100)
# Container-level
sum by (container, pod, namespace) (rate(container_cpu_usage_seconds_total{pod=~".*server.*"}[5m])) * 100

Identify CPU-intensive operations such as job scheduling or runner health checks, analyze API request patterns and volume, and add more servers for high availability if load is consistently high across the fleet.

Memory usage

Servers keep job state, runner registrations, client sessions, API connection pools, and job queues in memory. High usage can indicate job state accumulation, a memory leak in server code, too many client connections, or a large job definition. Exhaustion causes server OOM kills and restarts that lose in-flight job state, which can mean a job’s status becomes unknown until it’s manually reconciled.

PromQL:

# System-level
100 * (1 - ((node_memory_MemAvailable_bytes{instance=~".*server.*"} or
(node_memory_MemFree_bytes{instance=~".*server.*"} +
node_memory_Cached_bytes{instance=~".*server.*"} +
node_memory_Buffers_bytes{instance=~".*server.*"})) /
node_memory_MemTotal_bytes{instance=~".*server.*"}))
# Container-level
sum by (container, pod, namespace) (container_memory_usage_bytes{pod=~".*server.*"}) /
sum by (container, pod, namespace) (container_spec_memory_limit_bytes{pod=~".*server.*"}) * 100

Identify whether the growth is job state, connections, or cache, review job queue depth, and check client connection counts before increasing server memory or adding servers for redundancy.

Disk usage

Servers write configuration data, job metadata, and logs, and if using a local storage backend, state information, to disk. Running out of space prevents job state from persisting, breaks the audit trail in logs, and can fail configuration updates.

PromQL:

# System-level
100 - ((node_filesystem_avail_bytes{instance=~".*server.*",mountpoint=~"/data.*|/var.*"} /
node_filesystem_size_bytes{instance=~".*server.*",mountpoint=~"/data.*|/var.*"}) * 100)
# Container-level
sum by (pod, namespace) (kubelet_volume_stats_used_bytes{persistentvolumeclaim=~".*server.*"}) /
sum by (pod, namespace) (kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~".*server.*"}) * 100

Identify the largest disk consumers (logs, job history, configuration), implement log rotation and retention policies, and increase disk allocation ahead of running out rather than after.

Clients

Gateway clients handle API interactions with Platform, job result polling, webhook processing, and event streaming. Client resource pressure shows up as integration problems between Gateway and Platform rather than as job execution failures.

CPU usage

High CPU can come from heavy API request load, inefficient client code, excessive polling frequency, or a webhook or event stream processing bottleneck. Sustained high CPU delays API requests, causes missed job status updates, and can lead to missed webhook deliveries.

PromQL:

# System-level
100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle",instance=~".*client.*"}[5m])) * 100)
# Container-level
sum by (container, pod, namespace) (rate(container_cpu_usage_seconds_total{pod=~".*client.*"}[5m])) * 100

Review API request volume and polling frequency, and check whether events can be batched instead of processed one at a time.

Memory usage

Clients maintain API connection state, cache job results, buffer webhook payloads, and queue event streams. High usage can indicate result caching accumulation, a memory leak in client code, or webhook payload buffering that isn’t draining. Exhaustion causes client OOM kills and restarts that lose job state synchronization with Platform, which can surface as inconsistent job status between the two systems.

PromQL:

# System-level
100 * (1 - ((node_memory_MemAvailable_bytes{instance=~".*client.*"} or
(node_memory_MemFree_bytes{instance=~".*client.*"} +
node_memory_Cached_bytes{instance=~".*client.*"} +
node_memory_Buffers_bytes{instance=~".*client.*"})) /
node_memory_MemTotal_bytes{instance=~".*client.*"}))
# Container-level
sum by (container, pod, namespace) (container_memory_usage_bytes{pod=~".*client.*"}) /
sum by (container, pod, namespace) (container_spec_memory_limit_bytes{pod=~".*client.*"}) * 100

Review caching and webhook buffering behavior, check event queue depth, and implement a cache eviction policy if result caching is accumulating unbounded.

Disk usage

Clients write logs and cache data, and may buffer webhook payloads or events, to disk. Running out of space breaks the audit trail in logs, fails cache writes, and can overflow a webhook buffer, losing events.

PromQL:

# System-level
100 - ((node_filesystem_avail_bytes{instance=~".*client.*",mountpoint=~"/data.*|/var.*"} /
node_filesystem_size_bytes{instance=~".*client.*",mountpoint=~"/data.*|/var.*"}) * 100)
# Container-level
sum by (pod, namespace) (kubelet_volume_stats_used_bytes{persistentvolumeclaim=~".*client.*"}) /
sum by (pod, namespace) (kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~".*client.*"}) * 100

Identify what’s consuming disk space (logs, cache, buffers), implement log rotation, and clean up old cache data before increasing disk allocation.