Manage logs
This guide covers how to centralize and search logs across your whole Itential deployment. For Platform’s own log format, levels, and rotation behavior, see Logging overview.
The Itential monitoring stack supports two approaches to log management. These approaches aren’t mutually exclusive, and in larger deployments both are often present, serving different needs.
Log file locations
To point a log shipper such Alloy or Filebeat at the right place, you need to know where every component in the stack writes its log data, not just Platform.
Each adapter and application within Platform writes to its own file. This means log aggregation tools need to monitor an entire directory, not a single file, to capture the full Platform log stream.
Loki and Alloy
Loki is a log aggregation system built by Grafana Labs. Alloy is the agent that collects logs on each host and forwards them to Loki. Together, they provide centralized log storage and search that integrates natively with Grafana.
Loki
Loki stores log streams indexed by labels rather than by full-text indexing. A label is a key-value pair attached to a stream of log lines. Examples from an Itential deployment:
This label-based model uses significantly less storage and compute than full-text search systems, because Loki only indexes the labels, not the content of each log line. The trade-off is that log content searches run at query time across the raw text of matching streams, rather than against a pre-built index. For typical operational use cases, such as finding errors in a specific component over a specific time range, this is fast enough in practice.
Loki uses a query language called LogQL, which is intentionally similar to PromQL. Teams already familiar with Prometheus queries can write LogQL queries with minimal additional learning. LogQL can also derive time-series metrics from log content, enabling dashboard panels that show error rates or request rates calculated directly from log lines, without a separate metrics pipeline.
Impact
Loki is the lower-overhead path to centralized log management for Itential deployments. It uses the same Grafana instance already in place for metrics, adding log queries and log-derived metrics to existing dashboards without a separate visualization tool. Log-based alerts use the same Grafana alerting infrastructure as metric-based alerts. Kubernetes deployments benefit particularly because Loki with Alloy handles automatic pod log discovery without per-service configuration.
See Centralize logs with Loki and Alloy for deployment, configuration, security, and troubleshooting guidance for both the VM and Kubernetes paths.
Alloy
Alloy is the official Grafana observability agent and the successor to Promtail. Promtail has entered maintenance-only mode, and Alloy is the actively developed replacement. Alloy uses a component-based configuration model where discrete components, such as file discovery, log tailing, label transformations, and forwarding, are wired together in an .alloy configuration file.
On virtual machine hosts
Alloy runs as a systemd service. It discovers log files using path patterns, tails them continuously, and forwards new lines to Loki. It maintains a position file that records how far it has read through each log file, so no lines are lost or duplicated across agent restarts. For Platform, this means Alloy is configured to watch the /var/log/itential/platform/ directory and forward everything in it to Loki with appropriate labels.
On Kubernetes
Alloy runs as a DaemonSet, with one Alloy pod per Kubernetes node. It uses the Kubernetes API to discover running pods and tails their log output from the node filesystem under /var/log/pods/. Pod metadata (namespace, pod name, container name, application labels) is automatically attached to the log streams. As new Itential services are deployed to Kubernetes, their logs are collected automatically without any Alloy configuration change.
Benefits summary:
- Single agent for logs, metrics, and traces (metrics and traces collection is available in addition to logs, enabling future consolidation of agent tooling)
- Automatic Kubernetes log discovery
- Native Grafana integration with no bridging layers
- Ansible-managed deployment consistent with Itential’s existing automation practices
Elastic Stack
The Elastic Stack combines Elasticsearch, Logstash, Kibana, and Filebeat into a unified log management and observability platform. For Itential deployments, the stack provides centralized log search, end-to-end workflow tracing, audit trail capabilities, and proactive alerting across all Platform components.
Filebeat
Filebeat is the log collection agent for the Elastic Stack. Like Alloy, it runs on each host, monitors configured log file paths, and forwards new log lines to a central destination. Filebeat is lightweight and adds no measurable overhead to the hosts it runs on.
In the Itential deployment model, Filebeat agents run on every Platform, Gateway, MongoDB, and Redis host. Each agent monitors the log files for the component it runs alongside and forwards entries to Logstash for processing.
Logstash
Logstash is the processing pipeline for the Elastic Stack. It receives raw log events from Filebeat agents, applies transformations, enriches records with structured fields, and routes the processed output to the appropriate Elasticsearch index.
For Itential deployments, the Logstash pipeline:
- Parses the structured JSON log output from Platform and Gateway into discrete fields
- Tags records with component identity (which service produced this log line)
- Extracts and normalizes severity levels across components that use different naming conventions
- Routes records to separate Elasticsearch indices by component and date
- Handles backpressure when Elasticsearch is temporarily slow
Logstash acts as a buffer and normalizer. It ensures that logs from different components, which may use slightly different formats and field names, arrive in Elasticsearch in a consistent schema that Kibana dashboards can query uniformly.
Elasticsearch
Elasticsearch is the storage and search backend. It stores all log events in time-based indices, provides full-text search across the content of log lines, and supports aggregation queries that power Kibana dashboards.
Key capabilities for Itential deployments:
- Full-text search: Search for any string across all log data from all components, with results appearing in milliseconds. If a workflow fails, you can search for the job ID across Platform and Gateway logs simultaneously.
- Index lifecycle management: Automatically ages and expires old log data based on configurable retention policies, keeping storage costs predictable without manual intervention.
- Field-based querying: Because Logstash parses logs into structured fields, queries can target specific attributes, such as all error-level events from the Platform GatewayManager component in the last 24 hours.
Kibana
Kibana is the visualization and operations interface for the Elastic Stack. It provides dashboards, saved searches, and alerting for log data stored in Elasticsearch.
For Itential deployments, Kibana provides:
- Workflow health dashboards: Job execution rates, success and failure ratios, and duration trends across time.
- Gateway operations visibility: Gateway service invocations, device interaction records, and connectivity status.
- End-to-end job tracing: A single workflow execution can be traced from the initial API call in Platform’s web server log, through the gateway dispatch record in Platform’s GatewayManager log, to the device-level execution result in Gateway. Two correlation identifiers make this possible: the Platform Job ID (assigned when the workflow is submitted) and the Gateway Dispatch UUID (assigned when the job is dispatched to Gateway). Searching for either identifier in Kibana surfaces the complete execution trail across all components without switching between tools.
- Audit trail: A complete, searchable record of which user submitted which automation, when, against which devices, and with what outcome. This record supports both internal audit reviews and regulatory compliance requirements.
- Proactive alerting: Kibana’s alerting engine sends notifications by email, Slack, PagerDuty, or webhook when failure rates spike, gateway connectivity is lost, or authentication anomalies are detected.
Choosing an approach
Loki and the Elastic Stack aren’t competing choices. They address different parts of the observability problem:
For teams getting started with log aggregation, Loki with Alloy is the lower-friction path. It integrates with an existing Grafana deployment and can be operational quickly with modest resource requirements.
For teams with compliance requirements, a need for end-to-end workflow tracing, or an existing Elastic Stack deployment, adding the Elastic Stack to the Itential monitoring stack unlocks capabilities that Loki isn’t designed to provide.
Both stacks can be deployed simultaneously. Prometheus and Grafana handle infrastructure and job metrics. The Elastic Stack handles log aggregation, audit, and workflow tracing. Because Grafana supports Elasticsearch as a data source, it can surface Elastic log data alongside Prometheus metrics in the same dashboard panel.