Centralize logs with Loki and Alloy

This page is a deep dive on Grafana Loki and Grafana Alloy as a log management solution for Itential Platform and Itential Gateway. It’s the companion to Manage logs and is intended for operators who have already decided to adopt Loki and Alloy and need to deploy, configure, and operate the stack day to day.

If you haven’t read the Loki and Alloy overview in Manage logs yet, start there. This page picks up where that overview leaves off.

Overview

Loki is a horizontally scalable log aggregation system built by Grafana Labs. It indexes log streams by a small set of labels, such as job, env, and host, rather than by full-text content, which keeps storage and operational costs significantly lower than search-engine-based alternatives such as Elasticsearch.

Alloy is Grafana’s component-based observability collector and the official successor to Promtail. In this deployment, Alloy is scoped to log collection, where it plays the same role Promtail used to: discover log sources on each host, tail them continuously, attach labels, and forward each line to Loki over HTTP.

This combination gives you:

• A single log aggregation backend for both VM-based and Kubernetes-based Itential deployments.
• Native Grafana integration. Log queries, log-derived metrics, and log-based alerts all use the same Grafana instance and the same alerting infrastructure you already use for Prometheus metrics.
• A query language, LogQL, that’s intentionally similar to PromQL, so if your team already writes Prometheus queries, there’s very little new to learn.
• A forward-looking agent. Promtail is in maintenance-only status. Alloy is the actively developed path, and the same binary can also collect Prometheus metrics and OpenTelemetry traces if you choose to consolidate agents later.

Architecture

A single Loki instance is the central destination for log streams from every Itential host. Alloy runs differently on virtual machines than it does on Kubernetes, but the data flow is identical from Loki onward.

Every arrow from Alloy to Loki is an HTTP POST to Loki’s /loki/api/v1/push endpoint, optionally over TLS. Grafana queries Loki using Loki’s own HTTP API. No agent on an Itential host ever talks to Grafana directly.

Why Alloy and not Promtail

Promtail and Alloy serve the same role in this stack: discover logs, tail them, and push to Loki. Alloy is the supported path going forward. Promtail has entered maintenance-only mode and receives only security fixes from Grafana Labs.

For this deployment, Alloy is scoped identically to Promtail’s former role. Consolidating metrics and trace collection into the same agent is a follow-on benefit, not a requirement.

Label convention

Loki’s storage and query performance depend on keeping the label set small and low-cardinality. Every log stream from an Itential host carries the following labels.

Recommended job values for Itential components

Using these values keeps your LogQL queries and Grafana dashboards portable across environments and consistent with Itential’s reference implementation.

Cardinality rule

Never use a high-cardinality value as a label. Specifically, don’t put any of the following into a Loki label:

• Request IDs, job IDs, or correlation UUIDs
• URLs or URL paths
• User identifiers or session IDs
• Client request IP addresses
• Any field whose value space keeps growing over time

Each unique combination of label values creates a new stream in Loki. A single high-cardinality label can produce millions of streams and degrade query performance or exhaust your ingestion limits. Parse those fields out of the log line at query time instead, using LogQL’s pattern, regexp, or json parsers. The Loki query layer is designed for this; the index layer isn’t.

Deploy Loki

For a typical Itential deployment, Loki runs as a single-node monolithic instance. This mode uses filesystem storage and an in-memory ring, which is the simplest operational footprint and is sufficient for most platform monitoring workloads. If you need high availability or object storage, refer to Loki’s distributed deployment topology in the upstream documentation.

You have two practical choices for where to run Loki.

Option 1: Loki on a dedicated VM (systemd)

Choose this option when most of your Itential deployment runs on VMs and you don’t have a Kubernetes cluster available for the monitoring stack itself. Loki is distributed as a single Go binary and runs as a systemd service.

• Install on RHEL/Rocky Linux 9 or an equivalent distribution.
• Co-locating Loki with Grafana on the same VM is supported and common for development or smaller production deployments.
• Storage is local filesystem. Size it for your retention target; see Plan retention and sizing.
• Open TCP 3100 (the HTTP API and push endpoint) to every Alloy host. TCP 9096 (gRPC) is only needed for multi-component topologies.

Refer to Grafana’s Loki installation documentation for the binary download and systemd setup.

Option 2: Loki in Kubernetes (Helm)

Choose this option when Itential runs in Kubernetes, or when a separate Kubernetes cluster is available for the monitoring stack.

• Deploy Grafana Labs’ official Loki Helm chart in monolithic mode.
• Back the StatefulSet with a PersistentVolumeClaim sized for your retention target.
• Expose Loki through a ClusterIP service for in-cluster Alloy, or through a LoadBalancer if Grafana lives outside the cluster.

Refer to Grafana’s Loki Helm installation documentation.

Minimal Loki configuration

A monolithic Loki instance reads its configuration from a loki-config.yml file. The relevant sections look like this:

1
auth_enabled: false
2
3
server:
4
  http_listen_port: 3100
5
  grpc_listen_port: 9096
6
  log_level: info
7
8
common:
9
  instance_addr: 127.0.0.1
10
  path_prefix: /var/lib/loki
11
  storage:
12
    filesystem:
13
      chunks_directory: /var/lib/loki/chunks
14
      rules_directory: /var/lib/loki/rules
15
  replication_factor: 1
16
  ring:
17
    kvstore:
18
      store: inmemory
19
20
schema_config:
21
  configs:
22
    - from: 2024-01-01
23
      store: tsdb
24
      object_store: filesystem
25
      schema: v13
26
      index:
27
        prefix: index_
28
        period: 24h
29
30
limits_config:
31
  reject_old_samples: true
32
  reject_old_samples_max_age: 168h     # 7 days retention
33
  ingestion_rate_mb: 16
34
  ingestion_burst_size_mb: 32
35
  max_entries_limit_per_query: 5000
36
37
analytics:
38
  reporting_enabled: false

The Helm chart maps these same settings to chart values; the underlying configuration is identical.

Deploy Alloy on virtual machines

On VM-based Itential deployments, Alloy runs as a systemd service on every host that produces logs: Platform, Gateway, MongoDB, and Redis. Alloy is distributed by Grafana Labs as an official RPM and DEB package.

Install Alloy

Add the Grafana RPM repository (or the DEB equivalent on Debian/Ubuntu) and install the alloy package with your distribution’s package manager. Refer to Grafana’s Alloy installation documentation.

After installation:

• Add the alloy system user to the systemd-journal group so it can read the journal.
• For log files that aren’t world-readable, add the alloy user to the appropriate group. The MongoDB log (/var/log/mongodb/mongod.log) is owned by mongod:mongod, and the Gateway log is owned by itential:itential. Without group membership, Alloy fails to tail those files without raising an error you’d notice right away.
• Open TCP 12345 in the host firewall if you want access to Alloy’s local UI for diagnostics.
• If a Promtail service already exists on the host, stop and disable it before starting Alloy. Running both produces duplicate log streams.

Configure Alloy

Alloy’s configuration is a single file at /etc/alloy/config.alloy, written in Alloy’s component-based DSL. The pipeline for VM-based collection has three parts: the Loki push destination, the systemd journal source, and the application log file sources.

1
// Loki push endpoint. Applies to all sources.
2
loki.write "default" {
3
  endpoint {
4
    url = "http://<LOKI-HOST-PRIVATE-IP>:3100/loki/api/v1/push"
5
  }
6
}
7
8
// Systemd journal, all units on this host.
9
loki.source.journal "journal" {
10
  max_age    = "12h"
11
  labels     = {
12
    job  = "journal",
13
    host = constants.hostname,
14
    env  = "production",
15
  }
16
  forward_to = [loki.relabel.journal_unit.receiver]
17
}
18
19
loki.relabel "journal_unit" {
20
  forward_to = [loki.write.default.receiver]
21
  rule {
22
    source_labels = ["__journal__systemd_unit"]
23
    target_label  = "unit"
24
  }
25
}
26
27
// Application log files. Set the path and job label per file.
28
local.file_match "app_logs" {
29
  path_targets = [
30
    {
31
      __path__ = "/var/log/itential/platform/webserver.log",
32
      job      = "iap-http",
33
      host     = constants.hostname,
34
      env      = "production",
35
    },
36
    {
37
      __path__ = "/var/log/itential/platform/*.log",
38
      job      = "iap",
39
      host     = constants.hostname,
40
      env      = "production",
41
    },
42
  ]
43
}
44
45
loki.source.file "app_logs" {
46
  targets    = local.file_match.app_logs.targets
47
  forward_to = [loki.write.default.receiver]
48
}

The same pattern applies to Gateway, MongoDB, and Redis hosts. Change the __path__ and job label for each file.

Alloy stores its read position for each file at /var/lib/alloy/positions.yaml. This file survives restarts and ensures no log lines are duplicated or lost when the Alloy service restarts.

Deploy Alloy on Kubernetes

On Kubernetes-based Itential deployments, Alloy runs as a DaemonSet, with one pod per node. Each pod uses the Kubernetes API to discover the pods running on its own node and tails their container logs from the node filesystem under /var/log/pods/.

Install Alloy

Deploy Grafana Labs’ official Alloy Helm chart, or the Helm chart that ships with the Itential reference implementation. Refer to Grafana’s Alloy on Kubernetes documentation.

The DaemonSet requires:

• A ServiceAccount with a ClusterRole that grants get, list, and watch on nodes, namespaces, and pods. This is the minimum scope needed for pod discovery.
• HostPath mounts for /var/log/pods (read-only) and /var/lib/alloy (read-write, for the positions file).
• runAsUser: 0 on most container runtimes, because kubelet writes container logs as root.

Configure Alloy

The Alloy configuration for Kubernetes has four parts: a Loki push destination, pod discovery through the Kubernetes API, a relabel block that maps Kubernetes metadata onto the label convention above, and a file-tailing block that reads the discovered log files.

1
// Loki push endpoint.
2
loki.write "default" {
3
  endpoint {
4
    url = "http://loki.observability.svc:3100/loki/api/v1/push"
5
  }
6
  external_labels = {
7
    env     = "production",
8
    cluster = "eks-prod-us-east-1",
9
  }
10
}
11
12
// Discover only pods running on this node.
13
discovery.kubernetes "pods" {
14
  role = "pod"
15
  selectors {
16
    role  = "pod"
17
    field = "spec.nodeName=" + sys.env("HOSTNAME_NODE")
18
  }
19
}
20
21
// Map Kubernetes metadata onto the label convention, drop
22
// non-running pods, and build the on-disk log path.
23
discovery.relabel "pods" {
24
  targets = discovery.kubernetes.pods.targets
25
26
  rule {
27
    source_labels = ["__meta_kubernetes_pod_phase"]
28
    regex         = "Running"
29
    action        = "keep"
30
  }
31
  rule { source_labels = ["__meta_kubernetes_namespace"]              target_label = "namespace" }
32
  rule { source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"] target_label = "app" }
33
  rule { source_labels = ["__meta_kubernetes_pod_node_name"]          target_label = "host" }
34
  rule { source_labels = ["__meta_kubernetes_pod_name"]               target_label = "pod" }
35
  rule { source_labels = ["__meta_kubernetes_pod_container_name"]     target_label = "container" }
36
  rule {
37
    source_labels = ["__meta_kubernetes_pod_annotation_logs_itential_com_job"]
38
    target_label  = "job"
39
  }
40
  rule {
41
    source_labels = [
42
      "__meta_kubernetes_pod_uid",
43
      "__meta_kubernetes_pod_container_name",
44
    ]
45
    target_label = "__path__"
46
    separator    = "/"
47
    regex        = "(.+)/(.+)"
48
    replacement  = "/var/log/pods/*${1}/${2}/*.log"
49
  }
50
}
51
52
// Glob-expand the __path__ template into concrete file targets.
53
local.file_match "pods" {
54
  path_targets = discovery.relabel.pods.output
55
}
56
57
// Tail each file and forward to Loki.
58
loki.source.file "pods" {
59
  targets    = local.file_match.pods.targets
60
  forward_to = [loki.write.default.receiver]
61
}

The HOSTNAME_NODE environment variable is set on each Alloy container from the Kubernetes downward API (spec.nodeName). The field selector ensures each Alloy pod only watches its own node’s workloads, which scales linearly with cluster size.

Set the job label on Itential pods

The relabel rule above reads the logs.itential.com/job pod annotation and uses its value as the Loki job label. To label a Platform pod’s logs as iap, add this to the pod template, not the Deployment metadata:

1
spec:
2
  template:
3
    metadata:
4
      annotations:
5
        logs.itential.com/job: iap

Apply the same pattern to Gateway pods (iag), and to any Kubernetes-hosted MongoDB or Redis (mongodb, redis, redis-sentinel). A pod without this annotation has no job label set, so a query filtering on {job="iap"} won’t match it.

Connect Loki to Grafana

Loki is a first-class data source in Grafana. No plugins are required.

Configure manually through the UI

1. In Grafana, go to Connections > Data sources > Add data source > Loki.
2. Set the URL to your Loki endpoint:
   • VM-based Loki: http://<LOKI-HOST-PRIVATE-IP>:3100
   • Kubernetes-based Loki in the same cluster: http://loki.<namespace>.svc:3100
3. Save and test. A green “Data source is working” indicator confirms connectivity.

Provision the data source (recommended for production)

Provision the data source through Grafana’s configuration files so the connection survives Grafana restarts and is reproducible across environments. Drop a YAML file under Grafana’s provisioning/datasources/ directory:

1
apiVersion: 1
2
3
datasources:
4
  - name: Loki
5
    type: loki
6
    access: proxy
7
    url: http://<LOKI-HOST-PRIVATE-IP>:3100
8
    isDefault: false
9
    editable: false
10
    jsonData:
11
      maxLines: 1000

For Helm-deployed Grafana, the official chart includes a sidecar (kiwigrid/k8s-sidecar) that auto-mounts any ConfigMap labeled grafana_datasource=1 as a data source. This is the cleanest way to wire Loki up when both Grafana and Loki run in the same cluster.

Once the data source is connected, log queries become available in Explore, and you can add Loki-backed panels to any dashboard. A single dashboard can mix Prometheus and Loki panels freely.

Query logs with LogQL

LogQL is Loki’s query language. It has two forms: log queries, which return matching log lines, and metric queries, which derive time-series numbers from log content. Both start with a label selector in curly braces.

Log queries

The basic shape is {label="value"} | filter | parse.

Find every error line in the Platform web server log over the time range selected in Grafana:

{job="iap-http", env="production"} |= "ERROR"

Find Gateway errors that mention a specific device name, excluding noisy heartbeat lines:

{job="iag"} |= "core-router-42" |= "error" != "heartbeat"

Parse the HTTP status code out of a Platform web server log line, then filter on it:

{job="iap-http"} | pattern `<_> "<method> <_>" <status> <_>` | status =~ "5.."

Metric queries

A metric query wraps a log query in an aggregation function. The result is a time series that Grafana can plot like any Prometheus metric.

HTTP 5xx error rate per host, over the last 5 minutes:

sum by (host) (
  rate({job="iap-http"} |= " 5" [5m])
)

Top 10 URLs returning a 5xx status, over the last hour:

topk(10,
  sum by (url) (
    count_over_time({job="iap-http"}
      | pattern `<_> "<method> <url>" <status> <_>`
      | status =~ "5.."
      [1h])
  )
)

Error rate compared across environments:

sum by (env) (
  rate({job="iap"} |~ "(?i)error|exception" [5m])
)

Gateway device interaction throughput, by gateway host:

sum by (host) (
  rate({job="iag"} |= "task complete" [5m])
)

MongoDB connection-rejected events per minute:

sum(rate({job="mongodb"} |= "connection refused" [1m]))

For the full LogQL reference, including operators, parsers, and aggregations, see Grafana’s LogQL documentation.

Create recording rules for high-volume streams

A LogQL metric query is evaluated by scanning matching log streams at query time. For a dashboard that refreshes frequently, or a query over a high-volume stream such as the Platform web server log or a large Gateway fleet, that scan can become slow or expensive.

A Loki recording rule pre-computes a metric query on a schedule and stores the result. Dashboard panels then query the pre-computed metric instead of scanning raw logs. This is the same pattern Prometheus uses for its own recording rules.

A recording rule for the Platform HTTP 5xx error rate looks like this:

1
groups:
2
  - name: iap_recording_rules
3
    interval: 30s
4
    rules:
5
      - record: itential:iap_http_5xx_rate:5m
6
        expr: |
7
          sum by (host, env) (
8
            rate({job="iap-http"} |= " 5" [5m])
9
          )

Drop this file under Loki’s rules/ directory, or configure the ruler component to load it from object storage. A Grafana panel can then query the recorded metric directly:

itential:iap_http_5xx_rate:5m

Use a recording rule when:

• A LogQL metric query powers a panel that refreshes every 10 seconds or faster.
• A panel aggregates over a window longer than 15 minutes.
• An alert rule needs to evaluate consistently regardless of dashboard load.

Plan retention and sizing

Loki enforces retention through two complementary mechanisms.

reject_old_samples_max_age

This is the primary retention control. Loki rejects writes for log lines older than this age. The default is 168h (7 days). To retain logs for 30 days, set 744h. To retain for 90 days, set 2160h.

Ingestion rate limits

1
limits_config:
2
  ingestion_rate_mb: 16        # MB/s of log volume accepted per tenant
3
  ingestion_burst_size_mb: 32  # short-burst tolerance

The defaults handle most Itential deployments. Increase them only when Loki starts rejecting writes with 429 Too Many Requests errors, which Alloy surfaces in its own logs. A reasonable starting point for a busy production deployment is 32 / 64.

Estimate storage

Loki compresses log chunks aggressively, typically at a 5:1 to 10:1 ratio against raw log volume. To estimate disk usage:

1. Measure raw log volume per host per day, in bytes. For a typical Platform host, this is in the range of 100 MB to 1 GB per day depending on traffic and adapter activity.
2. Multiply by the number of hosts sending logs.
3. Divide by the expected compression ratio. Use 5 for a conservative estimate.
4. Multiply by the retention period in days.

For example, a 10-host deployment averaging 500 MB per host per day with 30-day retention: 10 × 500 MB × 30 / 5 = 30 GB. Provision 60 to 90 GB to leave headroom for the index and future growth.

Secure the Alloy-to-Loki connection

By default, Alloy ships logs to Loki over plain HTTP. For a production deployment where that traffic crosses a network boundary (between subnets, VPCs, or clusters), encrypt the connection with TLS.

Configure the Loki side

Provision a server certificate and private key for the Loki host. Place them at predictable paths and enable TLS in loki-config.yml:

1
server:
2
  http_listen_port: 3100
3
  http_tls_config:
4
    cert_file: /etc/loki/certs/loki.crt
5
    key_file:  /etc/loki/certs/loki.key

Restart Loki. The HTTP push endpoint now requires HTTPS, and so do the /ready and /metrics endpoints.

Configure the Alloy side

Place the CA certificate that signed Loki’s server certificate on every Alloy host, or mount it as a Kubernetes secret in the DaemonSet case. Update the Alloy loki.write block to use HTTPS and reference the CA:

1
loki.write "default" {
2
  endpoint {
3
    url = "https://<LOKI-HOST-PRIVATE-IP>:3100/loki/api/v1/push"
4
    tls_config {
5
      ca_file = "/etc/alloy/certs/ca.crt"
6
    }
7
  }
8
}

For mutual TLS, where Loki also verifies the Alloy client, add cert_file and key_file to the tls_config block and configure Loki’s http_tls_config with client_auth_type: RequireAndVerifyClientCert and a client_ca_file.

Certificate distribution is intentionally out of scope for these tools. Use your existing certificate management process, such as cert-manager in Kubernetes, an internal PKI, or Ansible Vault.

Verify the pipeline

After deployment, verify each stage of the pipeline, from Alloy through Loki to Grafana.

Verify Loki is healthy

$
# On the Loki host, or through a port-forward in Kubernetes
$
curl -sf http://<LOKI-HOST>:3100/ready
$
# Expected: "ready"
$
$
curl -s http://<LOKI-HOST>:3100/metrics | grep loki_build_info
$
# Expected: a single line with the running Loki version

Verify Alloy is tailing files

On a VM:

$
systemctl status alloy
$
journalctl -u alloy --since "5 minutes ago" | grep -E "tail routine|Seeked"
$
# Expected: one "tail routine: started" line per configured log file

On Kubernetes:

$
kubectl -n observability rollout status daemonset/alloy
$
kubectl -n observability logs daemonset/alloy --tail=100 | grep -E "tail routine|Seeked"

Check Alloy’s own logs for errors:

$
journalctl -u alloy --since "5 minutes ago" | grep -i error
$
# Or in Kubernetes:
$
kubectl -n observability logs daemonset/alloy --tail=200 | grep -i error

Alloy also exposes a local UI at http://<host>:12345 showing each component’s state, target counts, and any pipeline errors. This is the fastest way to confirm discovery is working.

Verify logs are landing in Loki

Query the list of labels Loki has seen:

$
curl -s "http://<LOKI-HOST>:3100/loki/api/v1/labels" | jq
$
# Expected: a JSON array including "job", "host", "env"

Query the values for the job label:

$
curl -s "http://<LOKI-HOST>:3100/loki/api/v1/label/job/values" | jq
$
# Expected: ["iap", "iap-http", "iag", "mongodb", "redis", ...]

Pull a few recent lines for Platform:

$
curl -sG "http://<LOKI-HOST>:3100/loki/api/v1/query_range" \
>
  --data-urlencode 'query={job="iap"}' \
>
  --data-urlencode 'limit=5' | jq

Verify Grafana can query Loki

Open Grafana, go to Explore, select the Loki data source, and run:

{job="iap"}

Recent Platform log lines should appear within seconds. If they don’t, work backward through the verification steps above.

Troubleshoot

No logs are appearing in Grafana

1. Confirm Loki is reachable from the Alloy host: run curl http://<LOKI-HOST>:3100/ready from the Alloy host itself, not from your workstation. Cloud security groups and VPC firewalls block this most often.
2. Confirm <LOKI-HOST> resolves to the private IP, not the public one. EC2 and Azure VMs can’t route to their own public addresses.
3. Look at Alloy’s own logs for push errors (status=4xx or status=5xx from loki.write).
4. Check Loki’s logs for ingestion rejections.

Alloy starts, but no application log streams appear

1. Verify the file paths in the Alloy config match the actual files on disk.
2. Verify the alloy user has read access. A permissions issue on /var/log/mongodb/mongod.log (mode 0640, owner mongod:mongod) or /var/log/gateway/gateway.log (mode 0660, owner itential:itential) is a common cause.
3. Add the alloy user to the file owner’s group, restart Alloy, and check again.

Loki rejects writes with “429 Too Many Requests”

You’ve hit the ingestion rate limit. Either reduce log volume by filtering at the source with a loki.relabel drop rule, or raise ingestion_rate_mb and ingestion_burst_size_mb in Loki’s limits_config.

Query performance degrades over time

This is almost always a label cardinality problem. Use Loki’s series API to find the offending label:

$
curl -s "http://<LOKI-HOST>:3100/loki/api/v1/series?match[]={job=\"iap\"}" | jq 'length'

A value in the tens of thousands or higher for a single job indicates a high-cardinality value has leaked into a label. Identify the offending label, remove it from the Alloy config, and parse the field at query time instead.

Static pods don’t appear in Kubernetes logs

This is expected with the default configuration. A static pod (a control-plane component on a Kubernetes control-plane node) writes to a directory that uses a config hash rather than the pod UID, and the default __path__ template doesn’t match it. Add a second loki.source.file block tailing /var/log/containers/*.log if you need control-plane log coverage.

The Alloy positions file grows large

The positions file at /var/lib/alloy/positions.yaml (or the hostPath mount in Kubernetes) grows as Alloy tracks more files. Growth up to a few megabytes is normal. If it grows past 100 MB, a path pattern is likely matching far more files than you expect. Check for log rotation files (*.log.1, *.log.gz) being unintentionally tailed, and restrict the __path__ glob to current files only.

Further reading

• Loki documentation
• Alloy documentation
• LogQL reference
• Grafana alerting on Loki queries