> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.itential.com/itential-platform/2023-2/monitor/log/loki-alloy/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.itential.com/_mcp/server. # Centralize logs with Loki and Alloy > Deploy, configure, secure, and troubleshoot Grafana Loki and Alloy for centralized log management across Platform, Gateway, MongoDB, and Redis. This page is a deep dive on Grafana Loki and Grafana Alloy as a log management solution for Itential Platform and Itential Gateway. It's the companion to [Manage logs](/itential-platform/monitor/log/manage) and is intended for operators who have already decided to adopt Loki and Alloy and need to deploy, configure, and operate the stack day to day. If you haven't read the Loki and Alloy overview in [Manage logs](/itential-platform/monitor/log/manage) yet, start there. This page picks up where that overview leaves off. ## Overview Loki is a horizontally scalable log aggregation system built by Grafana Labs. It indexes log streams by a small set of labels, such as `job`, `env`, and `host`, rather than by full-text content, which keeps storage and operational costs significantly lower than search-engine-based alternatives such as Elasticsearch. Alloy is Grafana's component-based observability collector and the official successor to Promtail. In this deployment, Alloy is scoped to log collection, where it plays the same role Promtail used to: discover log sources on each host, tail them continuously, attach labels, and forward each line to Loki over HTTP. This combination gives you: * A single log aggregation backend for both VM-based and Kubernetes-based Itential deployments. * Native Grafana integration. Log queries, log-derived metrics, and log-based alerts all use the same Grafana instance and the same alerting infrastructure you already use for Prometheus metrics. * A query language, LogQL, that's intentionally similar to PromQL, so if your team already writes Prometheus queries, there's very little new to learn. * A forward-looking agent. Promtail is in maintenance-only status. Alloy is the actively developed path, and the same binary can also collect Prometheus metrics and OpenTelemetry traces if you choose to consolidate agents later. ## Architecture A single Loki instance is the central destination for log streams from every Itential host. Alloy runs differently on virtual machines than it does on Kubernetes, but the data flow is identical from Loki onward. ![Alloy running as a systemd service on VMs and as a DaemonSet on Kubernetes both push logs to a single Loki instance. Grafana queries Loki, surfacing logs through Explore and metrics and alerts through Dashboards.](/_fern-files/itential.docs.buildwithfern.com/3ae55cb2b2f8e134136b4f974c5fec2b0642c4ca5c55d6f35f83414e51c358b5/docs/assets/images/platform/loki-alloy-architecture.drawio.svg) Every arrow from Alloy to Loki is an HTTP `POST` to Loki's `/loki/api/v1/push` endpoint, optionally over TLS. Grafana queries Loki using Loki's own HTTP API. No agent on an Itential host ever talks to Grafana directly. ## Why Alloy and not Promtail Promtail and Alloy serve the same role in this stack: discover logs, tail them, and push to Loki. Alloy is the supported path going forward. Promtail has entered maintenance-only mode and receives only security fixes from Grafana Labs. | Capability | Promtail | Alloy | | ------------------------------------- | ---------------- | ------------------------------- | | Log collection and forwarding to Loki | Yes | Yes | | Kubernetes pod discovery | Yes | Yes | | Linux file tailing | Yes | Yes | | Configuration language | YAML | Alloy DSL (component-based HCL) | | Prometheus metrics collection | No | Yes | | OpenTelemetry trace collection | No | Yes | | Active development | Maintenance only | Actively developed | | Multi-pipeline composition | No | Yes (wired components) | For this deployment, Alloy is scoped identically to Promtail's former role. Consolidating metrics and trace collection into the same agent is a follow-on benefit, not a requirement. If you have an existing Promtail deployment, plan a transition rather than running both agents side by side. A host running both Promtail and Alloy produces duplicate log streams in Loki. ## Label convention Loki's storage and query performance depend on keeping the label set small and low-cardinality. Every log stream from an Itential host carries the following labels. | Label | Description | Example | | ----------- | ----------------------------------------------------------- | -------------------------------------------------------------- | | `job` | The application or service that produced the log | `iap`, `iap-http`, `iag`, `mongodb`, `redis`, `redis-sentinel` | | `env` | Deployment environment | `production`, `staging`, `dev` | | `host` | VM hostname or Kubernetes node name | `platform01`, `worker-node-03` | | `cluster` | Kubernetes cluster identifier (Kubernetes only) | `eks-prod-us-east-1` | | `namespace` | Kubernetes namespace (Kubernetes only) | `itential-prod` | | `app` | Application name from `app.kubernetes.io/name` (Kubernetes) | `platform-core` | ### Recommended job values for Itential components | Component | `job` label | | ----------------------------------------- | ---------------- | | Platform web server (`webserver.log`) | `iap-http` | | Platform adapters and other platform logs | `iap` | | Gateway (servers, runners, clients) | `iag` | | MongoDB (`mongod.log`) | `mongodb` | | Redis data instances | `redis` | | Redis Sentinel | `redis-sentinel` | Using these values keeps your LogQL queries and Grafana dashboards portable across environments and consistent with Itential's reference implementation. ### Cardinality rule Never use a high-cardinality value as a label. Specifically, don't put any of the following into a Loki label: * Request IDs, job IDs, or correlation UUIDs * URLs or URL paths * User identifiers or session IDs * Client request IP addresses * Any field whose value space keeps growing over time Each unique combination of label values creates a new stream in Loki. A single high-cardinality label can produce millions of streams and degrade query performance or exhaust your ingestion limits. Parse those fields out of the log line at query time instead, using LogQL's `pattern`, `regexp`, or `json` parsers. The Loki query layer is designed for this; the index layer isn't. ## Deploy Loki For a typical Itential deployment, Loki runs as a single-node monolithic instance. This mode uses filesystem storage and an in-memory ring, which is the simplest operational footprint and is sufficient for most platform monitoring workloads. If you need high availability or object storage, refer to Loki's distributed deployment topology in the upstream documentation. You have two practical choices for where to run Loki. ### Option 1: Loki on a dedicated VM (systemd) Choose this option when most of your Itential deployment runs on VMs and you don't have a Kubernetes cluster available for the monitoring stack itself. Loki is distributed as a single Go binary and runs as a systemd service. * Install on RHEL/Rocky Linux 9 or an equivalent distribution. * Co-locating Loki with Grafana on the same VM is supported and common for development or smaller production deployments. * Storage is local filesystem. Size it for your retention target; see [Plan retention and sizing](#plan-retention-and-sizing). * Open TCP `3100` (the HTTP API and push endpoint) to every Alloy host. TCP `9096` (gRPC) is only needed for multi-component topologies. Refer to Grafana's [Loki installation documentation](https://grafana.com/docs/loki/latest/setup/install/) for the binary download and systemd setup. ### Option 2: Loki in Kubernetes (Helm) Choose this option when Itential runs in Kubernetes, or when a separate Kubernetes cluster is available for the monitoring stack. * Deploy Grafana Labs' official Loki Helm chart in `monolithic` mode. * Back the StatefulSet with a PersistentVolumeClaim sized for your retention target. * Expose Loki through a `ClusterIP` service for in-cluster Alloy, or through a `LoadBalancer` if Grafana lives outside the cluster. Refer to Grafana's [Loki Helm installation documentation](https://grafana.com/docs/loki/latest/setup/install/helm/). ### Minimal Loki configuration A monolithic Loki instance reads its configuration from a `loki-config.yml` file. The relevant sections look like this: ```yaml auth_enabled: false server: http_listen_port: 3100 grpc_listen_port: 9096 log_level: info common: instance_addr: 127.0.0.1 path_prefix: /var/lib/loki storage: filesystem: chunks_directory: /var/lib/loki/chunks rules_directory: /var/lib/loki/rules replication_factor: 1 ring: kvstore: store: inmemory schema_config: configs: - from: 2024-01-01 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h limits_config: reject_old_samples: true reject_old_samples_max_age: 168h # 7 days retention ingestion_rate_mb: 16 ingestion_burst_size_mb: 32 max_entries_limit_per_query: 5000 analytics: reporting_enabled: false ``` The Helm chart maps these same settings to chart values; the underlying configuration is identical. ## Deploy Alloy on virtual machines On VM-based Itential deployments, Alloy runs as a systemd service on every host that produces logs: Platform, Gateway, MongoDB, and Redis. Alloy is distributed by Grafana Labs as an official RPM and DEB package. ### Install Alloy Add the Grafana RPM repository (or the DEB equivalent on Debian/Ubuntu) and install the `alloy` package with your distribution's package manager. Refer to Grafana's [Alloy installation documentation](https://grafana.com/docs/alloy/latest/set-up/install/). After installation: * Add the `alloy` system user to the `systemd-journal` group so it can read the journal. * For log files that aren't world-readable, add the `alloy` user to the appropriate group. The MongoDB log (`/var/log/mongodb/mongod.log`) is owned by `mongod:mongod`, and the Gateway log is owned by `itential:itential`. Without group membership, Alloy fails to tail those files without raising an error you'd notice right away. * Open TCP `12345` in the host firewall if you want access to Alloy's local UI for diagnostics. * If a Promtail service already exists on the host, stop and disable it before starting Alloy. Running both produces duplicate log streams. ### Configure Alloy Alloy's configuration is a single file at `/etc/alloy/config.alloy`, written in Alloy's component-based DSL. The pipeline for VM-based collection has three parts: the Loki push destination, the systemd journal source, and the application log file sources. ```hcl // Loki push endpoint. Applies to all sources. loki.write "default" { endpoint { url = "http://:3100/loki/api/v1/push" } } // Systemd journal, all units on this host. loki.source.journal "journal" { max_age = "12h" labels = { job = "journal", host = constants.hostname, env = "production", } forward_to = [loki.relabel.journal_unit.receiver] } loki.relabel "journal_unit" { forward_to = [loki.write.default.receiver] rule { source_labels = ["__journal__systemd_unit"] target_label = "unit" } } // Application log files. Set the path and job label per file. local.file_match "app_logs" { path_targets = [ { __path__ = "/var/log/itential/platform/webserver.log", job = "iap-http", host = constants.hostname, env = "production", }, { __path__ = "/var/log/itential/platform/*.log", job = "iap", host = constants.hostname, env = "production", }, ] } loki.source.file "app_logs" { targets = local.file_match.app_logs.targets forward_to = [loki.write.default.receiver] } ``` The same pattern applies to Gateway, MongoDB, and Redis hosts. Change the `__path__` and `job` label for each file. | Host | `__path__` | `job` | | ------------------------------ | ------------------------------------------ | ---------------- | | Platform (web server) | `/var/log/itential/platform/webserver.log` | `iap-http` | | Platform (other platform logs) | `/var/log/itential/platform/*.log` | `iap` | | Gateway | `/var/log/gateway/gateway.log` | `iag` | | MongoDB | `/var/log/mongodb/mongod.log` | `mongodb` | | Redis data instance | `/var/log/redis/redis.log` | `redis` | | Redis Sentinel | `/var/log/redis/sentinel.log` | `redis-sentinel` | In cloud environments, set `loki.write` to the private or VPC-internal address of the Loki host. EC2 and Azure VMs can't route to their own public IP addresses. Using the public IP is the single most common misconfiguration in this stack. Alloy stores its read position for each file at `/var/lib/alloy/positions.yaml`. This file survives restarts and ensures no log lines are duplicated or lost when the Alloy service restarts. ## Deploy Alloy on Kubernetes On Kubernetes-based Itential deployments, Alloy runs as a DaemonSet, with one pod per node. Each pod uses the Kubernetes API to discover the pods running on its own node and tails their container logs from the node filesystem under `/var/log/pods/`. ### Install Alloy Deploy Grafana Labs' official Alloy Helm chart, or the Helm chart that ships with the Itential reference implementation. Refer to Grafana's [Alloy on Kubernetes documentation](https://grafana.com/docs/alloy/latest/set-up/install/kubernetes/). The DaemonSet requires: * A `ServiceAccount` with a `ClusterRole` that grants `get`, `list`, and `watch` on `nodes`, `namespaces`, and `pods`. This is the minimum scope needed for pod discovery. * HostPath mounts for `/var/log/pods` (read-only) and `/var/lib/alloy` (read-write, for the positions file). * `runAsUser: 0` on most container runtimes, because kubelet writes container logs as root. ### Configure Alloy The Alloy configuration for Kubernetes has four parts: a Loki push destination, pod discovery through the Kubernetes API, a relabel block that maps Kubernetes metadata onto the label convention above, and a file-tailing block that reads the discovered log files. ```hcl // Loki push endpoint. loki.write "default" { endpoint { url = "http://loki.observability.svc:3100/loki/api/v1/push" } external_labels = { env = "production", cluster = "eks-prod-us-east-1", } } // Discover only pods running on this node. discovery.kubernetes "pods" { role = "pod" selectors { role = "pod" field = "spec.nodeName=" + sys.env("HOSTNAME_NODE") } } // Map Kubernetes metadata onto the label convention, drop // non-running pods, and build the on-disk log path. discovery.relabel "pods" { targets = discovery.kubernetes.pods.targets rule { source_labels = ["__meta_kubernetes_pod_phase"] regex = "Running" action = "keep" } rule { source_labels = ["__meta_kubernetes_namespace"] target_label = "namespace" } rule { source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"] target_label = "app" } rule { source_labels = ["__meta_kubernetes_pod_node_name"] target_label = "host" } rule { source_labels = ["__meta_kubernetes_pod_name"] target_label = "pod" } rule { source_labels = ["__meta_kubernetes_pod_container_name"] target_label = "container" } rule { source_labels = ["__meta_kubernetes_pod_annotation_logs_itential_com_job"] target_label = "job" } rule { source_labels = [ "__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name", ] target_label = "__path__" separator = "/" regex = "(.+)/(.+)" replacement = "/var/log/pods/*${1}/${2}/*.log" } } // Glob-expand the __path__ template into concrete file targets. local.file_match "pods" { path_targets = discovery.relabel.pods.output } // Tail each file and forward to Loki. loki.source.file "pods" { targets = local.file_match.pods.targets forward_to = [loki.write.default.receiver] } ``` The `HOSTNAME_NODE` environment variable is set on each Alloy container from the Kubernetes downward API (`spec.nodeName`). The field selector ensures each Alloy pod only watches its own node's workloads, which scales linearly with cluster size. ### Set the job label on Itential pods The relabel rule above reads the `logs.itential.com/job` pod annotation and uses its value as the Loki `job` label. To label a Platform pod's logs as `iap`, add this to the pod template, not the Deployment metadata: ```yaml spec: template: metadata: annotations: logs.itential.com/job: iap ``` Apply the same pattern to Gateway pods (`iag`), and to any Kubernetes-hosted MongoDB or Redis (`mongodb`, `redis`, `redis-sentinel`). A pod without this annotation has no `job` label set, so a query filtering on `{job="iap"}` won't match it. Static pods on control-plane nodes (`etcd`, `kube-apiserver`, `kube-controller-manager`, `kube-scheduler`) write to a directory that uses a config hash rather than the pod UID, so the default `__path__` template above doesn't match them. Real workloads, meaning any Deployment, StatefulSet, DaemonSet, or Job, all work correctly. If you need control-plane log coverage, add a second `loki.source.file` block tailing `/var/log/containers/*.log`. ## Connect Loki to Grafana Loki is a first-class data source in Grafana. No plugins are required. ### Configure manually through the UI 1. In Grafana, go to **Connections > Data sources > Add data source > Loki**. 2. Set the URL to your Loki endpoint: * VM-based Loki: `http://:3100` * Kubernetes-based Loki in the same cluster: `http://loki..svc:3100` 3. Save and test. A green "Data source is working" indicator confirms connectivity. ### Provision the data source (recommended for production) Provision the data source through Grafana's configuration files so the connection survives Grafana restarts and is reproducible across environments. Drop a YAML file under Grafana's `provisioning/datasources/` directory: ```yaml apiVersion: 1 datasources: - name: Loki type: loki access: proxy url: http://:3100 isDefault: false editable: false jsonData: maxLines: 1000 ``` For Helm-deployed Grafana, the official chart includes a sidecar (`kiwigrid/k8s-sidecar`) that auto-mounts any `ConfigMap` labeled `grafana_datasource=1` as a data source. This is the cleanest way to wire Loki up when both Grafana and Loki run in the same cluster. Once the data source is connected, log queries become available in **Explore**, and you can add Loki-backed panels to any dashboard. A single dashboard can mix Prometheus and Loki panels freely. ## Query logs with LogQL LogQL is Loki's query language. It has two forms: log queries, which return matching log lines, and metric queries, which derive time-series numbers from log content. Both start with a label selector in curly braces. ### Log queries The basic shape is `{label="value"} | filter | parse`. Find every error line in the Platform web server log over the time range selected in Grafana: ```logql {job="iap-http", env="production"} |= "ERROR" ``` Find Gateway errors that mention a specific device name, excluding noisy heartbeat lines: ```logql {job="iag"} |= "core-router-42" |= "error" != "heartbeat" ``` Parse the HTTP status code out of a Platform web server log line, then filter on it: ```logql {job="iap-http"} | pattern `<_> " <_>" <_>` | status =~ "5.." ``` ### Metric queries A metric query wraps a log query in an aggregation function. The result is a time series that Grafana can plot like any Prometheus metric. HTTP 5xx error rate per host, over the last 5 minutes: ```logql sum by (host) ( rate({job="iap-http"} |= " 5" [5m]) ) ``` Top 10 URLs returning a 5xx status, over the last hour: ```logql topk(10, sum by (url) ( count_over_time({job="iap-http"} | pattern `<_> " " <_>` | status =~ "5.." [1h]) ) ) ``` Error rate compared across environments: ```logql sum by (env) ( rate({job="iap"} |~ "(?i)error|exception" [5m]) ) ``` Gateway device interaction throughput, by gateway host: ```logql sum by (host) ( rate({job="iag"} |= "task complete" [5m]) ) ``` MongoDB connection-rejected events per minute: ```logql sum(rate({job="mongodb"} |= "connection refused" [1m])) ``` For the full LogQL reference, including operators, parsers, and aggregations, see [Grafana's LogQL documentation](https://grafana.com/docs/loki/latest/query/). ## Create recording rules for high-volume streams A LogQL metric query is evaluated by scanning matching log streams at query time. For a dashboard that refreshes frequently, or a query over a high-volume stream such as the Platform web server log or a large Gateway fleet, that scan can become slow or expensive. A Loki recording rule pre-computes a metric query on a schedule and stores the result. Dashboard panels then query the pre-computed metric instead of scanning raw logs. This is the same pattern Prometheus uses for its own recording rules. A recording rule for the Platform HTTP 5xx error rate looks like this: ```yaml groups: - name: iap_recording_rules interval: 30s rules: - record: itential:iap_http_5xx_rate:5m expr: | sum by (host, env) ( rate({job="iap-http"} |= " 5" [5m]) ) ``` Drop this file under Loki's `rules/` directory, or configure the ruler component to load it from object storage. A Grafana panel can then query the recorded metric directly: ```logql itential:iap_http_5xx_rate:5m ``` Use a recording rule when: * A LogQL metric query powers a panel that refreshes every 10 seconds or faster. * A panel aggregates over a window longer than 15 minutes. * An alert rule needs to evaluate consistently regardless of dashboard load. ## Plan retention and sizing Loki enforces retention through two complementary mechanisms. ### reject\_old\_samples\_max\_age This is the primary retention control. Loki rejects writes for log lines older than this age. The default is `168h` (7 days). To retain logs for 30 days, set `744h`. To retain for 90 days, set `2160h`. This setting rejects new writes of old data; it doesn't delete existing chunks. With filesystem storage and the schema in this guide, old chunks stay on disk until you remove them or run Loki's compactor with a retention policy enabled. For most VM-based deployments, the simpler approach is to size the disk for your retention target and let the natural ingest cadence keep storage bounded. ### Ingestion rate limits ```yaml limits_config: ingestion_rate_mb: 16 # MB/s of log volume accepted per tenant ingestion_burst_size_mb: 32 # short-burst tolerance ``` The defaults handle most Itential deployments. Increase them only when Loki starts rejecting writes with `429 Too Many Requests` errors, which Alloy surfaces in its own logs. A reasonable starting point for a busy production deployment is `32` / `64`. ### Estimate storage Loki compresses log chunks aggressively, typically at a 5:1 to 10:1 ratio against raw log volume. To estimate disk usage: 1. Measure raw log volume per host per day, in bytes. For a typical Platform host, this is in the range of 100 MB to 1 GB per day depending on traffic and adapter activity. 2. Multiply by the number of hosts sending logs. 3. Divide by the expected compression ratio. Use `5` for a conservative estimate. 4. Multiply by the retention period in days. For example, a 10-host deployment averaging 500 MB per host per day with 30-day retention: `10 × 500 MB × 30 / 5 = 30 GB`. Provision 60 to 90 GB to leave headroom for the index and future growth. ## Secure the Alloy-to-Loki connection By default, Alloy ships logs to Loki over plain HTTP. For a production deployment where that traffic crosses a network boundary (between subnets, VPCs, or clusters), encrypt the connection with TLS. ### Configure the Loki side Provision a server certificate and private key for the Loki host. Place them at predictable paths and enable TLS in `loki-config.yml`: ```yaml server: http_listen_port: 3100 http_tls_config: cert_file: /etc/loki/certs/loki.crt key_file: /etc/loki/certs/loki.key ``` Restart Loki. The HTTP push endpoint now requires HTTPS, and so do the `/ready` and `/metrics` endpoints. ### Configure the Alloy side Place the CA certificate that signed Loki's server certificate on every Alloy host, or mount it as a Kubernetes secret in the DaemonSet case. Update the Alloy `loki.write` block to use HTTPS and reference the CA: ```hcl loki.write "default" { endpoint { url = "https://:3100/loki/api/v1/push" tls_config { ca_file = "/etc/alloy/certs/ca.crt" } } } ``` For mutual TLS, where Loki also verifies the Alloy client, add `cert_file` and `key_file` to the `tls_config` block and configure Loki's `http_tls_config` with `client_auth_type: RequireAndVerifyClientCert` and a `client_ca_file`. Certificate distribution is intentionally out of scope for these tools. Use your existing certificate management process, such as cert-manager in Kubernetes, an internal PKI, or Ansible Vault. ## Verify the pipeline After deployment, verify each stage of the pipeline, from Alloy through Loki to Grafana. ### Verify Loki is healthy ```bash # On the Loki host, or through a port-forward in Kubernetes curl -sf http://:3100/ready # Expected: "ready" curl -s http://:3100/metrics | grep loki_build_info # Expected: a single line with the running Loki version ``` ### Verify Alloy is tailing files On a VM: ```bash systemctl status alloy journalctl -u alloy --since "5 minutes ago" | grep -E "tail routine|Seeked" # Expected: one "tail routine: started" line per configured log file ``` On Kubernetes: ```bash kubectl -n observability rollout status daemonset/alloy kubectl -n observability logs daemonset/alloy --tail=100 | grep -E "tail routine|Seeked" ``` Check Alloy's own logs for errors: ```bash journalctl -u alloy --since "5 minutes ago" | grep -i error # Or in Kubernetes: kubectl -n observability logs daemonset/alloy --tail=200 | grep -i error ``` Alloy also exposes a local UI at `http://:12345` showing each component's state, target counts, and any pipeline errors. This is the fastest way to confirm discovery is working. ### Verify logs are landing in Loki Query the list of labels Loki has seen: ```bash curl -s "http://:3100/loki/api/v1/labels" | jq # Expected: a JSON array including "job", "host", "env" ``` Query the values for the `job` label: ```bash curl -s "http://:3100/loki/api/v1/label/job/values" | jq # Expected: ["iap", "iap-http", "iag", "mongodb", "redis", ...] ``` Pull a few recent lines for Platform: ```bash curl -sG "http://:3100/loki/api/v1/query_range" \ --data-urlencode 'query={job="iap"}' \ --data-urlencode 'limit=5' | jq ``` ### Verify Grafana can query Loki Open Grafana, go to **Explore**, select the **Loki** data source, and run: ```logql {job="iap"} ``` Recent Platform log lines should appear within seconds. If they don't, work backward through the verification steps above. ## Troubleshoot ### No logs are appearing in Grafana 1. Confirm Loki is reachable from the Alloy host: run `curl http://:3100/ready` from the Alloy host itself, not from your workstation. Cloud security groups and VPC firewalls block this most often. 2. Confirm `` resolves to the private IP, not the public one. EC2 and Azure VMs can't route to their own public addresses. 3. Look at Alloy's own logs for push errors (`status=4xx` or `status=5xx` from `loki.write`). 4. Check Loki's logs for ingestion rejections. ### Alloy starts, but no application log streams appear 1. Verify the file paths in the Alloy config match the actual files on disk. 2. Verify the `alloy` user has read access. A permissions issue on `/var/log/mongodb/mongod.log` (mode `0640`, owner `mongod:mongod`) or `/var/log/gateway/gateway.log` (mode `0660`, owner `itential:itential`) is a common cause. 3. Add the `alloy` user to the file owner's group, restart Alloy, and check again. ### Loki rejects writes with "429 Too Many Requests" You've hit the ingestion rate limit. Either reduce log volume by filtering at the source with a `loki.relabel` drop rule, or raise `ingestion_rate_mb` and `ingestion_burst_size_mb` in Loki's `limits_config`. ### Query performance degrades over time This is almost always a label cardinality problem. Use Loki's series API to find the offending label: ```bash curl -s "http://:3100/loki/api/v1/series?match[]={job=\"iap\"}" | jq 'length' ``` A value in the tens of thousands or higher for a single `job` indicates a high-cardinality value has leaked into a label. Identify the offending label, remove it from the Alloy config, and parse the field at query time instead. ### Static pods don't appear in Kubernetes logs This is expected with the default configuration. A static pod (a control-plane component on a Kubernetes control-plane node) writes to a directory that uses a config hash rather than the pod UID, and the default `__path__` template doesn't match it. Add a second `loki.source.file` block tailing `/var/log/containers/*.log` if you need control-plane log coverage. ### The Alloy positions file grows large The positions file at `/var/lib/alloy/positions.yaml` (or the hostPath mount in Kubernetes) grows as Alloy tracks more files. Growth up to a few megabytes is normal. If it grows past 100 MB, a path pattern is likely matching far more files than you expect. Check for log rotation files (`*.log.1`, `*.log.gz`) being unintentionally tailed, and restrict the `__path__` glob to current files only. ## Further reading * [Loki documentation](https://grafana.com/docs/loki/latest/) * [Alloy documentation](https://grafana.com/docs/alloy/latest/) * [LogQL reference](https://grafana.com/docs/loki/latest/query/) * [Grafana alerting on Loki queries](https://grafana.com/docs/grafana/latest/alerting/) > Deploy, configure, secure, and troubleshoot Grafana Loki and Alloy for centralized log management across Platform, Gateway, MongoDB, and Redis.