Loki: Logs by Stream, Not Full-Text Index

Source: Observability Platform source — section 11, “Logs: Loki as the Log Backend”

Published 04 Aug 2026

The enterprise problem and today’s slice

Enterprise problem: Engineers need detailed event evidence during incidents, but indexing every word and every unique identifier can make log cost and query latency grow faster than the systems being observed.

Whole-course context: The incoming evidence proves metrics can move from local Prometheus through remote write to shared storage and alert routing; logs now add the forensic record behind a metric symptom.

Today’s slice: Design Loki streams, structured metadata, object-storage flow, and LogQL queries while keeping source-owned business data and high-cardinality request identifiers out of the stream-label index.

End-of-day evidence: Produce a reviewed label contract plus one bounded LogQL investigation that finds a known checkout failure without creating an unbounded stream dimension.

Still unsolved: Trace causality, code-level profiles, Grafana navigation, and the complete cross-signal architecture remain deferred.

The smallest complete model

Searching raw checkout text across every host is slow, expensive, and difficult to isolate by tenant. Indexing every field instead creates too many streams and moves the failure into ingestion cost.

Thesis: Loki should index a small set of stable source labels while keeping high-cardinality event context in structured metadata or the log body. Why this matters: responders retain useful detail without turning each request into a new stream.

This smallest model needs only emitted entries, a stable stream identity, and a bounded query. It establishes where selection happens before introducing parsing, object storage, or deployment topology.

High-cardinality labels create many tiny streams and damage cost and performance, so labels should describe stable sources rather than unique events. A Loki stream is every log entry sharing one exact label set.

Indexed labels: service_name, deployment_environment, cluster, namespace
Structured metadata: trace_id, request_id, user_id, deployment identifier
Log body: message, exception, business-safe details

The current Loki documentation explicitly recommends structured metadata for frequently searched high-cardinality data and advises new users to demote k8s.pod.name and service.instance.id from default index labels. A field may still be sensitive even when it is not indexed; redact credentials, tokens, and unnecessary personal data before export.

Expand the model one boundary at a time

Expand the same entries–streams–query model in dependency order: normalize and redact at collection, assign bounded labels, persist compressed chunks and their index, then narrow a query before parsing fields. Each added boundary must preserve tenant and absolute time.

Boundary	Purpose, input, and transformation	Output or interface	Scaling constraint and failure mode	Alternative guidance
Collector policy	Receive application records, redact sensitive fields, and normalize stable attributes	Tenant-scoped entries for Loki	Refused records, queue pressure, or accidental secret export	Use collector normalization for shared policy; avoid duplicating inconsistent label logic in every service
Stream assignment	Convert the exact bounded label set into stream identity	Stream fingerprint plus ordered entries	Active-stream explosion from unique labels	Use labels for stable source dimensions; keep request, trace, and user identifiers as structured metadata
Chunk and index storage	Compress entry time ranges and index stream selectors	Retained chunks reachable by tenant, labels, and time	Many tiny chunks, object-store errors, or retention backlog	Use distributed mode when measured ingest/query scale requires it; avoid production complexity for a syntax-only lab
LogQL query	Select tenant, time, and labels, then line-filter and parse reduced records	Bounded rows or metric result	Bytes scanned and parser cost	Use native metrics for recurring aggregates; use log scans for irregular forensic questions

Parsing every record before narrowing streams wastes compute and can time out during an incident. Start with tenant, time, and precise labels; then apply cheap line filters before parsers and field comparisons.

{service_name="checkout", deployment_environment="production"}
  |= "payment authorization failed"
  | json
  | duration_ms > 1000

For a trace identifier stored as structured metadata, use the syntax supported by your Loki/collector version or filter the reduced log body. Validate the query plan and bytes scanned; a correct result with an unbounded selector is still an unsafe operational query.

Logs can also produce metrics:

sum by (service_name) (
  rate({deployment_environment="production"} |= "payment authorization failed" [5m])
)

Instrument recurring aggregate questions as native metrics when practical; reserve repeated log scans for irregular forensic questions.

Run the model through one incident

The general rule is to narrow by tenant, absolute time, and stable source labels before searching event detail. A simple example selects production checkout logs for 15 minutes, filters one error phrase, and only then parses duration_ms. A realistic checkout incident follows a payment timeout through the telemetry pipeline while a staging control proves that environment isolation still works.

A laptop monolith proves syntax but not distributed availability, so deployment evidence must match the promised scale. Current Loki supports monolithic and distributed modes; Simple Scalable is deprecated and documented for removal in Loki 4.0, so new production designs should not treat it as the durable destination.

Run these checks in a test tenant:

Seed two production timeout records and one staging control with a unique trace_id in structured metadata.
Query a 15-minute production window and prove the two records match while staging does not.
Submit a policy that promotes trace_id; verify policy denial or controlled demotion.
Compare stream cardinality before and after the rejected revision.
Delete the test tenant or wait through its retention policy and prove index/chunk expiration.

The observed evidence is the exact selector, scanned time range, matching production records, excluded staging control, stream-cardinality count, and tenant deletion result. Together they distinguish an application with no matching logs from a collector rejection, stream-policy mistake, chunk failure, or unsafe query.

Failure modes, trade-offs, and decision rules

The dominant failure mode is promoting unique event values into labels, which multiplies active streams and small chunks. The central trade-off is faster indexed selection versus higher ingestion, storage, and query overhead; keeping fewer labels lowers index cost but requires bounded parsing after selection.

index a field only when it is stable, bounded, commonly used to narrow queries, and justified by measured cardinality; otherwise keep it as structured metadata or in the redacted body. Use monolithic mode for small or learning environments and distributed mode when tested ingest, availability, or query objectives require independent scaling. Avoid the deprecated Simple Scalable path for a new durable production design, and avoid Loki as an authorization or business-record database.

Before changing capacity, identify whether the constraint is collector refusal, active streams, chunk flush, object storage, query bytes, or tenant limits. A scale change is justified only by the matching observed saturation signal.

Close the loop

The logging loop is Observe → Interpret → Decide → Act → Measure. Observe refused entries, active streams, chunk health, query bytes, latency, and retention; interpret which boundary changed; decide whether to fix policy, narrow a query, or scale storage; act on one test tenant; then measure both result quality and platform cost.

For the next bounded action, keep trace_id as structured metadata and run the seeded checkout query before and after a proposed label-policy revision. The falsifiable probe succeeds only if the same two production records remain discoverable, staging stays excluded, active-stream cardinality does not increase materially, and the unauthorized tenant remains denied.

Key takeaways

Loki’s value depends on disciplined indexing rather than collecting every possible field. Preserve these rules:

Loki indexes label sets for streams, not every word in each log line.
Stable, bounded source attributes belong in labels; unique identifiers usually belong in structured metadata.
Query from narrow time and labels toward expensive parsing.
Object storage owns retained chunks, while the source system still owns business truth.
Monolithic is suitable for getting started; distributed mode is the production-scale topology to evaluate.

Checklist

Log search can appear functional while the schema is already unsustainable. Complete these checks before onboarding a service:

[ ] Published an approved label, metadata, body, and redaction contract.
[ ] Measured active streams and cardinality by tenant and service.
[ ] Proved a bounded LogQL query finds a seeded failure and excludes a control.
[ ] Rejected or demoted one high-cardinality label.
[ ] Verified tenant denial and retention/deletion behaviour.
[ ] Selected a current supported deployment mode from measured scale.

Sources

Loki label defaults and deployment guidance change, so use current primary documentation for implementation. These sources support this lesson: