Loki: Logs by Stream, Not Full-Text Index

Enterprise problem: Engineers need detailed event evidence during incidents, but indexing every word and every unique identifier can make log cost and query latency grow faster than the systems being observed.

Whole-course context: The incoming evidence proves metrics can move from local Prometheus through remote write to shared storage and alert routing; logs now add the forensic record behind a metric symptom.

Today’s slice: Design Loki streams, structured metadata, object-storage flow, and LogQL queries while keeping source-owned business data and high-cardinality request identifiers out of the stream-label index.

End-of-day evidence: Produce a reviewed label contract plus one bounded LogQL investigation that finds a known checkout failure without creating an unbounded stream dimension.

Still unsolved: Trace causality, code-level profiles, Grafana navigation, and the complete cross-signal architecture remain deferred.

If the label contract mirrors every JSON field, customers pay for an explosive stream index and still receive slow investigations. These use cases separate repeatable service filtering from request-specific forensic search.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D11-UC-01	Incident responder	Find payment timeout logs for checkout in production during a bounded window	A stream selector plus filters returns the known event and preserves its trace/request metadata	An over-broad or unauthorized tenant query is rejected or bounded, while a precise control query completes
D11-UC-02	Observability platform operator	Keep log streams queryable without cardinality growth from unique identifiers	Label-analysis evidence shows bounded streams and high-cardinality fields stored as structured metadata	A policy test rejects a forbidden label and confirms the same field remains searchable as metadata

“Centralize logs” is not testable until query scope and index policy are observable. These stories require both a useful investigation and a rejected unsafe schema.

Story ID	Use case IDs	User story	Observable acceptance conditions
D11-US-01	D11-UC-01	As an incident responder, I want to narrow checkout logs by service and environment before filtering details, so that I can find the failure without scanning unrelated tenants	A fixed-window LogQL query returns the seeded timeout with trace ID and excludes staging data
D11-US-02	D11-UC-02	As a platform operator, I want request IDs and pod-instance IDs outside the stream-label index, so that stream count and query cost stay bounded	Ingestion policy rejects or demotes the forbidden label, stream cardinality stays within budget, and metadata search still finds the event

A log line is useful only if it remains attributable, retained, and queryable without violating tenant or cardinality policy. The flow begins with an application event and ends with a bounded result or explicit policy denial.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D11-FLOW-01	D11-UC-01, D11-UC-02	Happy	Responder opens logs for checkout production and a fixed incident window	1. Application emits structured stdout.<br>2. Alloy or an OpenTelemetry Collector parses and enriches it.<br>3. Policy promotes bounded resource attributes to labels and keeps unique IDs as metadata.<br>4. Loki ingests the stream and persists index/chunks.<br>5. LogQL selects streams, filters lines, and parses the reduced set.	Tenant, selector, time bounds, matching line, trace/request metadata, environment, timestamp, and query ID
D11-FLOW-02	D11-UC-01, D11-UC-02	Denied	Operator submits a pipeline revision that promotes trace_id to an indexed label	1. Schema policy detects an unbounded field.<br>2. Deployment is denied or the field is demoted to structured metadata.<br>3. Existing streams remain queryable.<br>4. A metadata query finds the seeded trace ID.	Denial reason, rejected revision, unchanged stream count, successful metadata lookup, and policy-run ID

Calling Loki a full-text search database leads teams to index the wrong data and overload ingestion. The system instead indexes stream label sets and stores compressed log content for parallel filtering at query time.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D11-UC-01	Grafana Explore or Loki query API	Query frontend, query scheduler, queriers, index gateway/store, object-store readers	Loki index and compressed chunks in object storage	Empty selector, query limit, chunk read error, parser error, tenant denial, or timed-out query
D11-UC-02	Collector pipeline and Loki push endpoint	Alloy/OTel processors, Loki distributor, stream policy, ingesters, compactor	Versioned pipeline policy plus Loki index/chunks	Rejected label, active-stream limit, distributor rejection, ingester error, or policy-test failure

When a stream label becomes an accidental identity database, deletion, authorization, and cost controls become unreliable. Loki owns telemetry copies and indexes; the source application remains authoritative for orders, users, and payments.

Generated-application database: Not created in this slice — Loki stores operational log evidence, not generated-application business records or user authorization state.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
Label-policy revision	Git/configuration store owned by platform operations	policy_revision_sha	None — root policy revision has no parent	organization_id	Only approved bounded attributes become index labels; secrets are redacted	Review, deploy, roll back, retain history, then expire by audit policy	D11-UC-02
Log stream	Loki index owned by logging platform	tenant_id/label_fingerprint	References policy revision and chunk set	tenant_id	Exact label set identifies one stream; tenant isolation and active-stream limits apply	Create on first entry, age out index records after retention, delete with tenant data	D11-UC-01, D11-UC-02
Log chunk	Object store owned by Loki	tenant_id/chunk_id	Foreign reference to stream fingerprint	tenant_id	Compressed immutable time range; retention and legal-hold policy apply	Flush, compact where supported, retain, tombstone, and delete	D11-UC-01
Query evidence	Platform evidence store owned by operations	log_query_run_id	Opaque reference to policy revision, selector, and result digest	organization_id	Time-bounded query and denial result are immutable	Create per test, retain for audit objective, then expire	D11-UC-01, D11-UC-02

High-cardinality labels create many tiny streams and damage cost and performance, so labels should describe stable sources rather than unique events. A Loki stream is every log entry sharing one exact label set.

Indexed labels: service_name, deployment_environment, cluster, namespace
Structured metadata: trace_id, request_id, user_id, deployment identifier
Log body: message, exception, business-safe details

The current Loki documentation explicitly recommends structured metadata for frequently searched high-cardinality data and advises new users to demote k8s.pod.name and service.instance.id from default index labels. A field may still be sensitive even when it is not indexed; redact credentials, tokens, and unnecessary personal data before export.

Parsing every record before narrowing streams wastes compute and can time out during an incident. Start with tenant, time, and precise labels; then apply cheap line filters before parsers and field comparisons.

{service_name="checkout", deployment_environment="production"}
  |= "payment authorization failed"
  | json
  | duration_ms > 1000

For a trace identifier stored as structured metadata, use the syntax supported by your Loki/collector version or filter the reduced log body. Validate the query plan and bytes scanned; a correct result with an unbounded selector is still an unsafe operational query.

Logs can also produce metrics:

sum by (service_name) (
  rate({deployment_environment="production"} |= "payment authorization failed" [5m])
)

Instrument recurring aggregate questions as native metrics when practical; reserve repeated log scans for irregular forensic questions.

A laptop monolith proves syntax but not distributed availability, so deployment evidence must match the promised scale. Current Loki supports monolithic and distributed modes; Simple Scalable is deprecated and documented for removal in Loki 4.0, so new production designs should not treat it as the durable destination.

Run these checks in a test tenant:

1. Seed two production timeout records and one staging control with a unique trace_id in structured metadata.
2. Query a 15-minute production window and prove the two records match while staging does not.
3. Submit a policy that promotes trace_id; verify policy denial or controlled demotion.
4. Compare stream cardinality before and after the rejected revision.
5. Delete the test tenant or wait through its retention policy and prove index/chunk expiration.

Loki’s value depends on disciplined indexing rather than collecting every possible field. Preserve these rules:

• Loki indexes label sets for streams, not every word in each log line.
• Stable, bounded source attributes belong in labels; unique identifiers usually belong in structured metadata.
• Query from narrow time and labels toward expensive parsing.
• Object storage owns retained chunks, while the source system still owns business truth.
• Monolithic is suitable for getting started; distributed mode is the production-scale topology to evaluate.

Log search can appear functional while the schema is already unsustainable. Complete these checks before onboarding a service:

• [ ] Published an approved label, metadata, body, and redaction contract.
• [ ] Measured active streams and cardinality by tenant and service.
• [ ] Proved a bounded LogQL query finds a seeded failure and excludes a control.
• [ ] Rejected or demoted one high-cardinality label.
• [ ] Verified tenant denial and retention/deletion behaviour.
• [ ] Selected a current supported deployment mode from measured scale.

Loki label defaults and deployment guidance change, so use current primary documentation for implementation. These sources support this lesson:

• Grafana Loki: Understand labels
• Grafana Loki label best practices
• Grafana Loki structured metadata
• Grafana Loki deployment modes
• Grafana Loki LogQL