Tempo: Distributed Traces and Causal Evidence

Enterprise problem: Logs from five healthy-looking services cannot prove which dependency made one customer request slow, so responders lose causal order and may remediate the wrong component.

Whole-course context: Metrics now expose aggregate symptoms and Loki preserves bounded forensic events; today consumes a seeded checkout failure and adds propagated request context that joins the distributed call chain.

Today’s slice: Instrument, collect, sample, store, and query traces in Tempo while keeping trace retention, tenant isolation, and source application authority explicit.

End-of-day evidence: Produce one complete checkout trace with correct parent-child relationships plus a TraceQL result that finds a slow or failed request without knowing its trace ID first.

Still unsolved: Code-level resource attribution, Grafana navigation across signals, and the full platform operating architecture remain deferred.

Trace volume alone does not create causal evidence; broken propagation or biased sampling can make the most important request disappear. These use cases require a complete trace and a governed retention decision.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D12-UC-01	Incident responder	Identify which checkout dependency caused a slow failed request	TraceQL finds the trace and its span tree attributes latency and error to the payment call	A malformed propagation test exposes a new root and a corrected run restores one connected trace
D12-UC-02	Observability platform operator	Retain high-value traces within tenant and cost budgets	Sampling evidence preserves seeded errors/slow traces and enforces tenant isolation and volume limits	Unauthorized lookup is denied; dropped/late spans and sampling decisions are measured with an unaffected control

“Tracing enabled” can still mean disconnected spans and invisible sampling loss. These stories make topology, search, isolation, and retention observable.

Story ID	Use case IDs	User story	Observable acceptance conditions
D12-US-01	D12-UC-01	As an incident responder, I want TraceQL to find slow checkout traces and display their connected spans, so that I can name the responsible dependency	The result contains gateway, checkout, and payment spans under one trace ID with timestamps, status, and a dominant payment duration
D12-US-02	D12-UC-02	As a platform operator, I want sampling and tenant policies to preserve rare failures without retaining every ordinary request, so that tracing remains useful and affordable	Seeded errors survive the policy, ordinary traffic is sampled at the declared rate, a foreign tenant is denied, and decisions carry run evidence

One missing propagation header breaks the causal graph, so the flow follows context from the customer request through storage and search. Tempo 3.0 microservices mode durably separates accepted writes from recent and historical reads through a Kafka-compatible system.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D12-FLOW-01	D12-UC-01, D12-UC-02	Happy	Responder searches for production checkout traces slower than two seconds	1. Instrumented services create spans and propagate W3C trace context.<br>2. Collector enriches, batches, and applies sampling.<br>3. Tempo distributor validates and shards by trace ID.<br>4. Kafka durably acknowledges.<br>5. Live-stores and block-builders serve recent and retained data.<br>6. TraceQL returns the matching connected trace.	Query, tenant, trace ID, span count/tree, sampling decision, environment, timestamp, and run ID
D12-FLOW-02	D12-UC-01, D12-UC-02	Recovery	Test request omits trace context on the checkout-to-payment call	1. Payment creates an unrelated root span.<br>2. Completeness check detects the missing child relationship.<br>3. Operator corrects propagation.<br>4. A new request produces one connected tree.<br>5. Foreign-tenant lookup remains denied.	Broken and corrected trace IDs, topology diff, denial response, collector/Tempo counters, and immutable run ID

If ingestion and search are treated as one synchronous path, expensive scans can threaten accepted trace traffic and recent-data gaps become hard to classify. The design separates collection policy, durable acceptance, recent reads, block construction, object storage, and query planning.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D12-UC-01	Grafana Tempo data source or Tempo query API	Query frontend, queriers, live-stores, object-store readers, TraceQL engine	Kafka-fed live-store for recent traces and Tempo object-store blocks for retained traces	Empty TraceQL result, partial span tree, query limit, object-store error, or topology assertion failure
D12-UC-02	OTLP receiver through collector and Tempo distributor	SDKs, Alloy/OTel Collector, sampling processor, distributor, Kafka, block-builders, metastore/backend workers	Kafka for accepted trace records; object storage for durable blocks; versioned policy store for sampling	Refused/dropped spans, exporter failure, Kafka append error, consumer lag, block-build failure, or tenant denial

If spans, sampling decisions, and application records share an implied owner, deletion and access reviews become unsafe. Tempo stores observational copies; the application remains authoritative for the order and payment.

Generated-application database: Not created in this slice — Tempo’s trace blocks and platform evidence are durable telemetry state, not business-domain storage.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
Sampling-policy revision	Git/config store owned by observability operations	sampling_revision_sha	None — root policy revision has no parent	organization_id	Deterministic rules, bounded memory, redaction, and declared keep/drop priorities	Review, deploy, roll back, retain history, then expire by audit policy	D12-UC-02
Trace	Tempo live/object storage owned by tracing platform	trace_id	Opaque request/order references only; no application authority	tenant_id	Spans sharing trace ID must preserve tenant; late spans may extend a trace	Ingest, retain, compact, apply tenant retention, then tombstone/delete	D12-UC-01, D12-UC-02
Span	Tempo trace block owned by tracing platform	trace_id/span_id	parent_span_id within the same trace or None — root span has no parent	tenant_id	Span ID unique within trace; resource and status attributes follow schema policy	Append to trace, retain with block, delete with trace retention	D12-UC-01
Trace proof run	Platform evidence store owned by operations	trace_run_id	Opaque references to policy revision, trace IDs, query, and topology digest	organization_id	Positive, broken-propagation, corrected, and denial observations are immutable	Create per test, retain for audit objective, then expire	D12-UC-01, D12-UC-02

Independent timestamps do not establish causality, so services must propagate shared trace context across every supported transport. A trace is a set of spans under one trace ID; each non-root span names its parent span, while resource attributes identify the emitting service.

W3C Trace Context commonly travels in traceparent. The receiving service extracts it and creates a child span. Test HTTP, messaging, retry, and asynchronous boundaries separately; do not infer propagation from one synchronous route.

Minimum useful fields include trace and span IDs, parent span ID, service name, operation, start/duration, status, attributes, events, and links. Never put secrets or unnecessary personal data in attributes.

Searching only by trace ID forces responders to obtain that identifier elsewhere, so Tempo also supports TraceQL selection over trace and span structure. In current Tempo 3.0 microservices mode, distributors write trace-ID-sharded records to Kafka; live-stores serve recent queries, block-builders create object-store blocks, and queriers combine recent and retained results.

{ resource.service.name = "checkout" && status = error && trace:duration > 2s }

Prefer scoped attributes and conjunctive filters that can use Parquet predicate pushdown. Bound the time range and result limit, then confirm the returned trace’s actual span tree; a matching attribute does not guarantee complete propagation.

Keeping every trace can exceed budget, while naive head sampling can discard the rare failure discovered only at the end of a request. Head sampling decides early and cheaply; tail sampling buffers enough spans to decide from completed-trace properties such as error, latency, or a deployment attribute.

Tail sampling requires state and trace-aware routing so all spans for one trace reach the same decision point. Test the policy with fixed traffic:

1. Send 100 ordinary successful requests, 10 slow requests, and 5 errors with known IDs.
2. Record expected and observed keep rates by class.
3. Verify every retained trace has the expected service path.
4. Introduce one missing traceparent edge and prove the topology assertion fails.
5. Correct propagation and repeat with the same tenant and environment.

This falsifies two dangerous claims: that sampling preserved important traces and that instrumentation produced connected causality.

Trace storage is valuable only when instrumentation and propagation preserve the request graph. Keep these conclusions:

• Trace context, not timestamp proximity, connects distributed work.
• Tempo stores and searches traces; it does not own application business records.
• Tempo 3.0 microservices mode uses Kafka-compatible durable ingest, live-stores, block-builders, and object storage.
• TraceQL finds traces without a known ID, but results still require completeness checks.
• Sampling is an explicit cost and evidence policy that must be tested with rare failures.

A rendered waterfall can still omit the decisive service. Attach evidence before declaring tracing complete:

• [ ] Published propagation and attribute contracts for every transport.
• [ ] Proved one connected gateway-to-payment trace.
• [ ] Used TraceQL to find a seeded slow/error trace without its ID.
• [ ] Measured accepted, refused, dropped, late, and sampled spans.
• [ ] Falsified propagation by breaking and then repairing one edge.
• [ ] Proved tenant denial, retention, and deletion behaviour.

Tempo’s architecture changed materially in version 3.0, so primary documentation must govern deployment. These sources support the lesson:

• Grafana Tempo architecture
• Grafana Tempo 3.0 migration architecture
• Grafana TraceQL
• Grafana TraceQL query construction
• OpenTelemetry sampling
• W3C Trace Context