Tempo: Distributed Traces and Causal Evidence

Enterprise problem: Logs from five healthy-looking services cannot prove which dependency made one customer request slow, so responders lose causal order and may remediate the wrong component.

Whole-course context: Metrics now expose aggregate symptoms and Loki preserves bounded forensic events; today consumes a seeded checkout failure and adds propagated request context that joins the distributed call chain.

Today’s slice: Instrument, collect, sample, store, and query traces in Tempo while keeping trace retention, tenant isolation, and source application authority explicit.

End-of-day evidence: Produce one complete checkout trace with correct parent-child relationships plus a TraceQL result that finds a slow or failed request without knowing its trace ID first.

Still unsolved: Code-level resource attribution, Grafana navigation across signals, and the full platform operating architecture remain deferred.

Separate checkout service timestamps show sequence, but not causal parentage across network and queue boundaries. Losing trace context turns one request into unrelated fragments.

Thesis: a useful distributed trace preserves one trace identity and explicit parent–child relationships from request entry to downstream work. Why this matters: Tempo can only reconstruct causality that instrumentation actually propagates.

This smallest model contains one root span, one downstream child, and one trace view. It is enough to falsify propagation before adding storage architecture, search, or sampling policy.

Independent timestamps do not establish causality, so services must propagate shared trace context across every supported transport. A trace is a set of spans under one trace ID; each non-root span names its parent span, while resource attributes identify the emitting service.

W3C Trace Context commonly travels in traceparent. The receiving service extracts it and creates a child span. Test HTTP, messaging, retry, and asynchronous boundaries separately; do not infer propagation from one synchronous route.

Minimum useful fields include trace and span IDs, parent span ID, service name, operation, start/duration, status, attributes, events, and links. Never put secrets or unnecessary personal data in attributes.

Expand the same root–child–view model in dependency order: propagate context, export tenant-scoped spans, durably ingest trace-ID-sharded records, build retained blocks, then search and reconstruct the tree. Sampling is added only after the unsampled topology is proven.

The general rule is to prove topology before interpreting duration: first confirm that every expected boundary shares the trace and parent chain, then use span timing and status as evidence. A simple example sends one checkout request and verifies a root span with one payment child. A realistic recurring checkout incident mixes ordinary, slow, and failed requests so the telemetry pipeline must retain the important traces without inventing continuity across a missing edge.

Keeping every trace can exceed budget, while naive head sampling can discard the rare failure discovered only at the end of a request. Head sampling decides early and cheaply; tail sampling buffers enough spans to decide from completed-trace properties such as error, latency, or a deployment attribute.

Tail sampling requires state and trace-aware routing so all spans for one trace reach the same decision point. Test the policy with fixed traffic:

1. Send 100 ordinary successful requests, 10 slow requests, and 5 errors with known IDs.
2. Record expected and observed keep rates by class.
3. Verify every retained trace has the expected service path.
4. Introduce one missing traceparent edge and prove the topology assertion fails.
5. Correct propagation and repeat with the same tenant and environment.

This falsifies two dangerous claims: that sampling preserved important traces and that instrumentation produced connected causality. The observed evidence must include expected and retained IDs by traffic class, the actual service topology, the broken edge, collector refusal or lag, and the corrected rerun.

The most damaging failure mode is a broken propagation edge that still leaves individually valid spans, because the query can look populated while causality is false. The main trade-off is head sampling’s low state and early cost control versus tail sampling’s ability to retain errors and slow traces at the price of buffering, trace-aware routing, and delayed decisions.

prove complete propagation with fixed fixtures first; then choose the cheapest sampling method that meets explicit retention goals for errors, latency, and representative success traffic. Use head sampling when early attributes are sufficient and loss is acceptable. Use tail sampling when completed-trace properties drive the keep decision and the platform can bound memory and route all spans for a trace consistently. Avoid sampling policies whose keep rates and topology cannot be measured.

When a trace is missing, compare producer export, collector refusal, ingest lag, recent-store visibility, block completion, and query scope before increasing retention or replicas. Scale only the boundary whose measured saturation explains the evidence.

The tracing loop is Observe → Interpret → Decide → Act → Measure. Observe exporter success, refused spans, ingest lag, retained IDs, span topology, and query latency; interpret whether evidence was never created, dropped, delayed, or merely excluded by a query; decide on propagation, sampling, or capacity; act on one fixture; then measure the corrected topology and keep rates.

Bound the next action to one deliberately broken checkout traceparent edge followed by one correction. The falsifiable probe succeeds only if the first topology assertion fails at the expected boundary, the second produces the complete service path under one trace ID, important slow/error keep rates meet policy, and a cross-tenant lookup remains denied.

Trace storage is valuable only when instrumentation and propagation preserve the request graph. Keep these conclusions:

• Trace context, not timestamp proximity, connects distributed work.
• Tempo stores and searches traces; it does not own application business records.
• Tempo 3.0 microservices mode uses Kafka-compatible durable ingest, live-stores, block-builders, and object storage.
• TraceQL finds traces without a known ID, but results still require completeness checks.
• Sampling is an explicit cost and evidence policy that must be tested with rare failures.

A rendered waterfall can still omit the decisive service. Attach evidence before declaring tracing complete:

• [ ] Published propagation and attribute contracts for every transport.
• [ ] Proved one connected gateway-to-payment trace.
• [ ] Used TraceQL to find a seeded slow/error trace without its ID.
• [ ] Measured accepted, refused, dropped, late, and sampled spans.
• [ ] Falsified propagation by breaking and then repairing one edge.
• [ ] Proved tenant denial, retention, and deletion behaviour.

Tempo’s architecture changed materially in version 3.0, so primary documentation must govern deployment. These sources support the lesson:

• Grafana Tempo architecture
• Grafana Tempo 3.0 migration architecture
• Grafana TraceQL
• Grafana TraceQL query construction
• OpenTelemetry sampling
• W3C Trace Context