Telemetry Signals as Complementary Evidence
Source: “Building an Observability Platform: From Prometheus to Mimir, Loki, Tempo, and Grafana” — Chapter 2, “The First Expansion: Telemetry Signals”
The enterprise problem and today’s slice
Enterprise problem: Operators cannot answer detection, event, causality, and code-cost questions from one data model, so forcing every investigation through a single signal creates blind spots or uncontrolled cost.
Whole-course context: The control-loop map from Day 01 is the incoming artifact; today expands its telemetry-system box into distinct but correlatable evidence types.
Today’s slice: Define metrics, logs, traces, and profiles, assign each an initial operational question, and establish shared resource and request identity.
End-of-day evidence: A signal-selection matrix and a correlated incident packet that moves from an elevated metric to a log event, trace span, and profile observation.
Still unsolved: Instrumentation pipelines, collector placement, storage backends, retention economics, sampling policy, and alert delivery remain deferred.
The smallest complete model
One telemetry signal cannot preserve both cheap fleet-wide summaries and every event, causal edge, and code sample. Forcing it to do so either removes the detail an operator needs or creates cost the platform cannot bound.
Thesis: Metrics, logs, traces, and profiles are complementary projections of one running system, joined by safe resource and request identity rather than collapsed into one universal schema. Why this matters: operators can start with the cheapest evidence that answers a question and cross a signal boundary only when the next question requires it.
The smallest model includes the system being observed, one representation chosen for the current question, and the answer that changes a decision. Backend internals and retention policy remain outside this boundary; the first task is to choose the right projection and preserve a safe way to correlate it.
Expand the model one boundary at a time
Expand the signal box in the order an investigation usually needs it: population symptom, individual event, distributed cause, then code-level cost. The representations complement each other because each deliberately trades away detail that another can preserve.
| Signal | Representation | Best initial question | Detail deliberately traded |
|---|---|---|---|
| Metrics | Aggregated numerical measurements over time | Is something wrong? | Individual event history |
| Logs | Individual structured events | What happened? | Complete distributed causality |
| Traces | A request path composed of spans | Where did it happen? | Unsampled aggregate population shape |
| Profiles | Resource samples attributed to code | Which code consumed resources? | Full request and business-event detail |
The relationship is not a mandatory query sequence. A known event may begin in logs, and sustained CPU may begin in profiles. Shared service.name, environment, release, time scope, and authorized trace context make those transitions bounded without copying every payload into a global index.
| Component | Purpose, input, and transformation | Output and interface | Scaling constraint and failure mode | Alternatives, use when, avoid when |
|---|---|---|---|---|
| Metrics | Aggregate repeated measurements into counters, gauges, and distributions | Labelled time series for rates, saturation, ratios, and latency trends | Every label set creates series cost; unique IDs cause cardinality growth | Use for fleet prevalence with bounded service, region, operation, and status dimensions; avoid customer, request, order, session, or trace IDs |
| Logs | Preserve individual state transitions as structured fields plus human text | Searchable events with timestamp, severity, resource identity, event name, and correlation context | Volume, indexing cost, sensitive data, and retention constrain detail; free text alone destroys schema | Use for a specific event and authorized business identity; avoid secrets, unnecessary personal data, and logs as a causal graph |
| Traces | Propagate context and connect spans into a request path | Trace and span interfaces that locate latency or failure across processes | Sampling limits population conclusions; broken propagation splits the causal tree | Use for distributed causality; avoid treating one sampled request as fleet-wide frequency |
| Profiles | Sample stacks and attribute CPU or memory consumption to functions | Comparable code-cost profiles scoped by service, release, and time | Workload mix, duration, instance shape, and sample rate affect comparison | Use under matched load to find expensive code; avoid claiming regression from unmatched profiles |
Structured logs make the event interface explicit rather than relying on prose alone.
{
"timestamp": "2026-07-30T10:42:18.131Z",
"severity": "ERROR",
"service.name": "checkout",
"deployment.environment.name": "production",
"service.version": "2026.07.30.3",
"trace_id": "5df6c1e47a724f62",
"event.name": "payment.authorization.failed",
"provider": "payment-gateway",
"reason": "upstream_timeout"
}
The World Wide Web Consortium (W3C) Trace Context standard carries trace and parent identifiers across process boundaries. A healthy propagation path produces a causal shape such as this one; losing context creates a new trace and breaks the relationship.
Run the model through one incident
The general rule is to begin with the representation that establishes the hypothesis, then move only to a signal that can answer the next unresolved question. In a simple example, a CPU metric proves sustained pressure, while a matched profile shows whether one function or increased traffic explains it.
For a recurring checkout incident, a metric shows p99 latency at 2.4 seconds and errors at 4.8 percent for one release. An exemplar or bounded time correlation leads to a trace in which 2.01 seconds of a 2.41-second request occur at the payment provider. Logs for that trace preserve the provider timeout event and authorized order reference. If CPU also rises, a profile can show whether retry policy and regular-expression validation dominate samples under matched load.
The observed evidence bounds the claim. Metrics establish prevalence but not the fate of a particular order. A captured trace explains one request but not the whole fleet. A log explains an event but does not reconstruct every dependency. A profile attributes sampled resource cost but does not prove customer impact. Together they support the action without pretending any one signal is complete.
Failure modes, trade-offs, and decision rules
Cross-signal correlation introduces both value and risk. The common failure mode is a broken or unsafe join: a service loses trace propagation, resource attributes disagree, metric labels contain unique identity, profile windows are incomparable, or a convenience index crosses a tenant boundary.
The main trade-off is summary cost versus diagnostic specificity. Aggregation is cheap to scan but discards events; detailed signals retain context but cost more to ingest, index, secure, and query. Sampling reduces trace or profile cost but also limits what absence means.
| Need | Use when | Avoid when |
|---|---|---|
| Add a metric dimension | It is bounded and supports a named aggregate question | It identifies an event, person, request, order, or session |
| Preserve a structured log | A specific state transition or authorized business reference matters | The data is secret, unnecessary, or intended to replace metrics at fleet scale |
| Follow or add a trace | The unresolved question concerns distributed ordering or latency | There is no propagated context or one sample is being used to claim frequency |
| Compare profiles | Service, release, request mix, traffic, duration, and instance shape are comparable | The workload changed enough to explain the observed function share |
choose the lowest-cost signal that can falsify the current hypothesis, and add a correlation boundary only when the next question needs detail that the current representation deliberately discarded.
Close the loop
Use Observe → Interpret → Decide → Act → Measure across signals without turning correlation into an unbounded search. Observe the checkout metric, interpret request and event evidence, decide whether the provider or release hypothesis survives, act on the smallest reversible boundary, and measure the original metric plus the same workflow probe.
Run one bounded correlation probe through two services. The test passes only if the metric changes using bounded labels, one trace ID crosses both services, logs agree on service and release, a profile can be scoped to the same release and window, and an unauthorized tenant receives no signal. Record “no matching log” as one of no event, ingestion delay, retention expiry, access denial, or missing identity; after the action, require checkout latency and completion to return within the stated objective.
Key takeaways
Signal quality comes from using each representation for the question it can answer and preserving safe links between them.
- Metrics detect fleet-level change efficiently.
- Logs preserve detailed events and state transitions.
- Traces reconstruct causality across component boundaries.
- Profiles attribute resource consumption to code.
- Shared resource identity enables correlation; unique IDs stay out of metric labels.
- A captured example explains an instance, while aggregates establish prevalence.
Checklist
Use this checklist before calling a service multi-signal observable.
- [ ] Assigned an initial operational question to each collected signal.
- [ ] Standardized service, environment, and release resource attributes.
- [ ] Verified trace context crosses every synchronous and asynchronous boundary.
- [ ] Kept unique request and customer identifiers out of metric labels.
- [ ] Tested positive correlation and cross-tenant denial.
- [ ] Documented sampling, retention, and profile-comparison assumptions.
Sources
These primary references describe current OpenTelemetry signal scope and correlation semantics.