Telemetry Signals as Complementary Evidence

Enterprise problem: Operators cannot answer detection, event, causality, and code-cost questions from one data model, so forcing every investigation through a single signal creates blind spots or uncontrolled cost.

Whole-course context: The control-loop map from Day 01 is the incoming artifact; today expands its telemetry-system box into distinct but correlatable evidence types.

Today’s slice: Define metrics, logs, traces, and profiles, assign each an initial operational question, and establish shared resource and request identity.

End-of-day evidence: A signal-selection matrix and a correlated incident packet that moves from an elevated metric to a log event, trace span, and profile observation.

Still unsolved: Instrumentation pipelines, collector placement, storage backends, retention economics, sampling policy, and alert delivery remain deferred.

When a team chooses a signal by habit, it either loses the needed detail or pays to preserve detail that no decision consumes. These use cases require intentional signal choice and cross-signal correlation.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D02-UC-01	On-call engineer	Move from a checkout symptom to the request and dependency that caused it	A metric exemplar or shared attributes lead to a trace and structured log for the same failure	A correlation gap is recorded with the missing identity field and owning team
D02-UC-02	Performance engineer	Attribute sustained CPU growth to code paths without adding request IDs to metrics	A profile identifies the costly functions for the affected service and release	A control profile or unchanged workload disproves the suspected code path

Signal names alone do not establish usefulness, so each story describes the investigation transition that must be observable.

Story ID	Use case IDs	User story	Observable acceptance conditions
D02-US-01	D02-UC-01	As an on-call engineer, I want aggregated symptoms linked to request-level evidence, so that I can locate a failing dependency without putting unique IDs in metric labels	The incident packet contains metric window, trace ID, slow span, log event, service, environment, and release
D02-US-02	D02-UC-02	As a performance engineer, I want profiles correlated by service and release, so that I can distinguish expensive code from traffic growth	Before-and-after profiles use comparable load and show the function-level resource change

Cross-signal investigation fails when every handoff requires a new unbounded search. These flows preserve bounded aggregate dimensions while carrying request and code identity in the signals designed for them.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D02-FLOW-01	D02-UC-01	Happy	Operator opens a checkout p99 latency alert	1. Inspect the metric by service, region, and release.<br>2. Follow an exemplar or time-window correlation to a trace.<br>3. Find the dominant span.<br>4. Open logs using trace identity.<br>5. Record the failing dependency.	Correlated packet with alert, query, trace ID, span ID, log event, actor, environment, timestamp, and immutable incident ID
D02-FLOW-02	D02-UC-02	Recovery	CPU remains elevated after a release rollback	1. Compare metric traffic and CPU windows.<br>2. Capture profiles under matched load.<br>3. Attribute samples to functions.<br>4. Compare against a control release.<br>5. Reject or retain the code-regression hypothesis.	Profile artifact IDs, matched-load assumptions, observed function shares, and explicit falsification result

If correlation is implemented as a global full-text search, investigations become slow and access controls become ambiguous. The design uses shared resource attributes and explicit request context while each backend retains authority over its own signal.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D02-UC-01	Alert detail and explore links	Metrics query service, trace query service, log query service, correlation resolver	Signal indexes in their owning backends plus evidence packet store	Missing exemplar, trace context, resource identity, or permission-safe link
D02-UC-02	Performance comparison workspace	Metrics query service, profiling backend, release catalogue	Profile store owned by performance platform	Incomparable load windows, absent release identity, or no control profile

Correlation becomes unsafe when a convenience index copies sensitive payloads across stores. This model persists references and shared identifiers while leaving raw signal data with its signal backend.

Generated-application database: Not created in this slice — observability control records and opaque signal references are sufficient; application domain data remains in its source system.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
SignalPolicy	Telemetry policy store, observability platform	signal_policy_id	Opaque service catalogue reference	organization_id	One active policy revision per service and environment	Versioned; old revisions retained for audit; deleted with service retirement policy	D02-UC-01, D02-UC-02
CorrelationPacket	Evidence packet store, incident management	packet_id	Opaque metric, trace, log, profile, and incident references	organization_id	References must share authorized tenant and time scope	Created during investigation; retained with incident; expires by incident policy	D02-UC-01
ProfileComparison	Performance evidence store, performance platform	comparison_id	Opaque release, profile, and metric-window references	organization_id	Baseline and candidate must record comparable workload assumptions	Retained through performance review; source profiles expire independently	D02-UC-02

Choosing the wrong signal either discards essential detail or creates explosive storage and query cost. Treat each signal as a projection optimized for a different first question.

Signal	Representation	Best initial question	Detail deliberately traded
Metrics	Aggregated numerical measurements over time	Is something wrong?	Individual event history
Logs	Individual structured events	What happened?	Complete distributed causality
Traces	A request path composed of spans	Where did it happen?	Unsampled aggregate population shape
Profiles	Resource samples attributed to code	Which code consumed resources?	Full request and business-event detail

A metric can show checkout p99 latency at 2.4 seconds and errors at 4.8 percent. A log can preserve the order identifier, provider response, and timeout reason. A trace can show that 2.01 seconds of a 2.41-second request occurred in the payment provider. A profile can show that retry policy and regular-expression validation dominate CPU. None invalidates the others.

Individual events are too numerous for every overview, so metrics aggregate observations into bounded time series. Counters, gauges, and distributions make rates, saturation, error ratios, and latency trends cheap to scan across a fleet.

Keep dimensions bounded: service, environment, region, operation, and status class are often useful. Unique customer, request, order, session, or trace identifiers do not belong in metric labels because every distinct label set becomes a new series. Preserve those identities in logs or traces instead.

Aggregates cannot explain a particular state transition, so structured logs preserve event detail with explicit fields. A useful event has timestamp, severity, service, environment, release, stable event name, and correlation context; free text remains a human explanation rather than the only schema.

{
  "timestamp": "2026-07-30T10:42:18.131Z",
  "severity": "ERROR",
  "service.name": "checkout",
  "deployment.environment.name": "production",
  "service.version": "2026.07.30.3",
  "trace_id": "5df6c1e47a724f62",
  "event.name": "payment.authorization.failed",
  "provider": "payment-gateway",
  "reason": "upstream_timeout"
}

Avoid secrets and unnecessary personal data. Redaction should happen before export where possible, and access to business identifiers must follow the application’s tenant boundary.

A slow request crossing several services cannot be reconstructed from timestamps alone, so traces represent the operation as spans connected by parent-child context. W3C Trace Context commonly carries trace and parent identifiers across process boundaries; losing propagation creates a new trace and breaks the causal tree.

Sampling changes what can be concluded. A sampled trace can explain one captured operation; it does not by itself prove fleet-wide frequency. Pair it with metrics for population-level impact.

CPU and memory metrics identify pressure but not the responsible functions, so profiles sample stacks and attribute resource consumption to code. Continuous profiling makes regressions visible across releases, while trace or resource correlation narrows the profile to the affected service and window.

Comparisons require controlled assumptions: similar request mix, traffic, duration, instance shape, and release identity. Without those controls, a larger function share may reflect a different workload rather than worse code.

Cross-signal links often look complete until one service loses context, so run a known request through at least two services and inspect each backend independently.

1. Confirm the metric changes using only bounded labels.
2. Locate a captured trace and verify one trace ID spans both services.
3. Query logs by that trace ID and verify service and release attributes agree.
4. Open a profile for the same service, release, and time window.
5. Repeat with an unauthorized tenant and confirm no cross-tenant signal is returned.

Record absence as evidence. “No matching log” should distinguish no event, ingestion delay, retention expiry, access denial, and missing correlation fields.

Signal quality comes from using each representation for the question it can answer and preserving safe links between them.

• Metrics detect fleet-level change efficiently.
• Logs preserve detailed events and state transitions.
• Traces reconstruct causality across component boundaries.
• Profiles attribute resource consumption to code.
• Shared resource identity enables correlation; unique IDs stay out of metric labels.
• A captured example explains an instance, while aggregates establish prevalence.

Use this checklist before calling a service multi-signal observable.

• [ ] Assigned an initial operational question to each collected signal.
• [ ] Standardized service, environment, and release resource attributes.
• [ ] Verified trace context crosses every synchronous and asynchronous boundary.
• [ ] Kept unique request and customer identifiers out of metric labels.
• [ ] Tested positive correlation and cross-tenant denial.
• [ ] Documented sampling, retention, and profile-comparison assumptions.

These primary references describe current OpenTelemetry signal scope and correlation semantics.

• OpenTelemetry signals
• W3C Trace Context
• OpenTelemetry profiles specification
• Prometheus metric and label naming