Cross-Signal Correlation: One Investigation, Four Databases
Source: Observability Platform source — section 15, “Correlation: Turning Four Databases into One Investigation”
The enterprise problem and today’s slice
Enterprise problem: Four healthy telemetry backends still force responders to copy identifiers, guess time windows, and reconcile checkout, checkout-api, and commerce-checkout, which lengthens incidents and creates false joins.
Whole-course context: The incoming artifact is a reproducible fixed-time Grafana investigation across Mimir, Loki, Tempo, and Pyroscope; today standardizes the identity and links that preserve context between those views.
Today’s slice: Define canonical resource attributes, exemplars, trace/log/profile links, deployment context, and correlation tests without merging backend authorization or ownership.
End-of-day evidence: Produce one metric-to-trace-to-log-to-profile-to-deployment investigation whose links preserve tenant, service, environment, and time, plus a deliberately mismatched control that fails safely.
Still unsolved: Final capacity, availability, alerting, governance, disaster-recovery, and retirement decisions remain deferred to the complete architecture.
The smallest complete model
Two telemetry records that both say “checkout” are not safely related unless their service, environment, tenant, and time meanings agree. A convenient link can otherwise cross the wrong deployment or authorization boundary.
Thesis: cross-signal correlation is a governed query transformation over canonical identity, not a database join or access grant. Why this matters: each hop can preserve context, be independently authorized, and fail visibly when identity does not match.
The smallest model has one source result, one canonical mapping, and one independently authorized target query. It is complete enough to test a positive hop and a deliberate mismatch before chaining every signal.
Similar strings are not reliable joins, so choose canonical resource attributes and define their scope. OpenTelemetry semantic conventions provide shared names across traces, metrics, logs, profiles, and resources.
| Attribute | Meaning and constraint |
|---|---|
service.name | Stable logical service name; identical for horizontally scaled instances |
service.namespace | Stable grouping that makes service names unambiguous |
service.instance.id | Unique live instance identity; usually metadata, not a Loki index label |
deployment.environment.name | Stable environment value such as production or staging |
service.version | Deployed code/artifact version used to resolve change context |
k8s.cluster.name | Cluster identity where relevant to telemetry and access policy |
Normalize at the producer or collector boundary, publish aliases only for migrations, and reject simultaneous conflicting canonical/legacy values. Do not use deployment.environment.name as part of service uniqueness; the OpenTelemetry (OTel) service identity contract is namespace, name, and instance.
Expand the model one boundary at a time
Expand the same source–identity–target model in dependency order: standardize producer attributes, validate them at collection, preserve tenant and absolute time, transform the source fields into a bounded target query, then let the target backend authorize and execute it. Add another signal only after the preceding hop has a positive and negative fixture.
| Boundary | Purpose, inputs, and transformation | Output or interface | Scaling constraint and failure mode | Alternative guidance |
|---|---|---|---|---|
| Producer and collector schema | Normalize canonical resource attributes and reject conflicts | Versioned tenant-scoped identity on telemetry | Schema drift, missing attributes, or high-cardinality misuse | Use canonical OTel names; use aliases only for bounded migrations and avoid silent dual meanings |
| Source evidence | Expose a trace ID, service/version, or other governed field from a bounded result | Selected exemplar, log row, trace span, or profile window | Sparse exemplars or missing context | Use exact trace ID when available; use service/time evidence when the signal is aggregate |
| Correlation definition | Transform source fields, tenant, and absolute time into a target selector | Reviewable Grafana target query | Over-broad time, mismatched environment, or stale aliases | Use provisioned, versioned definitions; avoid manual copy/paste for recurring paths |
| Target backend | Re-authorize and execute the bounded selector | Result, explicit no-match, or denial | Query cost, backend unavailability, or permission rejection | Use direct backend policy as authority; never treat a link as an access grant |
Copying a trace ID manually is slow and error-prone, so use explicit mechanisms: metric exemplars to traces, derived fields or correlations from logs to traces, trace-to-logs queries, trace-to-profile queries, and deployment links from service.version.
Exemplars attach representative trace context to metric observations; they do not imply every metric sample has a trace or that the selected trace explains the aggregate. Profiles are aggregate windows, not request-specific proof. Each hop supplies evidence for or against a hypothesis.
Run the model through one incident
The general rule is to carry canonical service identity, tenant, environment, and absolute time into each hop, then re-authorize at the target. A simple example turns one metric exemplar into an exact Tempo trace query. A realistic recurring checkout incident continues through payment logs, the same-window profile, and service.version deployment context while the telemetry pipeline records each transformation and denial.
Happy-path navigation can hide overly broad joins. Seed two tenants with the same service.name, two environments, and one trace ID present only in the authorized tenant.
- Click a production checkout exemplar and record the target Tempo query.
- Open related payment logs and verify trace ID, tenant, service, and absolute time.
- Open a profile narrowed to service and span window.
- Resolve
service.versionto the deployment catalog. - Change one fixture to
env=prodand verify explicit no-match or schema rejection. - Attempt the other tenant and verify denial without leaking whether its trace exists.
Observed evidence must capture each source field, generated target query, tenant, service, environment, absolute window, result digest, explicit no-match, and denial. That chain distinguishes an absent signal from a broken mapping, a broad selector, or an authorization defect.
Failure modes, trade-offs, and decision rules
The dangerous failure mode is a broad but plausible match that crosses tenant, environment, service version, or time. The main trade-off is navigation convenience versus the stricter identity, query cost, and independent authorization required to keep every hop trustworthy.
add a correlation only when its source fields have canonical meanings, its target query is bounded, and both a positive fixture and a conflicting fixture produce the expected match/no-match; require target authorization on every execution. Use exemplars for representative metric-to-trace navigation and exact IDs for direct causality. Use service/time windows for aggregate profiles with an explicit evidence limit. Avoid aliases without expiry, links that hide generated queries, and any mapping that transfers credentials.
If a hop fails, inspect schema conformance, source fields, transformation, time normalization, target permissions, and target query cost in that order. Scale a backend only when observed query saturation—not identity mismatch or denial—explains the failure.
Close the loop
The correlation loop is Observe → Interpret → Decide → Act → Measure. Observe one bounded source result; interpret its canonical identity; decide which target can test the current hypothesis; act by running the authorized transformed query; then measure whether the returned evidence narrows or falsifies the hypothesis.
For the next bounded action, run the six-step checkout fixture with one conflicting environment and one foreign tenant. The falsifiable probe succeeds only if the authorized chain reaches trace, logs, profile, and deployment with exact recorded context, the environment conflict returns no match or schema rejection, and the foreign tenant receives a denial without result leakage.
Key takeaways
Correlation is governed identity plus query transformation, not physical database unification. Keep these rules:
- Canonical resource semantics matter more than superficially similar label values.
- Every target backend re-evaluates authorization; links never grant access.
- Exemplars, logs, traces, profiles, and deployments provide different evidence strengths.
- Absolute time, tenant, service, environment, and version must survive each hop.
- A mismatched fixture and cross-tenant denial are required falsifiers.
Checklist
Correlation is unsafe until wrong matches fail visibly. Complete these checks:
- [ ] Published versioned canonical attributes, aliases, sensitivity, and cardinality classes.
- [ ] Added producer conformance fixtures for positive and conflicting inputs.
- [ ] Proved metric-to-trace-to-log-to-profile-to-deployment navigation.
- [ ] Recorded exact source and target queries plus absolute time at every hop.
- [ ] Proved mismatched environment no-match and cross-tenant denial.
- [ ] Verified correlation definitions are versioned, revocable, and do not contain credentials.
Sources
Attribute stability and Grafana link behaviour evolve, so primary specifications and documentation govern implementation. These sources support the lesson: