Cross-Signal Correlation: One Investigation, Four Databases

Enterprise problem: Four healthy telemetry backends still force responders to copy identifiers, guess time windows, and reconcile checkout, checkout-api, and commerce-checkout, which lengthens incidents and creates false joins.

Whole-course context: The incoming artifact is a reproducible fixed-time Grafana investigation across Mimir, Loki, Tempo, and Pyroscope; today standardizes the identity and links that preserve context between those views.

Today’s slice: Define canonical resource attributes, exemplars, trace/log/profile links, deployment context, and correlation tests without merging backend authorization or ownership.

End-of-day evidence: Produce one metric-to-trace-to-log-to-profile-to-deployment investigation whose links preserve tenant, service, environment, and time, plus a deliberately mismatched control that fails safely.

Still unsolved: Final capacity, availability, alerting, governance, disaster-recovery, and retirement decisions remain deferred to the complete architecture.

Correlation that merely opens another page can silently select the wrong service or tenant. These use cases require context-preserving navigation and governed schema rollout.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D15-UC-01	Incident responder	Move from a checkout latency symptom to causal and code evidence without manual re-entry	Metric exemplar opens the correct trace, related logs, profile window, and deployment revision	Missing/mismatched identity produces an explicit no-match or warning rather than a cross-tenant or wrong-service result
D15-UC-02	Observability schema owner	Keep service and deployment identity consistent across all signals and producers	Conformance tests pass for canonical attributes and approved mappings	A producer with env=prod or conflicting service identity is quarantined/rejected while conforming telemetry remains queryable

Shared field names do not guarantee shared meaning, so acceptance must validate actual link inputs and outputs. The stories cover both responder navigation and schema governance.

Story ID	Use case IDs	User story	Observable acceptance conditions
D15-US-01	D15-UC-01	As an incident responder, I want a latency exemplar to carry me through trace, log, profile, and deployment evidence, so that I can test one causal hypothesis without losing context	Every hop preserves tenant, service.name, environment, absolute time, and trace/deployment identifiers; seeded controls do not join
D15-US-02	D15-UC-02	As a schema owner, I want producers validated against canonical OpenTelemetry resource attributes, so that differently named records cannot impersonate the same service	CI or collector policy rejects an incompatible fixture, accepts the canonical fixture, versions aliases, and records rollout evidence

A correlation is a query transformation across ownership boundaries, so each hop must carry only the identifiers needed by the target and re-evaluate authorization there. The flow ends in a reproducible operational action, not merely an open tab.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D15-FLOW-01	D15-UC-01, D15-UC-02	Happy	Responder clicks an exemplar on checkout p99 latency	1. Grafana opens the Tempo trace for the exemplar trace ID.<br>2. Trace attributes identify payment as the slow service.<br>3. Derived Loki query uses canonical service/environment and trace ID.<br>4. Trace-to-profile link opens the same service and span window.<br>5. Deployment catalog resolves service.version to a reviewed change.<br>6. Responder records action and evidence.	Source/destination queries, attributes, trace ID, deployment revision, absolute window, actor, environment, and investigation ID
D15-FLOW-02	D15-UC-01, D15-UC-02	Denied	A test producer emits conflicting service/environment attributes	1. Conformance policy detects the conflict.<br>2. Telemetry is rejected, quarantined, or marked non-correlatable.<br>3. The correlation link shows explicit no-match.<br>4. Canonical control telemetry still completes the chain.<br>5. Cross-tenant access remains denied.	Policy reason, rejected fixture, no-match result, successful control chain, tenant denial, and run ID

Hard-coded links scattered through dashboards drift whenever a label changes. Central schema policy, versioned correlation definitions, and conformance fixtures make transformations reviewable and independently revocable.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D15-UC-01	Grafana exemplar, data link, or correlation action	Grafana correlations/data links, Mimir, Tempo, Loki, Pyroscope, deployment catalog, backend authorization	Each signal backend owns its records; deployment catalog owns release metadata; Grafana owns link definitions	Missing exemplar, wrong mapped field, empty target query, broadened time, permission denial, or mismatched service/tenant assertion
D15-UC-02	Instrumentation CI and collector policy	OTel SDK/resource config, schema registry, collector transform/validation, conformance runner	Versioned semantic contract and mapping registry owned by observability governance	Unknown alias, conflicting attributes, cardinality-policy failure, rejected telemetry, or control-fixture regression

An alias table can aid navigation without becoming an authorization bridge. The target backend must still evaluate the user or service credential independently, and opaque references never transfer authority.

Generated-application database: Not created in this slice — the schema registry, correlation definitions, deployment catalog, and proof records are platform control/evidence state rather than application-domain data.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
Identity-contract revision	Schema registry owned by observability governance	contract_version	None — root semantic contract has no parent	organization_id	Stable canonical meanings, compatibility rule, cardinality class, and sensitivity classification	Propose, approve, deprecate, migrate, retain history, then expire	D15-UC-02
Producer conformance result	Evidence store owned by schema governance	conformance_run_id	Foreign reference to contract version and opaque producer revision	organization_id	Positive and conflicting fixtures are immutable; environment and timestamp required	Create in CI/deploy, retain for audit objective, expire with revision policy	D15-UC-02
Correlation definition	Grafana provisioning store owned by admins	correlation_uid/revision	Opaque source/target data-source UIDs and contract version	organization_id	Explicit field mapping, bounded time transform, target authorization, independently revocable	Review, provision, roll back, disable, export, delete	D15-UC-01, D15-UC-02
Investigation chain proof	Platform evidence store owned by operations	correlation_run_id	Opaque references to source result, target results, trace ID, and deployment revision	organization_id	Every hop records input/output context, denial, and control result	Create per exercise, retain for incident/audit objective, then expire	D15-UC-01, D15-UC-02

Similar strings are not reliable joins, so choose canonical resource attributes and define their scope. OpenTelemetry semantic conventions provide shared names across traces, metrics, logs, profiles, and resources.

Attribute	Meaning and constraint
service.name	Stable logical service name; identical for horizontally scaled instances
service.namespace	Stable grouping that makes service names unambiguous
service.instance.id	Unique live instance identity; usually metadata, not a Loki index label
deployment.environment.name	Stable environment value such as production or staging
service.version	Deployed code/artifact version used to resolve change context
k8s.cluster.name	Cluster identity where relevant to telemetry and access policy

Normalize at the producer or collector boundary, publish aliases only for migrations, and reject simultaneous conflicting canonical/legacy values. Do not use deployment.environment.name as part of service uniqueness; the OTel service identity contract is namespace, name, and instance.

Copying a trace ID manually is slow and error-prone, so use explicit mechanisms: metric exemplars to traces, derived fields or correlations from logs to traces, trace-to-logs queries, trace-to-profile queries, and deployment links from service.version.

Exemplars attach representative trace context to metric observations; they do not imply every metric sample has a trace or that the selected trace explains the aggregate. Profiles are aggregate windows, not request-specific proof. Each hop supplies evidence for or against a hypothesis.

Happy-path navigation can hide overly broad joins. Seed two tenants with the same service.name, two environments, and one trace ID present only in the authorized tenant.

1. Click a production checkout exemplar and record the target Tempo query.
2. Open related payment logs and verify trace ID, tenant, service, and absolute time.
3. Open a profile narrowed to service and span window.
4. Resolve service.version to the deployment catalog.
5. Change one fixture to env=prod and verify explicit no-match or schema rejection.
6. Attempt the other tenant and verify denial without leaking whether its trace exists.

Correlation is governed identity plus query transformation, not physical database unification. Keep these rules:

• Canonical resource semantics matter more than superficially similar label values.
• Every target backend re-evaluates authorization; links never grant access.
• Exemplars, logs, traces, profiles, and deployments provide different evidence strengths.
• Absolute time, tenant, service, environment, and version must survive each hop.
• A mismatched fixture and cross-tenant denial are required falsifiers.

Correlation is unsafe until wrong matches fail visibly. Complete these checks:

• [ ] Published versioned canonical attributes, aliases, sensitivity, and cardinality classes.
• [ ] Added producer conformance fixtures for positive and conflicting inputs.
• [ ] Proved metric-to-trace-to-log-to-profile-to-deployment navigation.
• [ ] Recorded exact source and target queries plus absolute time at every hop.
• [ ] Proved mismatched environment no-match and cross-tenant denial.
• [ ] Verified correlation definitions are versioned, revocable, and do not contain credentials.

Attribute stability and Grafana link behaviour evolve, so primary specifications and documentation govern implementation. These sources support the lesson:

• OpenTelemetry semantic conventions
• OpenTelemetry service resource conventions
• OpenTelemetry deployment attributes
• Grafana correlations
• Grafana exemplars
• Grafana traces to profiles