The Telemetry Schema Is an API
Source: Observability Platform source notes, section 20, “The Telemetry Schema Is an API”
The enterprise problem and today’s slice
Enterprise problem: An unreviewed field rename can silently break dashboards, alerts, service-level objectives (SLOs), automation, cost allocation, and security investigations even though the application deployment appears healthy.
Whole-course context: The incoming cost ledger identifies expensive dimensions and critical evidence; this day turns shared telemetry names, units, types, semantics, and deprecations into a versioned consumer contract.
Today’s slice: We govern service identity plus metric, log, and trace conventions, then prove compatible rollout and rejection paths.
End-of-day evidence: A schema revision, compatibility report, dual-read migration, consumer inventory, and rollback probe demonstrate that producers and consumers evolve safely.
Still unsolved: Data minimization, endpoint access, tenant isolation, and retention enforcement remain the next security-focused slice.
The smallest complete model
A telemetry field is production-facing interface state because independently owned dashboards, alerts, service-level objectives, and automation depend on it. A healthy application deployment can therefore break operations without returning a single application error.
Thesis: Treat telemetry evolution as producer contract → compatibility proof → consumer migration, with immutable revisions and evidence-driven removal. Why this matters: a rename, type change, or unit change is safe only when every affected consumer is known and the old interpretation remains recoverable.
The boundary includes canonical identity, field semantics, types, units, dimensions, schema revisions, consumer fixtures, deprecation, and rollback. It excludes data-access policy, which is the next layer after a compatible field exists.
Expand the model one boundary at a time
Start with the producer’s immutable meaning, add a registry and compatibility gate, then add independently versioned consumers and a measured deprecation window. Each stage expands the same contract-to-migration model rather than introducing a disconnected governance system.
Shared identity breaks correlation when each signal uses a different service or environment name. Define canonical resource identity once and state signal-specific constraints separately.
| Contract area | Required decisions |
|---|---|
| Shared identity | service.name, service.namespace, service.version, deployment.environment.name, region, cluster, owner, and tenant semantics |
| Metrics | Stable name and meaning, base unit, type, required and forbidden labels, histogram boundaries, owner, and deprecation policy |
| Logs | Timestamp, severity, event name, service identity, trace reference, error type, schema version, structured fields, and sensitive-data classification |
| Traces | Span naming, resource attributes, parent/context propagation, status/error semantics, business attributes, sampling, and sensitive-attribute classification |
The schema is an API because dashboards, alerts, capacity models, service-level objective calculations, incident automation, cost allocation, deployment analysis, and security tools depend on it. OpenTelemetry’s stable schema specification exists precisely because source and consumer assumptions otherwise change together; a schema URL can identify an immutable version and describe supported transformations. It does not replace an organization’s complete field and value contract.
Infrastructure can capture mechanics such as HTTP status or CPU time. Only application code can state that a checkout failed, a policy was rejected, a model response was unsafe, a tenant boundary was violated, or a payment was duplicated. A collector may validate, redact, enrich, or perform a documented lossless transform; it must not invent that missing meaning.
The expanded boundaries need interfaces and selection rules that distinguish producer responsibility from platform convenience.
| Boundary | Purpose, inputs, and transformation | Output or interface | Scaling constraint and failure mode | Alternatives, use when, and avoid when |
|---|---|---|---|---|
| Business operation → instrumentation | Convert domain events into named, typed, unit-bearing telemetry with canonical identity | Metrics, structured logs, spans, and schema reference | Release velocity and instrumentation overhead; missing domain meaning cannot be recovered downstream | Use producer code for business semantics; avoid asking collectors to infer checkout outcomes from generic HTTP fields |
| Instrumentation → schema gate | Compare proposed names, types, units, dimensions, and compatibility mode with an immutable revision | Publish or denial with affected consumer owners | Number of schemas and consumers; incomplete inventory permits silent breakage | Use contract checks before release; avoid mutable wiki conventions as the source of truth |
| Gate → compatibility transform | Dual-emit, dual-read, or apply a documented lossless transformation during migration | Old and new queryable forms with revision evidence | Migration duration and duplicate volume; lossy transform changes interpretation | Use transforms for representational compatibility; avoid them when semantics or units cannot be preserved exactly |
| Schema → consumers | Execute representative dashboard, alert, SLO, cost, and security queries | Compatibility verdict and last-seen usage | Consumer count and query fixture quality; unregistered consumers evade deprecation | Use a consumer registry and real query shapes; avoid removal based only on the announced deadline |
Run the model through one incident
The general rule is to reproduce producer and consumer assumptions from immutable fixtures before blaming runtime health. A simple example is a metric unit changing from seconds to milliseconds: both values exist, but an alert threshold becomes wrong unless the revision or a lossless transform preserves interpretation.
During a recurring checkout incident, a new producer renames checkout.result while the error dashboard, alert rule, cost job, and trace search still query the old field. The deployment is healthy, yet the observed checkout error rate drops implausibly to zero. The compatibility run should emit old and new fixtures, execute every registered query, deny the breaking revision with named owners, and prove the replacement field works as a positive control. If a dual-emission migration is selected, evidence includes producer build, schema revision, consumer versions, fixture hash, expected and observed query results, and the rollback probe.
This incident teaches a reusable distinction: data presence does not prove contract compatibility; only consumer-shaped evidence does.
Failure modes, trade-offs, and decision rules
The primary failure mode is mutating a shared name, type, unit, or meaning in place and discovering dependants only after an incident. The central trade-off is migration safety versus temporary dual-write, dual-read, registry, and fixture maintenance cost.
publish an additive immutable revision by default; use a lossless transform only when old and new meanings are equivalent; remove the old field only after every registered consumer passes the replacement and observed usage is zero for the published window.
Happy-path fixtures miss silent consumer breakage, so test old/new producers against real query shapes and a deliberately incompatible change.
- Inventory dashboards, alerts, SLO rules, cost jobs, security queries, and automation with owners.
- Freeze old and new producer fixtures plus consumer query fixtures.
- Validate identity, field presence, type, unit, histogram boundaries, and cardinality policy.
- Run consumers against old, dual-emitted or transformed, and new telemetry.
- Attempt a breaking rename without migration; require the gate to deny it and name every affected consumer.
- Prove an unaffected field and replacement query still work.
- Remove the old field only after usage is zero for the published window; falsify readiness if any live consumer or unexplained query remains.
Use dual emission when rollback speed is critical and temporary volume is affordable. Use dual read when consumer migration can be centralized. Avoid transforms that invent business semantics, and avoid permanent compatibility layers whose owner and removal date are unknown.
Close the loop
Schema governance succeeds when an observed incompatibility becomes a safe migration and a measured removal. Apply Observe → Interpret → Decide → Act → Measure at the producer-consumer boundary.
Observe the checkout dashboard reporting zero errors after the producer revision while a raw new-field query shows failures. Interpret the mismatch as a consumer contract break rather than customer recovery. Decide on a bounded rollback or dual-emission migration. Act by restoring the last published revision or enabling the documented compatibility path. Measure every registered consumer against the same fixture and require old-field usage to reach zero before removal. If any consumer still fails or usage reappears, the removal hypothesis is falsified and the old contract remains active.
Key takeaways
Telemetry is production-facing interface state even when customers never see its field names directly.
- Shared identity makes metrics, logs, traces, profiles, deployments, and owners correlatable.
- Producers own business meaning; collectors cannot invent semantics.
- Published revisions are immutable and deprecations are evidence-driven.
- Compatibility requires consumer fixtures, negative tests, a migration window, and rollback.
Checklist
A schema change is ready only when both producers and consumers can prove their expectations.
- [ ] Every field has meaning, type, unit, owner, and cardinality or sensitivity policy.
- [ ] Canonical service identity is consistent across signals.
- [ ] Consumer inventory includes dashboards, alerts, SLOs, cost, security, and automation.
- [ ] A deliberately breaking fixture is denied with affected owners.
- [ ] Deprecation has replacement, deadline, zero-use evidence, and rollback.
Sources
Schema mechanics and naming conventions evolve, so current statements were checked against official specifications and documentation.