Observability as an Operational Control Loop

Enterprise problem: Operators must restore customer workflows when behaviour changes, but disconnected dashboards, alerts, and logs hide the relationship between evidence and action, extending impact and recovery time.

Whole-course context: This opening day establishes the control-loop contract that every later signal, collector, backend, and notification path must serve.

Today’s slice: Define the running-system, telemetry-system, and decision-system boundaries, then connect them with an evidence path owned by the platform team.

End-of-day evidence: A reviewed control-loop map plus a drill record showing that a synthetic checkout failure produces attributable evidence and a named operator action.

Still unsolved: Signal selection, collection topology, Prometheus mechanics, alert routing, cardinality control, and distributed retention remain deliberately deferred.

If the platform begins with tools instead of operator jobs, teams collect data that cannot shorten an incident. These use cases make customer-visible detection and evidence-based recovery the design constraints.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D01-UC-01	Service operator	Detect and diagnose a customer-facing checkout regression	The operator links the symptom to a changed service and selects a bounded response	A drill shows whether missing identity or telemetry prevents attribution
D01-UC-02	Incident commander	Verify that corrective action restored the customer workflow	A recovery probe passes and the incident timeline records the evidence used	A failed probe keeps the incident open and identifies the unresolved dependency

Vague goals such as “improve visibility” cannot be accepted or rejected, so the stories name observable outcomes at the point where people make decisions.

Story ID	Use case IDs	User story	Observable acceptance conditions
D01-US-01	D01-UC-01	As a service operator, I want a symptom to lead to service-owned evidence, so that I can choose an action without searching unrelated dashboards	A synthetic failure is detected, attributed to checkout, and linked to a deployment or dependency within the response objective
D01-US-02	D01-UC-02	As an incident commander, I want recovery to be proved by the affected workflow, so that an incident is not closed on infrastructure health alone	The checkout probe changes from failed to passing and its immutable run ID is attached to the timeline

A control loop is broken if it ends at a chart or notification, because neither artifact proves that a customer outcome improved. The flows therefore begin and end at a visible workflow.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D01-FLOW-01	D01-UC-01, D01-UC-02	Happy	Operator starts an approved checkout failure drill	1. Inject a bounded failure.<br>2. Observe the checkout symptom.<br>3. Correlate telemetry with the changed component.<br>4. Apply the runbook action.<br>5. Re-run the checkout probe.	Drill ID, actor, service, environment, timestamps, alert, correlated evidence, action, and passing recovery probe
D01-FLOW-02	D01-UC-01, D01-UC-02	Failure	The same drill produces an alert without attributable evidence	1. Record the visible failure.<br>2. Attempt service and release attribution.<br>3. Mark the evidence gap.<br>4. Stop automatic remediation.<br>5. preserve the failed probe and gap owner.	Failed probe, missing-field list, unaffected control probe, owner, and immutable drill ID

Without explicit boundaries, observability quietly becomes a shared-data dumping ground and nobody owns a broken link. This design assigns evidence production, interpretation, and response to separate components while preserving a single operational loop.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D01-UC-01	Failure-drill console and customer probe	Drill controller, checkout service instrumentation, telemetry gateway, correlation API	Evidence ledger owned by observability platform	Probe failure with absent service, environment, release, or trace correlation
D01-UC-02	Incident timeline recovery action	Incident service, runbook executor, workflow probe service	Incident timeline store owned by incident management	Action recorded without a subsequent passing customer probe

If evidence has no durable identity, incident conclusions cannot be reviewed and the same failure will be debated again. The model stores operational evidence and decisions without copying application business data.

Generated-application database: Not created in this slice — the platform needs durable evidence and incident-control records, while checkout business records remain owned by the application.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
DrillRun	Evidence ledger, observability platform	drill_run_id	Opaque service and deployment references	organization_id	Environment and failure scope are immutable after start	Created at drill start; retained for audit; expired by evidence policy	D01-UC-01
EvidenceItem	Evidence ledger, observability platform	evidence_id	drill_run_id local FK	organization_id	Source, timestamp, service identity, and immutable artifact URI are required	Appended during drill; retained with parent; deleted on parent expiry	D01-UC-01, D01-UC-02
IncidentAction	Incident timeline store, incident management	action_id	Opaque drill_run_id and runbook revision	organization_id	Actor, reason, expected effect, and observed result cannot be edited in place	Appended to timeline; corrections are new records; retained by incident policy	D01-UC-02
WorkflowProbe	Probe-result store, reliability engineering	probe_run_id	Opaque incident and drill references	organization_id	Result includes workflow, environment, expected state, and observed state	Retained through incident review; then aggregated and expired	D01-UC-01, D01-UC-02

Tool inventories obscure why observability exists, so start with three responsibilities rather than product names. The running system performs customer work, the telemetry system preserves evidence of behaviour, and the decision system turns that evidence into a human or automated response.

The distinction exposes failure modes. A dashboard without an owner or action is decoration. An alert without diagnostic context is interruption. Logs without stable service identity are text storage. Metrics with uncontrolled dimensions are an expensive collection of numbers.

Telemetry that only describes a failure does not restore service, so the platform must connect a change to its observed effect and then to a verified response. The loop is complete only when the same customer-facing measurement proves recovery.

A practical review asks six questions:

1. What customer workflow can fail?
2. Which system change or dependency can alter it?
3. What evidence appears before and during impact?
4. Who or what receives that evidence?
5. Which bounded action is available?
6. Which probe proves the action worked?

This is also why “collect everything” is not a strategy. Collection consumes CPU, network, storage, and human attention. Evidence earns that cost when it supports a decision; otherwise it needs a named future use, reduced retention, aggregation, or removal.

A diagram can conceal missing ownership, so validate the model with a small failure drill. Choose one reversible condition, such as a controlled latency injection in staging, and record the expected customer symptom before running it.

Use this evidence envelope for both the negative and recovery paths:

Field	Required observation
Actor and scope	Who initiated the drill; service, workflow, and environment affected
Precondition	Passing checkout probe and known release identifier
Expected result	Alert plus attributable telemetry within the response objective
Observed result	Exact symptom, evidence links, decision, and action
Positive control	An unrelated workflow remains healthy
Immutable identity	Drill, evidence, incident, and probe run IDs

The drill fails usefully when it finds a gap. A missing service label, a dashboard that cannot identify the release, or a recovery action without a workflow probe is concrete platform backlog, not an invitation to add another generic dashboard.

The platform succeeds when evidence changes an operational decision and that decision is verified against the customer workflow.

• Observability is a control loop, not a catalogue of telemetry products.
• Running, telemetry, and decision systems have separate responsibilities and owners.
• Detection, diagnosis, action, and recovery proof must remain connected.
• Evidence without a decision path should be justified, reduced, or removed.
• Controlled failure drills reveal broken links more reliably than dashboard reviews.

Use this checklist to determine whether the first platform boundary is operational rather than merely documented.

• [ ] Named one customer workflow and its accountable operator.
• [ ] Mapped running-system, telemetry-system, and decision-system ownership.
• [ ] Identified a bounded response action and approval boundary.
• [ ] Defined a customer-facing recovery probe.
• [ ] Recorded positive, negative, and recovery evidence with immutable IDs.
• [ ] Turned every discovered evidence gap into an owned platform requirement.

These primary references define observability signals and the platform concepts used in this day.

• OpenTelemetry observability primer
• OpenTelemetry signals