Observability as an Operational Control Loop
Source: “Building an Observability Platform: From Prometheus to Mimir, Loki, Tempo, and Grafana” — Chapter 1, “Observability Is an Operational Control Loop”
The enterprise problem and today’s slice
Enterprise problem: Operators must restore customer workflows when behaviour changes, but disconnected dashboards, alerts, and logs hide the relationship between evidence and action, extending impact and recovery time.
Whole-course context: This opening day establishes the control-loop contract that every later signal, collector, backend, and notification path must serve.
Today’s slice: Define the running-system, telemetry-system, and decision-system boundaries, then connect them with an evidence path owned by the platform team.
End-of-day evidence: A reviewed control-loop map plus a drill record showing that a synthetic checkout failure produces attributable evidence and a named operator action.
Still unsolved: Signal selection, collection topology, Prometheus mechanics, alert routing, cardinality control, and distributed retention remain deliberately deferred.
The smallest complete model
Operators do not restore a service by collecting more data; they restore it by turning trustworthy evidence into a verified action. A telemetry path that ends at a chart leaves the customer problem unsolved.
Thesis: Observability is a control loop in which a running system produces evidence, a decision system chooses a bounded response, and the customer workflow proves the result. Why this matters: the model makes every signal and tool justify its place by the operational decision it improves.
The boundary is deliberately small. The running system includes applications and dependencies that perform customer work; evidence describes their behaviour; the response includes investigation, a human or automated decision, a bounded action, and a recovery measurement. Product selection, every possible signal, and storage internals remain outside this first model.
Expand the model one boundary at a time
Expand the evidence box only after its relationship to customer work is clear. The dependency order is runtime identity, collection and protection, durable evidence, interpretation, action, and finally a customer-facing probe.
Each component has a different responsibility; treating them as one generic “observability stack” hides broken interfaces.
| Component | Purpose and inputs | Transformation, output, and interface | Scaling constraint and failure mode | Alternatives, use when, avoid when |
|---|---|---|---|---|
| Running system | Performs checkout using requests, releases, databases, queues, and dependencies | Emits measurements and context through instrumentation interfaces | Load and dependency fan-out grow evidence volume; missing service or release identity prevents attribution | Use direct instrumentation for application semantics; use exporters for existing systems; avoid backend-specific response logic in the application |
| Telemetry path | Accepts runtime evidence and applies identity, protection, routing, and retention policy | Produces queryable metrics, logs, traces, or profiles plus stable artifact references | Central processing unit (CPU), network, storage, and cardinality constrain collection; overload may drop, delay, or corrupt evidence | Use bounded evidence that supports a named question; aggregate, shorten retention, or remove unused data; avoid “collect everything” |
| Decision system | Consumes symptoms and diagnostic evidence | Produces a hypothesis, accountable actor, bounded action, and expected effect through alert, incident, or runbook interfaces | Human attention and automation safety are scarce; an alert without context becomes interruption | Use humans for ambiguous or high-blast-radius choices and automation for rehearsed reversible actions; avoid actions without approval and rollback boundaries |
| Workflow probe | Replays the affected customer outcome before and after action | Produces an immutable pass or fail result linked to the drill or incident | Probe coverage and representativeness limit what recovery claims mean; an infrastructure-only probe can report false recovery | Use a checkout journey for checkout impact and an unrelated positive control; avoid closing on host health alone |
A practical review now asks: which workflow can fail, which change can alter it, what evidence appears, who receives it, which safe action exists, and which probe can falsify the recovery claim. Record the answer in one evidence envelope rather than scattering it across dashboards.
| Field | Required observation |
|---|---|
| Actor and scope | Who initiated the drill; service, workflow, and environment affected |
| Precondition | Passing checkout probe and known release identifier |
| Expected result | Alert plus attributable telemetry within the response objective |
| Observed result | Exact symptom, evidence links, decision, and action |
| Positive control | An unrelated workflow remains healthy |
| Immutable identity | Drill, evidence, incident, and probe run IDs |
Run the model through one incident
The general rule is to follow evidence in the same direction as the control loop: establish customer impact, attribute it to a bounded change or dependency, act, and remeasure the original outcome. A simple example is a staging latency injection: the checkout probe slows, telemetry identifies the injected boundary, removing the injection restores the probe, and the unchanged catalogue probe acts as a control.
In a recurring production checkout incident, suppose a release changes payment-provider retry behaviour. The workflow probe records rising completion time, a service-level symptom shows the affected release, and request evidence locates repeated provider timeouts. The operator disables only the new retry policy, records the expected effect, and reruns the same checkout probe. A passing host-health check is supporting evidence; the passing checkout journey is the recovery proof.
Observed evidence also constrains the conclusion. If the alert fires but the release identifier is absent, attribution is not established. If checkout recovers before the action, the action did not prove causality. If the unrelated control probe also fails, the incident may be broader than checkout. These outcomes keep the investigation honest instead of forcing every drill to “succeed.”
Failure modes, trade-offs, and decision rules
Every control loop spends resources and can mislead its operator, so design choices must name the constraint they protect. The recurring failure mode is a broken link that remains hidden behind a healthy component: evidence without identity, a decision without ownership, an action without a rollback, or recovery without a customer probe.
The central trade-off is evidence richness versus collection and attention cost. More detail can shorten diagnosis, but it increases CPU, network, storage, access-control surface, and search effort. Automation can shorten response, but it also increases the blast radius of a wrong interpretation.
| Choice | Use when | Avoid when |
|---|---|---|
| Add or retain evidence | It answers a named operational question and has an accountable consumer | No decision, retention need, or future experiment is named |
| Automate the response | The condition is specific, the action is reversible, and the recovery probe is reliable | Evidence is ambiguous, blast radius is broad, or rollback is untested |
| Ask a human to decide | Diagnosis needs context or competing risks must be weighed | The action is routine, time-critical, bounded, and already rehearsed |
| Close the incident | The original customer workflow and an unaffected control support the claimed recovery | Only dashboards, process health, or notification delivery have recovered |
keep a control-loop component only when you can name its input, accountable output, failure signal, bounded action, and customer-facing measurement; otherwise simplify it or add the missing ownership before scaling it.
Close the loop
The operating sequence is Observe → Interpret → Decide → Act → Measure. Observe the customer symptom and attributable evidence; interpret a falsifiable cause; decide within an approval boundary; act with a reversible change; measure the same workflow that exposed the problem.
Run one bounded staging drill: inject no more than a documented latency threshold into checkout for a fixed window, then remove it using the approved runbook. The probe is falsifiable: the drill passes only if the alert and evidence identify checkout and the release within the response objective, the unrelated control stays healthy, and the checkout probe returns to its precondition after the action. Any missing link becomes an owned platform requirement rather than another generic dashboard.
Key takeaways
The platform succeeds when evidence changes an operational decision and that decision is verified against the customer workflow.
- Observability is a control loop, not a catalogue of telemetry products.
- Running, telemetry, and decision systems have separate responsibilities and owners.
- Detection, diagnosis, action, and recovery proof must remain connected.
- Evidence without a decision path should be justified, reduced, or removed.
- Controlled failure drills reveal broken links more reliably than dashboard reviews.
Checklist
Use this checklist to determine whether the first platform boundary is operational rather than merely documented.
- [ ] Named one customer workflow and its accountable operator.
- [ ] Mapped running-system, telemetry-system, and decision-system ownership.
- [ ] Identified a bounded response action and approval boundary.
- [ ] Defined a customer-facing recovery probe.
- [ ] Recorded positive, negative, and recovery evidence with immutable IDs.
- [ ] Turned every discovered evidence gap into an owned platform requirement.
Sources
These primary references define observability signals and the platform concepts used in this day.