The Final Mental Model

Enterprise problem: A product-name mental model fails during redesigns and incidents because it says where signals usually live but not how system behavior becomes trustworthy evidence, a decision, and accountable action.

Whole-course context: The incoming phase ledger proves one complete investigation and alert path; this final day compresses every prior contract into a reusable chain that survives product substitutions and scale changes.

Today’s slice: We assign creation, protection, storage, correlation, condition recognition, notification, decision, and action responsibilities, then falsify the entire loop.

End-of-day evidence: Two contrasting incidents traverse the same invariant behavior-to-action chain with positive, denied, and recovery observations and independently owned boundaries.

Still unsolved: No architecture is permanently finished; new customer outcomes, threats, cost limits, and product versions must continuously re-enter the same review loop.

A final model is useful only if different incidents can use it without collapsing back into product memorization. These cases test a customer failure and an evidence-pipeline failure.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D24-UC-01	Customer-impact on-call	Convert checkout behavior into a bounded operational action	Correlated evidence supports a decision, an owned action, and a verified customer recovery	Unauthorized tenant evidence is denied and an unaffected authorized investigation remains complete
D24-UC-02	Observability platform on-call	Restore the evidence path when collection or storage degrades	Platform health evidence identifies loss risk, triggers controlled recovery, and verifies replay or bounded loss	Failed export is contained, notification is delivered, backlog recovery is measured, and a healthy signal control remains available

An abstract chain can hide missing ownership, so each actor needs end-to-end acceptance from trigger through verified effect.

Story ID	Use case IDs	User story	Observable acceptance conditions
D24-US-01	D24-UC-01	As a customer-impact on-call, I want trustworthy evidence connected to an owned action, so that investigation ends in verified recovery rather than a dashboard observation	Seeded checkout failure creates metrics, trace, logs, decision, mitigation, and recovery probe; cross-tenant query is denied
D24-US-02	D24-UC-02	As an observability platform on-call, I want the evidence path to observe itself, so that telemetry loss becomes an actionable incident before buffers exhaust	Export fault raises outcome-risk alert, exposes component cause, recovers backlog, and proves completeness or bounded loss

If the chain omits denial and recovery, it describes a demo rather than a production operating model. Both flows end with a verified state change.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D24-FLOW-01	D24-UC-01	Happy	On-call acknowledges a sustained checkout-success alert	1. Application behavior emits domain evidence.<br>2. Collector protects, enriches, and routes it.<br>3. Specialized stores retain it.<br>4. Grafana correlates metric, trace, logs, profile, and deployment.<br>5. Human chooses a bounded rollback.<br>6. Customer recovery probe passes.	Actor, tenant, signal/schema/policy revisions, expected/observed investigation, decision, action revision, environment, timestamps, trace/alert/deployment/run IDs
D24-FLOW-02	D24-UC-02	Recovery	Fault drill blocks one regional export destination	1. Collector queue and drop-risk evidence changes.<br>2. Rule recognizes threatened evidence availability.<br>3. Alertmanager routes one controlled notification.<br>4. On-call restores or reroutes export.<br>5. Backlog drains.<br>6. Completeness and healthy-path controls are queried.	Fault interval, buffer maximum, dropped/replayed counts, notification ID, action, recovery objective, unaffected control, environment, timestamp, and drill ID

The final architecture must preserve responsibility even when individual products change. Each service below owns one transition in the evidence-to-action loop.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D24-UC-01	Outcome alert in Grafana and Alertmanager	Application instrumentation, collector policy, Prometheus or Mimir, Loki, Tempo, Pyroscope, Grafana, ruler, Alertmanager, deployment controller, recovery probe	Specialized telemetry stores plus incident decision log owned by incident management	Missing domain evidence, correlation break, tenant denial, unsafe action denial, or failed recovery probe
D24-UC-02	Evidence-availability alert	Collector self-monitoring, regional buffer, backend health, ruler, Alertmanager, recovery controller, completeness query	Platform-health evidence and drill log owned by observability reliability	Queue exhaustion, drops, backend/query failure, missing notification, replay gap, or recovery-objective breach

An operational loop cannot be audited if evidence, decisions, actions, and effects share an overwriteable record. Separate immutable observations from mutable execution state and tenant-scope every customer path.

Generated-application database: Not created in this slice — the model operates on observability, incident, and deployment control-plane records rather than new application domain data.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
OperationalEvidenceSet	Incident evidence store, incident tooling owner	evidence_set_id	Opaque metric query, trace, log query, profile, schema, policy, and deployment references	tenant_id	Source revisions, scope, environment, time range, expected and observed facts are immutable	Append during incident, seal at closure, export with incident, expire payloads by signal policy while retaining audit references	D24-UC-01, D24-UC-02
OperationalDecision	Incident decision log, incident commander owner	decision_id	evidence_set_id local FK	tenant_id	Actor, alternatives, rationale, authority, expected effect, and timestamp are mandatory	Append immutably, supersede with new decision, retain through incident audit, delete after governance retention	D24-UC-01, D24-UC-02
OperationalActionRun	Deployment or recovery controller, owning operator	action_run_id	decision_id local FK and opaque deployment or routing revision	tenant_id	Idempotency key, bounded scope, approval, rollback target, and observed effect are mandatory	Execute, complete or roll back, retain result, expire operational payload after audit window	D24-UC-01, D24-UC-02

Memorizing “Prometheus equals metrics” hides the customer-facing chain and breaks when architecture changes. Memorize responsibilities and evidence transitions instead.

Responsibility	Durable question	Common implementation
Applications create evidence	What customer or system behavior occurred, with what domain meaning?	Application instrumentation, exporters, OpenTelemetry SDKs
Collectors protect, transform, and route	Was evidence authenticated, minimized, normalized, limited, buffered, and delivered to the right tenant?	OpenTelemetry Collector or Grafana Alloy
Specialized stores preserve queryable shapes	Can the relevant metric, log stream, trace, or profile be retrieved within retention and cost limits?	Prometheus or Mimir, Loki, Tempo, Pyroscope, object storage
Query and correlation assemble an investigation	Can an operator move among signals, deployment, owner, and runbook without losing scope?	Grafana and backend query APIs
Rules recognize conditions	Does measured evidence threaten a customer or system objective?	Prometheus-compatible rulers and alert rules
Notification control routes attention	Is the condition grouped, inhibited, silenced, and delivered to the accountable receiver?	Alertmanager or equivalent routing layer
Humans and automation act	Is there authority, a bounded change, rollback, and an expected effect?	Incident process, deployment and recovery controllers
Follow-up evidence closes the loop	Did the action improve the customer outcome without violating another boundary?	Synthetic probes, SLO queries, audit and action-run evidence

The fundamental building block is not a product. It is a reliable path from system behavior to operational action and back to observed effect. Product substitutions are acceptable when they preserve the same owned interfaces, evidence quality, security, cost, and failure semantics.

Teams often test each component separately while the correlation and action handoffs fail, so exercise the loop as one product journey.

1. Seed two contrasting incidents: a customer checkout failure and a collector export failure.
2. Pin application build, schema, collector policy, backend, rule, notification, and action revisions.
3. For each incident, traverse behavior, evidence, collection, storage, correlation, decision, action, and observed effect.
4. Attempt a cross-tenant query and an unauthorized action; require independent denials.
5. Fail one component and prove the evidence names that boundary while a healthy control still works.
6. Roll back the action or reroute collection and measure recovery against its objective.
7. Falsify the architecture if any transition lacks an owner, immutable identifier, negative evidence, or follow-up observation.

Observability is a socio-technical control loop: software creates evidence, platforms preserve it, and accountable actors turn it into verified change.

• Product names are replaceable; owned transitions and customer outcomes are not.
• Applications create meaning, collectors enforce policy, and specialized stores retain queryable evidence.
• Grafana correlates, rules recognize, Alertmanager routes, and humans or automation act.
• The loop closes only when follow-up evidence verifies the effect and boundaries remain intact.

Use this final review whenever a signal, backend, team, or incident process changes.

• [ ] Every behavior-to-action transition has one owner and protocol.
• [ ] Domain meaning originates in application instrumentation.
• [ ] Collection proves redaction, tenant routing, limits, buffering, and delivery.
• [ ] Investigations include correlation, decision authority, rollback, and observed effect.
• [ ] Customer and platform incidents both pass happy, denial, failure, and recovery probes.

The mental model names current implementations but relies on official primary documentation for their present responsibilities.

• Grafana Alloy’s role between data sources and storage backends
• Grafana Mimir deployment modes and component scaling
• Prometheus Alertmanager configuration and routing behavior