The Full Observability Platform Architecture
Source: Observability Platform source — section 16, “The Full Architecture”
The enterprise problem and today’s slice
Enterprise problem: Independently capable collectors and backends can still fail as a platform when ownership, tenant boundaries, capacity signals, alert routing, recovery, and retirement are unspecified.
Whole-course context: The incoming evidence proves each signal path and one cross-signal investigation; today assembles those bounded components into an operable service with explicit failure domains rather than a decorative stack diagram.
Today’s slice: Define the complete running-system-to-action architecture, platform SLOs, multi-tenant controls, alert lifecycle, disaster recovery, cost governance, and self-observation.
End-of-day evidence: Produce a reviewable architecture decision record and game-day report proving ingestion, query, correlation, notification, isolation, and recovery objectives for a test tenant.
Still unsolved: Organization-specific capacity numbers, vendor contracts, production credentials, and irreversible data-retirement approvals remain deliberate owner decisions outside this reference design.
The smallest complete model
A catalog of observability products is not yet an operable platform. Checkout evidence creates value only when a running system produces it, a governed platform preserves and interprets it, and an accountable actor changes the system.
Thesis: the complete observability platform is a closed, governed path from running system to evidence to action, with separate objectives for every boundary. Why this matters: a healthy dashboard cannot conceal failed collection, stale data, broken correlation, or undelivered notification.
The three boxes are the smallest complete model: the source remains authoritative for business state, the platform owns telemetry and control evidence, and humans or automation own the reviewed response. The model becomes operational only when each transition has an owner and a measurable objective.
A single uptime percentage hides whether data was collected, queryable, correlated, or notified in time. Define objectives per customer-visible stage and tenant tier.
| Stage | Owner | Example indicators |
|---|---|---|
| Collection | Service team plus collector platform | scrape/receive success, refused/dropped items, queue age, schema rejection |
| Metrics delivery/storage | Prometheus and Mimir operators | remote-write lag, rejected samples, ingest partition health, block upload, query freshness |
| Logs | Loki operators | distributor rejection, active streams, chunk flush, query latency, retention |
| Traces | Tempo operators | accepted/refused spans, consumer lag, block builds, trace completeness, TraceQL latency |
| Profiles | Pyroscope operators | profile freshness, ingestion errors, overhead, query latency, symbol coverage |
| Investigation | Grafana administrators | data-source health, panel errors, permission denials, correlation conformance |
| Notification | Rule and Alertmanager owners | evaluation delay, pending/firing/resolved transitions, route delivery, grouping and inhibition |
Define error budgets separately for write acceptance, end-to-end freshness, query availability/latency, and critical notification delivery. A query service-level objective (SLO) must not permit silent write loss.
Expand the model one boundary at a time
Expand the same system–platform–action model in dependency order: create governed telemetry, collect and route it, store each signal in its specialized backend, query and correlate evidence, evaluate alert state, route notification, then record the action and its measured effect. Tenant identity, schema version, absolute time, and ownership must survive every interface.
| Boundary | Purpose, inputs, and transformation | Output or interface | Scaling constraint and failure mode | Alternative guidance |
|---|---|---|---|---|
| Instrumentation and collection | Turn runtime events into governed metrics, logs, traces, and profiles; redact, enrich, batch, and route them | Tenant-scoped telemetry for Prometheus, Mimir, Loki, Tempo, and Pyroscope | Refused items, queue age, memory limits, or schema rejection | Use agents for local reliability and gateways for shared policy; avoid a common gateway that becomes an untested global failure domain |
| Specialized backends | Validate, retain, compact, and query each signal using its native model | PromQL, LogQL, TraceQL, and profile query interfaces | Ingest lag, active-series/stream limits, object-store errors, or query saturation | Use separate signal stores when their economics and queries differ; avoid forcing every signal into one generic store |
| Grafana investigation | Preserve exact query context and transform governed identity across sources | Bounded dashboards, Explore queries, and correlations | Data-source failure, broad lookback, permission drift, or false joins | Use curated dashboards for known questions and Explore for hypotheses; avoid links that grant or imply backend access |
| Rules and notification | Evaluate metric conditions, then group, deduplicate, inhibit, silence, and route state changes | Firing and resolved receiver evidence | Evaluation delay, route failure, missing ownership, or notification flood | Use rules for state evaluation and Alertmanager for delivery policy; avoid merging the responsibilities into an opaque alert path |
| Accountable action | Convert evidence into a bounded human or automated change and record outcome | Reviewed remediation, run ID, and effect measurement | Unowned alerts, unsafe automation, or no post-action probe | Use automation only for reversible, scoped actions with guardrails; keep uncertain remediation human-owned |
An observability platform that pages only on application symptoms can fail silently. Monitor its collectors, queues, durable buses, object stores, query schedulers, rulers, Alertmanagers, Grafana data sources, and lifecycle workers from an independent enough path to detect common-mode failure.
Route alerts using stable ownership, service, environment, severity, and component labels. Rule evaluators determine alert state; Alertmanager manages grouping, deduplication, routing, inhibition, silences, and receiver delivery. Test firing and resolved notifications, route fallbacks, missing-owner handling, and receiver failure.
Capacity, tenancy, and cost controls
Unlimited telemetry lets one tenant degrade everyone, so limits are part of product correctness. Budget ingest rate, active series/streams, label cardinality, trace/profile volume, query concurrency, scanned bytes, retention, and object-store operations by tenant and signal.
Use rate limits, query fairness, bounded lookback, shuffle sharding or equivalent isolation where supported, and cost attribution. Do not solve budget pressure by silently dropping critical signals; publish priority and sampling/degradation policy, then alert when invoked.
Run the model through one incident
The general rule is to follow one customer symptom through creation, delivery, storage, query, correlation, notification, action, and measurement while recording partial failures. A simple example checks that one checkout error metric reaches Mimir, fires a rule, and produces firing and resolved notifications. A realistic recurring checkout incident adds logs, a payment trace, a same-window profile, and a telemetry-pipeline disruption so isolation and recovery are tested at the same time.
Backups are claims until restored, and deletion is a claim until every owning store reports completion. Run a game day that interrupts one collector and one read component, validates backlog/catch-up and unaffected tenants, restores metadata/configuration, and checks data/query continuity against declared recovery point objective (RPO) and recovery time objective (RTO).
For retirement, revoke access first, stop new ingestion, export approved telemetry/configuration, apply retention/legal holds, schedule per-backend deletion, verify tombstones/object removal, and retain a non-sensitive deletion report. Never recursively delete object-store prefixes without resolved tenant scope and provider safeguards.
Architecture review and practical proof
A design review should reject component catalogs with no measured paths. The final exercise must record:
- Approved tenant contract and canonical schema revision.
- One positive four-signal onboarding/control query.
- One denied cross-tenant query and one rejected schema fixture.
- One metric alert through pending, firing, receiver delivery, resolved, and resolved delivery.
- One metric-to-trace-to-log-to-profile-to-deployment chain.
- One collector/backend failure with detection, containment, drain, and recovery measurements.
- One export/deletion dry run with resolved object counts and immutable report ID.
These artifacts are the observed evidence for the whole model. They reveal whether the checkout symptom was absent, telemetry was rejected or delayed, a backend read failed, correlation broadened scope, notification stopped, or the action failed to improve the customer outcome.
Failure modes, trade-offs, and decision rules
The platform-wide failure mode is a common boundary—collector gateway, object store, identity provider, or Grafana surface—silently disabling multiple signals while component dashboards remain individually green. The governing trade-off is shared efficiency and consistent policy versus larger blast radius; more isolation improves containment but increases replicas, storage operations, configuration, and cost.
share a component only when tenants have compatible objectives and measured limits can contain noisy workloads; isolate a boundary when its plausible failure would breach another tenant’s write, query, notification, security, or recovery objective. Use independent capacity signals and error budgets per stage. Avoid one global availability number, unlimited tenants, silent sampling, untested restore claims, and irreversible automation without a bounded scope and probe.
Choose agents versus gateways, monoliths versus distributed backends, shared versus isolated tenants, and human versus automated action from explicit ingest, retention, latency, recovery, security, and cost constraints. Scale or split only the first boundary whose measured saturation or blast radius violates an objective.
Close the loop
The platform loop is Observe → Interpret → Decide → Act → Measure. Observe signal freshness, ingest and query health, correlation, alert delivery, tenant isolation, and customer symptoms; interpret the first broken boundary; decide on one reversible containment or remediation; act within a named tenant and component; then measure both platform recovery and checkout outcome.
Bound the final game day to one test tenant, one collector interruption, and one read-component interruption with declared recovery point and time objectives. The falsifiable probe succeeds only if all four signals recover without an unexplained gap, the alert reaches firing and resolved delivery, the correlation chain preserves identity, an unaffected tenant remains within objective, checkout health changes as predicted after remediation, and the export/deletion dry run reports the expected object counts.
Key takeaways
The complete platform is a set of explicit ownership and failure boundaries, not a single product. Preserve these conclusions:
- Collectors apply local transport and policy; specialized backends own each telemetry signal.
- Grafana owns investigation state, while rules and Alertmanager own distinct evaluation and notification duties.
- Write, freshness, query, correlation, and notification objectives require separate evidence.
- Tenant limits, independent authorization, self-observation, and cost attribution are product requirements.
- Restore, export, revoke, and delete complete the lifecycle and must be exercised.
Checklist
The architecture is not approved until its negative paths are demonstrated. Attach immutable evidence for each item:
- [ ] Published tenant contracts, owners, schema, limits, retention, RPO/RTO, and cost centre.
- [ ] Proved all four ingestion/query paths and one cross-signal investigation.
- [ ] Proved schema rejection and cross-tenant denial with unaffected controls.
- [ ] Proved alert pending, firing, routed, resolved, and resolved delivery.
- [ ] Measured a failure’s detection, containment, backlog, catch-up, and recovery.
- [ ] Restored configuration/metadata and checked data continuity.
- [ ] Exercised access revocation, export, retention, and deletion reporting.
- [ ] Verified the observability platform’s own critical signals use a survivable detection path.
Sources
Component architecture and feature status change across releases, so production implementation must follow current primary documentation. These sources support the integrated design: