The Full Observability Platform Architecture

Enterprise problem: Independently capable collectors and backends can still fail as a platform when ownership, tenant boundaries, capacity signals, alert routing, recovery, and retirement are unspecified.

Whole-course context: The incoming evidence proves each signal path and one cross-signal investigation; today assembles those bounded components into an operable service with explicit failure domains rather than a decorative stack diagram.

Today’s slice: Define the complete running-system-to-action architecture, platform SLOs, multi-tenant controls, alert lifecycle, disaster recovery, cost governance, and self-observation.

End-of-day evidence: Produce a reviewable architecture decision record and game-day report proving ingestion, query, correlation, notification, isolation, and recovery objectives for a test tenant.

Still unsolved: Organization-specific capacity numbers, vendor contracts, production credentials, and irreversible data-retirement approvals remain deliberate owner decisions outside this reference design.

A platform diagram is not a customer outcome unless teams can onboard safely and responders can diagnose failures during degraded operation. These use cases cover both the tenant journey and the platform’s recovery duty.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D16-UC-01	Service team and incident responder	Onboard checkout telemetry, detect a sustained failure, correlate evidence, and reach an owned action	Four signals meet freshness/retention objectives and one alert-to-investigation run identifies a tested remediation	Unauthorized tenant access and invalid schema are denied while authorized controls remain usable
D16-UC-02	Observability platform operator	Operate, scale, recover, export, and retire the platform without losing ownership or evidence	SLOs, limits, backups, restore drills, cost attribution, and deletion reports meet reviewed objectives	Injected backend/collector failure is contained and recovery proves declared RPO/RTO and unaffected tenants

“Highly available observability” hides different write, read, and notification objectives. These stories require customer-visible signal quality and operator-visible recovery proof.

Story ID	Use case IDs	User story	Observable acceptance conditions
D16-US-01	D16-UC-01	As a service team responder, I want one governed onboarding and incident path, so that I can trust freshness, correlation, ownership, and notification evidence	Onboarding validates schema/limits; game day records metric, log, trace, profile, alert, correlation, denial, and remediation evidence
D16-US-02	D16-UC-02	As a platform operator, I want independent capacity and recovery controls for every signal path, so that one query or tenant cannot disable collection for everyone	Load and failure tests meet write/read SLOs, isolate a noisy tenant, restore metadata/data within RPO/RTO, and prove deletion/export lifecycle

The platform is valuable only when a production symptom reaches an accountable action and the evidence survives partial failure. The flow therefore includes onboarding, steady operation, incident response, recovery, and retirement.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D16-FLOW-01	D16-UC-01, D16-UC-02	Happy	Service owner submits checkout onboarding revision then triggers the approved game day	1. Schema, tenant, limits, retention, ownership, and cost policy are reviewed.<br>2. SDKs/exporters and collectors emit four signals.<br>3. Prometheus remote-writes metrics; collectors route logs, traces, and profiles.<br>4. Backends retain/query within SLOs.<br>5. Rules create alerts and Alertmanager routes notification.<br>6. Grafana correlation reaches trace, logs, profile, deployment, runbook, and action.<br>7. Evidence and cost are attributed to the tenant.	Revision, actor, tenant, four-signal freshness, alert states, receiver payload, correlation chain, action, environment, timestamps, and game-day ID
D16-FLOW-02	D16-UC-01, D16-UC-02	Recovery	Operator injects a collector/back-end failure and later retires the test tenant	1. Self-observation detects the failed boundary.<br>2. Limits protect unaffected tenants and critical ingestion.<br>3. Operator restores or fails over the component.<br>4. Backlog drains and query continuity is checked.<br>5. Tenant export is verified.<br>6. Access is revoked and retained telemetry follows deletion policy.<br>7. Tombstone/deletion report closes the lifecycle.	Detection time, affected scope, backlog, RPO/RTO, recovered controls, tenant denial, export digest, deletion report, and run ID

One shared “observability cluster” failure domain makes capacity and recovery promises impossible to reason about. The architecture keeps collection policy, each signal backend, investigation, and notification independently scalable and observable.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D16-UC-01	Tenant onboarding API/repository, Grafana, and alert receiver	Schema/policy pipeline, SDK/exporters, Alloy/OTel, Prometheus, Mimir, Loki, Tempo, Pyroscope, Grafana, rules, Alertmanager, deployment/runbook catalogs	Versioned platform policy; specialized telemetry backends; Grafana UI state; alert/evidence stores	Schema rejection, freshness breach, query/correlation error, missing alert transition, route failure, or ownership gap
D16-UC-02	Platform operations, SLO dashboards, restore/export/delete workflows	Capacity controllers, backend operators, object stores, Kafka-compatible buses, secret/identity systems, backup/restore, cost attribution, lifecycle workers	Object storage and metadata stores per backend; policy/audit/evidence stores; external secret manager	Saturation, lag, limit rejection, unavailable partition, restore mismatch, RPO/RTO breach, orphaned access, or incomplete deletion

The platform cannot prove isolation or retirement if tenant policy, resource grants, telemetry objects, and lifecycle jobs are implicit. This model assigns durable control/evidence state without relocating application business authority.

Generated-application database: Not created in this slice — the observability platform stores telemetry and control-plane evidence only; each observed application keeps its own tenant, user, and domain data.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
Tenant observability contract	Platform policy store owned by governance	observability_tenant_id/contract_version	Opaque organization, service-catalog, and billing references	organization_id	Approved signals, schema, limits, retention, residency, owners, RPO/RTO, and cost centre	Draft, approve, revise, suspend, export, retire, retain audit history	D16-UC-01, D16-UC-02
Backend data object	Owning Mimir/Loki/Tempo/Pyroscope store	backend/tenant_id/object_id	Opaque source service and policy-version references	observability_tenant_id	Backend-specific integrity, encryption, tenant isolation, and retention	Ingest, compact, retain, export where supported, tombstone, delete	D16-UC-01, D16-UC-02
Access grant	Identity/policy store owned by platform security	grant_id	Opaque subject, role, tenant, and backend references	organization_id	Least privilege, expiry, audit, and independent revocation per boundary	Provision, review, rotate, revoke, retain audit event, delete after policy	D16-UC-01, D16-UC-02
Lifecycle job	Platform workflow/evidence store owned by operations	lifecycle_job_id	Foreign reference to tenant contract and opaque backend job IDs	organization_id	Idempotent state machine; export digest and per-backend deletion proof required	Request, approve, execute, retry, verify, close, retain report	D16-UC-02
Game-day evidence	Immutable evidence store owned by reliability engineering	game_day_id	Opaque references to contract, deployment, queries, alerts, and lifecycle jobs	organization_id	Positive/negative paths, actor, resource, scope, expected/observed, environment, time, IDs	Create per exercise, retain for assurance objective, then expire	D16-UC-01, D16-UC-02

A single uptime percentage hides whether data was collected, queryable, correlated, or notified in time. Define objectives per customer-visible stage and tenant tier.

Stage	Owner	Example indicators
Collection	Service team plus collector platform	scrape/receive success, refused/dropped items, queue age, schema rejection
Metrics delivery/storage	Prometheus and Mimir operators	remote-write lag, rejected samples, ingest partition health, block upload, query freshness
Logs	Loki operators	distributor rejection, active streams, chunk flush, query latency, retention
Traces	Tempo operators	accepted/refused spans, consumer lag, block builds, trace completeness, TraceQL latency
Profiles	Pyroscope operators	profile freshness, ingestion errors, overhead, query latency, symbol coverage
Investigation	Grafana administrators	data-source health, panel errors, permission denials, correlation conformance
Notification	Rule and Alertmanager owners	evaluation delay, pending/firing/resolved transitions, route delivery, grouping and inhibition

Define error budgets separately for write acceptance, end-to-end freshness, query availability/latency, and critical notification delivery. A query SLO must not permit silent write loss.

An observability platform that pages only on application symptoms can fail silently. Monitor its collectors, queues, durable buses, object stores, query schedulers, rulers, Alertmanagers, Grafana data sources, and lifecycle workers from an independent enough path to detect common-mode failure.

Route alerts using stable ownership, service, environment, severity, and component labels. Rule evaluators determine alert state; Alertmanager manages grouping, deduplication, routing, inhibition, silences, and receiver delivery. Test firing and resolved notifications, route fallbacks, missing-owner handling, and receiver failure.

Unlimited telemetry lets one tenant degrade everyone, so limits are part of product correctness. Budget ingest rate, active series/streams, label cardinality, trace/profile volume, query concurrency, scanned bytes, retention, and object-store operations by tenant and signal.

Use rate limits, query fairness, bounded lookback, shuffle sharding or equivalent isolation where supported, and cost attribution. Do not solve budget pressure by silently dropping critical signals; publish priority and sampling/degradation policy, then alert when invoked.

Backups are claims until restored, and deletion is a claim until every owning store reports completion. Run a game day that interrupts one collector and one read component, validates backlog/catch-up and unaffected tenants, restores metadata/configuration, and checks data/query continuity against declared recovery point objective (RPO) and recovery time objective (RTO).

For retirement, revoke access first, stop new ingestion, export approved telemetry/configuration, apply retention/legal holds, schedule per-backend deletion, verify tombstones/object removal, and retain a non-sensitive deletion report. Never recursively delete object-store prefixes without resolved tenant scope and provider safeguards.

A design review should reject component catalogs with no measured paths. The final exercise must record:

1. Approved tenant contract and canonical schema revision.
2. One positive four-signal onboarding/control query.
3. One denied cross-tenant query and one rejected schema fixture.
4. One metric alert through pending, firing, receiver delivery, resolved, and resolved delivery.
5. One metric-to-trace-to-log-to-profile-to-deployment chain.
6. One collector/backend failure with detection, containment, drain, and recovery measurements.
7. One export/deletion dry run with resolved object counts and immutable report ID.

The complete platform is a set of explicit ownership and failure boundaries, not a single product. Preserve these conclusions:

• Collectors apply local transport and policy; specialized backends own each telemetry signal.
• Grafana owns investigation state, while rules and Alertmanager own distinct evaluation and notification duties.
• Write, freshness, query, correlation, and notification objectives require separate evidence.
• Tenant limits, independent authorization, self-observation, and cost attribution are product requirements.
• Restore, export, revoke, and delete complete the lifecycle and must be exercised.

The architecture is not approved until its negative paths are demonstrated. Attach immutable evidence for each item:

• [ ] Published tenant contracts, owners, schema, limits, retention, RPO/RTO, and cost centre.
• [ ] Proved all four ingestion/query paths and one cross-signal investigation.
• [ ] Proved schema rejection and cross-tenant denial with unaffected controls.
• [ ] Proved alert pending, firing, routed, resolved, and resolved delivery.
• [ ] Measured a failure’s detection, containment, backlog, catch-up, and recovery.
• [ ] Restored configuration/metadata and checked data continuity.
• [ ] Exercised access revocation, export, retention, and deletion reporting.
• [ ] Verified the observability platform’s own critical signals use a survivable detection path.

Component architecture and feature status change across releases, so production implementation must follow current primary documentation. These sources support the integrated design:

• Grafana Mimir architecture
• Grafana Loki deployment modes
• Grafana Tempo architecture
• Grafana Pyroscope v2 architecture
• Grafana correlations
• Grafana Alerting fundamentals
• Prometheus Alertmanager
• OpenTelemetry semantic conventions