Where Local Prometheus Stops

Enterprise problem: Independent Prometheus servers preserve local autonomy but fragment long-term history and global queries, forcing operators to guess which cluster holds evidence during cross-region incidents.

Whole-course context: The local Prometheus engine, Alertmanager path, and cardinality budget are incoming evidence; today identifies the measured boundary at which a shared distributed metrics backend becomes justified.

Today’s slice: Define local-versus-global responsibilities, scaling triggers, remote-delivery requirements, migration stages, and failure-isolation proof without prematurely replacing local collection.

End-of-day evidence: A metrics architecture decision record with inventory, query and retention objectives, migration stages, rollback criteria, and a cross-cluster query drill.

Still unsolved: The internal design of Mimir, long-term object storage, distributed query execution, and the later logs, traces, profiles, and Grafana correlation layers belong to subsequent course days.

Centralization is worthwhile only when it solves observed customer and operator problems without removing local failure independence. These use cases test global investigation and durable history.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D08-UC-01	Global service operator	Query one service across clusters and regions without selecting each Prometheus manually	One authorized query returns consistently labelled series from every expected environment	Missing or late sources are named explicitly, and unauthorized tenant data is denied
D08-UC-02	Reliability and capacity lead	Retain shared metrics history beyond local disk life and survive one local Prometheus loss	Historical queries remain available centrally while healthy local environments continue scraping	Remote delivery lag and data gaps are measured, and a failed migration can roll back without stopping local monitoring

“We need scale” hides whether the real need is retention, global query, tenancy, or availability, so stories require a visible cross-environment outcome.

Story ID	Use case IDs	User story	Observable acceptance conditions
D08-US-01	D08-UC-01	As a global operator, I want one PromQL-compatible query across the fleet, so that I can scope an incident without hunting for the correct server	Results identify all expected clusters, preserve tenant isolation, and disclose source gaps and query partiality
D08-US-02	D08-UC-02	As a reliability lead, I want durable shared history while local collection remains autonomous, so that loss of one local disk does not erase fleet evidence	Retention exceeds local policy, remote delivery lag is within objective, and local scrape and rule paths survive central unavailability

A global query can look complete while omitting a cluster, so flows carry source inventory and degraded-state evidence to the terminal result.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D08-FLOW-01	D08-UC-01	Happy	Operator runs a fleet-wide checkout error query	1. Authenticate tenant and scope.<br>2. Resolve expected clusters.<br>3. execute one compatible query.<br>4. merge labelled results.<br>5. compare returned sources with inventory.<br>6. display completeness.	Actor, tenant, query, range, expected and returned clusters, result digest, completeness status, environment, timestamp, and query ID
D08-FLOW-02	D08-UC-01, D08-UC-02	Failure	One cluster’s remote delivery falls behind	1. Detect lag against objective.<br>2. mark central results partial.<br>3. retain local scrape and query.<br>4. recover delivery from bounded buffer.<br>5. verify historical continuity.<br>6. probe an unaffected cluster.	Lag window, partial-response marker, local query proof, recovered time range, control-cluster result, and drill ID
D08-FLOW-03	D08-UC-02	Recovery	Central backend is unavailable during rollout	1. Stop migration expansion.<br>2. keep local Prometheus collection and rules.<br>3. buffer within capacity.<br>4. restore or roll back central delivery.<br>5. reconcile gaps.<br>6. record decision.	Rollout revision, local health, buffered or dropped samples, recovery objective, gap report, and decision record

Replacing local Prometheus outright would couple target discovery and immediate rules to a remote dependency. The design keeps local acquisition autonomous and adds a shared backend for durable ingestion and global query.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D08-UC-01	Global PromQL-compatible query endpoint	Tenant gateway, query frontend, distributed query service, source inventory resolver	Shared metrics backend owns centralized series; inventory service owns expected sources	Partial query, missing cluster, tenant denial, timeout, or inconsistent external labels
D08-UC-02	Metrics architecture and migration console	Local Prometheus, remote-write sender, delivery buffer, shared ingestion, rollout controller	Local TSDB owns short local history; shared backend owns accepted long-term samples; rollout store owns migration state	Remote-write lag, buffer exhaustion, rejected samples, retention gap, or rollback event

Migration decisions become irreversible folklore unless their triggers, source inventory, and delivery evidence are durable. This model distinguishes local samples, centrally accepted samples, and rollout control state.

Generated-application database: Not created in this slice — metrics backends and platform control stores provide all required durable state; no application database is needed.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
MetricsSource	Fleet inventory, metrics platform	metrics_source_id	Opaque cluster, region, and local Prometheus references	organization_id	External labels uniquely identify source and environment	Active while enrolled; tombstone retained beyond metrics retention	D08-UC-01, D08-UC-02
ArchitectureDecision	Decision registry, observability architecture	decision_id	Opaque capacity reports and source inventory revision	organization_id	Requirements, alternatives, thresholds, rollback, and approvers are immutable	Permanently versioned; superseded rather than edited	D08-UC-01, D08-UC-02
DeliveryCheckpoint	Remote-delivery state, local metrics operator	checkpoint_id	Opaque metrics source and backend references	organization_id	Monotonic progress per source shard and destination	Advances continuously; old checkpoints compacted after reconciliation	D08-UC-02
GlobalQueryEvidence	Query evidence store, reliability engineering	global_query_id	Opaque source inventory revision and backend query reference	organization_id	Expected sources, returned sources, partiality, and result digest are required	Retained through review; raw samples follow metrics retention	D08-UC-01
MigrationRun	Rollout store, metrics platform	migration_run_id	decision_id local FK	organization_id	Stage transitions, gates, and rollback criteria are append-only	Retained for audit; detailed probe data expires by evidence policy	D08-UC-02

Prometheus should not be discarded merely because the organization grows. Within a bounded environment it combines discovery, scraping, local storage, PromQL, recording rules, and alert evaluation with few network dependencies.

That autonomy is valuable during a central outage. Cluster operators can still inspect current targets, query recent local history, and evaluate local rules. A scalable architecture preserves this property where the operational requirements justify it.

Independent servers become an operator problem when the evidence needed for one decision spans them. Common triggers are multiple clusters or regions, long retention beyond local disks, one global query endpoint, tenant isolation, independent ingestion and query scaling, durable shared history, and failure isolation between teams.

Measure the trigger before selecting a product:

Requirement	Evidence to collect
Global query	Fraction of incidents that require more than one Prometheus and time lost switching sources
Long retention	Required analysis horizon versus affordable local retention
Availability	Local disk or server loss events and tolerated evidence gap
Tenant isolation	Teams, access boundaries, noisy-neighbour incidents, and chargeback needs
Query scale	Concurrency, latency percentiles, scanned series, and peak review windows
Ingestion scale	Samples per second, active series, churn, and regional peaks

If these requirements are absent, operationally mature local Prometheus may be simpler and safer than a distributed backend.

A distributed Prometheus-compatible backend does not make local collection expertise disappear. A common division keeps Prometheus or an agent close to targets for discovery, scraping, relabeling, and bounded delivery buffering, while the shared backend receives many sources, enforces tenant limits, stores durable history, and executes distributed PromQL.

This relationship is complementary, not “Prometheus versus distributed storage.” The exact local rule placement depends on the failure objective: some alerts must continue through central outages, while fleet-wide rules inherently depend on shared data.

Centralization risks losing data or changing query results, so migrate by evidence. Inventory sources and external labels, add remote delivery for a canary, compare local and central query results over the same windows, exercise central failure, and expand only when delivery lag and query completeness meet objectives.

Define rollback before rollout. Keep application metrics endpoints and local scrape paths stable. Bound remote queues, expose rejected and dropped samples, and mark central responses partial when expected sources are absent. A global graph that silently omits a region is more dangerous than an explicit error.

Architecture should follow measured pain and objectives, so complete this review before choosing a shared backend.

1. Inventory every Prometheus, owner, tenant, external label, retention, and rule dependency.
2. Measure active series, samples per second, churn, query concurrency, and remote-delivery bandwidth.
3. Replay representative cross-cluster queries and define completeness semantics.
4. Model a central outage and prove local monitoring continues where required.
5. Canary one source, compare local and shared results, then force delivery lag.
6. Record product-neutral requirements and only then evaluate a distributed backend such as Mimir.

Local Prometheus has a clear strength and a measurable boundary; scaling should extend that architecture rather than erase it by reflex.

• Local Prometheus provides autonomous discovery, scraping, query, storage, and rules.
• Multi-cluster decisions, durable retention, tenancy, and independent scaling justify a shared backend.
• Global results need expected-source and partiality evidence.
• Local collection and distributed storage are commonly complementary.
• Canary delivery, query comparison, outage drills, and rollback make migration safe.

Use this checklist to decide whether the platform has actually reached Prometheus’s local boundary.

• [ ] Quantified cross-cluster investigation frequency and delay.
• [ ] Defined retention, ingestion, query, tenancy, and availability objectives.
• [ ] Inventoried every source and made external labels globally unambiguous.
• [ ] Preserved required local query and rule autonomy.
• [ ] Defined partial global-query semantics and remote-delivery gap evidence.
• [ ] Proved canary comparison, central outage, recovery, and rollback.

These official references describe Prometheus local operation and remote-write integration; later days will introduce the chosen distributed backend in detail.

• Prometheus overview
• Prometheus storage
• Prometheus remote write
• Prometheus federation