Where Local Prometheus Stops
Source: “Building an Observability Platform: From Prometheus to Mimir, Loki, Tempo, and Grafana” — Chapter 8, “Where Prometheus Stops”
The enterprise problem and today’s slice
Enterprise problem: Independent Prometheus servers preserve local autonomy but fragment long-term history and global queries, forcing operators to guess which cluster holds evidence during cross-region incidents.
Whole-course context: The local Prometheus engine, Alertmanager path, and cardinality budget are incoming evidence; today identifies the measured boundary at which a shared distributed metrics backend becomes justified.
Today’s slice: Define local-versus-global responsibilities, scaling triggers, remote-delivery requirements, migration stages, and failure-isolation proof without prematurely replacing local collection.
End-of-day evidence: A metrics architecture decision record with inventory, query and retention objectives, migration stages, rollback criteria, and a cross-cluster query drill.
Still unsolved: The internal design of Mimir, long-term object storage, distributed query execution, and the later logs, traces, profiles, and Grafana correlation layers belong to subsequent course days.
The smallest complete model
Local Prometheus is strong inside one bounded environment, but an operator cannot make a fleet-wide decision if history and queries are scattered across unknown servers. Scaling should extend local autonomy with a shared responsibility, not erase it by reflex.
Thesis: Keep Prometheus close to targets for autonomous collection and local decisions, and add a shared PromQL-compatible backend only when measured requirements demand global query, durable history, tenancy, or independent scale. Why this matters: the platform gains a fleet view without making every scrape and urgent local rule depend on the central service.
The smallest model preserves two authorities: local Prometheus owns immediate discovery, scraping, recent history, and required local rules; the shared backend owns accepted central samples, durable retention, and global query. Product internals remain outside the boundary until the requirements justify them.
Expand the model one boundary at a time
Expand the model from measured need. First inventory local sources and prove their autonomy; then add bounded remote delivery; then add tenant-aware durable ingestion; finally expose one global query that reports which expected sources did or did not contribute.
Prometheus should not be discarded simply because the organization grows. Within a bounded environment it combines discovery, scraping, a local time-series database (TSDB), Prometheus Query Language (PromQL), recording rules, and alert evaluation with few network dependencies. During a central outage, cluster operators can still inspect targets, query recent local history, and evaluate local rules.
| Component | Purpose and inputs | Transformation, output, and interface | Scaling constraint and failure mode | Alternatives, use when, avoid when |
|---|---|---|---|---|
| Local Prometheus | Discover and scrape local targets, relabel, store recent samples, and evaluate required local rules | Produces local PromQL and alert state plus bounded remote delivery | Local disk, active series, query concurrency, and one environment’s failure domain constrain growth | Use for autonomous local operation; avoid forcing cross-cluster decisions through manual server selection |
| Remote-delivery path | Send accepted local samples and track progress per source | Produces lag, retry, rejection, drop, and checkpoint evidence | Bandwidth, queue capacity, backend latency, and outage duration constrain buffering | Use for staged central ingestion; avoid unbounded queues or coupling scrape success to central availability |
| Shared backend | Receive many sources, isolate tenants, retain durable history, and execute distributed PromQL | Produces authorized global results and partiality signals | Samples per second, active series, churn, storage, and query fan-out require independent scaling | Use when measured retention, query, tenancy, or availability objectives require it; avoid it when mature local Prometheus already meets needs |
| Source inventory | Define expected clusters, regions, tenants, owners, and unique external labels | Lets each global query compare expected and returned sources | Stale inventory creates false completeness; ambiguous labels merge unrelated sources | Use for every global result; avoid claims of completeness without it |
The scaling trigger must be evidence rather than “we need scale.”
| Requirement | Evidence to collect |
|---|---|
| Global query | Fraction of incidents that require more than one Prometheus and time lost switching sources |
| Long retention | Required analysis horizon versus affordable local retention |
| Availability | Local disk or server loss events and tolerated evidence gap |
| Tenant isolation | Teams, access boundaries, noisy-neighbour incidents, and chargeback needs |
| Query scale | Concurrency, latency percentiles, scanned series, and peak review windows |
| Ingestion scale | Samples per second, active series, churn, and regional peaks |
The responsibilities remain complementary rather than “Prometheus versus distributed storage.” Some alerts must continue during a central outage; fleet-wide rules inherently depend on shared data.
Run the model through one incident
The general rule is to make global incompleteness visible while preserving the local path that can diagnose it. In a simple example, one canary source stops remote delivery: the global query names that source as missing, while its local Prometheus still returns current data and local rules continue.
During a recurring fleet-wide checkout incident, an operator queries error ratio across clusters. The inventory expects clusters A, B, and C, but remote-delivery lag shows C has fallen behind. The shared result is marked partial rather than treating two healthy responses as the whole fleet. The operator checks C locally, confirms checkout impact there, and avoids declaring a regional recovery from incomplete central evidence.
Bounded delivery buffers the outage within its objective, then resumes from the recorded checkpoint. The operator compares local and central results over the same recovered window and records any irreconcilable gap. A healthy control cluster proves central query and tenant authorization still work while C recovers.
Failure modes, trade-offs, and decision rules
Centralization creates new ways to be confidently wrong. The recurring failure mode is a global graph that silently omits a source; others include ambiguous external labels, remote queues exhausting memory, rejected samples without ownership, central failure stopping local rules, and a migration that changes query results without comparison evidence.
The main trade-off is shared durability and query convenience versus distributed-system cost and dependency. A central backend provides retention, tenancy, global PromQL, and independent scaling, but it adds delivery lag, partial results, capacity planning, and another failure domain. Local autonomy reduces those dependencies but fragments decisions that genuinely span environments.
| Choice | Use when | Avoid when |
|---|---|---|
| Stay local | Retention, query scope, availability, and tenancy objectives fit bounded Prometheus servers | Incidents repeatedly require manual cross-server investigation or history outlives local disks |
| Add federation | Selected aggregates need hierarchical access and its limits match the query model | Durable raw history, strong tenancy, or independently scaled global query is required |
| Add a shared distributed backend | Measured global query, retention, tenancy, ingestion, or query scale exceeds the local boundary | Requirements are vague and operationally mature local Prometheus already meets them |
| Keep critical rules local | They must continue during central unavailability | The rule inherently depends on fleet-wide data |
add shared metrics infrastructure only when a quantified requirement exceeds local Prometheus, and accept it only if source completeness, bounded delivery loss, local-outage independence, and rollback are proven.
Close the loop
Migrate through Observe → Interpret → Decide → Act → Measure. Observe cross-cluster investigation delay, retention gaps, active series, delivery lag, query concurrency, and partial results; interpret which measured boundary is exceeded; decide on a product-neutral requirement; act with one canary source; and measure local autonomy plus central completeness before expanding.
Run one bounded migration probe: inventory every Prometheus, owner, tenant, external label, retention, and rule dependency; canary remote delivery from one source; compare local and shared queries; then make the central backend unavailable long enough to exercise the declared buffer. The falsifiable result passes only if local scraping and required rules continue, central responses identify missing sources, rejected or dropped samples are explicit, the recovered historical window matches within the defined tolerance, an unauthorized tenant is denied, and rollback leaves application metrics endpoints unchanged.
Key takeaways
Local Prometheus has a clear strength and a measurable boundary; scaling should extend that architecture rather than erase it by reflex.
- Local Prometheus provides autonomous discovery, scraping, query, storage, and rules.
- Multi-cluster decisions, durable retention, tenancy, and independent scaling justify a shared backend.
- Global results need expected-source and partiality evidence.
- Local collection and distributed storage are commonly complementary.
- Canary delivery, query comparison, outage drills, and rollback make migration safe.
Checklist
Use this checklist to decide whether the platform has actually reached Prometheus’s local boundary.
- [ ] Quantified cross-cluster investigation frequency and delay.
- [ ] Defined retention, ingestion, query, tenancy, and availability objectives.
- [ ] Inventoried every source and made external labels globally unambiguous.
- [ ] Preserved required local query and rule autonomy.
- [ ] Defined partial global-query semantics and remote-delivery gap evidence.
- [ ] Proved canary comparison, central outage, recovery, and rollback.
Sources
These official references describe Prometheus local operation and remote-write integration; later days will introduce the chosen distributed backend in detail.