Mimir: The Distributed Metrics Backend
Source: Observability Platform source — section 9, “Mimir: The Distributed Metrics Backend”
The enterprise problem and today’s slice
Enterprise problem: A platform team must retain and query metrics from many clusters, but a single Prometheus server eventually exposes one-node retention, availability, and query-capacity limits that can hide production history during an incident.
Whole-course context: The incoming evidence is a production Prometheus design that discovers targets, scrapes samples, evaluates rules, and writes a local write-ahead log; this slice turns that local evidence into a shared metrics service.
Today’s slice: Build the Mimir storage and query boundary without transferring target discovery or scraping responsibility away from Prometheus or Alloy.
End-of-day evidence: Produce a reviewed write/read-path map plus a query proving that a remotely written series is available through Mimir’s Prometheus-compatible API.
Still unsolved: Remote-write backlog behaviour, logs, traces, profiles, cross-signal navigation, and the complete operating model remain deliberately deferred.
The smallest complete model
A local Prometheus can collect checkout metrics correctly and still lose the shared history responders need after that node or cluster disappears. The smallest useful shared model therefore separates the local writer from one durable metrics service.
Thesis: Prometheus should own local collection while Mimir owns durable, tenant-scoped metric storage and distributed querying. Why this matters: that boundary preserves local autonomy while making long-range evidence independently available.
The three boxes answer the first architectural question: who creates samples, who retains them, and who consumes query results. This model is complete enough to test one write and one read, but not yet detailed enough to assign capacity or diagnose partial failures.
Expand the model one boundary at a time
Scaling all Mimir components together wastes capacity and couples unrelated failures. Expand the same writer–service–reader model in dependency order: first the write entrance, then durable acceptance, then recent and historical storage, and finally query coordination.
On the write path, distributors validate and shard series to Kafka partitions; successful persistence lets the distributor acknowledge the sender before ingesters process the records. On the read path, query frontends split and cache work, schedulers queue it, and queriers combine recent samples from ingesters with historical blocks reached through store-gateways. Compactors reduce block count and apply retention in object storage. Optional rulers evaluate recording and alerting rules per tenant.
| Component | Primary responsibility | Scale or failure signal |
|---|---|---|
| Distributor | Authenticate, validate, limit, transform, and shard writes | Request rate, rejection reason, Kafka append latency |
| Kafka-compatible ingest storage | Durable accepted-write pipeline | Partition throughput, unavailable partitions, consumer lag |
| Ingester | Consume partitions and serve recent samples | Lag, memory/disk use, partition ownership |
| Query frontend and scheduler | Split, queue, cache, retry, and fairly schedule queries | Queue length, cache hit rate, rejected or retried work |
| Querier and store-gateway | Evaluate PromQL across recent and block data | Query latency, object-store errors, fetched bytes |
| Compactor | Compact blocks and enforce retention | Backlog, failed jobs, deletion-marker age |
Use Mimir when the sender count, retention period, tenant isolation, or query load requires independent scaling. Avoid it when a small environment can meet its recovery and retention objectives with one well-operated Prometheus; the extra durable bus, object store, and query services then add failure surface without a customer benefit.
Run the model through one incident
The general rule is to follow evidence across the same boundaries that acknowledge and later serve a sample. A simple example is one synthetic counter: if Prometheus sees it locally, Mimir accepts it, and a PromQL query returns it, the minimal path works. A realistic checkout incident adds continuous traffic while one recent-data consumer fails, so operators can distinguish delayed reading from lost writing.
A configuration file is only intended state, so it cannot prove that a sample crossed every boundary. Reuse the Kubernetes Day 56 checkout lab or an equivalent isolated environment and capture a runtime query plus failure counters.
Configure the writer with the in-cluster Mimir endpoint:
prometheus:
prometheusSpec:
retention: 2h
remoteWrite:
- url: http://mimir.observability.svc:9009/api/v1/push
After generating checkout traffic, query Mimir directly:
kubectl port-forward -n observability service/mimir 9009:9009
curl -fsS -G http://localhost:9009/prometheus/api/v1/query \
--data-urlencode 'query=sum(checkout_requests_total)' \
| jq '.data.result'
A non-empty vector proves export, scrape, remote write, Mimir acceptance, and Mimir read-path availability for that series. During the checkout disruption, correlate the query window with distributor acceptance, Kafka partition lag, ingester catch-up, block upload, and query latency. That observed evidence identifies whether the telemetry pipeline stored the samples but could not yet read them. It does not prove long retention, cross-zone survival, or tenant isolation; record separate tests for those claims.
Failure modes, trade-offs, and decision rules
Every production choice exchanges simplicity, cost, and isolation. The main failure mode is treating Mimir as one opaque service, because operators then cannot tell durable write acceptance from recent-query or historical-query health. The central trade-off is more independent scaling and recovery control in exchange for Kafka-compatible ingest storage, object storage, tenant policy, and additional components to operate.
use distributed Mimir only when measured sender scale, retention, multi-tenancy, or query demand exceeds a simpler Prometheus design; then scale and recover the first saturated boundary instead of the whole platform. Use separate availability zones and tenant limits when the stated objectives require failure containment. Avoid speculative replicas, retention, or query capacity that no objective or load test justifies.
A development single-process deployment can teach API flow but cannot prove production durability. Production design must explicitly choose tenant authentication, object storage, ingest-storage capacity, availability zones, limits, and recovery objectives.
Run these bounded drills:
- Deny a request with a missing or incorrect tenant identity and verify the authorized tenant still queries successfully.
- Stop one recent-data consumer while writes continue; record append success, lag, catch-up time, and query recovery.
- Apply a deliberately expensive bounded query; verify query fairness and that write acceptance stays within its objective.
- Temporarily deny object-store access in a test tenant; verify historical read and block-upload errors are distinguishable from live ingestion.
Choose Mimir when multi-cluster aggregation, long retention, multi-tenancy, independent scaling, or centralized governance justify the added operational system. For a small environment, one well-operated Prometheus can remain the safer design.
Close the loop
The operating loop is Observe → Interpret → Decide → Act → Measure. Observe write acknowledgements, partition lag, recent and historical query results; interpret which boundary is impaired; decide whether to restore, isolate, or scale it; act on one bounded test tenant; then measure catch-up time and query continuity.
For the next drill, stop one test ingester for no more than five minutes while checkout writes continue, then restore it. The falsifiable probe is a range query over the interruption: the change succeeds only if accepted-write counters continue, lag returns to baseline within the recovery objective, the expected samples remain continuous, and a foreign-tenant query is still denied.
Key takeaways
Missing component boundaries make failures look like generic “Mimir outages,” which slows recovery. Keep these decisions explicit:
- Prometheus or Alloy scrapes and forwards; Mimir stores and queries at shared scale.
- Mimir 3.x ingest storage durably separates write acceptance from recent-query consumers.
- Recent data comes from ingesters; historical data comes from object-store blocks through store-gateways.
- Tenant identity, limits, retention, and evidence are owned platform state, not application authority.
- A successful query is necessary evidence, but resilience and isolation require independent negative-path drills.
Checklist
An architecture review can pass while its operating evidence is missing. Complete this checklist with links to actual artifacts:
- [ ] Named the tenant model, authentication boundary, limits, and retention policy.
- [ ] Diagrammed separate write, recent-read, and historical-read paths.
- [ ] Queried a known remotely written series through Mimir.
- [ ] Captured rejected-write, ingest-lag, block-upload, and query-failure signals.
- [ ] Proved one cross-tenant denial with an unaffected authorized control.
- [ ] Recorded why Mimir is justified instead of assuming every cluster needs it.
Sources
Architecture claims drift as major versions change, so implementation must be checked against current primary documentation. These sources define the responsibilities used in this lesson: