Mimir: The Distributed Metrics Backend

Enterprise problem: A platform team must retain and query metrics from many clusters, but a single Prometheus server eventually exposes one-node retention, availability, and query-capacity limits that can hide production history during an incident.

Whole-course context: The incoming evidence is a production Prometheus design that discovers targets, scrapes samples, evaluates rules, and writes a local write-ahead log; this slice turns that local evidence into a shared metrics service.

Today’s slice: Build the Mimir storage and query boundary without transferring target discovery or scraping responsibility away from Prometheus or Alloy.

End-of-day evidence: Produce a reviewed write/read-path map plus a query proving that a remotely written series is available through Mimir’s Prometheus-compatible API.

Still unsolved: Remote-write backlog behaviour, logs, traces, profiles, cross-signal navigation, and the complete operating model remain deliberately deferred.

Without explicit customer jobs, a metrics backend can become expensive shared infrastructure with no proof that it improves diagnosis or governance. These use cases bind Mimir to durable multi-cluster history and tenant-safe querying.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D09-UC-01	Service owner	Compare current checkout latency with the same service across clusters and prior releases	One PromQL result returns the authorized long-range series with cluster and version dimensions	An unauthorized tenant query is rejected and a same-tenant control query still returns data
D09-UC-02	Observability platform operator	Keep metric writes available while query load or an ingester failure occurs	Write acknowledgements continue and recent/historical query probes recover within the objective	Queue lag, rejected samples, partition health, and a recovery probe identify the failed boundary

Vague requirements make availability and isolation impossible to test. These stories translate the two customer jobs into observable positive and negative outcomes.

Story ID	Use case IDs	User story	Observable acceptance conditions
D09-US-01	D09-UC-01	As a service owner, I want one long-range PromQL view across my authorized clusters, so that I can distinguish a release regression from seasonal behaviour	The query spans the requested window, includes both clusters, and the same credentials cannot read another tenant
D09-US-02	D09-UC-02	As an observability platform operator, I want durable ingestion separated from expensive reads, so that query pressure does not silently discard new samples	A load test records successful writes, ingest lag, query latency, an injected read-path failure, and a recovered positive probe

A metrics query can appear healthy even while writes are being lost, so the product flow must follow both ingestion and reading to terminal evidence. Mimir’s current ingest-storage architecture uses Kafka or a compatible system as the durable handoff between distributors and read-path consumers.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D09-FLOW-01	D09-UC-01, D09-UC-02	Happy	Operator enables a tenant’s Mimir remote-write destination and the owner runs a long-range query	1. Prometheus scrapes and queues samples.<br>2. The distributor authenticates, validates, and shards the request.<br>3. Kafka durably acknowledges the write.<br>4. Ingesters expose recent samples and upload blocks.<br>5. Queriers combine recent and historical results.	Tenant, series selector, time range, write status, query result, environment, timestamp, and immutable test-run ID
D09-FLOW-02	D09-UC-01, D09-UC-02	Recovery	Operator injects a querier or ingester disruption during continuous writes	1. Writes continue through distributors and Kafka.<br>2. Health and lag signals identify the impaired read component.<br>3. Replacement consumers catch up.<br>4. The owner repeats the bounded query.<br>5. A foreign-tenant query remains denied.	Failure interval, Kafka lag, recovered query result, denied cross-tenant response, unaffected write counter, and run ID

Combining every role into one unnamed “Mimir cluster” hides which failure can affect writes, reads, or both. The design therefore separates the write entry, durable ingest boundary, recent-data consumers, block storage, and query coordination.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D09-UC-01	Prometheus-compatible query API through query frontend	Query frontend, query scheduler, querier, ingester, store-gateway	Object store for durable blocks; Kafka-backed ingesters for the recent query window	Query error class, split-query retries, store-gateway failures, missing-window comparison, and tenant-denial response
D09-UC-02	Remote-write or OTLP metrics endpoint through distributor	Distributor, Kafka-compatible ingest storage, ingesters, compactor, partition ring	Kafka for accepted recent writes and object storage for long-term blocks	Rejected samples, distributor errors, partition lag, unhealthy consumer, block-upload failure, and recovery probe

Mimir and Prometheus are complementary. Prometheus or Alloy discovers and scrapes local targets, applies relabeling, and buffers delivery; Mimir receives many writers, enforces tenant policy, stores blocks, and executes distributed PromQL.

If tenant identity or block ownership is ambiguous, a valid query can leak another customer’s metrics or deletion can leave billable data behind. The ownership model keeps application data at its source while Mimir owns metric samples, blocks, policy, and operational evidence.

Generated-application database: Not created in this slice — durable Mimir, Kafka, object-storage, and audit records are sufficient because the observed application remains the authority for business data.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
Tenant policy	Mimir runtime configuration owned by the platform team	tenant_id	None — tenant policy is the root isolation record	tenant_id	Authenticated tenant must match limits and query scope	Provision, version, revoke, retain audit record, then remove after data-retention policy	D09-UC-01, D09-UC-02
Accepted write batch	Kafka topic owned by Mimir ingestion	topic_partition_offset	Opaque sender and request references; no authority transferred	tenant_id	Acknowledged only after durable Kafka acceptance; series consistently map to partitions	Append, consume, retain for configured window, then expire by topic policy	D09-UC-02
Metric block	Object store owned by Mimir storage	tenant_id/block_ulid	References accepted samples through block metadata	tenant_id	Immutable block identity; compaction preserves samples and tenant boundary	Upload, compact, apply retention, tombstone, and delete according to tenant policy	D09-UC-01, D09-UC-02
Query evidence	Platform evidence store owned by operations	query_run_id	Opaque reference to tenant, query, and deployment revision	tenant_id	Result digest and denial status are immutable for the test run	Create per probe, retain for audit window, then expire without deleting source metrics	D09-UC-01, D09-UC-02

Scaling all Mimir components together wastes capacity and couples unrelated failures. The split paths let operators scale ingestion, recent reads, historical reads, and query coordination from their own saturation signals.

On the write path, distributors validate and shard series to Kafka partitions; successful persistence lets the distributor acknowledge the sender before ingesters process the records. On the read path, query frontends split and cache work, schedulers queue it, and queriers combine recent samples from ingesters with historical blocks reached through store-gateways. Compactors reduce block count and apply retention in object storage. Optional rulers evaluate recording and alerting rules per tenant.

Component	Primary responsibility	Scale or failure signal
Distributor	Authenticate, validate, limit, transform, and shard writes	Request rate, rejection reason, Kafka append latency
Kafka-compatible ingest storage	Durable accepted-write pipeline	Partition throughput, unavailable partitions, consumer lag
Ingester	Consume partitions and serve recent samples	Lag, memory/disk use, partition ownership
Query frontend and scheduler	Split, queue, cache, retry, and fairly schedule queries	Queue length, cache hit rate, rejected or retried work
Querier and store-gateway	Evaluate PromQL across recent and block data	Query latency, object-store errors, fetched bytes
Compactor	Compact blocks and enforce retention	Backlog, failed jobs, deletion-marker age

A configuration file is only intended state, so it cannot prove that a sample crossed every boundary. Reuse the Kubernetes Day 56 checkout lab or an equivalent isolated environment and capture a runtime query plus failure counters.

Configure the writer with the in-cluster Mimir endpoint:

prometheus:
  prometheusSpec:
    retention: 2h
    remoteWrite:
      - url: http://mimir.observability.svc:9009/api/v1/push

After generating checkout traffic, query Mimir directly:

kubectl port-forward -n observability service/mimir 9009:9009
curl -fsS -G http://localhost:9009/prometheus/api/v1/query \
  --data-urlencode 'query=sum(checkout_requests_total)' \
  | jq '.data.result'

A non-empty vector proves export, scrape, remote write, Mimir acceptance, and Mimir read-path availability for that series. It does not prove long retention, cross-zone survival, or tenant isolation; record separate tests for those claims.

A development single-process deployment can teach API flow but cannot prove production durability. Production design must explicitly choose tenant authentication, object storage, ingest-storage capacity, availability zones, limits, and recovery objectives.

Run these bounded drills:

1. Deny a request with a missing or incorrect tenant identity and verify the authorized tenant still queries successfully.
2. Stop one recent-data consumer while writes continue; record append success, lag, catch-up time, and query recovery.
3. Apply a deliberately expensive bounded query; verify query fairness and that write acceptance stays within its objective.
4. Temporarily deny object-store access in a test tenant; verify historical read and block-upload errors are distinguishable from live ingestion.

Choose Mimir when multi-cluster aggregation, long retention, multi-tenancy, independent scaling, or centralized governance justify the added operational system. For a small environment, one well-operated Prometheus can remain the safer design.

Missing component boundaries make failures look like generic “Mimir outages,” which slows recovery. Keep these decisions explicit:

• Prometheus or Alloy scrapes and forwards; Mimir stores and queries at shared scale.
• Mimir 3.x ingest storage durably separates write acceptance from recent-query consumers.
• Recent data comes from ingesters; historical data comes from object-store blocks through store-gateways.
• Tenant identity, limits, retention, and evidence are owned platform state, not application authority.
• A successful query is necessary evidence, but resilience and isolation require independent negative-path drills.

An architecture review can pass while its operating evidence is missing. Complete this checklist with links to actual artifacts:

• [ ] Named the tenant model, authentication boundary, limits, and retention policy.
• [ ] Diagrammed separate write, recent-read, and historical-read paths.
• [ ] Queried a known remotely written series through Mimir.
• [ ] Captured rejected-write, ingest-lag, block-upload, and query-failure signals.
• [ ] Proved one cross-tenant denial with an unaffected authorized control.
• [ ] Recorded why Mimir is justified instead of assuming every cluster needs it.

Architecture claims drift as major versions change, so implementation must be checked against current primary documentation. These sources define the responsibilities used in this lesson:

• Grafana Mimir ingest-storage architecture
• Grafana Mimir ingester
• Grafana Mimir architecture
• Grafana Mimir HTTP API