Prometheus and Mimir: From Metric to Incident

Prometheus is excellent at discovering nearby targets, scraping metrics, and evaluating alerts. A single Prometheus server is deliberately simple, though: its local disk and one-node query capacity become limits when retention, tenant count, or cluster count grows.

Solution: keep Prometheus close to the workloads as the scraper and rule evaluator, then use remote write to send every sample to Grafana Mimir. Mimir exposes a Prometheus-compatible query API while scaling ingestion, query, and object storage independently. Alertmanager remains the notification router.

The important boundary is:

application -> Prometheus -> Mimir
                    |
                    +-> alert rule -> Alertmanager -> receiver

Mimir does not create an incident merely because a sample arrived. A rule evaluates a PromQL condition; Alertmanager routes the resulting alert; an external incident system may then turn that notification into an incident.

A supermarket has checkout counters, a shift supervisor, a central archive, and an incident desk. Each counter reports throughput and failed payments. The shift supervisor watches the live counters and reacts quickly (Prometheus). The central archive keeps every store's history and answers long-range questions (Mimir). When a sustained failure breaches policy, the supervisor hands a structured alarm to the incident desk (Alertmanager), which groups duplicates and calls the correct team.

Graph legend

The Prometheus Operator watches monitoring CRDs. It converts a ServiceMonitor into scrape configuration and a PrometheusRule into loaded rule groups. Prometheus sends samples to Mimir and alerts to Alertmanager.

Graph legend

Do not treat Mimir as a replacement for the scrape loop. They solve different parts of the metrics path.

For this localhost lab, Mimir runs as one process with filesystem storage. That is intentionally a development topology. Production Mimir normally separates the distributor, ingesters, query frontend, queriers, store gateways, compactor, and ruler, with durable object storage.

Graph legend

An alert is not a log line and not an incident. It is a state machine crossing several boundaries.

The for: 30s clause filters brief spikes. The expression must remain true through the entire window. If it becomes false at 29 seconds, the pending alert returns to inactive and no firing notification is sent.

Metrics labels make series queryable. Alert labels also become the contract between Prometheus, Alertmanager, notification systems, and status pages.

labels:
  severity: critical
  environment: lab
  owner: payments
  component: storefront

Typical uses:

• severity selects paging versus ticketing.
• environment prevents development alerts from paging production on-call.
• owner routes to the team responsible for remediation.
• component maps an incident to a user-visible capability.

If component is missing, downstream automation cannot infer the affected capability safely. A status page may show impact unknown even though the alert and incident are valid. That is expected behavior, not proof that Alertmanager failed.

Prefer explicit labels in the rule. Do not try to recover ownership by parsing free-text descriptions downstream.

The lab uses a single-node Kind cluster and two namespaces:

observability
  mimir
  Prometheus Operator
  Prometheus
  Alertmanager
  Grafana

demo
  checkout-api
  ServiceMonitor
  PrometheusRule
  alert-sink

Prerequisites:

docker version
kind version
kubectl version --client
helm version
curl --version
jq --version

The manifests use fictional names and values. They are not copied from a production system.

Use a dedicated name so cleanup cannot delete another local cluster.

kind create cluster --name metrics-lab --wait 120s
kubectl cluster-info --context kind-metrics-lab
kubectl create namespace observability
kubectl create namespace demo

Expected:

Kubernetes control plane is running at https://127.0.0.1:...
namespace/observability created
namespace/demo created

This is the official single-process filesystem pattern adapted to a Kubernetes Deployment. It is small enough for a laptop and deliberately not a production configuration.

# mimir.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: mimir
  namespace: observability
data:
  mimir.yaml: |
    multitenancy_enabled: false
    usage_stats:
      enabled: false
    blocks_storage:
      backend: filesystem
      bucket_store:
        sync_dir: /tmp/mimir/tsdb-sync
      filesystem:
        dir: /tmp/mimir/data/tsdb
      tsdb:
        dir: /tmp/mimir/tsdb
    compactor:
      data_dir: /tmp/mimir/compactor
      sharding_ring:
        kvstore:
          store: memberlist
    distributor:
      ring:
        instance_addr: 127.0.0.1
        kvstore:
          store: memberlist
    ingester:
      ring:
        instance_addr: 127.0.0.1
        kvstore:
          store: memberlist
        replication_factor: 1
    ruler_storage:
      backend: filesystem
      filesystem:
        dir: /tmp/mimir/rules
    server:
      http_listen_port: 9009
      log_level: warn
    store_gateway:
      sharding_ring:
        replication_factor: 1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mimir
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mimir
  template:
    metadata:
      labels:
        app: mimir
    spec:
      containers:
        - name: mimir
          image: grafana/mimir:3.0.6
          args:
            - -config.file=/etc/mimir/mimir.yaml
          ports:
            - name: http
              containerPort: 9009
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 3
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 1Gi
          volumeMounts:
            - name: config
              mountPath: /etc/mimir
      volumes:
        - name: config
          configMap:
            name: mimir
---
apiVersion: v1
kind: Service
metadata:
  name: mimir
  namespace: observability
spec:
  selector:
    app: mimir
  ports:
    - name: http
      port: 9009
      targetPort: http

kubectl apply -f mimir.yaml
kubectl rollout status deployment/mimir -n observability --timeout=180s
kubectl port-forward -n observability service/mimir 9009:9009

In another terminal:

curl -fsS http://localhost:9009/ready
# ready

The chart installs the Prometheus Operator, Prometheus, Alertmanager, and Grafana. The selectors below allow our independently-authored ServiceMonitor and PrometheusRule objects to be discovered.

# monitoring-values.yaml
defaultRules:
  create: false
kubeStateMetrics:
  enabled: false
nodeExporter:
  enabled: false

prometheus:
  prometheusSpec:
    retention: 2h
    scrapeInterval: 5s
    evaluationInterval: 5s
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    remoteWrite:
      - url: http://mimir.observability.svc:9009/api/v1/push

alertmanager:
  config:
    global:
      resolve_timeout: 1m
    route:
      receiver: local-webhook
      group_by: [alertname, environment, component]
      group_wait: 5s
      group_interval: 10s
      repeat_interval: 1h
    receivers:
      # The chart's built-in Watchdog route targets this inert receiver.
      - name: "null"
      - name: local-webhook
        webhook_configs:
          - url: http://alert-sink.demo.svc:8080/alerts
            send_resolved: true

grafana:
  adminPassword: admin
  additionalDataSources:
    - name: Mimir
      type: prometheus
      access: proxy
      url: http://mimir.observability.svc:9009/prometheus
      isDefault: false

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade --install monitoring \
  prometheus-community/kube-prometheus-stack \
  --version 86.0.0 \
  --namespace observability \
  --values monitoring-values.yaml \
  --wait --timeout 10m

kubectl get pods -n observability

The application uses only Python's standard library. /checkout increments a success or error counter; /metrics exports both counters in Prometheus text format.

# checkout-api.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: checkout-api
  namespace: demo
data:
  server.py: |
    from http.server import BaseHTTPRequestHandler, HTTPServer
    from urllib.parse import parse_qs, urlparse

    counts = {"success": 0, "error": 0}

    class Handler(BaseHTTPRequestHandler):
        def do_GET(self):
            parsed = urlparse(self.path)
            if parsed.path == "/checkout":
                failed = parse_qs(parsed.query).get("fail", ["0"])[0] == "1"
                result = "error" if failed else "success"
                counts[result] += 1
                body = f'{{"result":"{result}"}}\n'.encode()
                self.send_response(503 if failed else 200)
                self.send_header("Content-Type", "application/json")
            elif parsed.path == "/metrics":
                text = (
                    "# HELP checkout_requests_total Checkout attempts by result.\n"
                    "# TYPE checkout_requests_total counter\n"
                    f'checkout_requests_total{{result="success"}} {counts["success"]}\n'
                    f'checkout_requests_total{{result="error"}} {counts["error"]}\n'
                )
                body = text.encode()
                self.send_response(200)
                self.send_header("Content-Type", "text/plain; version=0.0.4")
            else:
                body = b"not found\n"
                self.send_response(404)
                self.send_header("Content-Type", "text/plain")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)

        def log_message(self, *_):
            pass

    HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: app
          image: python:3.12-alpine
          command: [python, /app/server.py]
          ports:
            - name: metrics
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /metrics
              port: metrics
          volumeMounts:
            - name: code
              mountPath: /app
          resources:
            requests:
              cpu: 20m
              memory: 32Mi
            limits:
              memory: 128Mi
      volumes:
        - name: code
          configMap:
            name: checkout-api
---
apiVersion: v1
kind: Service
metadata:
  name: checkout-api
  namespace: demo
  labels:
    app: checkout-api
spec:
  selector:
    app: checkout-api
  ports:
    - name: metrics
      port: 8080
      targetPort: metrics

kubectl apply -f checkout-api.yaml
kubectl rollout status deployment/checkout-api -n demo --timeout=180s

The receiver prints every Alertmanager POST to its pod logs. It stands in for chat, paging, incident, or status-page automation without requiring credentials.

# alert-sink.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: alert-sink
  namespace: demo
data:
  server.py: |
    from http.server import BaseHTTPRequestHandler, HTTPServer

    class Handler(BaseHTTPRequestHandler):
        def do_GET(self):
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"ok\n")

        def do_POST(self):
            size = int(self.headers.get("Content-Length", "0"))
            body = self.rfile.read(size).decode()
            print("ALERT_WEBHOOK " + body, flush=True)
            self.send_response(200)
            self.end_headers()

        def log_message(self, *_):
            pass

    HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: alert-sink
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alert-sink
  template:
    metadata:
      labels:
        app: alert-sink
    spec:
      containers:
        - name: sink
          image: python:3.12-alpine
          command: [python, /app/server.py]
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /
              port: http
          volumeMounts:
            - name: code
              mountPath: /app
          resources:
            requests:
              cpu: 10m
              memory: 24Mi
            limits:
              memory: 64Mi
      volumes:
        - name: code
          configMap:
            name: alert-sink
---
apiVersion: v1
kind: Service
metadata:
  name: alert-sink
  namespace: demo
spec:
  selector:
    app: alert-sink
  ports:
    - name: http
      port: 8080
      targetPort: http

kubectl apply -f alert-sink.yaml
kubectl rollout status deployment/alert-sink -n demo --timeout=180s

The ServiceMonitor selects the Service's labels, then refers to its named port, not the container port number. The rule calculates the fraction of checkout attempts returning errors.

# checkout-monitoring.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout-api
  namespace: demo
spec:
  namespaceSelector:
    matchNames: [demo]
  selector:
    matchLabels:
      app: checkout-api
  endpoints:
    - port: metrics
      path: /metrics
      interval: 5s
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: checkout-api
  namespace: demo
spec:
  groups:
    - name: checkout.rules
      rules:
        - alert: CheckoutErrorRateHigh
          expr: |
            sum(rate(checkout_requests_total{result="error"}[30s]))
            /
            clamp_min(sum(rate(checkout_requests_total[30s])), 0.001)
            > 0.20
          for: 30s
          labels:
            severity: critical
            environment: lab
            owner: payments
            component: storefront
          annotations:
            summary: Checkout error rate is above 20 percent
            description: More than one in five checkout attempts failed for 30 seconds.
            runbook_url: https://example.invalid/runbooks/checkout-errors

kubectl apply -f checkout-monitoring.yaml
kubectl get servicemonitor,prometheusrule -n demo
kubectl port-forward -n observability \
  service/monitoring-kube-prometheus-prometheus 9090:9090

Verify target discovery:

curl -fsS -G http://localhost:9090/api/v1/query \
  --data-urlencode 'query=up{service="checkout-api"}' \
  | jq '.data.result[] | {metric, value}'

Expected value:

{
  "metric": {
    "namespace": "demo",
    "service": "checkout-api"
  },
  "value": ["...", "1"]
}

If the result is empty, inspect the rendered configuration boundary:

kubectl get servicemonitor checkout-api -n demo -o yaml
curl -fsS http://localhost:9090/api/v1/targets \
  | jq '.data.activeTargets[] | select(.labels.service=="checkout-api")'

Generate a baseline, wait for two scrapes, and query Mimir's Prometheus-compatible API:

kubectl run healthy-baseline -n demo --rm -i --restart=Never \
  --image=curlimages/curl:8.12.1 -- \
  sh -c 'for i in $(seq 1 20); do
    curl -sf http://checkout-api.demo.svc:8080/checkout >/dev/null
  done'

sleep 15

curl -fsS -G http://localhost:9009/prometheus/api/v1/query \
  --data-urlencode 'query=sum(checkout_requests_total)' \
  | jq '.data.result'

A non-empty vector proves:

1. the app exported the series;
2. Prometheus discovered and scraped it;
3. Prometheus remote-wrote it;
4. Mimir ingested it;
5. Mimir's query path returned it.

Check the remote-write queue directly if the query is empty:

curl -fsS -G http://localhost:9090/api/v1/query \
  --data-urlencode \
  'query=prometheus_remote_storage_samples_failed_total' \
  | jq '.data.result'

kubectl logs -n observability deployment/mimir --tail=100

Send sustained failures for long enough to cover the 30-second range vector and the separate 30-second for period:

kubectl delete pod failure-load -n demo --ignore-not-found
kubectl run failure-load -n demo --restart=Never \
  --image=curlimages/curl:8.12.1 -- \
  sh -c 'for i in $(seq 1 320); do
    curl -s http://checkout-api.demo.svc:8080/checkout?fail=1 >/dev/null
    sleep 0.25
  done'

Watch the state move from pending to firing:

watch -n 5 "curl -fsS http://localhost:9090/api/v1/alerts \
  | jq -r '.data.alerts[] |
  select(.labels.alertname==\"CheckoutErrorRateHigh\") |
  [.state,.labels.owner,.labels.component] | @tsv'"

Expected:

pending  payments  storefront
firing   payments  storefront

Then prove Alertmanager crossed the webhook boundary:

kubectl logs -n demo deployment/alert-sink --since=5m \
  | grep 'CheckoutErrorRateHigh'

The JSON contains:

{
  "status": "firing",
  "commonLabels": {
    "alertname": "CheckoutErrorRateHigh",
    "component": "storefront",
    "environment": "lab",
    "owner": "payments",
    "severity": "critical"
  }
}

Wait for the failure pod to finish, then make the range window healthy:

kubectl wait -n demo --for=jsonpath='{.status.phase}'=Succeeded \
  pod/failure-load --timeout=180s

kubectl run recovery-load -n demo --rm -i --restart=Never \
  --image=curlimages/curl:8.12.1 -- \
  sh -c 'for i in $(seq 1 240); do
    curl -sf http://checkout-api.demo.svc:8080/checkout >/dev/null
    sleep 0.1
  done'

sleep 60

kubectl logs -n demo deployment/alert-sink --since=10m \
  | grep '"status":"resolved"'

The resolved webhook proves the entire lifecycle, not merely alert creation. send_resolved: true is what tells Alertmanager to deliver this transition.

Run each port-forward in its own terminal:

# Grafana: admin / admin
kubectl port-forward -n observability \
  service/monitoring-grafana 3001:80

# Prometheus
kubectl port-forward -n observability \
  service/monitoring-kube-prometheus-prometheus 9090:9090

# Alertmanager
kubectl port-forward -n observability \
  service/monitoring-kube-prometheus-alertmanager 9093:9093

# Mimir
kubectl port-forward -n observability service/mimir 9009:9009

Open:

• Grafana: http://localhost:3001 and select the Mimir data source.
• Prometheus: http://localhost:9090/alerts.
• Alertmanager: http://localhost:9093.
• Mimir readiness: http://localhost:9009/ready.

Try this PromQL in Grafana Explore:

sum by (result) (rate(checkout_requests_total[5m]))

The lab stops at a local webhook because incident products require credentials. In a real system the next boundary commonly looks like this:

Graph legend

Two investigations require different evidence:

Repository access can prove the intended configuration. It cannot prove current runtime state. Conversely, a screenshot can prove a visible incident but not which committed rule generated it. Reliable incident analysis joins both.

This local lab deliberately removes production complexity. Before production:

• deploy distributed Mimir with object storage and multiple replicas;
• enable authentication and tenant IDs (X-Scope-OrgID);
• use TLS and NetworkPolicies between writers and Mimir;
• persist Prometheus WAL and Mimir state;
• set Mimir ingestion/query limits and capacity alerts;
• add HA Prometheus replicas with replica labels and deduplication;
• version and test rules as code;
• keep secrets in Kubernetes Secrets or an external secret manager;
• monitor the monitoring stack itself;
• use real runbooks and owned component labels;
• test both firing and resolved receiver behavior.

Delete only the dedicated cluster:

kind delete cluster --name metrics-lab

Extend the lab with a warning tier and a critical tier:

1. warning when the checkout error rate is above 10% for 2 minutes;
2. critical when it is above 25% for 1 minute;
3. warnings go only to a second local webhook receiver;
4. critical alerts keep the component: storefront label;
5. remove component from the warning and explain the downstream result;
6. prove firing and resolved delivery for both routes.

Use two rules with different thresholds, durations, and severity labels:

- alert: CheckoutErrorRateWarning
  expr: |
    sum(rate(checkout_requests_total{result="error"}[2m]))
    /
    clamp_min(sum(rate(checkout_requests_total[2m])), 0.001)
    > 0.10
  for: 2m
  labels:
    severity: warning
    environment: lab
    owner: payments
  annotations:
    summary: Checkout error rate is above 10 percent

- alert: CheckoutErrorRateCritical
  expr: |
    sum(rate(checkout_requests_total{result="error"}[1m]))
    /
    clamp_min(sum(rate(checkout_requests_total[1m])), 0.001)
    > 0.25
  for: 1m
  labels:
    severity: critical
    environment: lab
    owner: payments
    component: storefront
  annotations:
    summary: Checkout error rate is above 25 percent

Add severity-first child routes:

route:
  receiver: local-webhook
  group_by: [alertname, environment, component]
  routes:
    - matchers:
        - severity="warning"
      receiver: warning-webhook
    - matchers:
        - severity="critical"
      receiver: local-webhook

The critical notification can map directly to storefront. The warning lacks a component, so downstream automation must leave impact unknown or require a human to attach the component. That behavior is safer than guessing from the alert name.

• Prometheus Operator kube-prometheus-stack
• Prometheus Operator API reference
• Grafana Mimir get started
• Grafana Mimir architecture
• Grafana Mimir HTTP API
• Kind quick start