Prometheus Remote Write: The Delivery Contract
Source: Observability Platform source — section 10, “Remote Write: The Bridge Between Prometheus and Mimir”
The enterprise problem and today’s slice
Enterprise problem: A service owner can see healthy local scrapes while the shared metrics backend silently falls behind, causing central dashboards and later incident analysis to omit the very samples needed during an outage.
Whole-course context: The incoming artifact is a Mimir write/read-path map and a direct Mimir query; today makes the Prometheus-to-Mimir delivery boundary measurable and recoverable.
Today’s slice: Configure and test the Prometheus write-ahead-log queue, authentication, batching, retry, backlog, and the metric-to-alert lifecycle without changing Mimir into a scraper or Alertmanager into a rule evaluator.
End-of-day evidence: Produce a firing-and-resolved alert run plus remote-write queue telemetry that proves accepted delivery and a bounded recovery from receiver failure.
Still unsolved: Logs, traces, profiles, Grafana investigation workflows, and full cross-signal platform governance remain deferred.
The smallest complete model
Healthy local scrapes do not prove that the shared backend has received those samples. If the delivery queue falls behind long enough, later checkout investigations can contain an invisible gap.
Thesis: Prometheus remote write is a bounded delivery contract from the local write-ahead log (WAL) to one authenticated receiver, not a second scrape. Why this matters: freshness and backlog must be measured independently from collection health.
The smallest model has one durable local source, one destination-specific queue, and one receiver. It is sufficient to reason about accepted, pending, retried, and expired samples before adding alert evaluation or notification routing.
If one shard fills, delivery can stall even though new samples continue to enter the local system, so backlog is a data-loss risk rather than a cosmetic latency metric. Each remote destination owns a queue that reads the WAL, places samples in per-shard memory queues, and sends batches to the receiver.
Prometheus adjusts shard count using incoming rate, unsent samples, and send duration. More shards can increase throughput but also CPU, memory, network load, and pressure on the receiver. The official tuning guide warns that prolonged receiver unavailability can outlive the WAL’s unsent-data window; design the recovery objective from the configured Prometheus version and retention rather than assuming indefinite buffering.
Monitor at least:
prometheus_remote_storage_samples_pending
rate(prometheus_remote_storage_samples_failed_total[5m])
rate(prometheus_remote_storage_samples_retried_total[5m])
Also watch send latency, highest sent timestamp versus wall clock, shard count, receiver status codes, Prometheus memory, CPU, disk, and network saturation.
Expand the model one boundary at a time
Expand the same WAL–queue–receiver model in dependency order: configure the destination, prove local collection, prove receiver acceptance, then prove time continuity in the backend. Only after those interfaces work should alert evaluation and notification be attached.
A successful POST proves only one batch, so the practical check must exercise scraping, queued delivery, shared querying, and time continuity. Start with conservative defaults and change queue parameters only from observed throughput and resource evidence.
remote_write:
- url: http://mimir.observability.svc:9009/api/v1/push
queue_config:
capacity: 10000
max_shards: 20
max_samples_per_send: 2000
Query local scrape health and Mimir’s shared result independently:
curl -fsS -G http://localhost:9090/api/v1/query \
--data-urlencode 'query=up{service="checkout-api"}' | jq '.data.result'
curl -fsS -G http://localhost:9009/prometheus/api/v1/query \
--data-urlencode 'query=sum(checkout_requests_total)' | jq '.data.result'
For a recovery drill, block only the test destination, capture queue metrics every 15 seconds, restore it before the tested safety window, and use query_range across the outage. A healthy local up series plus growing remote backlog proves the application and scraper are not the failed boundary.
The expanded boundary contract is explicit: the WAL supplies ordered local samples; the queue transforms them into bounded batches and retries; Mimir returns acceptance or rejection; the Mimir query API exposes delivered samples. More shards are an alternative for measured throughput pressure, but avoid them when CPU, memory, network, or receiver saturation is already the constraint.
Run the model through one incident
The general rule is to compare local collection, queued delivery, backend freshness, rule state, and receiver evidence on one clock. A simple example blocks Mimir for two minutes and expects pending samples to rise while local up stays healthy. A realistic checkout incident sustains an error rate long enough to fire an alert while the telemetry pipeline recovers, proving both evidence delivery and notification lifecycle.
Metrics in Mimir do not create incidents by themselves, so responders need the ownership chain from sample through rule and routing policy. Prometheus or a compatible ruler evaluates the expression; Alertmanager groups, deduplicates, inhibits, silences, and routes resulting alerts; an external system may then create an incident.
- alert: CheckoutErrorRateHigh
expr: |
sum(rate(checkout_requests_total{result="error"}[5m]))
/
clamp_min(sum(rate(checkout_requests_total[5m])), 0.001)
> 0.20
for: 10m
labels:
severity: critical
owner: payments
component: storefront
annotations:
runbook_url: https://example.invalid/runbooks/checkout-errors
Prove pending, firing, notification delivery, recovery traffic, resolved state, and resolved notification. Repository configuration proves intended thresholds and routes; live query and receiver payloads prove runtime state. Neither evidence class substitutes for the other. Record checkout request timestamps beside queue lag and highest-sent timestamps so observed evidence shows whether alert delay came from the application, delivery path, rule, or notification route.
Failure modes, trade-offs, and decision rules
Without boundary-specific symptoms, operators may scale Mimir when the actual fault is a selector, credential, or local disk. The common failure mode is tuning shards before identifying the constrained interface. The main trade-off is queue capacity and concurrency versus Prometheus memory, CPU, network use, and receiver pressure.
first compare local scrape health, pending samples, send latency, receiver responses, and backend freshness; repair the first broken boundary, and increase queue capacity or shards only when measured throughput—not authentication, disk, or receiver health—is limiting delivery. Use a larger recovery buffer when the tested receiver outage fits within WAL retention and local resources. Avoid claiming durable delivery when an outage can exceed that bounded safety window.
Classify before changing capacity.
| Symptom | Boundary to inspect | Safe first action |
|---|---|---|
Local up is empty | Discovery and scrape | Inspect ServiceMonitor selectors, named port, target API, and scrape error |
| Local samples exist; Mimir is empty | WAL queue, network, authentication, distributor | Inspect failed/retried/pending samples and receiver response |
| Pending samples grow with high send latency | Receiver or network capacity | Bound receiver health, lower risk of overload, then tune from measured throughput |
| Mimir query works; alert absent | Rule discovery/evaluation | Inspect rendered rule, labels, expression value, and for state |
| Alert fires; receiver is silent | Alertmanager route/contact point | Inspect routing tree, grouping, DNS/TLS, and receiver response |
| Firing arrives; resolved does not | Notification policy | Verify recovery condition and send_resolved behaviour |
Close the loop
The delivery loop is Observe → Interpret → Decide → Act → Measure. Observe local scrape success, queue depth, send timestamps, Mimir freshness, alert state, and receiver payloads; interpret the first divergent boundary; decide on restoration or measured tuning; act on one test destination; then measure backlog drain and data continuity.
Bound the next action to a two-minute receiver interruption during steady checkout traffic. The falsifiable probe succeeds only if pending samples rise, local scrapes remain healthy, the queue drains within the recovery objective, a Mimir range query contains the expected test-window samples, and both firing and resolved notifications carry the recorded run ID.
Key takeaways
Remote write can fail independently of both the application and Mimir query layer, so its queue is part of the production data path. Preserve these conclusions:
- The WAL and sharded queues buffer delivery; they are not infinite durable storage.
- Local scrape success, remote delivery, backend query, rule evaluation, and notification are separate proofs.
- Queue tuning trades throughput against Prometheus resources and receiver pressure.
- Alert labels are a routing contract; rules create alert instances, Alertmanager routes notifications, and incident tools own incidents.
- Test both firing and resolved paths, plus a receiver outage and recovery.
Checklist
A remote-write rollout is incomplete until the negative path is observable. Attach evidence for every checked item:
- [ ] Recorded destination revision, tenant mapping, TLS/auth method, and secret owner.
- [ ] Graphed pending, failed, retried, sent-timestamp, shard, and resource signals.
- [ ] Proved local scrape health and Mimir query success independently.
- [ ] Injected a receiver outage and measured backlog drain within the objective.
- [ ] Verified pending, firing, and resolved alert states.
- [ ] Captured firing and resolved receiver payloads with owner/component labels.
Sources
Remote-write semantics and available metrics change over time, so use primary documentation as the implementation authority. These sources support the protocol, queue, and notification boundaries: