Prometheus Remote Write: The Delivery Contract

Enterprise problem: A service owner can see healthy local scrapes while the shared metrics backend silently falls behind, causing central dashboards and later incident analysis to omit the very samples needed during an outage.

Whole-course context: The incoming artifact is a Mimir write/read-path map and a direct Mimir query; today makes the Prometheus-to-Mimir delivery boundary measurable and recoverable.

Today’s slice: Configure and test the Prometheus write-ahead-log queue, authentication, batching, retry, backlog, and the metric-to-alert lifecycle without changing Mimir into a scraper or Alertmanager into a rule evaluator.

End-of-day evidence: Produce a firing-and-resolved alert run plus remote-write queue telemetry that proves accepted delivery and a bounded recovery from receiver failure.

Still unsolved: Logs, traces, profiles, Grafana investigation workflows, and full cross-signal platform governance remain deferred.

A remote-write URL alone gives customers no assurance about delivery or alert freshness. These use cases require both normal delivery and a visible recovery path.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D10-UC-01	Service owner	See fresh checkout metrics and receive a sustained-error notification	Mimir returns recent samples and the receiver records firing then resolved notifications	A malformed or unauthorized write is rejected without corrupting the authorized series
D10-UC-02	Observability platform operator	Detect and recover a backed-up remote-write destination before WAL retention loses data	Pending samples return to baseline within the recovery objective and no test-window gap appears	Receiver outage, retry/backlog signals, recovery time, and an unaffected local scrape prove the failed boundary

If acceptance stops at “configuration loaded,” transport loss can remain invisible. The stories therefore require runtime values and immutable receiver evidence.

Story ID	Use case IDs	User story	Observable acceptance conditions
D10-US-01	D10-UC-01	As a service owner, I want sustained checkout failures to produce firing and resolved notifications, so that I can act and confirm recovery	Prometheus shows pending then firing, the webhook records both states with owner/component labels, and Mimir returns the underlying metric
D10-US-02	D10-UC-02	As a platform operator, I want queue backlog and rejection reasons exposed, so that I can restore delivery before unsent WAL data is compacted away	A controlled receiver outage increases pending samples, local scraping stays healthy, restoration drains backlog, and the run records timestamps and IDs

Remote write is a queued delivery protocol rather than a second scrape, so failure can occur after local collection looks healthy. The flow starts with an operator-visible configuration and ends with backend query and notification evidence.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D10-FLOW-01	D10-UC-01, D10-UC-02	Happy	Owner generates healthy then failing checkout requests	1. Prometheus scrapes samples and appends them to its WAL.<br>2. Per-destination shards read, batch, and send samples.<br>3. Mimir accepts the authenticated batch.<br>4. The rule evaluates local series through pending and firing.<br>5. Alertmanager routes firing and resolved states.<br>6. Mimir answers the evidence query.	Query result, queue counters, alert state transitions, receiver payloads, environment, timestamps, and run ID
D10-FLOW-02	D10-UC-01, D10-UC-02	Recovery	Operator makes the Mimir receiver temporarily unavailable	1. Remote-write requests fail and retry with backoff.<br>2. Pending samples rise while scrape success remains healthy.<br>3. Operator restores the receiver before the WAL safety window is exceeded.<br>4. Shards drain the backlog.<br>5. A Mimir range query checks continuity and an invalid credential remains denied.	Outage interval, pending/failed/retried counters, drain duration, range-query continuity, denial response, and run ID

Treating remote write, rule evaluation, and notification as one service obscures independent failure modes. This design keeps the local WAL/queue, remote receiver, rule engine, and notification router separately observable.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D10-UC-01	Prometheus scrape configuration and PrometheusRule	Prometheus scraper/WAL/queue, Mimir distributor, Prometheus rule evaluator, Alertmanager, webhook receiver	Prometheus WAL for pending local delivery; Mimir storage for shared metrics; Alertmanager state for notification routing	Missing target, rejected batch, absent Mimir series, rule state, route mismatch, or missing resolved webhook
D10-UC-02	Remote-write queue configuration and operational dashboard	Prometheus queue manager, network/auth boundary, Mimir distributor, platform alerting	Prometheus WAL and queue metrics plus immutable test-run evidence	Pending/failed/retried samples, CPU/network saturation, receiver status code, backlog age, and continuity gap

Without durable configuration and evidence records, operators cannot distinguish intended routing from what actually happened. The model preserves local delivery state, shared metric state, and external notification evidence under their real owners.

Generated-application database: Not created in this slice — the checkout application owns its business counters in process, while Prometheus, Mimir, Alertmanager, and the evidence store own telemetry delivery and lifecycle state.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
Remote-write destination revision	Git configuration owned by platform operations	destination_revision_sha	Opaque receiver URL and credential reference	organization_id	Reviewed endpoint, tenant mapping, TLS, and queue policy; secrets never stored in Git	Propose, approve, deploy, roll back, retain revision history	D10-UC-01, D10-UC-02
Pending WAL sample	Prometheus WAL and queue manager	series_ref/timestamp/destination_id	Local reference to scraped series	organization_id	Ordered per series; retained only within WAL lifecycle and queue capacity	Append, retry, acknowledge, compact, and eventually expire if outage exceeds retention	D10-UC-02
Alert instance	Prometheus rule state owned by local monitoring	rule_uid/label_fingerprint	References metric labels and rule revision	organization_id	One state machine per complete label set; for must remain true before firing	Inactive, pending, firing, resolved; history retained in evidence store	D10-UC-01
Delivery test run	Platform evidence store owned by operations	remote_write_run_id	Opaque references to destination revision, alert, query, and receiver payload	organization_id	Positive and negative observations are immutable and timestamped	Create per drill, retain for audit objective, then expire	D10-UC-01, D10-UC-02

If one shard fills, delivery can stall even though new samples continue to enter the local system, so backlog is a data-loss risk rather than a cosmetic latency metric. Each remote destination owns a queue that reads the Prometheus write-ahead log (WAL), places samples in per-shard memory queues, and sends batches to the receiver.

Prometheus adjusts shard count using incoming rate, unsent samples, and send duration. More shards can increase throughput but also CPU, memory, network load, and pressure on the receiver. The official tuning guide warns that prolonged receiver unavailability can outlive the WAL’s unsent-data window; design the recovery objective from the configured Prometheus version and retention rather than assuming indefinite buffering.

Monitor at least:

prometheus_remote_storage_samples_pending

rate(prometheus_remote_storage_samples_failed_total[5m])

rate(prometheus_remote_storage_samples_retried_total[5m])

Also watch send latency, highest sent timestamp versus wall clock, shard count, receiver status codes, Prometheus memory, CPU, disk, and network saturation.

A successful POST proves only one batch, so the practical check must exercise scraping, queued delivery, shared querying, and time continuity. Start with conservative defaults and change queue parameters only from observed throughput and resource evidence.

remote_write:
  - url: http://mimir.observability.svc:9009/api/v1/push
    queue_config:
      capacity: 10000
      max_shards: 20
      max_samples_per_send: 2000

Query local scrape health and Mimir’s shared result independently:

curl -fsS -G http://localhost:9090/api/v1/query \
  --data-urlencode 'query=up{service="checkout-api"}' | jq '.data.result'

curl -fsS -G http://localhost:9009/prometheus/api/v1/query \
  --data-urlencode 'query=sum(checkout_requests_total)' | jq '.data.result'

For a recovery drill, block only the test destination, capture queue metrics every 15 seconds, restore it before the tested safety window, and use query_range across the outage. A healthy local up series plus growing remote backlog proves the application and scraper are not the failed boundary.

Metrics in Mimir do not create incidents by themselves, so responders need the ownership chain from sample through rule and routing policy. Prometheus or a compatible ruler evaluates the expression; Alertmanager groups, deduplicates, inhibits, silences, and routes resulting alerts; an external system may then create an incident.

- alert: CheckoutErrorRateHigh
  expr: |
    sum(rate(checkout_requests_total{result="error"}[5m]))
    /
    clamp_min(sum(rate(checkout_requests_total[5m])), 0.001)
    > 0.20
  for: 10m
  labels:
    severity: critical
    owner: payments
    component: storefront
  annotations:
    runbook_url: https://example.invalid/runbooks/checkout-errors

Prove pending, firing, notification delivery, recovery traffic, resolved state, and resolved notification. Repository configuration proves intended thresholds and routes; live query and receiver payloads prove runtime state. Neither evidence class substitutes for the other.

Without boundary-specific symptoms, operators may scale Mimir when the actual fault is a selector, credential, or local disk. Classify before changing capacity.

Symptom	Boundary to inspect	Safe first action
Local up is empty	Discovery and scrape	Inspect ServiceMonitor selectors, named port, target API, and scrape error
Local samples exist; Mimir is empty	WAL queue, network, authentication, distributor	Inspect failed/retried/pending samples and receiver response
Pending samples grow with high send latency	Receiver or network capacity	Bound receiver health, lower risk of overload, then tune from measured throughput
Mimir query works; alert absent	Rule discovery/evaluation	Inspect rendered rule, labels, expression value, and for state
Alert fires; receiver is silent	Alertmanager route/contact point	Inspect routing tree, grouping, DNS/TLS, and receiver response
Firing arrives; resolved does not	Notification policy	Verify recovery condition and send_resolved behaviour

Remote write can fail independently of both the application and Mimir query layer, so its queue is part of the production data path. Preserve these conclusions:

• The WAL and sharded queues buffer delivery; they are not infinite durable storage.
• Local scrape success, remote delivery, backend query, rule evaluation, and notification are separate proofs.
• Queue tuning trades throughput against Prometheus resources and receiver pressure.
• Alert labels are a routing contract; rules create alert instances, Alertmanager routes notifications, and incident tools own incidents.
• Test both firing and resolved paths, plus a receiver outage and recovery.

A remote-write rollout is incomplete until the negative path is observable. Attach evidence for every checked item:

• [ ] Recorded destination revision, tenant mapping, TLS/auth method, and secret owner.
• [ ] Graphed pending, failed, retried, sent-timestamp, shard, and resource signals.
• [ ] Proved local scrape health and Mimir query success independently.
• [ ] Injected a receiver outage and measured backlog drain within the objective.
• [ ] Verified pending, firing, and resolved alert states.
• [ ] Captured firing and resolved receiver payloads with owner/component labels.

Remote-write semantics and available metrics change over time, so use primary documentation as the implementation authority. These sources support the protocol, queue, and notification boundaries:

• Prometheus remote-write tuning
• Prometheus remote-write configuration
• Prometheus Remote-Write 2.0 specification
• Prometheus alerting overview
• Grafana Mimir push API