Alertmanager and Notification Control
Source: “Building an Observability Platform: From Prometheus to Mimir, Loki, Tempo, and Grafana” — Chapter 6, “Alertmanager: From Conditions to Notifications”
The enterprise problem and today’s slice
Enterprise problem: Hundreds of duplicate or derivative alerts can overwhelm responders while a misrouted page delays customer recovery, so active conditions need a governed path to the right action owner.
Whole-course context: Prometheus rule state and the synthetic symptom packet are incoming; today turns firing alerts into grouped, deduplicated, inhibited, silenced, and routed notifications.
Today’s slice: Design Alertmanager routing and suppression around user-impacting symptoms, ownership labels, high availability, and notification evidence.
End-of-day evidence: A notification drill showing one grouped page from duplicate alerts, correct team routing, dependency inhibition, planned silence, and resolved delivery.
Still unsolved: Fleet-wide cardinality enforcement, multi-cluster metrics history, distributed queries, and long-term backend architecture remain deferred.
The smallest complete model
A firing alert is machine state, not yet a useful interruption. It becomes operationally valuable only when notification policy turns it into one contextual demand on the correct action owner.
Thesis: Alertmanager controls human attention by transforming active alert conditions into grouped, suppressed, routed, and evidenced notifications while leaving condition evaluation to Prometheus or another ruler. Why this matters: teams can change notification policy without changing symptom logic, and they can prove that suppression did not hide independent customer impact.
The boundary begins when Alertmanager receives active alerts and ends with receiver delivery and acknowledgement evidence. Metric evaluation stays upstream; the human or automated remediation stays downstream.
Expand the model one boundary at a time
Expand notification control in the order required to protect attention: deduplicate equivalent producer output, group alerts that describe one incident, evaluate bounded silences and causal inhibition, route by ownership labels, deliver to a receiver, and record acknowledgement separately.
| Component | Purpose and inputs | Transformation, output, and interface | Scaling constraint and failure mode | Alternatives, use when, avoid when |
|---|---|---|---|---|
| Deduplication | Accept equivalent alerts from replicated producers | Produce one logical alert identity | Producer and label inconsistency defeats matching; stale state can repeat delivery | Use for replicated rulers or Prometheus servers; avoid assuming it groups distinct instances |
| Grouping | Combine related alert instances into one notification | Emit a group keyed by incident-level dimensions | Fan-out and group timing constrain urgency; pod-level keys preserve noise while region-wide keys hide unrelated incidents | Start with alert name, cluster, and service; avoid grouping dimensions people do not act on |
| Silence | Match a bounded operator-created exception | Suppress notifications for a declared scope and time | Broad matchers can hide impact; missing expiry creates permanent blindness | Use for planned work with actor, reason, scope, and expiry; avoid open-ended or convenience silences |
| Inhibition | Use an active source alert to suppress derivative targets | Apply causal source, target, and equality matchers | Wrong equality scope can mute independent incidents | Use when the source explains the target, such as cluster failure causing pod alerts; avoid suppression based only on temporal coincidence |
| Routing and receiver | Match labels to an owned destination and send context | Produce provider delivery and acknowledgement evidence | Receiver throttling, retries, provider limits, and unmatched labels constrain reliability | Use a visible catch-all plus specific owned leaves; avoid routes that can silently discard alerts |
A default receiver prevents unmatched alerts from disappearing. This routing tree catches everything, then directs commerce pages and scopes cluster inhibition with an equality label.
route:
receiver: observability-dead-letter
group_by: [alertname, cluster, service]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- receiver: commerce-pager
matchers:
- team="commerce"
- severity="page"
inhibit_rules:
- source_matchers:
- alertname="ClusterUnavailable"
target_matchers:
- alertname=~"Pod.*"
equal: [cluster]
receivers:
- name: observability-dead-letter
- name: commerce-pager
Page-level symptoms should describe customer impact and a known response. High checkout error ratio, sustained latency beyond an objective, or inability to complete payment may justify interruption. A single restarted pod usually belongs on a dashboard or ticket unless it causes impact. Every page should name outcome, location, duration, owner, change, safe action, and evidence or runbook link.
Alertmanager can run as a high-availability (HA) cluster. Current Prometheus guidance says Prometheus should send alerts to every Alertmanager instance rather than through a load balancer; peers coordinate notification state while each processes alerts. Receiver APIs can still throttle or fail, so delivery status, retries, latency, provider IDs, and human acknowledgement remain separate evidence.
Run the model through one incident
The general rule is to suppress repetition and derivation, never independent impact. In a simple example, two Prometheus replicas send the same alert; deduplication and grouping should produce one notification whose affected-count field still exposes the fan-out.
During a recurring checkout incident, several checkout replicas exceed the error-ratio threshold while a cluster alert also fires. Alertmanager groups the checkout instances by alert name, cluster, and service and routes one commerce page with runbook and release context. Pod-level alerts in the failed cluster are inhibited because the cluster condition explains them, but an independent checkout symptom in another cluster remains delivered as the positive control.
Observed evidence continues beyond a successful receiver request. The route revision and receiver identify policy, the provider delivery ID proves handoff, and on-call acknowledgement proves human receipt where supported. An unmatched test alert reaching the dead-letter receiver is a visible configuration defect; disappearing without evidence would be an unsafe false success.
Failure modes, trade-offs, and decision rules
Notification policy can reduce noise or hide reality. The worst failure mode is silent suppression or misrouting: an overbroad silence, an inhibition rule without causal scope, a missing catch-all route, a duplicate flood, or provider success mistaken for human acknowledgement.
The main trade-off is interruption speed versus context and noise reduction. A longer grouping wait gathers related instances but delays the first page. Broad grouping lowers notification count but can combine unrelated incidents. Inhibition protects responders during a root-cause failure but can conceal a separate symptom if matchers are weak.
| Choice | Use when | Avoid when |
|---|---|---|
| Page | Customer impact or urgent risk has a known immediate action | The condition is informational or no safe action exists |
| Ticket or dashboard | Work is important but not urgent, or evidence needs trend review | Customer recovery requires immediate ownership |
| Silence | Planned, narrow, temporary work has an accountable actor | Scope or expiry cannot be stated |
| Inhibition | A source condition causally explains scoped target alerts | The relationship is merely correlated or equality labels are absent |
suppress or group an alert only when the remaining notification still preserves customer impact, accountable ownership, and an independent positive control outside the matcher scope.
Close the loop
Operate notifications as Observe → Interpret → Decide → Act → Measure. Observe active conditions and routing evidence; interpret whether they describe one incident or derivative noise; decide the owner and suppression boundary; act through the runbook; then measure delivery, acknowledgement, customer recovery, and resolved notification.
Run one bounded notification drill with duplicate checkout alerts, commerce and platform routes, a cluster source plus pod targets, and a short silence. The falsifiable result passes only if one contextual page reaches commerce, the correct pod alerts are inhibited, an unrelated checkout control still delivers, the silence expires automatically, normal delivery resumes, and the resolved state reaches the receiver with immutable delivery evidence.
Key takeaways
Alertmanager converts active conditions into governed demands on human attention.
- Deduplication, grouping, routing, silencing, and inhibition solve different problems.
- Prometheus evaluates alert conditions; Alertmanager controls notification delivery.
- Routing needs a catch-all path and owned receivers.
- Inhibition must encode a scoped causal dependency.
- Useful pages describe customer impact and a meaningful action.
- Positive controls are required when testing suppression.
Checklist
Use this checklist before relying on an alert route in production.
- [ ] Every page-level alert represents user impact or required urgent action.
- [ ] Root routing catches unmatched alerts visibly.
- [ ] Grouping reduces instance noise without combining unrelated incidents.
- [ ] Silences require actor, reason, scope, and expiry.
- [ ] Inhibition uses explicit shared labels and an independent positive control.
- [ ] Duplicate, delivery-failure, resolved, and post-silence paths are tested.
Sources
These official Prometheus references define current Alertmanager behaviour and deployment guidance.