Alertmanager and Notification Control

Enterprise problem: Hundreds of duplicate or derivative alerts can overwhelm responders while a misrouted page delays customer recovery, so active conditions need a governed path to the right action owner.

Whole-course context: Prometheus rule state and the synthetic symptom packet are incoming; today turns firing alerts into grouped, deduplicated, inhibited, silenced, and routed notifications.

Today’s slice: Design Alertmanager routing and suppression around user-impacting symptoms, ownership labels, high availability, and notification evidence.

End-of-day evidence: A notification drill showing one grouped page from duplicate alerts, correct team routing, dependency inhibition, planned silence, and resolved delivery.

Still unsolved: Fleet-wide cardinality enforcement, multi-cluster metrics history, distributed queries, and long-term backend architecture remain deferred.

An alert is valuable only when a responsible person can take a meaningful action. These cases cover actionable delivery and safe suppression during known or derivative conditions.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D06-UC-01	On-call engineer	Receive one contextual notification for a user-impacting checkout condition	Duplicate instances group into one page routed to commerce with runbook and impact context	A deliberately unmatched alert reaches a dead-letter route and produces a configuration defect
D06-UC-02	Incident commander	Suppress derivative noise without hiding independent customer impact	Cluster-level failure inhibits pod alerts; planned maintenance silence expires automatically	An independent checkout symptom remains delivered as a positive control

Notification count is not a reliability outcome, so the stories require correct ownership, context, and unaffected controls.

Story ID	Use case IDs	User story	Observable acceptance conditions
D06-US-01	D06-UC-01	As an on-call engineer, I want duplicate alerts grouped and routed by ownership, so that I receive one actionable page rather than instance noise	Multiple replicas produce one group with service, environment, severity, runbook, affected count, and delivery ID
D06-US-02	D06-UC-02	As an incident commander, I want bounded inhibition and silences, so that known derivative noise is muted without concealing separate impact	Matching child alerts are suppressed, an unrelated symptom delivers, and the silence has actor, reason, scope, and expiry

Routing and suppression rules can fail silently, so flows include receiver delivery and negative controls rather than stopping at Alertmanager state.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D06-FLOW-01	D06-UC-01	Happy	Duplicate checkout alerts arrive from Prometheus replicas	1. Receive alerts.<br>2. Deduplicate fingerprints.<br>3. Group by alert name, cluster, and service.<br>4. Match team and severity routes.<br>5. Send one page.<br>6. Record acknowledgement.	Actor, alert group, receiver, route revision, notification ID, delivery status, acknowledgement, environment, and timestamps
D06-FLOW-02	D06-UC-02	Denied	Cluster outage and child pod alerts fire during a maintenance window	1. Evaluate silence matchers.<br>2. Evaluate inhibition source and target.<br>3. Suppress matching notifications.<br>4. Send unrelated control alert.<br>5. Expire silence.<br>6. verify normal delivery resumes.	Suppressed alert IDs and reason, silence actor and expiry, inhibition rule, control delivery ID, and post-expiry delivery

If alert production and notification policy are treated as one component, teams cannot change delivery without changing symptom logic. Alertmanager owns notification policy while Prometheus or another ruler owns alert evaluation.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D06-UC-01	Alertmanager alert API	Alert receivers, grouping engine, routing tree, notification integrations	Alertmanager state plus versioned notification configuration	Unmatched route, receiver error, excessive group size, duplicate delivery, or missing acknowledgement
D06-UC-02	Silence API and inhibition engine	Authorization layer, silence store, matcher engine, notification log	Alertmanager silence state and versioned inhibition rules	Overbroad match, expired silence still active, independent control suppressed, or child alert delivered

Notification policy affects who is interrupted and what may be suppressed, so its revisions and temporary exceptions require durable ownership and audit.

Generated-application database: Not created in this slice — Alertmanager and platform control stores persist notification policy and evidence; customer applications remain independent.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
RouteRevision	Configuration repository, observability platform	route_revision_id	Opaque repository commit	organization_id	Root route catches all alerts; every leaf names an owned receiver	Immutable revisions retained; deployment metadata expires by audit policy	D06-UC-01
NotificationAttempt	Notification evidence store, incident platform	notification_attempt_id	Opaque alert group, route revision, and provider reference	organization_id	Receiver, status, retry, and provider response are append-only	Retained with incident; sensitive provider payload expires earlier	D06-UC-01, D06-UC-02
Silence	Alertmanager state, authorized operators	silence_id	Opaque incident or change reference	organization_id	Actor, reason, bounded matchers, start, and expiry are required	Expires automatically; metadata retained for audit then deleted	D06-UC-02
InhibitionRule	Configuration repository, observability platform	inhibition_rule_id	route_revision_id local FK	organization_id	Source and target matchers share explicit equality labels and cannot self-match	Versioned with routes; retired rules remain in history	D06-UC-02

Raw firing alerts map poorly to human attention, so Alertmanager performs distinct controls. Deduplication prevents equivalent alerts from replicated producers creating repeated pages. Grouping combines related instances into one notification. Routing chooses receivers from labels. Silences mute matching alerts for a bounded period. Inhibition suppresses derivative alerts while a higher-level cause is active.

Grouping dimensions should describe the incident people act on. Grouping by pod preserves noise; grouping every alert in a region can hide unrelated incidents. Begin with alert name, cluster, and service, then test with realistic fan-out.

A default receiver prevents unmatched alerts from disappearing, so every routing tree starts with a catch-all root and adds specific children.

route:
  receiver: observability-dead-letter
  group_by: [alertname, cluster, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - receiver: commerce-pager
      matchers:
        - team="commerce"
        - severity="page"

inhibit_rules:
  - source_matchers:
      - alertname="ClusterUnavailable"
    target_matchers:
      - alertname=~"Pod.*"
    equal: [cluster]

receivers:
  - name: observability-dead-letter
  - name: commerce-pager

An inhibition rule encodes causality, not convenience. The source alert must explain why the target is derivative, and equality labels such as cluster prevent one failure from muting alerts elsewhere.

Technically precise telemetry can still create a useless page, so page on user-impacting symptoms with a known response. High checkout error ratio, sustained latency beyond an objective, or inability to complete payment can justify interruption. A single restarted pod usually belongs on a dashboard or ticket unless it causes impact.

Every page should answer: what customer outcome is affected, where, for how long, who owns it, what changed, what action is safe, and which link opens evidence or a runbook. If no action exists, create the action or lower the notification urgency.

Notification control is itself critical infrastructure, so Alertmanager can run as a high-availability cluster. Current Prometheus guidance says Prometheus should send alerts to all Alertmanager instances rather than through a load balancer; peers coordinate notification state while each can process alerts.

Receiver APIs also fail or throttle. Observe delivery status, retries, and latency without logging secrets. A successful HTTP request is not always a human acknowledgement, so capture provider delivery IDs and, where available, on-call acknowledgement separately.

Configuration review cannot reveal every matcher interaction, so exercise the routing tree with controlled alerts.

1. Send equivalent alerts from two producers and expect one grouped notification.
2. Send commerce and platform alerts and verify distinct routes.
3. Fire one cluster source alert plus many pod targets and verify inhibition.
4. Create a narrow, short silence with actor, reason, and change reference.
5. Send an unrelated control alert and prove it still delivers.
6. Wait for expiry and prove matching alerts deliver again.

Alertmanager converts active conditions into governed demands on human attention.

• Deduplication, grouping, routing, silencing, and inhibition solve different problems.
• Prometheus evaluates alert conditions; Alertmanager controls notification delivery.
• Routing needs a catch-all path and owned receivers.
• Inhibition must encode a scoped causal dependency.
• Useful pages describe customer impact and a meaningful action.
• Positive controls are required when testing suppression.

Use this checklist before relying on an alert route in production.

• [ ] Every page-level alert represents user impact or required urgent action.
• [ ] Root routing catches unmatched alerts visibly.
• [ ] Grouping reduces instance noise without combining unrelated incidents.
• [ ] Silences require actor, reason, scope, and expiry.
• [ ] Inhibition uses explicit shared labels and an independent positive control.
• [ ] Duplicate, delivery-failure, resolved, and post-silence paths are tested.

These official Prometheus references define current Alertmanager behaviour and deployment guidance.

• Alertmanager concepts
• Alertmanager configuration
• Alertmanager high availability
• Prometheus alerting practices