Prometheus as the Local Metrics Engine
Source: “Building an Observability Platform: From Prometheus to Mimir, Loki, Tempo, and Grafana” — Chapter 5, “Metrics: Prometheus as the Local Metrics Engine”
The enterprise problem and today’s slice
Enterprise problem: Operators need low-latency rates, ratios, distributions, and alert conditions from changing targets, but raw events are too expensive to scan for every operational decision.
Whole-course context: The collector topology and signal contract are incoming; today builds the local metrics path from discovery and scraping through storage, PromQL, rules, dashboards, and alert production.
Today’s slice: Operate one autonomous Prometheus for a bounded environment, design metrics and labels, write reviewable PromQL, and validate recording and alerting rules.
End-of-day evidence: A scrape-and-query packet containing target health, four Prometheus Query Language (PromQL) results, evaluated rule states, and a synthetic symptom tied to a release.
Still unsolved: Notification routing, cardinality governance at fleet scale, multi-cluster querying, durable long retention, and distributed ingestion remain deferred.
The smallest complete model
Raw events are too expensive to scan every time an operator asks whether a service is healthy. A local metrics engine compresses repeated behaviour into labelled samples and evaluates the same questions consistently.
Thesis: Prometheus is a bounded control plane for local metrics: it discovers and scrapes targets, stores labelled samples, then evaluates PromQL queries and rules. Why this matters: a correct result depends on acquisition, series identity, query semantics, and rule state—not merely on having a database.
The boundary is one autonomous environment. Prometheus owns current discovery, Hypertext Transfer Protocol (HTTP) scraping, local time-series storage, query execution, and rule evaluation; global retention and notification delivery remain separate concerns.
Expand the model one boundary at a time
Expand Prometheus in the order that gives every result meaning: discover a target, scrape its endpoint, identify the resulting series, store samples in the local time-series database (TSDB), evaluate PromQL, and then expose a dashboard result or alert state.
Pull-based scraping lets Prometheus control interval and expose collection health. Targets come from static configuration or service discovery, and each scrape creates timestamped samples with the final label set. The up metric proves scrape reachability, not that an application metric has the correct meaning.
| Component | Purpose and inputs | Transformation, output, and interface | Scaling constraint and failure mode | Alternatives, use when, avoid when |
|---|---|---|---|---|
| Discovery and scraper | Resolve changing targets and fetch their /metrics endpoints | Produce timestamped samples plus target and scrape-health evidence | Target count, scrape interval, payload size, and timeouts constrain load; missing targets create false gaps | Use pull when Prometheus can reach targets and collection health must be explicit; avoid assuming up == 1 proves semantic correctness |
| Local TSDB | Store samples identified by metric name plus complete label set | Compact local blocks for recent query and rule evaluation | Active series, churn, disk, memory, and retention constrain scale; a new label value creates a new series | Use for autonomous local history; avoid unique event identity in labels |
| PromQL engine | Select, transform, aggregate, and compare series | Return instant or range query results with explicit windows and groupings | Series scanned, range, step, and concurrency drive cost; missing data can resemble zero | Use recorded windows and bounded groupings; avoid plausible numbers without target-coverage evidence |
| Rule manager | Evaluate recording and alerting expressions on a schedule | Produce precomputed series and pending, firing, or resolved alert state | Evaluation interval and expression cost constrain timeliness; wrong for or labels create noisy state | Use versioned rules for repeated logic and actionable symptoms; avoid treating alert state as notification delivery |
Series type determines valid transformation. A counter increases until restart and needs rate() over a window. A gauge rises or falls, such as queue depth. A histogram counts observations in buckets and aggregates across instances. A summary calculates client-side streaming statistics whose configured quantiles generally cannot be meaningfully aggregated across instances. Prefer histograms for fleet latency, choose buckets that cover the service-level objective, and retain the le label when aggregating classic buckets.
PromQL turns samples into operational scope. Record every window and grouping because both change the conclusion.
Request rate:
sum by (service) (
rate(http_server_requests_total[5m])
)
Error ratio:
sum by (service) (
rate(http_server_requests_total{status_code=~"5.."}[5m])
)
/
sum by (service) (
rate(http_server_requests_total[5m])
)
p99 latency from classic histogram buckets:
histogram_quantile(
0.99,
sum by (service, le) (
rate(http_server_request_duration_seconds_bucket[5m])
)
)
CPU non-idle ratio:
1 - avg by (instance) (
rate(node_cpu_seconds_total{mode="idle"}[5m])
)
Instant queries evaluate at one point in time; range queries repeat evaluation over an interval. A graph’s step changes evaluation density, not the underlying scrape frequency.
Recording rules precompute repeated expensive expressions into named series.
groups:
- name: service-rates
interval: 30s
rules:
- record: service:http_requests:rate5m
expr: |
sum by (service) (
rate(http_server_requests_total[5m])
)
Alerting rules evaluate symptom state. The for duration rejects short blips; labels identify responsibility; annotations explain impact and action.
groups:
- name: checkout-alerts
rules:
- alert: CheckoutHighErrorRatio
expr: |
sum(rate(http_server_requests_total{service="checkout",status_code=~"5.."}[5m]))
/
sum(rate(http_server_requests_total{service="checkout"}[5m]))
> 0.05
for: 10m
labels:
severity: page
team: commerce
annotations:
summary: Checkout error ratio is above 5 percent
runbook_url: https://runbooks.example.com/checkout-errors
Prometheus decides whether an alert is active. Alertmanager separately owns notification grouping, suppression, routing, and delivery.
Run the model through one incident
The general rule is to prove the input population before interpreting an aggregate. In a simple counter example, restart one target and use rate() across the window; a raw difference may look negative, while the rate function handles the reset semantics.
In a recurring checkout incident, the p99 panel rises after a release. The operator first compares expected replicas with up, then records the request-rate denominator, error-ratio numerator, histogram result, grouping, and time range. The rule remains pending for ten minutes and then fires because the synthetic error ratio stays above five percent. Release and region labels scope the symptom without introducing request identity.
Observed evidence changes the diagnosis. If a target disappeared, the apparent traffic drop may be incomplete coverage. If traffic stopped, no series or an absent denominator is not the same as a measured zero. If only one release has high latency at matched load, rollback is a testable action; if every region changes at once, a shared dependency becomes more plausible.
Failure modes, trade-offs, and decision rules
A number can be syntactically valid and operationally wrong. The recurring failure mode is silent incompleteness: undiscovered targets, scrape errors, stale series, counter resets, missing denominators, unsuitable histogram buckets, or aggregation that drops the le label.
The main trade-off is query fidelity versus resource cost and latency. More labels and longer ranges add scope but increase series and scan work. Recording rules reduce repeated query cost but create another named series and evaluation lifecycle. Longer alert for durations reject noise but delay detection.
| Choice | Use when | Avoid when |
|---|---|---|
Counter plus rate() | Measuring events that accumulate and may reset on restart | The value can legitimately rise and fall |
| Gauge | Measuring current state such as queue depth or connections | A rate of discrete events is the real question |
| Histogram | Fleet-wide latency or size quantiles must aggregate across instances | Buckets do not cover the decision threshold or cardinality is uncontrolled |
| Recording rule | A stable, reviewed expression is repeated frequently | The query is exploratory or its ownership and lifecycle are unclear |
| Alerting rule | A sustained symptom has an owner and bounded action | The condition is informational, unactionable, or cannot distinguish absence |
trust a PromQL result only when target coverage, series identity, type transformation, time window, grouping, and missing-data behaviour are recorded alongside the value.
Close the loop
Use Observe → Interpret → Decide → Act → Measure for local metrics. Observe discovery, up, request rate, error ratio, latency, and rule state; interpret whether the symptom is impact or missing data; decide on a bounded release or capacity action; act; then repeat the same queries and customer probe over a matched window.
Run one bounded checkout drill: generate controlled errors long enough to exercise pending and firing states, restart one target, then stop traffic briefly. The falsifiable packet passes only if all expected targets are accounted for, rate() survives the reset, zero is distinguished from absence, the alert carries owner, severity, service, environment, and runbook, and the original checkout metric plus workflow probe recover after the action.
Key takeaways
Prometheus is an autonomous local metrics engine, not merely a storage component.
- Discovery and scraping determine which samples exist.
- Metric name plus complete label set defines a time series.
- Counters need rates; fleet latency usually needs aggregatable histograms.
- PromQL semantics depend on windows, grouping, and missing-data treatment.
- Recording rules precompute queries; alerting rules produce alert state, not notifications.
Checklist
Use this checklist before depending on a local Prometheus for operational decisions.
- [ ] Discovery covers every expected target and exposes scrape failures.
- [ ] Metric names, units, types, and bounded labels are documented.
- [ ] PromQL handles resets, missing data, and aggregation correctly.
- [ ] Histograms have useful buckets and preserve
leduring aggregation. - [ ] Rules are versioned, tested, owned, and linked to action.
- [ ] Synthetic symptoms exercise pending, firing, and resolved states.
Sources
These official Prometheus references define the mechanics taught here.