Prometheus as the Local Metrics Engine

Enterprise problem: Operators need low-latency rates, ratios, distributions, and alert conditions from changing targets, but raw events are too expensive to scan for every operational decision.

Whole-course context: The collector topology and signal contract are incoming; today builds the local metrics path from discovery and scraping through storage, PromQL, rules, dashboards, and alert production.

Today’s slice: Operate one autonomous Prometheus for a bounded environment, design metrics and labels, write reviewable PromQL, and validate recording and alerting rules.

End-of-day evidence: A scrape-and-query packet containing target health, four PromQL results, evaluated rule states, and a synthetic symptom tied to a release.

Still unsolved: Notification routing, cardinality governance at fleet scale, multi-cluster querying, durable long retention, and distributed ingestion remain deferred.

Metrics become useful only when they answer an operational question with controlled dimensions. These cases cover service health and capacity decisions.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D05-UC-01	Service operator	Detect a sustained checkout error or latency regression and identify its scope	PromQL shows impact by service, region, and release and an alert enters pending then firing state	Missing targets, absent labels, or insufficient traffic produces explicit no-data evidence rather than a false healthy result
D05-UC-02	Capacity engineer	Measure CPU saturation and request demand before changing capacity	Comparable rate and saturation windows support a documented scaling decision	A scrape gap or counter reset is identified and excluded or handled in the query

Dashboard presence is not acceptance, so these stories require query results and their failure semantics.

Story ID	Use case IDs	User story	Observable acceptance conditions
D05-US-01	D05-UC-01	As a service operator, I want error ratio and p99 latency by bounded service dimensions, so that I can distinguish a release regression from a regional dependency failure	Queries return labelled series, target health is visible, and a synthetic symptom changes the matching rule state
D05-US-02	D05-UC-02	As a capacity engineer, I want request rate and CPU saturation over matched windows, so that I can scale from observed demand rather than snapshots	Query packet records range, step, target coverage, counter-reset handling, and before-and-after capacity decision

A Prometheus result can be wrong while still looking plausible, so the flows include discovery, scrape health, query semantics, and terminal evidence.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D05-FLOW-01	D05-UC-01	Happy	Operator opens a checkout symptom panel	1. Confirm target discovery and up.<br>2. Query request rate.<br>3. Compute error ratio.<br>4. Compute p99 from histogram buckets.<br>5. Group by bounded scope.<br>6. Inspect rule state.	Actor, environment, query strings, time range, series count, target health, values, rule state, release, and immutable packet ID
D05-FLOW-02	D05-UC-01, D05-UC-02	Recovery	A target disappears or restarts during analysis	1. Detect up == 0 or missing series.<br>2. Inspect scrape error.<br>3. Preserve counter-reset-safe rate.<br>4. Separate no-data from zero.<br>5. Restore target.<br>6. Re-run queries.	Scrape error, outage window, query treatment, recovered target, unaffected target control, and packet ID
D05-FLOW-03	D05-UC-02	Happy	Capacity review starts for checkout	1. Select matched traffic windows.<br>2. Query request rate.<br>3. Query CPU non-idle ratio.<br>4. Compare saturation and latency.<br>5. Record capacity action and threshold.	Matched-window queries, target coverage, observed saturation, decision owner, expected result, and review ID

Treating Prometheus as only a database hides the acquisition and evaluation stages that determine correctness. The system owns discovery, scrape scheduling, local time-series storage, PromQL, and rule evaluation as one bounded metrics engine.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D05-UC-01	Grafana panel or Prometheus query API	Service discovery, scraper, local TSDB, PromQL engine, rule manager	Prometheus local TSDB and rule-state memory; rule definitions in version control	Target down, scrape error, no data, query error, stale series, or non-firing synthetic symptom
D05-UC-02	Capacity review query packet	Scraper, TSDB, PromQL engine, capacity evidence service	Prometheus TSDB for samples; capacity evidence store for reviewed decisions	Incomplete target coverage, unmatched windows, resets mishandled, or query packet missing

Metrics are identified by complete label sets, so ownership must cover both series identity and the reviewed artifacts derived from it. Application records are never copied into this model.

Generated-application database: Not created in this slice — Prometheus owns time-series samples and the platform stores rule/query evidence; business data remains source-owned.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
ScrapeTarget	Prometheus service-discovery state, metrics platform	target_fingerprint	Opaque workload and discovery references	organization_id	Address plus final label set is unique within Prometheus	Appears and disappears with discovery; historical samples follow retention	D05-UC-01, D05-UC-02
TimeSeries	Prometheus local TSDB, metrics platform	series_fingerprint	Opaque metric-family reference	organization_id	Metric name plus complete label set uniquely identifies a series	Samples compacted into blocks and deleted by configured retention	D05-UC-01, D05-UC-02
RuleRevision	Version control and rule catalogue, service owner	rule_revision_id	Opaque repository commit	organization_id	Expression, duration, labels, annotations, and owner are immutable by revision	Versioned permanently; deployment status expires by policy	D05-UC-01
QueryPacket	Operational evidence store, reliability engineering	query_packet_id	Opaque Prometheus, rule, incident, and release references	organization_id	Query, range, step, target coverage, and result digest are required	Retained through review; source samples expire independently	D05-UC-01, D05-UC-02

Metrics operations fail when teams assume collection, storage, and alerting are separate services but configure only one of them. Prometheus combines target discovery, HTTP scraping, a local time-series database, PromQL query execution, recording-rule evaluation, and alert production.

Pull-based scraping lets Prometheus control interval and expose collection health. Targets come from static configuration or service discovery, and every scrape creates samples carrying timestamps and the final label set. The up metric makes reachability visible, but a healthy scrape does not prove the application metric is semantically correct.

Unbounded event detail makes aggregate analysis uneconomical, so a Prometheus series is identified by metric name plus the complete set of labels. A new label value creates a new series.

• A counter increases until process restart; apply rate() over a window.
• A gauge rises or falls, such as queue depth or active connections.
• A histogram counts observations in buckets and supports aggregation across instances.
• A summary calculates client-side streaming statistics; configured quantiles generally cannot be aggregated meaningfully across instances.

Prefer histograms for fleet latency. Ensure the selected bucket boundaries cover the service-level objective and keep the le label when aggregating classic histogram buckets.

Raw series are not operational conclusions, so PromQL selects, transforms, and aggregates them. Record the query window and grouping because both change meaning.

Request rate:

sum by (service) (
  rate(http_server_requests_total[5m])
)

Error ratio:

sum by (service) (
  rate(http_server_requests_total{status_code=~"5.."}[5m])
)
/
sum by (service) (
  rate(http_server_requests_total[5m])
)

p99 latency from classic histogram buckets:

histogram_quantile(
  0.99,
  sum by (service, le) (
    rate(http_server_request_duration_seconds_bucket[5m])
  )
)

CPU non-idle ratio:

1 - avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle"}[5m])
)

Instant queries evaluate at one time; range queries evaluate repeatedly across an interval. A graph’s step controls evaluation density, not raw scrape frequency.

Repeated expensive expressions slow dashboards and duplicate logic, so recording rules precompute a named series at evaluation time.

groups:
  - name: service-rates
    interval: 30s
    rules:
      - record: service:http_requests:rate5m
        expr: |
          sum by (service) (
            rate(http_server_requests_total[5m])
          )

Alerting rules turn a symptom expression into alert state. The for duration avoids paging on a short blip; labels route responsibility, while annotations explain impact and the next action.

groups:
  - name: checkout-alerts
    rules:
      - alert: CheckoutHighErrorRatio
        expr: |
          sum(rate(http_server_requests_total{service="checkout",status_code=~"5.."}[5m]))
          /
          sum(rate(http_server_requests_total{service="checkout"}[5m]))
          > 0.05
        for: 10m
        labels:
          severity: page
          team: commerce
        annotations:
          summary: Checkout error ratio is above 5 percent
          runbook_url: https://runbooks.example.com/checkout-errors

Prometheus determines whether the alert is active. Notification grouping, suppression, routing, and delivery belong to Alertmanager.

A query that returns a number may still omit targets or mishandle absence, so validate the whole path.

1. Inspect discovered targets and scrape errors.
2. Compare expected workload replicas with up series.
3. Restart a target and prove rate() handles the counter reset.
4. Stop traffic and distinguish a zero rate from missing data.
5. Generate bounded errors long enough to exercise pending and firing states.
6. Confirm the alert carries owner, severity, service, environment, and runbook.

Prometheus is an autonomous local metrics engine, not merely a storage component.

• Discovery and scraping determine which samples exist.
• Metric name plus complete label set defines a time series.
• Counters need rates; fleet latency usually needs aggregatable histograms.
• PromQL semantics depend on windows, grouping, and missing-data treatment.
• Recording rules precompute queries; alerting rules produce alert state, not notifications.

Use this checklist before depending on a local Prometheus for operational decisions.

• [ ] Discovery covers every expected target and exposes scrape failures.
• [ ] Metric names, units, types, and bounded labels are documented.
• [ ] PromQL handles resets, missing data, and aggregation correctly.
• [ ] Histograms have useful buckets and preserve le during aggregation.
• [ ] Rules are versioned, tested, owned, and linked to action.
• [ ] Synthetic symptoms exercise pending, firing, and resolved states.

These official Prometheus references define the mechanics taught here.

• Prometheus getting started
• Prometheus data model
• Prometheus metric types
• PromQL querying basics
• Prometheus recording rules
• Prometheus alerting rules