Metric Cardinality as a Platform Constraint
Source: “Building an Observability Platform: From Prometheus to Mimir, Loki, Tempo, and Grafana” — Chapter 7, “Cardinality: The Central Constraint of Metrics”
The enterprise problem and today’s slice
Enterprise problem: Every unique metric label combination consumes memory, CPU, disk, network, and query capacity, so one unbounded identifier can destabilize the shared metrics platform and delay incident response for every team.
Whole-course context: The working Prometheus queries and notification routes are incoming evidence; today governs the series identities those queries and rules depend on.
Today’s slice: Model cardinality multiplication, set per-metric and per-tenant budgets, move event identity to logs or traces, and contain unsafe instrumentation before ingestion.
End-of-day evidence: A cardinality review with estimated and observed series counts, rejected dangerous labels, approved alternatives, and a budget-enforcement drill with an unaffected tenant control.
Still unsolved: Central multi-cluster retention, distributed ingestion and query scaling, and the architecture beyond local Prometheus remain deferred.
The smallest complete model
A metric label is not free metadata; each distinct complete label set creates a time series. One unbounded value can multiply platform work until every tenant’s ingestion and queries slow down.
Thesis: Cardinality must be treated as a pre-release product constraint that starts with a named aggregate question, permits only bounded dimensions, and enforces a measured series budget at runtime. Why this matters: useful metrics remain queryable without letting event identity destabilize the shared telemetry path.
The model includes metric schema, estimated combinations, observed active series and churn, and tenant enforcement. It excludes individual event identity, which belongs in signals designed to preserve it.
Expand the model one boundary at a time
Expand the model by dependency: name the query first, enumerate each dimension’s bounded values, multiply the full deployment space, compare it with a tenant budget, then validate the estimate under representative traffic and enforce it at ingestion.
Label dimensions multiply rather than add. For one metric with 30 services, 4 environments, 5 regions, 200 routes, 10 status codes, and 6 methods, the theoretical upper bound is:
30 × 4 × 5 × 200 × 10 × 6 = 7,200,000 series
Adding 100,000 customer IDs can multiply that space by 100,000 rather than merely add 100,000 series. Real combinations may be sparse, but a review cannot assume sparsity without observed evidence and a reason the combinations remain bounded. Active series consume ingestion bookkeeping, current-chunk and index memory, disk, network, compaction, and query work. High churn adds pressure even when simultaneous count looks moderate.
| Component | Purpose and inputs | Transformation, output, and interface | Scaling constraint and failure mode | Alternatives, use when, avoid when |
|---|---|---|---|---|
| Metric schema | Connect a metric name, type, unit, labels, owner, and query | Produces an immutable reviewable schema revision | Every label’s value set multiplies the identity space; generated values escape estimates | Use bounded dimensions tied to aggregate questions; avoid open-ended label APIs |
| Cardinality estimator | Multiplies value bounds across replicas, clusters, regions, and tenants | Produces an upper bound and capacity comparison | Sparse assumptions can understate real combinations; estimates miss runtime generation | Use before release and compare with load evidence; avoid approving solely from current test data |
| Runtime analyzer | Measures active series, churn, sample volume, top metrics, labels, and growth | Produces attributable capacity and anomaly evidence | Scanning and storing analysis has cost; late alerts allow saturation first | Use continuously for drift from estimates; avoid dashboards without remediation ownership |
| Tenant limiter | Applies active-series and ingestion-rate budgets | Accepts, rejects, or relabels with explicit reason and owner notification | Silent removal creates false health; broad limits can punish unrelated services | Use to contain a noisy tenant; avoid correction without denial evidence and recovery path |
Event identity belongs in a different representation rather than a more permissive metric budget.
| Evidence | Correct home | Reason |
|---|---|---|
| Service, environment, region, operation, status class | Metric labels | Bounded dimensions support aggregation |
| User, email, order, request, session, raw URL | Structured logs with access controls | Individual event identity is searchable without one series per value |
| Request and parent relationship | Trace context | Preserves distributed causality |
| Function and stack resource attribution | Profiles | Maps cost to code locations |
Replace /orders/8128 with /orders/{order_id} and exception text with a bounded failure category, while preserving authorized detail in logs. Software development kit (SDK) views can drop unsafe attributes, collectors or Prometheus relabeling can remove known hazards, and ingestion systems can enforce tenant limits. Temporary exceptions need scope, owner, justification, capacity impact, and expiry.
Run the model through one incident
The general rule is to compare observed series growth with the schema change that could produce it before changing platform capacity. A simple example adds failure_category with six enumerated values: estimate the multiplier, load test it, and approve only if observed growth matches the expected order of magnitude.
In a recurring checkout telemetry pipeline incident, a release adds raw customer_id and request-shaped URL values to an HTTP metric. Active series and churn spike, ingestion latency grows, and unrelated queries begin slowing. The limiter crosses the checkout tenant’s budget and rejects new unsafe series with the offending label names and metric revision, while the control tenant remains queryable.
Observed evidence leads to a bounded response: roll back the instrumentation or remove the hazardous attributes at the collector, keep customer and request identity in authorized logs or traces, and replace the raw URL with a route template. Recovery requires active series, churn, and query latency to return toward the pre-release window; a lower sample rate alone would not fix series identity.
Failure modes, trade-offs, and decision rules
The characteristic failure mode is multiplicative growth hidden by a harmless-looking label. Other failures include many short-lived series, a limit that silently drops evidence, an estimate that ignores replicas or tenants, and a permanent exception with no owner or expiry.
The central trade-off is diagnostic segmentation versus shared-platform cost. A bounded label can reveal regional or operation-level impact cheaply; a unique identifier adds perfect event segmentation by converting every event into durable metrics state. Limits protect the platform but may remove useful visibility if the schema and denial path are poorly designed.
| Choice | Use when | Avoid when |
|---|---|---|
| Add a metric label | The value set is bounded, the aggregate query is named, and the complete multiplier fits budget | The value identifies a person, request, order, session, raw route, or exception text |
| Move detail to logs | Individual events need authorized search and retention | Fleet-wide rates or ratios are the primary question |
| Move causality to traces | Request and parent relationships matter | The question is only aggregate prevalence |
| Enforce a tenant limit | One tenant must be contained without harming others and rejection can be visible | The system cannot attribute or explain the denied series |
accept a label only when its operational query, bounded value set, worst-case multiplier, observed load result, owner, and runtime enforcement all fit the tenant budget; otherwise move the detail to logs or traces.
Close the loop
Govern cardinality as Observe → Interpret → Decide → Act → Measure. Observe series count, churn, sample rate, and query latency; interpret the metric revision and label causing growth; decide whether to reject, relabel, roll back, or grant a bounded exception; act on that schema; then measure recovery and tenant isolation.
Run one bounded review and enforcement probe: list each introduced metric and label, state the query, calculate deployment and tenant multipliers, compare representative-load observations with the estimate, and inject one unbounded test label. The falsifiable result passes only if the unsafe series is rejected or removed visibly, its owner receives evidence, checkout queries recover after remediation, and another tenant’s query latency stays within objective.
Key takeaways
Cardinality is a product and reliability constraint, not an after-the-fact storage concern.
- Every unique label set creates a time series.
- Dimensions multiply and scale again across replicas, clusters, and tenants.
- Unique event identity belongs in logs or traces, not metric labels.
- Review-time estimates need runtime series and churn measurements.
- Limits require visible denial evidence and tenant-isolation controls.
Checklist
Use this checklist for every instrumentation revision.
- [ ] Every label supports a named aggregate query.
- [ ] Value sets are bounded or the label is rejected.
- [ ] The estimate includes deployment and tenant multipliers.
- [ ] Raw URLs, IDs, emails, and exception messages are absent from labels.
- [ ] Observed active series and churn match the expected order of magnitude.
- [ ] Limits and remediation are tested with an unaffected tenant control.
Sources
These official references define Prometheus label identity and current cardinality guidance.