Metric Cardinality as a Platform Constraint

Enterprise problem: Every unique metric label combination consumes memory, CPU, disk, network, and query capacity, so one unbounded identifier can destabilize the shared metrics platform and delay incident response for every team.

Whole-course context: The working Prometheus queries and notification routes are incoming evidence; today governs the series identities those queries and rules depend on.

Today’s slice: Model cardinality multiplication, set per-metric and per-tenant budgets, move event identity to logs or traces, and contain unsafe instrumentation before ingestion.

End-of-day evidence: A cardinality review with estimated and observed series counts, rejected dangerous labels, approved alternatives, and a budget-enforcement drill with an unaffected tenant control.

Still unsolved: Central multi-cluster retention, distributed ingestion and query scaling, and the architecture beyond local Prometheus remain deferred.

Teams need useful dimensions, but an unrestricted label API transfers the cost and failure risk to everyone sharing the backend. These cases preserve operational questions while enforcing economic limits.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D07-UC-01	Service developer	Add a metric dimension that supports a named aggregate question	Review estimates its value set and combinations, then accepts the bounded label or directs detail to another signal	Labels such as customer ID, request ID, raw URL, or exception text are rejected with a replacement design
D07-UC-02	Metrics platform operator	Keep tenant series growth within capacity without hiding useful service health	Budgets, dashboards, and limits identify growth before platform saturation	A noisy tenant is contained and an unrelated tenant remains queryable within objective

“Low cardinality” is not measurable, so stories require bounded value sets, observed counts, and a safe denial path.

Story ID	Use case IDs	User story	Observable acceptance conditions
D07-US-01	D07-UC-01	As a service developer, I want a label review before release, so that my metric answers an aggregate question without creating a series per event	The review records dimension bounds, product estimate, owner, query examples, and alternative signal for rejected detail
D07-US-02	D07-UC-02	As a platform operator, I want per-tenant series budgets and growth alerts, so that one service cannot exhaust shared metrics capacity	The drill crosses one tenant’s threshold, blocks or drops unsafe series visibly, and leaves the control tenant healthy

Cardinality damage often appears only after deployment, so the flows combine pre-release estimation with observed production enforcement.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D07-FLOW-01	D07-UC-01	Happy	Developer proposes a failure_category label	1. Name the operational query.<br>2. Enumerate bounded values.<br>3. Multiply dimensions and deployment scope.<br>4. Compare with budget.<br>5. Run load test.<br>6. approve revision.	Actor, metric revision, value bounds, estimated and observed series, queries, budget, environment, timestamp, and approval ID
D07-FLOW-02	D07-UC-01	Denied	Developer proposes customer_id as a metric label	1. Calculate worst-case combinations.<br>2. Reject the label.<br>3. preserve customer ID in authorized logs or traces.<br>4. replace with bounded customer tier if needed.<br>5. re-run review.	Rejection reason, estimate, replacement schema, access boundary, revised series count, and review ID
D07-FLOW-03	D07-UC-02	Recovery	Tenant active-series count exceeds its budget	1. Identify new series source.<br>2. enforce tenant limit.<br>3. notify owner.<br>4. roll back instrumentation or relabel.<br>5. verify queries recover.<br>6. probe another tenant.	Limit event, offending label set, remediation, recovery time, control-tenant latency, and immutable incident ID

If enforcement exists only in documentation, one accidental deploy can create millions of series. The design joins review-time estimates, runtime observation, tenant limits, and reversible remediation.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D07-UC-01	Instrumentation review workflow	Metric catalogue, cardinality estimator, schema validator, load-test runner	Metric catalogue owned by metrics platform	Unbounded value set, budget excess, missing query, or observed count beyond estimate
D07-UC-02	Tenant capacity dashboard and limit API	Series analyzer, ingestion limiter, relabel controller, notification service	Tenant budget store owned by metrics platform	Limit rejection, active-series spike, query latency breach, or control-tenant impact

Budgets cannot be enforced consistently if metric schemas and exceptions live in chat messages. Durable revisions tie each label decision to an owner, limit, and lifecycle.

Generated-application database: Not created in this slice — metric governance and evidence live in platform control stores; no application-domain database is required.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
MetricSchemaRevision	Metric catalogue, metrics platform	metric_schema_revision_id	Opaque service and source commit references	organization_id	Metric name, type, unit, label names, bounded values, and owner are immutable per revision	Versioned through service life; tombstoned after retention and query migration	D07-UC-01
CardinalityReview	Governance store, metrics platform	cardinality_review_id	metric_schema_revision_id local FK	organization_id	Named query, formula, estimate, observed test count, and decision are required	Retained with schema history; raw load-test samples expire separately	D07-UC-01
TenantSeriesBudget	Capacity store, metrics platform	tenant_budget_id	Opaque plan or capacity revision	organization_id	One active budget per tenant and environment; exceptions require expiry	Versioned; superseded budgets retained for audit then expired	D07-UC-02
LimitEvent	Ingestion evidence store, metrics platform	limit_event_id	Opaque tenant budget and series fingerprint references	organization_id	Reason, measured count, threshold, and disposition are append-only	Retained through incident review; detailed label samples redacted and expired	D07-UC-02

Label costs are easy to underestimate because dimensions multiply rather than add. For one metric with 30 services, 4 environments, 5 regions, 200 routes, 10 status codes, and 6 methods, the theoretical upper bound is:

30 × 4 × 5 × 200 × 10 × 6 = 7,200,000 series

Adding 100,000 customer IDs does not add 100,000 series; it can multiply the existing space by 100,000. Real combinations may be sparse, but a safety review cannot assume sparsity without observed evidence and a reason it remains bounded.

Series cost is broader than storage. Each active series requires ingestion bookkeeping, memory for current chunks and indexes, network transfer, compaction work, and query matching. High churn—many short-lived label values—adds further pressure even when the simultaneous count seems tolerable.

Metrics become unsafe when they carry event identity, so divide evidence by representation.

Evidence	Correct home	Reason
Service, environment, region, operation, status class	Metric labels	Bounded dimensions support aggregation
User, email, order, request, session, raw URL	Structured logs with access controls	Individual event identity is searchable without one series per value
Request and parent relationship	Trace context	Preserves distributed causality
Function and stack resource attribution	Profiles	Maps cost to code locations

Replace raw routes such as /orders/8128 with templates such as /orders/{order_id}. Replace exception messages with a bounded failure category while keeping the detailed exception in logs.

Review alone cannot stop runtime churn, so govern at several layers. SDK views can drop unsafe attributes. Collector processors or Prometheus relabeling can remove known hazards. Ingestion systems can enforce active-series and rate limits per tenant. Platform dashboards should show top metrics and labels by series count, churn, sample volume, and growth.

Avoid silent correction. When a label is dropped or a tenant is limited, emit attributable evidence and notify the owner. A hidden drop can make a dashboard look healthy while discarding failures.

A temporary exception needs scope, owner, justification, capacity impact, and expiry. Permanent exemptions become undocumented platform architecture.

A static estimate can miss generated values, so combine code review with representative load.

1. List every metric and label introduced by the revision.
2. State the operational query each label enables.
3. Bound values per deployment unit and multiply combinations.
4. Include replicas, clusters, regions, and tenant count.
5. Run representative traffic and compare observed series with the estimate.
6. Inject an unbounded test label and prove the policy rejects or removes it visibly.
7. Verify another tenant’s query latency remains within objective.

Cardinality is a product and reliability constraint, not an after-the-fact storage concern.

• Every unique label set creates a time series.
• Dimensions multiply and scale again across replicas, clusters, and tenants.
• Unique event identity belongs in logs or traces, not metric labels.
• Review-time estimates need runtime series and churn measurements.
• Limits require visible denial evidence and tenant-isolation controls.

Use this checklist for every instrumentation revision.

• [ ] Every label supports a named aggregate query.
• [ ] Value sets are bounded or the label is rejected.
• [ ] The estimate includes deployment and tenant multipliers.
• [ ] Raw URLs, IDs, emails, and exception messages are absent from labels.
• [ ] Observed active series and churn match the expected order of magnitude.
• [ ] Limits and remediation are tested with an unaffected tenant control.

These official references define Prometheus label identity and current cardinality guidance.

• Prometheus metric and label naming
• Prometheus instrumentation practices
• OpenTelemetry metric cardinality limits