The Cost Model of Each Signal

Enterprise problem: Customers lose diagnostic coverage or face unpredictable bills when teams apply one retention and ingestion policy to metrics, logs, traces, profiles, and queries despite their different cost drivers.

Whole-course context: The incoming topology decision identifies collectors, stores, tenants, and failure domains; this day attaches measurable unit economics and quality constraints to every signal path.

Today’s slice: We model active series, bytes, spans, profile samples, retention, and query work, then govern changes with budgets rather than indiscriminate dropping.

End-of-day evidence: A per-tenant cost ledger and replayable budget experiment show unit cost, retained diagnostic value, limit behavior, and an unaffected control.

Still unsolved: Schema migration and sensitive-data policy remain deferred because cheap telemetry can still be incompatible or unsafe.

Cost reduction without a customer job can erase the only evidence needed during an incident. These cases require predictable spend and preserved diagnostic value.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D19-UC-01	Engineering manager	Attribute observability cost to a team and signal	The monthly forecast explains volume, retention, query, and shared overhead within an agreed error band	A budget breach identifies the dominant dimension and preserves a critical-path positive control
D19-UC-02	On-call engineer	Reduce noisy telemetry without losing rare failures	Sampling and retention changes lower cost while a seeded rare failure remains discoverable	Replay rejects a policy that hides the rare failure and restores the previous revision

“Optimize cost” is not testable unless actors can compare money and diagnostic outcomes under the same workload.

Story ID	Use case IDs	User story	Observable acceptance conditions
D19-US-01	D19-UC-01	As an engineering manager, I want cost attributed by tenant and signal, so that teams can make bounded trade-offs	Forecast and invoice-window measurements name volume, retention, queries, overhead, currency, and error band
D19-US-02	D19-UC-02	As an on-call engineer, I want budget policies tested against seeded incidents, so that savings do not silently remove causal evidence	A versioned replay finds all required incident clues; a deliberately over-aggressive policy fails and rolls back

A cost dashboard alone cannot prove causality, so each flow starts with a policy action and ends with measured spend plus signal quality.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D19-FLOW-01	D19-UC-01	Happy	Manager opens a tenant cost review	1. Ledger reads signal volumes and retention.<br>2. It applies versioned unit rates and shared allocation.<br>3. Forecast is compared with the prior invoice window.<br>4. Dominant drivers and owners are published.	Tenant, signal, measurement window, units, rate revision, forecast, observed cost, error, timestamp, and ledger run ID
D19-FLOW-02	D19-UC-02	Recovery	Operator proposes lower trace sampling and shorter log retention	1. Policy simulator replays a fixed incident corpus.<br>2. It measures stored bytes and query work.<br>3. Required clues are checked.<br>4. A missing clue rejects the policy.<br>5. Previous policy is restored and re-probed.	Policy revisions, corpus hash, expected clues, observed clues, cost delta, rollback result, environment, timestamp, and experiment ID

If accounting and quality checks live in separate systems, savings can be approved without seeing what evidence disappeared. The design joins volume, rates, policies, and replay verdicts through immutable references.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D19-UC-01	Tenant cost-review page	Usage metering, rate catalogue, allocation engine, report publisher	Cost ledger owned by FinOps with immutable raw usage references	Missing tenant attribution, stale rate, forecast error outside band, or unallocated shared cost
D19-UC-02	Budget-policy review action	Policy registry, telemetry collectors, replay runner, query probes	Policy registry owned by observability platform; corpus catalogue owned by reliability engineering	Lost required clue, unexpected cardinality, query-budget breach, or failed rollback probe

Cost claims become irreproducible when unit rates, usage windows, and policy revisions are overwritten. Durable control-plane records preserve the calculation without copying application data.

Generated-application database: Not created in this slice — cost, policy, and replay evidence belong to the observability control plane.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
SignalUsageWindow	Usage ledger, FinOps owner	usage_window_id	Opaque backend usage snapshot reference	tenant_id	Signal, unit, interval, source revision, and measured quantity are immutable	Append per window, retain through billing audit, expire raw detail after aggregate reconciliation	D19-UC-01, D19-UC-02
CostRateRevision	Rate catalogue, FinOps owner	rate_revision_id	None — root rate revision	None — shared rate schedule with explicit allocation scope	Currency, unit, effective interval, and source are required; intervals cannot overlap	Publish immutably, supersede prospectively, retain for financial audit	D19-UC-01, D19-UC-02
BudgetExperiment	Policy registry, platform owner	experiment_id	Opaque usage-window, policy-revision, and corpus references	tenant_id	Same corpus and workload seed are required for before/after comparison	Retain verdict and aggregates; expire replay payload by incident-data policy; delete after governance retention	D19-UC-01, D19-UC-02

One universal “gigabytes” number conceals the multiplier that an engineer can actually control. Use a separate first-order equation for each signal, then validate it against real backend measurements.

Signal	First-order driver	Dangerous multiplier	Quality guardrail
Metrics	Metrics per target × label combinations × targets × scrape frequency × retention	Unbounded label values create new active series	Required service-level indicators and aggregation queries still return valid results
Logs	Bytes per second × retention × replication and storage overhead	Verbose payloads, too many indexed labels, tiny streams, repeated parsing	Seeded error event remains findable by bounded labels and time
Traces	Requests per second × spans per request × retained sampling ratio × bytes per span × retention	Mesh, database, messaging, and application spans multiply per request	Rare error and long-tail traces remain represented by explicit sampling policy
Profiles	Processes × sample frequency × stack depth × profile types × retention	Excess dimensions fragment profiles and queries	Known hot function remains visible within an agreed error tolerance
Queries	Time range × selected data × fan-out × query complexity	Long unbounded scans across tenants or clusters	Query budgets reject or split work without hiding an allowed bounded query

Cheap object storage does not make a 30-day, all-cluster, unbounded-text investigation cheap. Ingestion, retention, indexing, compaction, caching, and query fan-out form one economic system. Prometheus documentation also warns that each unique label set is a new time series, making label cardinality a multiplicative concern rather than a cosmetic naming choice.

A model that only fits last month can still fail after traffic or schema changes, so test both forecast accuracy and diagnostic survival.

1. Freeze a representative workload and incident corpus with immutable hashes.
2. Measure raw accepted, rejected, and dropped telemetry by tenant and signal.
3. Change exactly one variable: label dimension, sampling ratio, retention, profile frequency, or query range.
4. Compare stored volume, compute/query work, and required investigation clues.
5. Run a negative case that deliberately exceeds the limit and a positive control under the limit.
6. Falsify the model if forecast error exceeds the agreed band or if a required clue disappears.
7. Recalibrate rates or reject the policy; never relabel lost evidence as savings.

The right optimization keeps the evidence needed for customer outcomes while removing dimensions that add cost without decision value.

• Active series dominate metric economics; volume and retention dominate logs, traces, and profiles.
• Span count, sampling, stack dimensions, and query fan-out need explicit ownership.
• Storage price is only one term; ingestion and query work matter too.
• Every policy change needs a fixed-corpus diagnostic-quality test and rollback.

A budget is safe only when finance and on-call evidence agree.

• [ ] Usage is measured by tenant, signal, unit, and window.
• [ ] Rate revisions and shared-cost allocation are immutable.
• [ ] Before/after experiments hold workload and incident corpus constant.
• [ ] A rare-failure positive control remains discoverable.
• [ ] Limit denials and rollback results are recorded.

Signal economics and label behavior evolve, so current guidance was checked against official primary documentation.

• Prometheus metric and label naming, including cardinality
• Prometheus instrumentation guidance
• Grafana Cloud guidance on sampling and telemetry-volume cost