The Cost Model of Each Signal
Source: Observability Platform source notes, section 19, “The Cost Model of Each Signal”
The enterprise problem and today’s slice
Enterprise problem: Customers lose diagnostic coverage or face unpredictable bills when teams apply one retention and ingestion policy to metrics, logs, traces, profiles, and queries despite their different cost drivers.
Whole-course context: The incoming topology decision identifies collectors, stores, tenants, and failure domains; this day attaches measurable unit economics and quality constraints to every signal path.
Today’s slice: We model active series, bytes, spans, profile samples, retention, and query work, then govern changes with budgets rather than indiscriminate dropping.
End-of-day evidence: A per-tenant cost ledger and replayable budget experiment show unit cost, retained diagnostic value, limit behavior, and an unaffected control.
Still unsolved: Schema migration and sensitive-data policy remain deferred because cheap telemetry can still be incompatible or unsafe.
The smallest complete model
Telemetry cost is not a storage bill; it is measured demand transformed by a policy and judged against diagnostic value. Cutting volume without the last box can make the platform cheaper precisely by making incidents unanswerable.
Thesis: Model each signal as measured usage → versioned budget policy → preserved decision evidence, because metrics, logs, traces, profiles, and queries have different multipliers. Why this matters: a team can reduce spend safely only when the same change also proves that required incident clues survive.
This model includes ingestion, retention, storage work, query work, tenant allocation, and diagnostic guardrails. It excludes invoice-provider details that do not change which engineering dimension drives cost.
Expand the model one boundary at a time
First measure signal-native units, then apply immutable rate and allocation rules, then test one candidate policy against a fixed incident corpus. The expanded nouns remain the same model: usage becomes per-signal windows, policy becomes rates plus limits, and proof becomes cost reconciliation plus investigation replay.
One universal “gigabytes” number conceals the multiplier that an engineer can actually control. Use a separate first-order equation for each signal, then validate it against real backend measurements.
| Signal | First-order driver | Dangerous multiplier | Quality guardrail |
|---|---|---|---|
| Metrics | Metrics per target × label combinations × targets × scrape frequency × retention | Unbounded label values create new active series | Required service-level indicators and aggregation queries still return valid results |
| Logs | Bytes per second × retention × replication and storage overhead | Verbose payloads, too many indexed labels, tiny streams, repeated parsing | Seeded error event remains findable by bounded labels and time |
| Traces | Requests per second × spans per request × retained sampling ratio × bytes per span × retention | Mesh, database, messaging, and application spans multiply per request | Rare error and long-tail traces remain represented by explicit sampling policy |
| Profiles | Processes × sample frequency × stack depth × profile types × retention | Excess dimensions fragment profiles and queries | Known hot function remains visible within an agreed error tolerance |
| Queries | Time range × selected data × fan-out × query complexity | Long unbounded scans across tenants or clusters | Query budgets reject or split work without hiding an allowed bounded query |
Cheap object storage does not make a 30-day, all-cluster, unbounded-text investigation cheap. Ingestion, retention, indexing, compaction, caching, and query fan-out form one economic system. Prometheus documentation also warns that each unique label set is a new time series, making label cardinality a multiplicative concern rather than a cosmetic naming choice.
The interfaces and constraints differ enough that each signal needs explicit selection guidance.
| Boundary | Purpose, inputs, and transformation | Output or interface | Scaling constraint and failure mode | Alternatives, use when, and avoid when |
|---|---|---|---|---|
| Producers → usage meter | Count accepted, rejected, and dropped records in signal-native units by tenant and window | Immutable usage windows | Attribution gaps and late data; missing tenant identity creates unallocated cost | Use backend and collector measurements together; avoid estimating only from application request volume |
| Usage → cost ledger | Join quantities to effective rate revisions and shared-cost allocation | Forecast, observed cost, error band, and dominant driver | Rate drift and allocation rules; overwritten rates make history irreproducible | Use immutable effective-dated rates; avoid one blended gigabyte rate for unlike signals |
| Policy → telemetry path | Apply bounded labels, sampling, retention, profile frequency, and query limits | Accepted telemetry plus policy decision counters | Cardinality, bytes, spans, samples, and fan-out; aggressive policy removes rare evidence | Use the cheapest control that targets the dominant multiplier; avoid random dropping without a quality objective |
| Policy → replay | Run the same seeded incident corpus before and after the change | Required-clue verdict and rollback evidence | Corpus representativeness; stale fixtures can approve harmful savings | Use when changing any material budget control; avoid claiming savings from volume alone |
Run the model through one incident
The general rule is to hold workload and evidence expectations constant while changing one cost variable. In a simple example, remove one unbounded metric label and confirm active series fall while the required service-level indicator still answers the same query.
In a recurring checkout incident, the candidate policy lowers trace sampling and shortens verbose-log retention. A fixed replay includes a common success, a rare payment failure, and a slow downstream provider. The team observes stored bytes, active series, spans retained, query work, forecast cost, and whether the metric anomaly still links to the rare error trace and structured log. If the rare failure disappears, the cost reduction fails even if the invoice forecast improves. Restoring the previous policy and re-running the exact trace and log queries provides recovery evidence rather than an assumption.
The same reasoning applies to a telemetry pipeline: find the multiplier, change one control, and require both financial and diagnostic outcomes under identical inputs.
Failure modes, trade-offs, and decision rules
The common failure mode is optimizing the easiest visible number—usually stored bytes—while cardinality, ingest work, query fan-out, or lost clues remain hidden. The essential trade-off is lower ongoing spend versus the probability and time cost of missing decisive evidence during a rare incident.
change one dominant signal multiplier at a time, approve it only when forecast error stays inside the agreed band and every required clue in the fixed corpus remains discoverable, and roll back on either failure.
A model that only fits last month can still fail after traffic or schema changes, so test both forecast accuracy and diagnostic survival.
- Freeze a representative workload and incident corpus with immutable hashes.
- Measure raw accepted, rejected, and dropped telemetry by tenant and signal.
- Change exactly one variable: label dimension, sampling ratio, retention, profile frequency, or query range.
- Compare stored volume, compute/query work, and required investigation clues.
- Run a negative case that deliberately exceeds the limit and a positive control under the limit.
- Falsify the model if forecast error exceeds the agreed band or if a required clue disappears.
- Recalibrate rates or reject the policy; never relabel lost evidence as savings.
Use aggregation for repeated metric questions, targeted structured logs for discrete events, explicit trace sampling for causality, bounded profile dimensions for code attribution, and query limits for fan-out. Avoid applying one retention or drop percentage uniformly: equal percentages do not preserve equal decision value across signals.
Close the loop
Cost governance is complete only when savings and investigation quality are measured together. Apply Observe → Interpret → Decide → Act → Measure for every budget revision.
Observe that checkout trace bytes dominate one tenant while the monthly forecast exceeds its band. Interpret whether span count, bytes per span, retained ratio, or query work is the true multiplier. Decide on one bounded sampling or attribute change. Act through a versioned policy with an immediate rollback target. Measure both the cost delta and discovery of the seeded rare failure across the next representative window. If cost falls but the rare trace or its correlation disappears—or if forecast accuracy does not improve—the experiment is falsified and the previous revision must be restored.
Key takeaways
The right optimization keeps the evidence needed for customer outcomes while removing dimensions that add cost without decision value.
- Active series dominate metric economics; volume and retention dominate logs, traces, and profiles.
- Span count, sampling, stack dimensions, and query fan-out need explicit ownership.
- Storage price is only one term; ingestion and query work matter too.
- Every policy change needs a fixed-corpus diagnostic-quality test and rollback.
Checklist
A budget is safe only when finance and on-call evidence agree.
- [ ] Usage is measured by tenant, signal, unit, and window.
- [ ] Rate revisions and shared-cost allocation are immutable.
- [ ] Before/after experiments hold workload and incident corpus constant.
- [ ] A rare-failure positive control remains discoverable.
- [ ] Limit denials and rollback results are recorded.
Sources
Signal economics and label behavior evolve, so current guidance was checked against official primary documentation.