Pyroscope: Continuous Profiles as Code-Level Evidence
Source: Observability Platform source — section 13, “Profiles: Pyroscope as the Code-Level Signal”
The enterprise problem and today’s slice
Enterprise problem: Metrics show a service consumes more CPU and traces show slow requests, but neither identifies which functions allocate memory, contend on locks, or burn processor time across the fleet.
Whole-course context: The incoming evidence is a connected slow checkout trace whose payment span dominates latency; today tests whether local code resource use explains that symptom or can be ruled out.
Today’s slice: Collect, retain, compare, and query continuous profiles in Pyroscope while controlling overhead, labels, tenant access, and source-code symbol exposure.
End-of-day evidence: Produce a bounded before/after flame graph with a named hot function, or a falsifiable result showing CPU remained normal during the slow-trace window.
Still unsolved: Unified Grafana navigation, cross-signal identity governance, and the complete platform operating architecture remain deferred.
The smallest complete model
A slow checkout trace says where time passed, but not which local code consumed runtime resources during the same window. A profile can answer that question only when its type, load, and time interval match the hypothesis.
Thesis: continuous profiling compares aggregated stack samples for one bounded service and workload, not individual-request latency. Why this matters: a wide flame-graph frame is evidence about sampled resource use, not automatic proof of the incident’s cause.
This smallest model contains a runtime, samples with declared units, and one comparable query. It is complete enough to reject an invalid CPU or allocation conclusion before adding distributed storage or trace links.
Choosing the wrong profile type answers a different question and can send remediation toward unrelated code. CPU profiles estimate where on-CPU samples occur; allocation profiles attribute allocation activity; heap profiles represent retained memory; mutex, block, wall-time, thread, or goroutine profiles depend on language/runtime support.
A wide flame box means more samples in that stack, not necessarily slow wall-clock latency for one request. Validate units, runtime support, sampling interval, symbolization, and load shape before comparing widths.
Expand the model one boundary at a time
Expand the same runtime–samples–comparison model in dependency order: select a supported profiler, attach bounded resource identity, ingest typed samples, store profile blocks and metadata, then compare matched windows. Add trace-to-profile navigation only after direct profile queries preserve tenant, service, environment, and version.
| Boundary | Purpose, inputs, and transformation | Output or interface | Scaling constraint and failure mode | Alternative guidance |
|---|---|---|---|---|
| Runtime profiler | Sample supported CPU, allocation, heap, mutex, block, wall-time, thread, or goroutine state | Typed stack samples with known units | Application overhead, missing symbols, or unsupported runtime mode | Use the lowest-overhead type that answers the hypothesis; avoid enabling every type without a budget |
| Identity and export | Add stable service, environment, version, tenant, and profile type | Bounded profile series | Cardinality from instance/request labels or credential failure | Use stable deployment dimensions; avoid request IDs in the main series identity |
| Pyroscope v2 storage | Write profile data to object storage and coordinate metadata/index and compaction through the metastore | Retained profile blocks and queryable metadata | Object-store latency, metastore saturation, or compaction backlog | Use distributed v2 when scale and recovery objectives demand it; avoid obsolete v1 storage assumptions |
| Query and comparison | Select equal load, time, type, service, environment, and versions | Flame graph or differential profile | Incomparable windows or broad selectors | Use matched baseline/candidate comparisons; avoid declaring causality from one unbounded flame graph |
Architecture guidance that assumes v1 ingester disks is now stale for new deployments. Pyroscope v2 writes profile data directly to object storage and uses a metastore as the stateful metadata/index and compaction coordinator; consult the current deployment and migration documentation for production topology.
Query by stable, bounded labels such as service, environment, and version. Keep instance IDs and request IDs out of the main series identity unless a measured use case and cardinality budget justify them.
Run the model through one incident
The general rule is to compare like with like and let the profile falsify as well as support a local-code hypothesis. A simple example compares checkout CPU profiles before and after one release under identical request rate. A realistic recurring checkout incident starts from a slow payment span, follows the telemetry pipeline to a profile for the same service and time, and checks whether local CPU or allocation actually changed.
Comparing different traffic invalidates the conclusion, so hold request rate, endpoint mix, duration, environment, and sampling policy constant. Capture a baseline, deploy the new release, repeat the same load, and compare.
For trace-to-profile navigation, configure the Tempo data source to derive a Pyroscope query from span/resource attributes. Verify that a selected checkout span opens the same service, environment, profile type, and time interval; a link that silently broadens the service or tenant is a correlation defect.
The profile may falsify a local-code hypothesis. If the slow payment trace coincides with normal checkout CPU and allocation profiles, investigate the upstream wait, network, retry, or provider rather than optimizing an unrelated local function. Record query selectors, absolute windows, load shape, profile units, overhead, version, trace-derived context, and result digest as the observed evidence.
Failure modes, trade-offs, and decision rules
The most common failure mode is comparing unmatched workloads or profile types and then treating a wider frame as causal proof. The central trade-off is richer or more frequent sampling versus runtime overhead, storage volume, series cardinality, and symbol exposure.
choose one profile type from the resource hypothesis, hold workload and scope constant, and accept a conclusion only when the matched comparison exceeds a declared effect threshold without breaching the overhead budget. Use CPU profiling for on-CPU work, allocation for allocation activity, and heap for retained memory. Avoid profiling modes unsupported by the runtime, high-cardinality labels without a measured need, and trace links that broaden tenant or service scope.
Scale object storage, the metastore, compaction, or queries only when the matching latency, backlog, or saturation evidence identifies that boundary. A missing result caused by export or symbolization is not a storage-capacity signal.
Close the loop
The profiling loop is Observe → Interpret → Decide → Act → Measure. Observe profile freshness, units, symbols, overhead, workload, and matched differences; interpret whether the evidence supports local resource pressure; decide on code, runtime, or upstream investigation; act on one bounded release or fixture; then measure the same profile and customer symptom again.
For the next action, run one fixed checkout load for ten minutes on baseline and candidate versions using one approved CPU profile policy. The falsifiable probe succeeds only if query scope and load match, profiler overhead stays within budget, the targeted stack’s sample share changes by the declared threshold, and checkout latency changes in the predicted direction; otherwise reject the local-code hypothesis.
Key takeaways
Profiles complement request traces by aggregating where code spends resources over time. Keep these distinctions:
- Traces explain one distributed request; profiles aggregate code-level resource use across a window.
- Comparable flame graphs require equal load, labels, units, and collection policy.
- A normal profile is useful negative evidence that can rule out local CPU or allocation regressions.
- Collection overhead, symbol access, tenant isolation, and freshness require explicit tests.
- New deployments should follow current Pyroscope v2 architecture documentation.
Checklist
A flame graph screenshot alone is not reproducible evidence. Complete these checks:
- [ ] Chose and documented the profile type, unit, profiler, and sampling interval.
- [ ] Measured profiler CPU and memory overhead against an objective.
- [ ] Compared equal pre/post windows under controlled traffic.
- [ ] Named a function-level delta or documented a falsified local-code hypothesis.
- [ ] Proved trace-to-profile mapping preserves service, tenant, and time.
- [ ] Proved tenant denial, freshness alerting, retention, and deletion.
Sources
Pyroscope v2 changes storage and component responsibilities, so current primary documentation is authoritative. These sources support the lesson: