Pyroscope: Continuous Profiles as Code-Level Evidence

Enterprise problem: Metrics show a service consumes more CPU and traces show slow requests, but neither identifies which functions allocate memory, contend on locks, or burn processor time across the fleet.

Whole-course context: The incoming evidence is a connected slow checkout trace whose payment span dominates latency; today tests whether local code resource use explains that symptom or can be ruled out.

Today’s slice: Collect, retain, compare, and query continuous profiles in Pyroscope while controlling overhead, labels, tenant access, and source-code symbol exposure.

End-of-day evidence: Produce a bounded before/after flame graph with a named hot function, or a falsifiable result showing CPU remained normal during the slow-trace window.

Still unsolved: Unified Grafana navigation, cross-signal identity governance, and the complete platform operating architecture remain deferred.

Profiles without a hypothesis become attractive pictures that do not guide action. These use cases require a code-level explanation or an explicit negative result tied to a known service window.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D13-UC-01	Performance engineer	Identify which checkout functions changed CPU or allocation cost after a release	A comparison profile attributes a material delta to named functions and deployment versions	A rollback or fixed build removes the delta; a normal profile can falsify local CPU as the cause
D13-UC-02	Observability platform operator	Provide low-overhead tenant-safe profiling with governed symbols and retention	Overhead stays within budget and authorized service/version queries return profiles	Unauthorized tenant/service access is denied and collector failure is detected with an unaffected control

“Profiling enabled” does not prove useful resolution, acceptable overhead, or safe access. These stories demand measured deltas and negative-path evidence.

Story ID	Use case IDs	User story	Observable acceptance conditions
D13-US-01	D13-UC-01	As a performance engineer, I want to compare checkout profiles before and after a release, so that I can name the function responsible for changed resource use	Identical time windows and load controls show function-level delta, version labels, sample counts, and a repeatable fix or falsifier
D13-US-02	D13-UC-02	As a platform operator, I want profile collection bounded by overhead and access policy, so that production diagnosis does not create a new reliability or code-disclosure risk	CPU/memory overhead remains within objective, tenant denial succeeds, ingestion loss is alerted, and authorized queries remain available

A flame graph can mislead when workloads or time windows differ, so the flow preserves comparable inputs from collection through remediation. The terminal evidence includes both the visible stack delta and the controlled conditions.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D13-FLOW-01	D13-UC-01, D13-UC-02	Happy	Engineer selects checkout CPU profiles for equal pre/post-release windows	1. Language agent or eBPF collector samples stacks.<br>2. Collector attaches canonical service/environment/version labels.<br>3. Pyroscope validates tenant and persists profile data.<br>4. Query frontend resolves the two windows.<br>5. Grafana renders comparison flame graphs.<br>6. Engineer verifies the hot function under controlled load.	Query, windows, labels, sample counts, flame-graph delta, overhead, environment, and run ID
D13-FLOW-02	D13-UC-01, D13-UC-02	Recovery	Collector stops or a foreign tenant requests checkout profiles	1. Ingest heartbeat or profile freshness fails.<br>2. Alert identifies the collection boundary.<br>3. Foreign request is denied.<br>4. Collector restarts and profile freshness recovers.<br>5. Authorized comparison repeats.	Gap interval, denial response, recovered freshness, unaffected service metrics, and run ID

Profiling agents run near production code, so collection overhead and symbol exposure are trust-boundary concerns rather than UI details. The design separates application sampling, collection policy, Pyroscope storage/query, and operator access.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D13-UC-01	Grafana Pyroscope data source	Query frontend, query backend, metastore, object-store readers, flame-graph renderer	Pyroscope object storage and metadata index	Empty comparison, mismatched labels/windows, missing symbols, query error, or non-repeatable delta
D13-UC-02	Profiling agent/export endpoint	Language profiler or eBPF collector, Alloy, Pyroscope distributor/write path, tenant auth, freshness monitor	Versioned collection policy, Pyroscope object storage, and evidence store	Refused profiles, stale freshness, overhead breach, tenant denial failure, or object-store error

Profile stacks can reveal proprietary symbol names and code paths, so ownership and deletion must be explicit. Pyroscope owns telemetry copies; the source repository and application runtime retain authority over code and business data.

Generated-application database: Not created in this slice — profiles, metadata, policies, and proof runs are sufficient telemetry state and must not become a business-data store.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
Profiling-policy revision	Git/config store owned by platform operations	profile_policy_sha	None — root policy revision has no parent	organization_id	Approved profiler, interval, labels, symbol policy, and overhead budget	Review, deploy, roll back, retain audit history, then expire	D13-UC-02
Profile series	Pyroscope metadata/object storage owned by profiling platform	tenant_id/profile_type/label_fingerprint/time	Opaque service and deployment references	tenant_id	Bounded labels; service/environment/version semantics remain stable	Ingest, compact, retain, tombstone, and delete by tenant policy	D13-UC-01, D13-UC-02
Stack sample aggregate	Pyroscope profile block owned by profiling platform	profile_block_id/stack_fingerprint	Foreign reference to profile series	tenant_id	Counts preserve stack ancestry and profile units	Merge into blocks, retain with series, delete with profile retention	D13-UC-01
Profile proof run	Platform evidence store owned by operations	profile_run_id	Opaque references to policy, queries, versions, and result digest	organization_id	Comparable windows/load plus denial and overhead observations are immutable	Create per experiment, retain for audit objective, then expire	D13-UC-01, D13-UC-02

Choosing the wrong profile type answers a different question and can send remediation toward unrelated code. CPU profiles estimate where on-CPU samples occur; allocation profiles attribute allocation activity; heap profiles represent retained memory; mutex, block, wall-time, thread, or goroutine profiles depend on language/runtime support.

A wide flame box means more samples in that stack, not necessarily slow wall-clock latency for one request. Validate units, runtime support, sampling interval, symbolization, and load shape before comparing widths.

Architecture guidance that assumes v1 ingester disks is now stale for new deployments. Pyroscope v2 writes profile data directly to object storage and uses a metastore as the stateful metadata/index and compaction coordinator; consult the current deployment and migration documentation for production topology.

Query by stable, bounded labels such as service, environment, and version. Keep instance IDs and request IDs out of the main series identity unless a measured use case and cardinality budget justify them.

Comparing different traffic invalidates the conclusion, so hold request rate, endpoint mix, duration, environment, and sampling policy constant. Capture a baseline, deploy the new release, repeat the same load, and compare.

For trace-to-profile navigation, configure the Tempo data source to derive a Pyroscope query from span/resource attributes. Verify that a selected checkout span opens the same service, environment, profile type, and time interval; a link that silently broadens the service or tenant is a correlation defect.

The profile may falsify a local-code hypothesis. If the slow payment trace coincides with normal checkout CPU and allocation profiles, investigate the upstream wait, network, retry, or provider rather than optimizing an unrelated local function.

Profiles complement request traces by aggregating where code spends resources over time. Keep these distinctions:

• Traces explain one distributed request; profiles aggregate code-level resource use across a window.
• Comparable flame graphs require equal load, labels, units, and collection policy.
• A normal profile is useful negative evidence that can rule out local CPU or allocation regressions.
• Collection overhead, symbol access, tenant isolation, and freshness require explicit tests.
• New deployments should follow current Pyroscope v2 architecture documentation.

A flame graph screenshot alone is not reproducible evidence. Complete these checks:

• [ ] Chose and documented the profile type, unit, profiler, and sampling interval.
• [ ] Measured profiler CPU and memory overhead against an objective.
• [ ] Compared equal pre/post windows under controlled traffic.
• [ ] Named a function-level delta or documented a falsified local-code hypothesis.
• [ ] Proved trace-to-profile mapping preserves service, tenant, and time.
• [ ] Proved tenant denial, freshness alerting, retention, and deletion.

Pyroscope v2 changes storage and component responsibilities, so current primary documentation is authoritative. These sources support the lesson:

• Grafana Pyroscope v2 architecture
• Grafana Pyroscope v2 metastore
• Grafana Pyroscope profile types
• Grafana: Configure traces to profiles