Security and Privacy Boundaries
Source: Observability Platform source notes, section 21, “Security and Privacy Boundaries”
The enterprise problem and today’s slice
Enterprise problem: Telemetry can capture credentials, identifiers, prompts, database statements, file paths, and customer payloads, so one leaked attribute or over-permissive endpoint can turn the diagnostic platform into a breach path.
Whole-course context: The incoming versioned schema classifies fields and consumers; this day enforces minimization, redaction, tenant identity, endpoint protection, retention, and independently revocable access.
Today’s slice: We place controls at application, collector, storage, and query boundaries and prove both authorized access and denied exfiltration.
End-of-day evidence: A canary-secret exercise, cross-tenant denial, endpoint authorization probe, deletion result, and immutable audit trail demonstrate layered protection.
Still unsolved: Broader architectural anti-patterns and the full implementation sequence remain deferred until these controls are explicit.
The smallest complete model
Telemetry is production data, and observability endpoints can also carry administrative authority. If sensitive evidence enters an over-broad path, storage encryption alone cannot undo collection or prevent an authorized credential from exceeding its intended purpose.
Thesis: Secure observability as minimize data → enforce tenant-scoped access → prove denial and deletion across independently revocable boundaries. Why this matters: prevention, containment, and recovery must still work when any single application, collector, store, or credential fails.
The boundary includes application emission, collector policy, durable stores, query and mutation access, retention, deletion, revocation, and audit. It excludes general organizational identity lifecycle except where a telemetry grant consumes it.
Expand the model one boundary at a time
Begin by preventing unnecessary data at the producer, then enforce declared policy at collection, preserve only tenant-scoped encrypted data, and finally separate query from administrative authority. These are successive controls on the same minimized-data-to-proof model.
Relying on engineers to remember every dangerous field fails under new libraries and incident pressure. Minimize first, then enforce overlapping controls.
| Boundary | Controls | Failure to test |
|---|---|---|
| Application | Data minimization, safe libraries, explicit field classification, consent where required | A forbidden field is emitted by a new code path |
| Collector | Allowlist, attribute deletion or transformation, redaction, tenant enrichment, authentication, limits, backpressure | Canary passes through, safe field is destroyed, or tenant identity is spoofed |
| Storage | Encryption, tenant isolation, retention, backend limits, backups, deletion | Cross-tenant read, expired data remains, or backup deletion diverges |
| Query and administration | Identity-aware proxy, least privilege, bounded queries, separate mutation permissions, auditing | A query credential creates a silence or reads another tenant |
OpenTelemetry’s official guidance says implementers remain responsible for deciding what is sensitive; processors can remove, filter, redact, or transform attributes but cannot determine business sensitivity automatically. Prometheus documents strong endpoint assumptions: monitoring endpoints should not be public, and access to Alertmanager can permit creating or deleting silences and creating or resolving alerts. Treat observability administration as privileged control-plane authority, not as a harmless dashboard permission.
Each boundary transforms or constrains data differently, so its interface and failure must be testable in isolation.
| Boundary | Purpose, inputs, and transformation | Output or interface | Scaling constraint and failure mode | Alternatives, use when, and avoid when |
|---|---|---|---|---|
| Application → collector | Emit only classified diagnostic fields and suppress known secrets at their source | Structured telemetry with privacy class and safe correlation fields | Number of code paths and libraries; a new path can emit a forbidden value | Use typed safe-logging and instrumentation libraries; avoid deny lists as the only control because unknown fields bypass them |
| Collector policy | Authenticate, allowlist, delete, redact, transform, enrich tenant identity, and enforce limits | Accepted records plus rejection, drop, and policy counters | Throughput and processor correctness; overload or bad rules can leak a canary or destroy safe evidence | Use a collector as a second independently owned boundary; avoid assuming it can infer business sensitivity |
| Store and retention | Encrypt, isolate tenants, retain by classification, and delete across hot data, objects, exports, and backups | Tenant-scoped query API and deletion evidence | Retained bytes and copy count; backup divergence or stale derived data survives deletion | Use separate retention classes and tested deletion; avoid one global retention period for unlike risk |
| Query and administration | Authenticate actor, authorize resource/action/scope/purpose, audit, and revoke independently | Bounded query or mutation decision with audit ID | Policy evaluation, credential scope, and audit volume; query authority may accidentally permit Alertmanager mutation | Use separate grants for read and mutation; avoid public endpoints, shared administrator tokens, or tenant identity supplied only by the client |
Run the model through one incident
The general rule is to trace sensitive data and authority through every boundary, using synthetic canaries rather than real secrets. A simple example emits a fake token and a safe trace reference: the token must disappear before durable storage while the trace reference remains queryable.
In a recurring checkout incident, a fixture carries a canary payment token, tenant ID, and safe trace ID through logs and spans. The collector records the active privacy-policy revision, removes the token, preserves correlation, and routes only to the intended tenant stores. An authorized responder can query the bounded checkout trace, but a cross-tenant request and a protected Alertmanager silence mutation are denied. After revocation, the formerly valid query must also fail. Observed evidence includes canary absence in hot storage, object storage, derived data, exports, and backups; safe-field presence; denial reasons; revocation version; and immutable audit IDs.
The incident succeeds only if both halves hold: forbidden data is absent and necessary evidence remains available to the right actor.
Failure modes, trade-offs, and decision rules
A common failure mode is relying on one control—application hygiene, collector redaction, storage encryption, or a perimeter proxy—and treating its configuration as proof. The governing trade-off is diagnostic richness and responder speed versus data minimization, narrower authority, processing cost, and deletion complexity.
collect and retain a field only when a named operational decision needs it, protect it at two independently owned boundaries where practical, and require canary absence plus a safe-field positive control before activating the policy.
Controls can pass configuration review while leaking in another signal or backup, so probe every durable destination and access path.
- Create synthetic canaries for token, personal identifier, prompt fragment, and safe correlation value; never use real secrets.
- Send the fixture through metrics, logs, and traces where the type permits.
- Query hot storage, long-term object storage, derived metrics, exports, and backups for forbidden canaries and safe controls.
- Use an authorized tenant token, then test cross-tenant read, excessive time range, protected mutation, and expired token.
- Revoke the grant independently and retry the formerly authorized request.
- Verify audit events contain actor, resource, scope, precondition, expected, observed, environment, timestamp, and immutable ID without copying the secret.
- Falsify the design if any forbidden canary persists, safe control disappears, mutation crosses authority, or revocation is delayed beyond policy.
Use allowlists when the permitted telemetry contract is small and stable; use targeted transformation for fields whose safe representation remains useful. Avoid collecting a sensitive field merely because redaction might remove it later, and avoid granting administrative mutation to credentials intended only for incident queries.
Close the loop
Security work must end in measured containment, not a policy document. Apply Observe → Interpret → Decide → Act → Measure to a bounded data class or access grant.
Observe a checkout canary appearing at collector input or a cross-tenant request reaching the proxy. Interpret which boundary should suppress the data or authority. Decide on one policy revision or independent grant revocation. Act by activating the versioned rule with a rollback target. Measure canary absence across every durable copy, safe trace-ID presence for the authorized tenant, denial of the unauthorized request, and post-revocation failure within the stated objective. Any surviving canary or delayed denial falsifies the control and keeps rollout blocked.
Key takeaways
Observability data is sensitive production data and its endpoints can carry administrative power.
- Prevent collection where possible; redact at the collector as a second boundary.
- Tenant identity, access, retention, encryption, scanning, and audit are independent controls.
- Collector processors enforce declared policy but cannot infer business sensitivity.
- Query and mutation authority must be separately scoped and revocable.
Checklist
Security is ready only when denials and deletion are observed, not merely configured.
- [ ] Synthetic canaries are absent from every durable and derived store.
- [ ] Safe diagnostic controls remain queryable.
- [ ] Cross-tenant reads and protected mutations are denied.
- [ ] Grant revocation is independently verified.
- [ ] Retention and deletion include backups and object storage.
Sources
Security behavior is high-stakes and version-sensitive, so claims were verified against official primary documentation.