Security and Privacy Boundaries

Enterprise problem: Telemetry can capture credentials, identifiers, prompts, database statements, file paths, and customer payloads, so one leaked attribute or over-permissive endpoint can turn the diagnostic platform into a breach path.

Whole-course context: The incoming versioned schema classifies fields and consumers; this day enforces minimization, redaction, tenant identity, endpoint protection, retention, and independently revocable access.

Today’s slice: We place controls at application, collector, storage, and query boundaries and prove both authorized access and denied exfiltration.

End-of-day evidence: A canary-secret exercise, cross-tenant denial, endpoint authorization probe, deletion result, and immutable audit trail demonstrate layered protection.

Still unsolved: Broader architectural anti-patterns and the full implementation sequence remain deferred until these controls are explicit.

Security controls that are not tied to real access paths can leave high-risk telemetry untouched. These cases cover data entry and data retrieval.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D21-UC-01	Application and platform engineer	Keep secrets and unnecessary personal data out of durable telemetry	A seeded canary is removed before storage while approved diagnostic fields remain	Scanner finds no canary in any backend and records the collector action plus an unaffected safe-field control
D21-UC-02	Authorized incident responder	Query only the tenant and fields permitted for an incident	An authorized bounded query succeeds and is audited	Cross-tenant, mutating, or over-broad access is denied; token revocation blocks the next request

Policy prose alone cannot prove containment, so the stories require observed positive and negative paths at separate boundaries.

Story ID	Use case IDs	User story	Observable acceptance conditions
D21-US-01	D21-UC-01	As an application and platform engineer, I want layered minimization and redaction, so that one missed safe-logging rule does not persist a secret	Canary appears at controlled input, is absent from every durable backend, and a safe correlation field remains queryable
D21-US-02	D21-UC-02	As an incident responder, I want least-privilege telemetry access, so that investigation authority cannot become tenant-wide administration	Authorized query, cross-tenant denial, protected mutation denial, audit event, and post-revocation denial are observed

A checkbox cannot show where sensitive data was removed or where access was denied. These flows preserve actor, resource, scope, and immutable evidence at each boundary.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D21-FLOW-01	D21-UC-01	Happy	Engineer emits a fixture containing a canary secret and allowed trace reference	1. Safe library suppresses known fields.<br>2. Collector allowlist and redaction process remaining data.<br>3. Tenant identity and limits are applied.<br>4. Backends ingest accepted fields.<br>5. Scanner queries every store for the canary and control.	Actor, fixture hash, policy revision, expected absence/presence, observed backend results, environment, timestamp, trace ID, and scan ID
D21-FLOW-02	D21-UC-02	Denied	Responder token requests another tenant and then attempts a protected Alertmanager mutation	1. Query proxy authenticates token.<br>2. Tenant and scope checks reject the query.<br>3. Mutation gateway rejects the silence action.<br>4. Token is revoked.<br>5. An allowed control query then fails as expected after revocation.	Subject, resource, requested scopes, denials, prior authorized control, revocation version, environment, timestamp, and audit event IDs

One perimeter control is insufficient because applications, collectors, stores, and query endpoints fail differently. Defense in depth keeps each boundary independently testable and revocable.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D21-UC-01	Telemetry fixture submission and privacy scan	Safe instrumentation library, OTel Collector or Alloy processors, tenant router, backend scanners	Versioned privacy-policy registry owned by security; backends own retained telemetry	Canary found, redaction error, unclassified attribute, dropped safe control, or limit breach
D21-UC-02	Identity-aware query and mutation proxies	Identity provider, authorization policy engine, query proxy, Alertmanager gateway, audit service	Access-policy registry and append-only audit store owned by security	Cross-tenant result, protected mutation accepted, missing audit event, or revoked token still accepted

Security evidence becomes unverifiable when policy revisions and access decisions are mutable. Control-plane records retain policy and audit facts without copying sensitive payloads.

Generated-application database: Not created in this slice — privacy policies, grants, scans, and audit events are observability control-plane state.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
PrivacyPolicyRevision	Security policy registry, privacy engineering owner	policy_revision_id	None — immutable root revision	organization_id	Allowed, transformed, forbidden, and retention classes are explicit; published revisions are immutable	Draft, approve, activate, supersede, retain for audit, delete only after regulatory retention	D21-UC-01, D21-UC-02
TelemetryAccessGrant	Identity store, security owner	grant_id	Opaque subject and policy-revision references	tenant_id	Resource, action, scope, purpose, expiry, and independent revocation are mandatory	Issue, expire or revoke, retain audit metadata, purge token material immediately	D21-UC-01, D21-UC-02
SecurityProbeRun	Evidence store, security validation owner	probe_run_id	Opaque fixture, grant, policy, and backend references	tenant_id	Expected and observed positive/negative results, environment, timestamp, and artifact hash are immutable	Retain verdict to compliance policy, expire canary payload promptly, preserve non-sensitive audit facts	D21-UC-01, D21-UC-02

Relying on engineers to remember every dangerous field fails under new libraries and incident pressure. Minimize first, then enforce overlapping controls.

Boundary	Controls	Failure to test
Application	Data minimization, safe libraries, explicit field classification, consent where required	A forbidden field is emitted by a new code path
Collector	Allowlist, attribute deletion or transformation, redaction, tenant enrichment, authentication, limits, backpressure	Canary passes through, safe field is destroyed, or tenant identity is spoofed
Storage	Encryption, tenant isolation, retention, backend limits, backups, deletion	Cross-tenant read, expired data remains, or backup deletion diverges
Query and administration	Identity-aware proxy, least privilege, bounded queries, separate mutation permissions, auditing	A query credential creates a silence or reads another tenant

OpenTelemetry’s official guidance says implementers remain responsible for deciding what is sensitive; processors can remove, filter, redact, or transform attributes but cannot determine business sensitivity automatically. Prometheus documents strong endpoint assumptions: monitoring endpoints should not be public, and access to Alertmanager can permit creating or deleting silences and creating or resolving alerts. Treat observability administration as privileged control-plane authority, not as a harmless dashboard permission.

Controls can pass configuration review while leaking in another signal or backup, so probe every durable destination and access path.

1. Create synthetic canaries for token, personal identifier, prompt fragment, and safe correlation value; never use real secrets.
2. Send the fixture through metrics, logs, and traces where the type permits.
3. Query hot storage, long-term object storage, derived metrics, exports, and backups for forbidden canaries and safe controls.
4. Use an authorized tenant token, then test cross-tenant read, excessive time range, protected mutation, and expired token.
5. Revoke the grant independently and retry the formerly authorized request.
6. Verify audit events contain actor, resource, scope, precondition, expected, observed, environment, timestamp, and immutable ID without copying the secret.
7. Falsify the design if any forbidden canary persists, safe control disappears, mutation crosses authority, or revocation is delayed beyond policy.

Observability data is sensitive production data and its endpoints can carry administrative power.

• Prevent collection where possible; redact at the collector as a second boundary.
• Tenant identity, access, retention, encryption, scanning, and audit are independent controls.
• Collector processors enforce declared policy but cannot infer business sensitivity.
• Query and mutation authority must be separately scoped and revocable.

Security is ready only when denials and deletion are observed, not merely configured.

• [ ] Synthetic canaries are absent from every durable and derived store.
• [ ] Safe diagnostic controls remain queryable.
• [ ] Cross-tenant reads and protected mutations are denied.
• [ ] Grant revocation is independently verified.
• [ ] Retention and deletion include backups and object storage.

Security behavior is high-stakes and version-sensitive, so claims were verified against official primary documentation.

• OpenTelemetry guidance for handling sensitive data
• OpenTelemetry Collector processor catalogue
• Prometheus security model, including Alertmanager authority