The Telemetry Collection Layer

Enterprise problem: Applications coupled directly to backend addresses, credentials, retries, redaction, and tenancy rules become fragile, and a storage migration then requires unsafe changes across the fleet.

Whole-course context: The incoming signal-selection matrix defines which evidence leaves each workload; today introduces the collection boundary that protects and routes it.

Today’s slice: Design instrumentation, exporters, and receive-process-export pipelines using the OpenTelemetry Collector or Grafana Alloy, including overload and sensitive-data controls.

End-of-day evidence: A versioned collector configuration plus positive and negative pipeline probes showing routing, redaction, batching, and rejection behaviour.

Still unsolved: Agent-versus-gateway placement, backend-specific storage, alert delivery, cardinality budgets, and multi-cluster scale remain deferred.

Without a stable collection contract, every service team must become an expert in every backend and outages propagate into application threads. These use cases separate telemetry production from policy-controlled delivery.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D03-UC-01	Application developer	Emit standard telemetry without embedding backend credentials or retry logic	Valid OTLP telemetry reaches the correct tenant backend with resource identity intact	Invalid or unauthorized telemetry is rejected with attributable collector evidence
D03-UC-02	Observability platform operator	Change routing, redaction, batching, or destination policy centrally	A canary pipeline proves the new revision before fleet rollout	Rollback restores the prior revision while applications continue emitting to the same endpoint

“Install a collector” is not an outcome, so the stories focus on a stable application contract and reviewable policy changes.

Story ID	Use case IDs	User story	Observable acceptance conditions
D03-US-01	D03-UC-01	As an application developer, I want to emit OTLP to a stable endpoint, so that storage changes do not require an application release	A test span and metric reach authorized backends, a sensitive attribute is absent, and bad tenant credentials are denied
D03-US-02	D03-UC-02	As a platform operator, I want collector policy deployed by revision, so that I can canary and roll back routing changes safely	Validation passes, canary probes identify the revision, and rollback restores delivery within the objective

A pipeline can silently drop or leak data between components, so every change needs an observed success path and a contained failure path.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D03-FLOW-01	D03-UC-01, D03-UC-02	Happy	Operator deploys a canary collector revision and developer sends a probe	1. Authenticate the source.<br>2. Receive OTLP.<br>3. enforce memory limits.<br>4. Remove sensitive attributes.<br>5. Batch and export by signal.<br>6. Query the destination.	Actor, tenant, revision, environment, probe IDs, redaction result, backend query result, and timestamps
D03-FLOW-02	D03-UC-01, D03-UC-02	Recovery	Exporter errors exceed the canary threshold	1. Preserve retryable batches within limits.<br>2. Reject new excess load explicitly.<br>3. Record drop and retry counters.<br>4. Roll back the revision.<br>5. Send a fresh probe.	Failed revision, exporter errors, drop reason, rollback event, unaffected tenant control, and successful post-rollback probe

If one opaque collector owns every concern, policy changes are hard to review and overload is hard to localize. The design exposes receiver, processor, exporter, and configuration ownership as a pipeline.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D03-UC-01	Stable OTLP endpoint	OTLP receiver, identity resolver, processors, signal exporters	Signal backends own telemetry; collector emits internal health metrics	Authentication rejection, processor drop counter, exporter failure, or missing destination probe
D03-UC-02	Collector policy release API	Configuration registry, validator, rollout controller, probe runner	Collector configuration registry owned by observability platform	Invalid graph, canary threshold breach, rollout pause, or rollback record

An unversioned configuration cannot explain why data was routed or dropped, so collection policy and rollout evidence must be durable even though telemetry remains in backend stores.

Generated-application database: Not created in this slice — durable collector-policy and rollout records are sufficient; no generated business application is being built.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
CollectorPolicyRevision	Configuration registry, observability platform	policy_revision_id	Opaque source repository commit	organization_id	Immutable content hash; receiver-to-exporter graph must validate	Created on proposal; retained for audit; tombstoned after policy retention	D03-UC-02
SourceGrant	Identity store, platform security	source_grant_id	Opaque workload identity and policy reference	organization_id	Signal, environment, and destination scopes are least privilege	Created by admin; revoked independently; retained as audit metadata	D03-UC-01
RolloutRun	Rollout store, observability platform	rollout_run_id	policy_revision_id local FK	organization_id	Only one active rollout per target fleet; state transitions are append-only	Retained through audit period; expired after fleet revision history compaction	D03-UC-02
PipelineProbe	Evidence store, observability platform	probe_id	rollout_run_id local FK and opaque backend references	organization_id	Must record expected redaction, route, and observed result	Retained with rollout; raw probe telemetry follows backend retention	D03-UC-01, D03-UC-02

Applications cannot be observed unless they produce evidence, but production code should not absorb backend-specific policy. Instrumentation libraries create metrics, logs, spans, and profiles; exporters translate existing systems such as operating-system statistics or database status into a supported telemetry format.

A Prometheus client may expose /metrics. An OpenTelemetry SDK may export spans over OTLP, the OpenTelemetry Protocol. A node exporter translates kernel and hardware measurements into Prometheus exposition format. A log agent reads container standard output or host journals. Automatic instrumentation can cover common frameworks, but semantic names and business boundaries still require deliberate design.

Telemetry needs policy before it crosses a trust boundary, so the Collector organizes components into signal-specific pipelines. Receivers accept protocols, processors transform or protect data, and exporters deliver it.

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    limit_percentage: 80
  attributes/remove-sensitive-data:
    actions:
      - key: user.email
        action: delete
      - key: db.statement
        action: delete
  batch:

exporters:
  otlp/tempo:
    endpoint: tempo:4317
  prometheusremotewrite/mimir:
    endpoint: https://mimir.example.com/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/remove-sensitive-data, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite/mimir]

Component availability depends on the chosen Collector distribution. Validate the actual binary, configuration graph, authentication, transport security, retry queues, and memory limits before rollout.

Collector distributions package different components, so choosing one is a support and capability decision rather than a change to the pipeline model. Grafana Alloy is an OpenTelemetry Collector distribution with built-in Prometheus pipelines and integrations that route metrics to Mimir, logs to Loki, traces to Tempo, and profiles to Pyroscope.

Current Grafana documentation directs Promtail migrations to Alloy, and Loki 3.7 release notes state that Promtail was removed in Loki 3.7.3 after deprecation. A new design should therefore use Alloy or another supported collector instead of introducing Promtail.

Collection and storage remain separate choices. An organization can run upstream OpenTelemetry Collector, Alloy, or another compatible agent while retaining the same backends, and it can change a backend without changing the application’s emission contract.

Collectors can fail while applications remain healthy, so overload behaviour must be designed explicitly. Memory limiters, bounded queues, retries with deadlines, batching, load shedding, and internal telemetry prevent a slow backend from consuming the workload’s resources indefinitely.

Decide which failure mode is acceptable for each signal. Blocking application work to preserve debug logs is usually wrong; silently dropping security-relevant audit events may be unacceptable. Record accepted loss, expose dropped-item counters by reason, and avoid logging the sensitive payload that a processor rejected.

A syntactically valid graph can still route to the wrong tenant or leak an attribute, so probe behaviour at each boundary.

1. Send one uniquely identifiable test item per signal.
2. Confirm resource identity and destination tenant.
3. Confirm sensitive fields are absent at the exporter and backend.
4. Use invalid credentials and verify explicit rejection without cross-tenant delivery.
5. Block one backend, observe retries and bounded memory, then restore it.
6. Roll back the policy revision and prove the stable application endpoint did not change.

The collection layer is the policy and isolation boundary between instrumented software and telemetry storage.

• Instrumentation produces evidence; exporters translate existing systems.
• Collector pipelines receive, process, and export signal-specific data.
• Central redaction, routing, batching, and credentials reduce application coupling.
• Alloy and the upstream Collector share the same conceptual pipeline model.
• Overload and loss policies must protect the running workload and remain observable.

Use this checklist before promoting a collection pipeline.

• [ ] Applications emit to a stable, authenticated endpoint.
• [ ] Every pipeline has explicit receivers, processors, and exporters.
• [ ] Sensitive attributes are removed before leaving the collection boundary.
• [ ] Tenant routing has positive and negative tests.
• [ ] Queues, retry limits, memory limits, and drop evidence are defined.
• [ ] Configuration revisions can be canaried and rolled back.

These official references verify the current Collector and Alloy architecture and the Promtail migration status.

• OpenTelemetry Collector
• OpenTelemetry Collector configuration
• Introduction to Grafana Alloy
• How Grafana Alloy works
• Grafana Loki 3.7 release notes