The Telemetry Collection Layer
Source: “Building an Observability Platform: From Prometheus to Mimir, Loki, Tempo, and Grafana” — Chapter 3, “The Collection Layer: How Telemetry Leaves the System”
The enterprise problem and today’s slice
Enterprise problem: Applications coupled directly to backend addresses, credentials, retries, redaction, and tenancy rules become fragile, and a storage migration then requires unsafe changes across the fleet.
Whole-course context: The incoming signal-selection matrix defines which evidence leaves each workload; today introduces the collection boundary that protects and routes it.
Today’s slice: Design instrumentation, exporters, and receive-process-export pipelines using stable OpenTelemetry Protocol (OTLP) endpoints, the OpenTelemetry Collector, or Grafana Alloy, including overload and sensitive-data controls.
End-of-day evidence: A versioned collector configuration plus positive and negative pipeline probes showing routing, redaction, batching, and rejection behaviour.
Still unsolved: Agent-versus-gateway placement, backend-specific storage, alert delivery, cardinality budgets, and multi-cluster scale remain deferred.
The smallest complete model
Applications need a stable way to emit telemetry even when storage, credentials, and routing policy change. Without a collection boundary, every backend migration becomes an application migration and every backend outage can consume application resources.
Thesis: A telemetry collector decouples evidence production from evidence storage by receiving a stable protocol, applying centralized safety policy, and exporting to authorized backends. Why this matters: application teams keep one emission contract while platform operators can change routing, redaction, batching, and destinations independently.
The boundary includes instrumentation and exporters that produce supported telemetry, the collector policy path, and backend delivery. It excludes the backend’s internal storage architecture and the physical placement of agents and gateways, which is a separate topology decision.
Expand the model one boundary at a time
Expand the collector in dependency order: a source first produces a supported signal, a receiver authenticates and decodes it, processors protect and shape it, and an exporter sends it to a destination. The backend remains downstream rather than leaking its policy into application code.
An OTLP endpoint can accept spans and metrics from an OpenTelemetry software development kit (SDK). A Prometheus client can expose /metrics; a node exporter translates kernel and hardware measurements into Prometheus format; and a log agent reads container output or host journals. Automatic instrumentation covers common frameworks, but service semantics and business boundaries still need deliberate names.
| Component | Purpose and inputs | Transformation, output, and interface | Scaling constraint and failure mode | Alternatives, use when, avoid when |
|---|---|---|---|---|
| Instrumentation or source exporter | Turns application operations or existing system state into supported telemetry | Emits OTLP, Prometheus exposition, structured logs, or another declared contract | Application CPU and latency are hard limits; missing semantic identity makes data ambiguous | Use SDKs for application semantics and exporters for existing systems; avoid backend credentials and retry policy in business code |
| Receiver | Authenticates and decodes a protocol at a stable endpoint | Produces accepted telemetry for one or more pipelines | Connection count and bytes per second limit capacity; bad identity can cause rejection or cross-tenant routing | Use explicit authenticated endpoints; avoid accepting unscoped sources |
| Processor | Limits memory, removes sensitive attributes, enriches, samples, batches, or routes | Emits protected, policy-compliant items to exporters | CPU, memory, queue depth, and statefulness constrain scale; wrong ordering can leak data or drop context | Use central processors for shared policy; avoid transformations whose loss semantics are unknown |
| Exporter | Encodes and sends batches to one authorized backend | Uses backend APIs such as OTLP or Prometheus remote write and reports delivery state | Backend latency, retry capacity, and network bandwidth constrain throughput; failure can delay or drop data | Use bounded queues and retries for transient faults; avoid unbounded retry or application blocking |
The same model is visible in a concrete configuration. This example limits memory before redaction and batching, then routes traces and metrics to separate backends.
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
limit_percentage: 80
attributes/remove-sensitive-data:
actions:
- key: user.email
action: delete
- key: db.statement
action: delete
batch:
exporters:
otlp/tempo:
endpoint: tempo:4317
prometheusremotewrite/mimir:
endpoint: https://mimir.example.com/api/v1/push
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, attributes/remove-sensitive-data, batch]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheusremotewrite/mimir]
Component availability depends on the chosen distribution, so validate the actual binary, configuration graph, authentication, transport security, retry queues, and limits. Grafana Alloy is an OpenTelemetry Collector distribution with built-in Prometheus pipelines and integrations for Mimir, Loki, Tempo, and Pyroscope. Current Grafana documentation directs Promtail migrations to Alloy, and Loki 3.7 release notes state that Promtail was removed in Loki 3.7.3 after deprecation. Use Alloy or another supported collector for new designs rather than introducing Promtail.
Collection and storage are separate choices. Upstream OpenTelemetry Collector, Alloy, or another compatible distribution can retain the same application contract while backends change.
Run the model through one incident
The general rule is to inspect a telemetry path at each interface instead of treating “data missing” as one undifferentiated symptom. A simple probe sends one known item, records its source identity and policy revision, then checks receiver acceptance, processor effects, exporter delivery, and backend queryability.
During a recurring checkout telemetry pipeline incident, spans disappear after a policy rollout while application requests continue normally. Receiver counters show accepted OTLP, processor counters show a sharp increase in dropped items, and the backend receives no probe. The canary evidence points to the new redaction or routing graph rather than the checkout service. The operator rolls back the collector revision; the application keeps emitting to the same endpoint; a fresh probe becomes queryable with sensitive fields absent.
If the backend is slow instead, exporter failures, queue utilization, retry age, and drop reason provide different observed evidence. Block the backend deliberately: bounded queues may preserve a short interruption, then explicit loss protects the workload. Silently consuming unbounded memory would turn an observability failure into a checkout failure.
Failure modes, trade-offs, and decision rules
Collectors sit on a trust and availability boundary, so the most dangerous failure mode is invisible corruption: wrong-tenant delivery, sensitive-data leakage, silent drops, or an overloaded collector consuming workload resources. Syntactic validation alone cannot detect these outcomes.
The core trade-off is central policy consistency versus another operational dependency. Central redaction, credentials, routing, and batching reduce application coupling, but collector CPU, memory, queues, and rollout safety must be operated explicitly. Different signals also need different loss choices: blocking application work to preserve debug logs is usually wrong, while silently dropping security-relevant audit events may be unacceptable.
| Choice | Use when | Avoid when |
|---|---|---|
| Upstream Collector | Its supported components and release model meet the required protocol and policy set | Required integrations or operational support are absent from the chosen distribution |
| Grafana Alloy | Prometheus and OpenTelemetry pipelines plus Grafana integrations fit the platform | The required component set has not been validated in the deployed binary |
| Retry with a bounded queue | A backend fault is transient and delayed delivery still has value | Queue growth threatens application resources or the retry deadline exceeds the evidence value |
| Drop or shed load visibly | Protecting the running workload is more important than preserving that signal | The signal has a stronger delivery requirement that needs a separate durable path |
place policy in the collector when it must be consistent across producers and can be tested at the receive, process, and export interfaces; keep it out when the collector cannot provide the required loss, isolation, or lifecycle guarantee.
Close the loop
Operate the collection boundary as Observe → Interpret → Decide → Act → Measure. Observe receiver, processor, queue, exporter, and backend evidence; interpret the first interface where expected state diverges; decide whether to contain load or roll back policy; act on one revision; then measure both application health and end-to-end telemetry delivery.
Run one bounded canary with a uniquely identifiable item per signal. The falsifiable probe passes only when resource identity and tenant are correct, sensitive fields are absent at exporter and backend, invalid credentials are rejected without cross-tenant delivery, blocked-backend memory stays within the declared limit, and rollback restores a fresh probe without changing the application endpoint.
Key takeaways
The collection layer is the policy and isolation boundary between instrumented software and telemetry storage.
- Instrumentation produces evidence; exporters translate existing systems.
- Collector pipelines receive, process, and export signal-specific data.
- Central redaction, routing, batching, and credentials reduce application coupling.
- Alloy and the upstream Collector share the same conceptual pipeline model.
- Overload and loss policies must protect the running workload and remain observable.
Checklist
Use this checklist before promoting a collection pipeline.
- [ ] Applications emit to a stable, authenticated endpoint.
- [ ] Every pipeline has explicit receivers, processors, and exporters.
- [ ] Sensitive attributes are removed before leaving the collection boundary.
- [ ] Tenant routing has positive and negative tests.
- [ ] Queues, retry limits, memory limits, and drop evidence are defined.
- [ ] Configuration revisions can be canaried and rolled back.
Sources
These official references verify the current Collector and Alloy architecture and the Promtail migration status.