Collector Deployment Patterns
Source: “Building an Observability Platform: From Prometheus to Mimir, Loki, Tempo, and Grafana” — Chapter 4, “Two Collector Deployment Patterns”
The enterprise problem and today’s slice
Enterprise problem: A collector placed only centrally cannot read node-local evidence, while collectors placed only beside workloads duplicate credentials and policy, so the wrong topology creates blind spots or uncontrolled operational burden.
Whole-course context: The versioned receive-process-export pipeline is the incoming artifact; today decides where its components run and which failure domain each placement owns.
Today’s slice: Compare agent, gateway, and combined topologies, including trace-aware routing, metric single-writer identity, capacity, and degraded operation.
End-of-day evidence: A topology decision record and a resilience drill proving local collection, gateway policy, backend delivery, and bounded behaviour during a gateway failure.
Still unsolved: Prometheus query mechanics, alert routing, cardinality budgets, and the transition from local to distributed metrics storage remain deferred.
The smallest complete model
Collector placement decides which evidence can be reached, which policy can be shared, and how far a failure spreads. One universal placement cannot satisfy local file access and regional policy without making one of those responsibilities implicit.
Thesis: Place collection beside the evidence that requires locality and place shared policy in a scalable gateway, connecting the two only when both responsibilities are needed. Why this matters: topology then follows evidence access and failure boundaries rather than a fashionable number of tiers.
This smallest model asks where the collector must run to acquire evidence and which failure domain owns delivery. It does not yet assume an agent, gateway, or combined topology; that choice follows from locality, central policy, scale, and resilience objectives.
Expand the model one boundary at a time
Expand placement in dependency order. First add an agent when evidence is node-local; then add a gateway when many sources need shared authentication, tenancy, redaction, sampling, or rate policy; finally preserve the backend as a separate durable boundary.
| Placement | Purpose, input, and transformation | Output and interface | Scaling constraint and failure mode | Alternatives, use when, avoid when |
|---|---|---|---|---|
| Agent | Reads local files and journals, scrapes node endpoints, receives workload telemetry, and adds node or cluster identity | Forwards enriched evidence through a bounded sender | Instance count grows with nodes; losing one agent creates a node-scoped gap | Use when evidence or metadata requires host locality; avoid a fleet when no local source or policy justifies it |
| Gateway | Receives many sources and centralizes credentials, tenant routing, redaction, rate limits, enrichment, and stateful processing | Exposes a shared OTLP endpoint and exports by policy | Peak items, bytes, processor state, queue depth, and exporter latency constrain scale; a regional failure can affect many workloads | Use for shared policy and horizontal traffic control; avoid a central hop whose only responsibility is “forwarding” |
| Combined path | Agents acquire locally and gateways enforce regional policy | Adds two explicit queues and failure boundaries before storage | Extra latency, capacity, rollout, and failure-management work must be justified | Use when both locality and central policy are required; avoid tiers without a named responsibility |
Kubernetes commonly realizes the agent tier as a DaemonSet on each eligible node. That makes the blast radius local but turns rollout, resource budgets, and health monitoring into fleet operations. A direct workload-to-gateway path can be correct when there is no local source, while an agent-only path can be correct for a small environment with trusted backend delivery.
Stateful processing tightens the interface. Tail sampling needs all spans for a trace at the same decision point, so use trace-aware routing rather than blind round robin. For metrics, OpenTelemetry gateway guidance emphasizes the single-writer principle: each metric data stream needs one writer with globally unique identity, because duplicate writers can create gaps, jumps, or out-of-order samples.
Run the model through one incident
The general rule is to locate the failed placement boundary before scaling or restarting everything. A simple example removes one node agent: only that node should lose local evidence, other nodes should continue, and the inventory should expose the exact coverage gap.
In a recurring checkout telemetry path incident, one regional gateway is terminated while agents continue receiving spans, metrics, and logs. Endpoint health drains the failed instance; bounded agent queues absorb the interruption; healthy gateways resume delivery. Trace-aware routing still assembles complete sampled traces, metric writer identity remains unique, and the checkout probe appears in the backend with the expected node, cluster, tenant, and policy revision.
Observed evidence distinguishes topology failures. Missing-agent heartbeat plus a node-scoped gap points to local acquisition. Rising queues across many agents plus exporter failures at the gateway points to regional delivery. Duplicate or out-of-order metric samples suggest writer identity or routing corruption, while incomplete traces suggest stateful traffic was split.
Failure modes, trade-offs, and decision rules
Topology makes failure domains explicit, but each added tier creates another failure mode: local evidence gaps, gateway overload, split trace state, duplicate metric writers, wrong-tenant routing, or queues that consume workload resources during backend failure.
The central trade-off is locality versus policy concentration. Agents reach evidence and isolate failures but multiply fleet operations. Gateways centralize sensitive credentials and expensive policy but create shared capacity and availability responsibilities. The combined topology gains both properties at the cost of another hop and two queues.
| Constraint | Use when | Avoid when |
|---|---|---|
| Agent-only | Sources are local, the environment is bounded, and direct backend delivery is trusted | Central tenancy, shared stateful processing, or credential isolation is required |
| Gateway-only | Workloads can reach a stable endpoint and no host-local source is needed | File, journal, or node evidence requires local access |
| Agent plus gateway | Local acquisition and regional governance are both named requirements | The second tier has no measurable policy, resilience, or cost benefit |
| Separate durable audit path | Loss is unacceptable and delivery guarantees exceed general telemetry semantics | Ordinary diagnostic data can tolerate bounded, visible loss |
add a collector tier only when it owns a distinct locality, policy, or failure-isolation responsibility, and accept it only after peak capacity and the loss behaviour of every queue are measurable.
Close the loop
Operate topology through Observe → Interpret → Decide → Act → Measure. Observe coverage, accepted and refused items, queue utilization, send failures, process memory, CPU, and time to export; interpret which placement boundary diverged; decide whether to drain, reroute, shed, or roll back; act on one failure domain; and measure backend completeness plus application safety.
Run one bounded regional canary: verify exactly one healthy agent per expected node, send known trace, metric, and log probes, terminate one gateway, and saturate only one tenant. The probe is falsifiable: it passes only if traffic recovers within the stated objective, trace-aware processing keeps traces complete, no duplicate or out-of-order metric samples appear, agent memory remains bounded, and an unrelated tenant stays within its query and delivery objective.
Key takeaways
Collector placement divides locality, policy, and storage responsibilities into explicit failure domains.
- Agents handle node-local acquisition and enrichment.
- Gateways handle shared authentication, routing, sampling, redaction, and traffic policy.
- Combined topology is common but adds capacity and failure-management work.
- Tail sampling needs trace-aware routing; metrics need single-writer identity.
- Bounded failure and tenant-isolation drills are part of the topology design.
Checklist
Use this checklist to review a collector topology before scaling it.
- [ ] Named every source that requires local access.
- [ ] Assigned each policy to the agent or gateway tier deliberately.
- [ ] Defined gateway load balancing for stateful processors.
- [ ] Guaranteed globally unique metric writer identity.
- [ ] Sized queues and memory for a documented backend outage window.
- [ ] Proved gateway failure and noisy-tenant isolation with immutable drill evidence.
Sources
These official references verify current OpenTelemetry deployment guidance.