Collector Deployment Patterns

Enterprise problem: A collector placed only centrally cannot read node-local evidence, while collectors placed only beside workloads duplicate credentials and policy, so the wrong topology creates blind spots or uncontrolled operational burden.

Whole-course context: The versioned receive-process-export pipeline is the incoming artifact; today decides where its components run and which failure domain each placement owns.

Today’s slice: Compare agent, gateway, and combined topologies, including trace-aware routing, metric single-writer identity, capacity, and degraded operation.

End-of-day evidence: A topology decision record and a resilience drill proving local collection, gateway policy, backend delivery, and bounded behaviour during a gateway failure.

Still unsolved: Prometheus query mechanics, alert routing, cardinality budgets, and the transition from local to distributed metrics storage remain deferred.

Topology is useful only when it preserves evidence and application safety through realistic failures. These cases cover local acquisition and centrally governed export.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D04-UC-01	Cluster platform operator	Collect node-local logs, host metrics, and workload telemetry without granting applications backend credentials	Each node’s agent enriches and forwards authorized evidence with stable resource identity	Loss of one agent affects only its node and produces a visible collection-gap signal
D04-UC-02	Observability platform operator	Enforce regional tenancy, redaction, sampling, and rate policy while scaling ingestion	Gateways distribute load without splitting trace state or creating duplicate metric writers	A failed gateway is drained, traffic recovers, and overload is rejected by tenant policy

“Highly available collectors” is too vague to verify, so these stories name locality, blast radius, and recovery evidence.

Story ID	Use case IDs	User story	Observable acceptance conditions
D04-US-01	D04-UC-01	As a cluster operator, I want one local collector per node, so that file logs and host evidence remain collectable without application privileges	Daemon coverage matches schedulable nodes, metadata is present, and deleting one agent creates only a node-scoped gap
D04-US-02	D04-UC-02	As an observability operator, I want horizontally scaled gateways with signal-aware routing, so that policy remains central without corrupting traces or metrics	Load distribution is visible, all spans for sampled traces meet the routing requirement, and each metric stream has one writer identity

An agent-to-gateway design introduces two queues and two failure boundaries, so its proof must follow an item through both and exercise a failed regional instance.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D04-FLOW-01	D04-UC-01, D04-UC-02	Happy	Operator deploys a workload and starts a topology probe	1. Local agent discovers the workload.<br>2. Collect local evidence.<br>3. Add node and cluster identity.<br>4. Forward to regional endpoint.<br>5. Gateway applies policy and routes.<br>6. Query backend.	Actor, node, cluster, tenant, agent and gateway revisions, signal IDs, backend result, environment, and timestamp
D04-FLOW-02	D04-UC-01, D04-UC-02	Recovery	One regional gateway is terminated during the probe	1. Detect unhealthy endpoint.<br>2. Stop new routing to it.<br>3. Keep bounded agent queues.<br>4. Route to healthy gateways.<br>5. Drain or expire queued items.<br>6. Verify an unaffected node and tenant.	Failover time, queue depth, drop and retry counts, recovered probe, unaffected control, and immutable drill ID

Mixing local acquisition and global policy into one undifferentiated tier makes failure impact unpredictable. Agents own locality; gateways own shared policy and traffic control; backends own durable storage and query.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D04-UC-01	Node agent endpoint and discovery API	DaemonSet or host agent, discovery, metadata processor, bounded sender	Topology inventory owned by cluster platform; signal backends own exported data	Missing-agent heartbeat, discovery gap, local queue saturation, or node-scoped drop counter
D04-UC-02	Regional OTLP endpoint	Load balancer, gateway pool, tenant policy, trace-aware router, exporters	Gateway policy registry owned by observability platform	Endpoint health failure, tenant throttle, split trace, out-of-order metric sample, or exporter failure

Topology state must be reviewable because a missing instance or wrong routing key can silently corrupt evidence. The records below describe desired placement, discovered instances, and resilience proof.

Generated-application database: Not created in this slice — collector topology and evidence records belong to platform control stores, not a generated application.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
TopologyRevision	Topology registry, observability platform	topology_revision_id	Opaque configuration commit	organization_id	Immutable agent, endpoint, gateway, and backend graph	Versioned and retained for audit; tombstoned after retirement policy	D04-UC-01, D04-UC-02
CollectorInstance	Fleet inventory, platform operations	collector_instance_id	topology_revision_id local FK and opaque node or region reference	organization_id	Instance identity and writer identity are globally unique	Created on registration; heartbeat expires; tombstone retained for investigations	D04-UC-01, D04-UC-02
TenantRoute	Policy registry, observability platform	tenant_route_id	topology_revision_id local FK	organization_id	Signal destination, rate limit, and routing key must be explicit	Versioned; revoked independently; retained as policy history	D04-UC-02
ResilienceDrill	Evidence store, reliability engineering	drill_id	Opaque instance, route, and backend references	organization_id	Records precondition, expected result, observed result, and control	Retained through review; raw telemetry expires by backend policy	D04-UC-01, D04-UC-02

Some evidence exists only beside the workload, so an agent runs on each host or node. It can read local files and journals, scrape node-local endpoints, attach node or cluster metadata, and absorb short backend interruptions without teaching the application those details.

In Kubernetes, a DaemonSet commonly places one agent on every eligible node. The blast radius is local: losing the agent on node A should not stop collection on node B. The trade-off is fleet size—configuration rollout, resource budgets, and health monitoring apply to many instances.

Shared policy is difficult to enforce independently in thousands of agents, so a gateway exposes one OTLP endpoint backed by one or more Collector instances. It centralizes credentials, tenant routing, redaction, rate limits, shared enrichment, and processing such as tail sampling.

The gateway is another service and can fail. Scale it horizontally, monitor queue and exporter saturation, define regional failure boundaries, and ensure load balancing matches stateful processing. Tail sampling needs all spans for a trace at the same decision point; use trace-aware routing rather than blind round robin between sampling collectors.

For metrics, OpenTelemetry’s gateway guidance emphasizes the single-writer principle: each metric data stream needs one writer with globally unique identity. Duplicate writers can create gaps, jumps, or out-of-order samples.

Large platforms need both locality and shared policy, so agents forward to a regional gateway tier. The extra hop is justified when it reduces application coupling, centralizes sensitive credentials, and avoids repeating expensive processing everywhere.

Do not add tiers without a named responsibility. A direct workload-to-gateway path can be correct when there is no node-local source or local policy need. An agent-only path can be correct for small environments with trusted backend delivery. Architecture follows evidence locality, policy centralization, scale, and failure objectives.

Healthy averages hide overload, so size each tier from peak items and bytes per second, processor cost, batch size, queue duration, exporter latency, and expected failure time. Watch refused items, accepted items, queue utilization, send failures, process memory, CPU, and time-to-export.

Decide how the workload is protected if every gateway is unavailable. Bounded queues plus explicit drops are safer than unbounded memory. Critical audit evidence may require a separate durable path rather than pretending a general telemetry collector provides guaranteed delivery.

Topology claims remain theoretical until a failure crosses real instances, so conduct a regional canary drill.

1. Verify every expected node has exactly one healthy agent.
2. Send trace, metric, and log probes with known resource identity.
3. Terminate one gateway and observe endpoint health, queue depth, and delivery.
4. Verify trace-aware processing still sees complete traces.
5. Check the backend for duplicate or out-of-order metric samples.
6. Saturate one tenant and prove an unrelated tenant remains within objective.

Collector placement divides locality, policy, and storage responsibilities into explicit failure domains.

• Agents handle node-local acquisition and enrichment.
• Gateways handle shared authentication, routing, sampling, redaction, and traffic policy.
• Combined topology is common but adds capacity and failure-management work.
• Tail sampling needs trace-aware routing; metrics need single-writer identity.
• Bounded failure and tenant-isolation drills are part of the topology design.

Use this checklist to review a collector topology before scaling it.

• [ ] Named every source that requires local access.
• [ ] Assigned each policy to the agent or gateway tier deliberately.
• [ ] Defined gateway load balancing for stateful processors.
• [ ] Guaranteed globally unique metric writer identity.
• [ ] Sized queues and memory for a documented backend outage window.
• [ ] Proved gateway failure and noisy-tenant isolation with immutable drill evidence.

These official references verify current OpenTelemetry deployment guidance.

• OpenTelemetry Collector deployment patterns
• OpenTelemetry gateway deployment pattern
• How Grafana Alloy works