Small, Medium, and Large Deployments

Enterprise problem: A team that deploys the most distributed topology before requirements demand it pays an on-call and upgrade tax, while a topology kept too small loses telemetry or misses availability targets as customers grow.

Whole-course context: The incoming component contract identifies each signal owner and edge; this day chooses the smallest deployment shape that satisfies measured volume, retention, tenancy, and failure-isolation requirements.

Today’s slice: We define evidence-based transitions from a single-cluster stack through regional gateways to a multi-region platform, including rollback conditions.

End-of-day evidence: A signed sizing decision records thresholds, load-test results, failure drills, cost, and a reversible migration plan.

Still unsolved: Per-signal unit economics and schema/privacy governance are deferred, although their limits become inputs to sizing.

Deployment labels such as “small” and “large” are subjective unless they resolve customer jobs and recovery expectations. These cases bind topology to observable service outcomes.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D18-UC-01	Service owner	Investigate one-cluster production failures without operating unnecessary distributed backends	Required retention and query latency are met by a simple topology	Peak-load test records accepted data, bounded loss, and successful rollback after a rejected scale-up
D18-UC-02	Observability platform lead	Scale shared telemetry across regions and tenants without a global collection outage	Regional ingestion survives a gateway or region failure within the agreed recovery objective	Fault drill shows isolated backlog, tenant limits, replay, and an unaffected region control

Sizing intent cannot guide an operator during overload, so each story names the customer-visible acceptance evidence.

Story ID	Use case IDs	User story	Observable acceptance conditions
D18-US-01	D18-UC-01	As a service owner, I want the smallest adequate topology, so that telemetry helps incidents without becoming another large platform	A load test meets ingest, query, retention, and recovery objectives; an oversized option is rejected with cost evidence
D18-US-02	D18-UC-02	As a platform lead, I want regional isolation and independent scaling, so that one failure cannot erase every tenant’s evidence	A regional fault causes bounded backlog and replay while the control region continues serving authorized queries

A scaling decision that stops at a diagram cannot be falsified, so the flow begins with measured demand and ends with an observed topology outcome.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D18-FLOW-01	D18-UC-01	Happy	Owner submits measured ingest, retention, and query requirements	1. Planner models headroom.<br>2. Team deploys a monolithic or single-binary candidate.<br>3. Load and query tests run.<br>4. Failure recovery is timed.<br>5. Review accepts the smallest passing shape.	Requirements, topology revision, workload fixture, p95 query result, accepted/rejected counts, recovery time, timestamp, and immutable test run ID
D18-FLOW-02	D18-UC-02	Recovery	Regional gateway or buffer is made unavailable during a controlled drill	1. Edge agents buffer within limits.<br>2. Healthy regions continue.<br>3. Failed region recovers.<br>4. Backlog replays.<br>5. Tenant queries verify completeness and bounded duplicates.	Region, tenant, fault interval, backlog maximum, drops, replay result, unaffected control, environment, timestamp, and drill ID

If scaling components are added without a corresponding flow step, they become permanent complexity without customer evidence. The design separates a local investigation path from a regional resilience path.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D18-UC-01	Sizing review form and test runner	Capacity planner, Alloy or OTel Collector, Prometheus, Loki, Tempo, Grafana	Versioned sizing-decision store owned by the platform team; local signal stores own telemetry	Failed load threshold, query timeout, excess loss, recovery-objective breach, or rejected oversized option
D18-UC-02	Regional fault-drill action	Edge collectors, regional gateway and buffer, distributed backends, tenant query gateway	Regional buffer state and backend object stores owned by their runtime teams	Queue overflow, dropped telemetry, replay gap, cross-tenant response, or healthy-region regression

Scaling from memory or fashion repeats old mistakes, so requirements, measurements, topology revisions, and drills must be durable. No generated customer application data is needed.

Generated-application database: Not created in this slice — control-plane sizing and drill evidence fully represent the customer decision.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
SizingRequirement	Capacity registry, platform capacity owner	requirement_id	None — root requirement revision	organization_id	Units, percentile, time window, headroom, retention, tenancy, and recovery objective are mandatory	Supersede by revision, retain through topology lifetime, delete after audit retention	D18-UC-01, D18-UC-02
TopologyDecision	Architecture registry, platform lead	decision_id	requirement_id local FK	organization_id	Exactly one active decision per environment; acceptance requires a passing run	Activate, supersede, roll back, retain rationale, tombstone after retirement	D18-UC-01, D18-UC-02
CapacityOrFaultRun	Evidence store, reliability engineering owner	run_id	decision_id local FK and opaque workload artifact reference	organization_id	Fixture hash, expected and observed results, environment, timestamps, and positive control are immutable	Retain with decision, expire bulky samples by policy, preserve aggregate verdict	D18-UC-01, D18-UC-02

Teams can mistake scale for maturity, which leads either to needless coordination or to hidden saturation. A deployment envelope is a bounded operating shape with explicit exit criteria, not a prestige tier.

Envelope	Typical shape	Good fit	Exit evidence
Small	Workloads feed Alloy; Prometheus, Loki, and Tempo run locally or in simple modes; Grafana queries them	One cluster, modest retention and volume, no strict multi-tenancy, a small owning team	Sustained capacity, retention, availability, tenant isolation, or recovery requirements fail with headroom applied
Growing	Per-cluster agents feed a regional gateway; shared Mimir, Loki, and Tempo provide central querying	Several clusters, shared standards, longer retention, centralized alerts, and a platform team	Regional blast radius, independent scaling, or tenant governance can no longer meet explicit objectives
Large multi-region	Edge agents and regional gateways buffer into independently scaled, tenant-aware global or federated backends	High ingest, multiple regions, formal compliance, many tenants, dedicated ownership	Exit means decomposition or federation driven by measured limits, not simply “more microservices”

Official documentation currently describes monolithic and microservices modes for Mimir, while Loki documents monolithic and microservices modes and is deprecating its simple-scalable mode before Loki 4.0. That is why a topology decision must pin product versions and migration assumptions rather than copying an old diagram indefinitely.

Without a repeatable experiment, a migration can improve a benchmark while degrading failure recovery or cost. Run the same protocol before scaling up and before scaling back.

1. Capture peak and normal ingest by signal, active series, spans per request, log bytes, query concurrency, and retention.
2. Add documented headroom and define loss, availability, recovery-time, and query-latency objectives.
3. Replay a versioned workload fixture against the smallest candidate topology.
4. Fail one collector, gateway, backend instance, and region separately; measure backlog, drops, replay, and healthy controls.
5. Compare total operational cost, including upgrades, on-call load, and object-store operations.
6. Falsify the candidate if any objective fails twice under the same fixture; add only the boundary that addresses that failure.
7. Keep rollback possible until the new topology passes production-shaped probes for the agreed observation window.

Distributed deployment is a response to measured requirements, not the starting definition of an observability platform.

• Start with the smallest topology that meets explicit service objectives.
• Separate regional collection failures from global storage and query failures only when the requirement warrants it.
• Pin version-specific deployment assumptions because supported modes evolve.
• Make every scale transition reversible and proven by load plus fault evidence.

A topology is ready only when its normal and failure behavior are both measured.

• [ ] Requirements use units, time windows, percentiles, retention, and headroom.
• [ ] The smallest candidate passes ingest, query, and recovery tests.
• [ ] Regional drills preserve an unaffected positive control.
• [ ] Tenant isolation is tested, not inferred from shared infrastructure.
• [ ] Upgrade, rollback, on-call, and object-storage costs are in the decision.

Deployment guidance is version-sensitive, so these statements were verified against official product documentation.

• Grafana Mimir deployment modes
• Grafana Loki deployment modes
• Grafana Alloy edge, gateway, and hybrid patterns