Common Architectural Mistakes
Source: Observability Platform source notes, section 22, “Common Architectural Mistakes”
The enterprise problem and today’s slice
Enterprise problem: Teams can buy and deploy every observability component yet still miss customer failures because dashboards lead instrumentation, logs substitute for metrics, high-cardinality IDs are indexed, alerts describe components, and the telemetry platform is not monitored.
Whole-course context: The incoming component, scale, cost, schema, and privacy contracts provide reviewable constraints; this day uses them to detect recurring design mistakes before production incidents expose them.
Today’s slice: We turn nine anti-patterns into operational questions, policy checks, and controlled failure exercises.
End-of-day evidence: An architecture review and seeded-incident run identify each applicable mistake, reject one unsafe design, and preserve one customer-outcome positive control.
Still unsolved: The ordered implementation program and final behavior-to-action mental model remain for the last two days.
The smallest complete model
Most observability mistakes begin when a visible artifact is built before the operational question it must answer. The result is more telemetry and more dashboards without a dependable path to a customer decision.
Thesis: Review every design as operational question → required evidence → outcome-based action, and reject any component that cannot defend its place in that chain. Why this matters: this three-part model exposes dashboard-first work, interchangeable-signal thinking, unsafe cardinality, unjustified scale, and unmonitored pipelines before they become incident failures.
The model covers artifact design, signal choice, labels, coverage, alert intent, deployment scale, and the telemetry path’s own health. It does not prescribe a particular product when several can preserve the same evidence contract.
Expand the model one boundary at a time
Expand the evidence box backward to owned instrumentation and forward through specialized storage, bounded query, and an outcome projection. Then add meta-monitoring around that same path so its disappearance becomes observable before the customer evidence is exhausted.
Similar symptoms can have different causes, so name the mistaken decision and the corrective mechanism rather than merely adding more telemetry.
| Mistake | Consequence | Better design |
|---|---|---|
| Start with dashboards | Teams search for data to justify a projection | Start with an operational question, then derive signal, instrumentation, store, query, and projection |
| Treat logs as universal | Repeated parsing is expensive and semantics drift | Use counters and histograms for repeated numeric questions; logs explain discrete events |
| Treat traces as expensive logs | Parent-child causality, timing, and boundaries are lost | Preserve context propagation, span relationships, status, and causal structure |
| Index trace IDs as labels | Near-unique values explode cardinality | Keep trace IDs as correlation metadata or structured fields, not indexed metric/log labels |
| Scale metrics before mastering semantics | Distributed storage amplifies bad names and series | Prove local metric meaning, labels, ownership, and rules before introducing Mimir |
| Instrument everything equally | Critical paths drown in low-value volume | Prioritize customer, revenue, security, irreversible, expensive, and newly risky paths |
| Ignore platform health | Telemetry disappears without warning | Monitor acceptance/rejection, queues, drops, object storage, queries, compaction, rules, notifications, and limits |
| Alert on components | Harmless restarts page people while customer loss is missed | Alert on user-visible failure, latency, capacity, backlog, data loss, and security outcomes; diagnose with components |
| Build distributed everything on day one | Capacity, upgrades, caching, recovery, security, and on-call complexity arrive before value | Purchase complexity only after simpler systems fail explicit requirements |
The mistakes can be grouped by the boundary they corrupt, which makes their constraints and alternatives easier to apply.
| Boundary | Purpose, inputs, and transformation | Output or interface | Scaling constraint and failure mode | Alternatives, use when, and avoid when |
|---|---|---|---|---|
| Question → signal | Translate a customer or system decision into the minimum evidence needed | Owned metric, log, trace, or profile contract | Number of questions and critical paths; dashboard-first work creates orphan artifacts | Use metrics for repeated numeric questions, logs for discrete events, traces for causality, and profiles for code cost; avoid treating any signal as universal |
| Signal → storage | Preserve the signal’s query shape with bounded dimensions and retention | Specialized query API | Cardinality, bytes, spans, samples, and retention; trace IDs as labels create near-unique series | Use structured fields for correlation identifiers; avoid indexing unbounded IDs or scaling storage before semantics are proven |
| Query → projection | Turn bounded queries into an investigation view or threatened-outcome condition | Dashboard, rule, and owner/runbook link | Query cost and human attention; component alerts page without customer risk | Use outcome alerts for paging and component evidence for diagnosis; avoid publishing artifacts without a question and owner |
| Telemetry path → meta-monitoring | Measure acceptance, rejection, queues, drops, storage, compaction, queries, rules, and notifications | Evidence-loss risk alert and drill record | Pipeline volume and buffer duration; silent export failure exhausts recoverable evidence | Use independent health controls and a healthy tenant path; avoid relying on the broken pipeline as the only source of its health proof |
Run the model through one incident
The general rule is to walk backward from the decision to the evidence that must support it, then forward through the live path to find the first broken contract. A simple example rejects a CPU-restart page that names no threatened customer outcome, while retaining CPU telemetry as diagnostic context.
In a realistic telemetry pipeline incident, an export path pauses while checkout traffic continues. Queue depth, accepted and rejected counts, and drop risk change before buffers exhaust. An outcome-risk rule pages the platform on-call once; component detail identifies the paused exporter; an unaffected tenant query remains the positive control. After export resumes, the team observes backlog drain, bounded loss or completeness, query recovery, rule evaluation, and notification delivery. If the platform produces no advance evidence, or if the alert merely says “collector restarted,” the design has reproduced the mistake rather than corrected it.
The same chain reviews product artifacts: a dashboard is accepted only when its query answers a named operational question and supports a bounded action.
Failure modes, trade-offs, and decision rules
The recurring failure mode is substituting a product, signal, or projection for an evidence-to-action contract. The important trade-off is broad telemetry coverage and architectural flexibility versus semantic quality, cardinality, cost, human attention, and operational complexity.
approve a signal, dashboard, alert, or topology component only when the team can name the customer decision it supports, the bounded evidence it consumes, its owner, its failure observation, and a positive control; otherwise remove or defer it.
Checklists can be gamed with plausible prose, so challenge each design with a seeded counterexample.
- Choose one important customer outcome and state the decision an operator must make.
- Walk backward from action through alert or dashboard, query, store, signal, and instrumentation.
- Calculate worst-case label cardinality and reject identifiers with unbounded values.
- Remove one component signal; confirm the outcome rule still describes customer risk while diagnostic detail narrows.
- Pause one export path and observe accepted, rejected, queued, dropped, query, rule, and notification behavior.
- Keep one unaffected tenant or signal path as a positive control.
- Falsify the design if a projection has no question, a page has no threatened outcome, or platform data can disappear without advance evidence.
Use a compact review gate before publication and recurring failure drills after publication. Avoid solving a semantic or sequencing mistake by purchasing more storage or introducing more components; those alternatives multiply the original defect.
Close the loop
An architectural correction is complete only when a real operational outcome improves. Apply Observe → Interpret → Decide → Act → Measure to one artifact or pipeline boundary.
Observe that a checkout dashboard has no actionable question or that the export queue approaches exhaustion without an outcome alert. Interpret the missing link in the question-to-evidence-to-action chain. Decide to remove the orphan artifact or add one bounded evidence-loss rule. Act through a reviewed revision with the existing healthy path as a control. Measure whether the seeded incident now produces one timely, owned action while harmless component events remain silent. If notification timing, backlog recovery, or the customer query does not improve, the correction is falsified.
Key takeaways
Most observability failures are sequencing and ownership failures rather than missing products.
- Questions precede signals, and signals precede dashboards.
- Each signal has a distinct job; correlation does not make them interchangeable.
- Outcome alerts page people; component telemetry explains causes.
- The observability platform needs its own loss, backlog, query, rule, and notification evidence.
Checklist
Use these checks to stop common mistakes before a production rollout.
- [ ] Every dashboard and alert names an operational question and owner.
- [ ] Metrics are not repeatedly reconstructed from logs without justification.
- [ ] Trace IDs are correlation fields, not unbounded indexed labels.
- [ ] Critical paths receive deliberate signal quality and retention.
- [ ] Platform-health drills cover queues, drops, stores, queries, rules, and notifications.
Sources
Several anti-patterns depend on current product behavior, so they were checked against official primary documentation.