Common Architectural Mistakes

Enterprise problem: Teams can buy and deploy every observability component yet still miss customer failures because dashboards lead instrumentation, logs substitute for metrics, high-cardinality IDs are indexed, alerts describe components, and the telemetry platform is not monitored.

Whole-course context: The incoming component, scale, cost, schema, and privacy contracts provide reviewable constraints; this day uses them to detect recurring design mistakes before production incidents expose them.

Today’s slice: We turn nine anti-patterns into operational questions, policy checks, and controlled failure exercises.

End-of-day evidence: An architecture review and seeded-incident run identify each applicable mistake, reject one unsafe design, and preserve one customer-outcome positive control.

Still unsolved: The ordered implementation program and final behavior-to-action mental model remain for the last two days.

A catalogue of mistakes becomes trivia unless it changes a design or investigation outcome. These cases make prevention and recovery observable.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D22-UC-01	Architecture reviewer	Reject a telemetry design that cannot answer its stated operational question	Every approved dashboard or alert traces back to a customer outcome, required signal, owner, and tested query	A dashboard-first or universal-log proposal is denied with the missing evidence chain
D22-UC-02	Observability on-call	Detect and recover from the platform’s own telemetry failure	Backlog, drops, query latency, compaction, rules, and notifications reveal the fault before customer evidence is exhausted	Seeded collector or backend failure alerts on outcome risk and shows an unaffected signal path

General warnings do not tell a reviewer what to accept, so the stories require explicit pass and fail observations.

Story ID	Use case IDs	User story	Observable acceptance conditions
D22-US-01	D22-UC-01	As an architecture reviewer, I want every visualization derived from an operational question, so that attractive dashboards cannot substitute for evidence	Review traces question to signal, instrumentation, store, query, and decision; an orphan dashboard fails
D22-US-02	D22-UC-02	As the observability on-call, I want outcome-based meta-monitoring, so that platform degradation is detected before evidence is irrecoverable	A seeded queue backlog triggers risk evidence, points to component telemetry, and leaves one healthy tenant query working

Anti-pattern detection must happen during a real customer action, not after an abstract checklist. The flows end in a review denial or recovery record.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D22-FLOW-01	D22-UC-01	Denied	Team requests approval for a new dashboard and alert	1. Reviewer reads the operational question.<br>2. Contract traces required signal and owner.<br>3. Cardinality, semantics, and outcome alert checks run.<br>4. Missing user outcome rejects publication.<br>5. A known-good dashboard remains published.	Proposal, question, missing link, policy revision, denied result, unaffected dashboard control, timestamp, and review ID
D22-FLOW-02	D22-UC-02	Recovery	Fault injector pauses a collector export path	1. Queue depth and accepted/rejected/dropped counters change.<br>2. Outcome-risk alert fires before buffer exhaustion.<br>3. On-call inspects component detail.<br>4. Export resumes and backlog drains.<br>5. Completeness and notification delivery are verified.	Fault interval, queue maximum, drops, alert timing, notification ID, recovered query, unaffected tenant control, environment, timestamp, and drill ID

If policy checks and meta-monitoring are bolted on after deployment, unsafe telemetry reaches production before anyone can reject it. The design puts review gates before publication and health probes around the live pipeline.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D22-UC-01	Dashboard and rule publication review	Question registry, schema/cardinality checker, rule tester, architecture reviewer	Versioned observability-artifact registry owned by platform governance	Orphan question, log-derived counter without justification, trace ID label, missing owner, or component-only page
D22-UC-02	Meta-monitoring alert and drill action	Collector self-metrics, backend health metrics, alert ruler, Alertmanager, drill controller	Drill and platform-health evidence store owned by observability reliability	Accepted/rejected/drop anomaly, backlog exhaustion risk, query or compaction failure, rule failure, notification failure

Lessons vanish when review decisions and drills are not durable, causing the same mistake to recur. These control-plane records preserve rationale and observed behavior.

Generated-application database: Not created in this slice — architecture review and platform-health evidence belong to the observability control plane.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
OperationalQuestion	Artifact registry, service owner	question_id	None — root question with service reference	organization_id	Customer outcome, owner, decision, required signal, and review interval are mandatory	Draft, approve, revise, retire with dependent artifacts, retain audit history	D22-UC-01, D22-UC-02
ObservabilityArtifactReview	Governance store, platform reviewer	review_id	question_id local FK and opaque dashboard or rule revision	organization_id	Every applicable mistake has pass, fail, or reasoned exclusion; verdict is immutable	Append per revision, supersede on change, retain to artifact retirement plus audit window	D22-UC-01, D22-UC-02
MetaMonitoringDrill	Reliability evidence store, observability on-call owner	drill_id	Opaque platform revision, fault fixture, alert, and notification references	tenant_id	Expected and observed results, positive control, environment, and timestamps are immutable	Retain aggregate evidence, expire raw telemetry by policy, delete after reliability retention	D22-UC-01, D22-UC-02

Similar symptoms can have different causes, so name the mistaken decision and the corrective mechanism rather than merely adding more telemetry.

Mistake	Consequence	Better design
Start with dashboards	Teams search for data to justify a projection	Start with an operational question, then derive signal, instrumentation, store, query, and projection
Treat logs as universal	Repeated parsing is expensive and semantics drift	Use counters and histograms for repeated numeric questions; logs explain discrete events
Treat traces as expensive logs	Parent-child causality, timing, and boundaries are lost	Preserve context propagation, span relationships, status, and causal structure
Index trace IDs as labels	Near-unique values explode cardinality	Keep trace IDs as correlation metadata or structured fields, not indexed metric/log labels
Scale metrics before mastering semantics	Distributed storage amplifies bad names and series	Prove local metric meaning, labels, ownership, and rules before introducing Mimir
Instrument everything equally	Critical paths drown in low-value volume	Prioritize customer, revenue, security, irreversible, expensive, and newly risky paths
Ignore platform health	Telemetry disappears without warning	Monitor acceptance/rejection, queues, drops, object storage, queries, compaction, rules, notifications, and limits
Alert on components	Harmless restarts page people while customer loss is missed	Alert on user-visible failure, latency, capacity, backlog, data loss, and security outcomes; diagnose with components
Build distributed everything on day one	Capacity, upgrades, caching, recovery, security, and on-call complexity arrive before value	Purchase complexity only after simpler systems fail explicit requirements

Checklists can be gamed with plausible prose, so challenge each design with a seeded counterexample.

1. Choose one important customer outcome and state the decision an operator must make.
2. Walk backward from action through alert or dashboard, query, store, signal, and instrumentation.
3. Calculate worst-case label cardinality and reject identifiers with unbounded values.
4. Remove one component signal; confirm the outcome rule still describes customer risk while diagnostic detail narrows.
5. Pause one export path and observe accepted, rejected, queued, dropped, query, rule, and notification behavior.
6. Keep one unaffected tenant or signal path as a positive control.
7. Falsify the design if a projection has no question, a page has no threatened outcome, or platform data can disappear without advance evidence.

Most observability failures are sequencing and ownership failures rather than missing products.

• Questions precede signals, and signals precede dashboards.
• Each signal has a distinct job; correlation does not make them interchangeable.
• Outcome alerts page people; component telemetry explains causes.
• The observability platform needs its own loss, backlog, query, rule, and notification evidence.

Use these checks to stop common mistakes before a production rollout.

• [ ] Every dashboard and alert names an operational question and owner.
• [ ] Metrics are not repeatedly reconstructed from logs without justification.
• [ ] Trace IDs are correlation fields, not unbounded indexed labels.
• [ ] Critical paths receive deliberate signal quality and retention.
• [ ] Platform-health drills cover queues, drops, stores, queries, rules, and notifications.

Several anti-patterns depend on current product behavior, so they were checked against official primary documentation.

• Prometheus guidance on measuring user outcomes and cardinality
• Prometheus metric and label naming
• Grafana Alloy health and troubleshooting endpoints