Observability as an Operating Loop

Enterprise problem: ParcelFlow can authenticate and isolate tenants, yet an operator still cannot safely answer whether checkout is failing, which boundary caused it, who changed the system, or whether a restart actually restored the customer outcome.

Whole-course context: The incoming system has tenant-safe human and workload authority plus durable sync and event flows; this day makes those flows diagnosable before the production deployment and final release proof.

Today’s slice: Define logs, metrics, traces, and audit as different evidence products; correlate them with stable IDs; add request-rate, error, duration, service-level objective, health, readiness, and graceful-stop behavior; and keep OpenTelemetry behind a tested adapter.

End-of-day evidence: One injected payment timeout yields a tenant-safe trace, bounded structured logs, RED metrics, a service-level-objective alert, an operator decision, a graceful restart, and a passing customer recovery probe.

Still unsolved: Cloud resources, immutable container release, canary promotion, rollback, full adversarial E2E, export, revocation, and teardown remain for the deployment and capstone days.

Telemetry is useful only when it helps accountable actors protect a customer outcome. These cases cover a customer-impact incident and failure of the evidence path itself.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D10-UC-01	ParcelFlow on-call engineer	Find and recover a rise in failed order confirmations	Correlated metrics, trace, logs, deployment revision, and audit references identify the payment timeout; a bounded restart or rollback restores the confirmation probe	Cross-tenant detail remains denied, secrets stay redacted, and a post-action probe plus unaffected control proves the result
D10-UC-02	Observability platform owner	Detect and restore a broken telemetry export path	Queue, export, and drop-risk evidence identifies the failed destination; buffering or rerouting restores delivery within the declared objective	Missing spans or rejected exports are visible rather than silently treated as healthy; a known-good signal remains queryable

Dashboards can display activity without supporting action, so the stories require a decision and observed effect rather than a graph alone.

Story ID	Use case IDs	User story	Observable acceptance conditions
D10-US-01	D10-UC-01	As a ParcelFlow on-call engineer, I want one order journey correlated across HTTP, database, event, and payment boundaries, so that I can change the smallest failing component and prove customer recovery	Trace and event IDs join RED metrics, bounded logs, release digest, decision, action, and pre/post confirmation probes without exposing another tenant
D10-US-02	D10-UC-02	As an observability platform owner, I want the telemetry pipeline to observe its own acceptance, queue, export, and drop behavior, so that evidence loss becomes an incident before investigation is blind	Export failure raises a platform-health alert, buffered data drains after recovery, dropped count is zero or bounded explicitly, and a healthy control signal remains available

An operating loop is incomplete if it ends at detection. Both flows continue through interpretation, authority, action, and measurement.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D10-FLOW-01	D10-UC-01	Recovery	Confirmation success SLO burns faster than its alert threshold	1. RED metrics identify elevated confirmation errors and duration.<br>2. On-call selects an example trace.<br>3. Trace links Orders, Payment, outbox, and consumer spans.<br>4. Structured logs add bounded error facts.<br>5. Audit reference identifies the active deployment and authorized action.<br>6. Operator gracefully restarts the failing adapter.<br>7. Pre/post probes measure recovery.	Alert ID, tenant-safe trace, request/order/event IDs, deployment digest, decision and action IDs, expected/observed SLO, environment, timestamps, recovery probe, and unaffected control
D10-FLOW-02	D10-UC-02	Failure	Test fault blocks the telemetry exporter	1. Exporter reports rejected or queued batches.<br>2. Pipeline-health metric threatens the evidence objective.<br>3. Alert routes to the platform owner.<br>4. Owner restores or reroutes destination.<br>5. Queue drains.<br>6. Completeness query compares sent, accepted, retried, and dropped counts.	Fault interval, queue high-water mark, sent/accepted/retried/dropped counts, alert and action IDs, recovery duration, healthy signal control, environment, and drill ID
D10-FLOW-03	D10-UC-01, D10-UC-02	Happy	Customer submits a valid in-stock order while the evidence pipeline is healthy	1. Orders accepts the request and starts a trace.<br>2. Inventory, fake Payment, outbox, and consumer spans propagate the same context.<br>3. Bounded logs record state transitions.<br>4. The terminal-duration histogram records accepted within two seconds.<br>5. Export counters confirm delivery with zero drops.<br>6. Customer observes confirmation and the operator can join the evidence without taking corrective action.	Confirmed order, request/trace/event IDs, duration bucket, accepted/exported counters, deployment digest, tenant-safe logs, environment, timestamp, and zero-drop control

Treating one dashboard as authoritative hides where data originated and who may act. Each transition retains its owner: applications create domain meaning, the telemetry pipeline protects and transports signals, stores answer queries, and incident tooling records decisions.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D10-UC-01	Confirmation SLO alert and correlated trace link	ParcelFlow instrumentation, telemetry adapter, metrics/trace/log backends, alert evaluator, incident service, deployment controller, recovery probe	Signal backends own telemetry; incident log owns decisions; deployment store owns active release; Orders owns domain outcome	Missing propagation, redaction failure, query denial, alert routing failure, unsafe action denial, or failed recovery probe
D10-UC-02	Telemetry-pipeline health alert	Export adapter, bounded queue, collector/export destination, health metrics, alert router, reroute controller, completeness query	Pipeline-health store owns accepted/rejected/queued/dropped counters; drill log owns fault and recovery decisions	Queue exhaustion, rejected batches, silent drops, missing alert, failed reroute, or incomplete drain

Mutable dashboards cannot serve as incident history, and audit records should not become a high-cardinality telemetry store. Separate observations, alerts, decisions, actions, and immutable audit references while preserving controlled joins.

Generated-application database: Not created in this slice — existing ParcelFlow stores continue to own orders and reservations, while telemetry, incident, deployment, and audit systems own operational evidence and decisions.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
SignalEnvelope	Telemetry backend owned by observability platform	Backend-native signal ID plus timestamp	Opaque request, trace, order, event, service, schema, and deployment references	tenant_id only where policy permits; otherwise tenant hash	Schema version, classification, redaction, environment, and bounded cardinality required	Ingest, compact or sample, retain by signal class, expire payload while preserving incident references	D10-UC-01, D10-UC-02
ServiceLevelObjective	Reliability store owned by product reliability	slo_id plus version	Opaque service and signal-query references	tenant_id or None — fleet objective aggregates only approved non-identifying labels	Indicator, target, window, owner, exclusions, and alert policy immutable per version	Version and supersede; retain with incidents; expire obsolete detail under policy	D10-UC-01
AlertOccurrence	Alert store owned by routing service	alert_occurrence_id	Local slo_id; opaque query and notification references	Tenant scope inherited from the objective	State transition, evidence window, dedup key, and receiver required	Open, acknowledge, resolve; retain through incident review, then expire	D10-UC-01, D10-UC-02
IncidentDecision	Incident log owned by incident commander	decision_id	Opaque alert, trace, deployment, and audit references	Authorized tenant or None — platform incident has no customer payload	Actor, evidence, alternatives, authority, expected effect, rollback, and timestamp required	Append-only; supersede with another decision; export and expire by incident policy	D10-UC-01, D10-UC-02
OperationalAction	Action-controller store owned by service operator	action_id	Local decision_id; opaque target revision and run reference	Target tenant or None — fleet action is bounded by service/environment	Idempotency key, exact target, authority, deadline, result, and follow-up probe required	Execute, finish or roll back; retain result, delete transient execution payload	D10-UC-01, D10-UC-02
AuditReference	Security audit index owned by audit service	audit_reference_id	Opaque source audit-event ID; no copied secret or customer payload	Owning audit tenant scope	Integrity digest, source owner, actor/action/time pointers required; cannot authorize an operation	Retain according to audit policy; source owns redaction and deletion; dangling reference is marked	D10-UC-01

The general rule is behavior → trustworthy evidence → verified action. Metrics, logs, and traces are useful only when they help a responsible actor choose a bounded action and measure its effect.

A simple example is a readiness failure that triggers a graceful restart and then passes. The realistic example is a payment timeout correlated across an order trace and event replay. The failure case is a green dashboard with silently dropped spans or a restart that never improves confirmation success. The reusable decision rule is to reject any signal that has no owner, decision it can support, failure observation, or follow-up measurement.

Logs record discrete bounded facts, metrics aggregate numeric behavior, traces connect causal work across boundaries, and audit records who attempted or authorized consequential actions. They complement rather than replace one another.

Evidence	Best question	ParcelFlow example	Misuse to avoid
Metrics	How often, how slow, how many?	Confirmation request rate, error ratio, duration histogram, outbox lag	Customer/order IDs as labels that explode cardinality
Logs	What bounded fact occurred here?	Payment deadline exceeded with error class and trace ID	Full token, secret, card, or unbounded request body
Traces	Which causal path and dependency consumed time?	Orders → Payment → outbox → fulfillment consumer	Sampling without retained error or high-latency paths
Audit	Who attempted or approved what under which policy?	Operator authorized adapter restart	Treating telemetry sampling as decision-grade history

Use a shared envelope to keep correlation DRY without forcing every signal into the same backend:

type EvidenceContext = Readonly<{
  requestId: string;
  traceId: string;
  tenantRef: string;
  service: "orders" | "inventory" | "fulfillment";
  deploymentDigest: string;
}>;

export function logFailure(ctx: EvidenceContext, error: unknown): void {
  const errorClass = error instanceof Error ? error.name : "UnknownFailure";
  console.error(JSON.stringify({
    severity: "error",
    event: "order_confirmation_failed",
    errorClass,
    ...ctx,
  }));
}

Instrumentation libraries and Bun's Node compatibility evolve, so business code should depend on a small telemetry port. OpenTelemetry defines vendor-neutral APIs and semantic conventions; the selected JavaScript SDK/exporter combination still requires a pinned-version compatibility test under Bun.

export interface Telemetry {
  span<T>(name: string, attributes: Readonly<Record<string, string>>, run: () => Promise<T>): Promise<T>;
  count(name: string, value: number, attributes: Readonly<Record<string, string>>): void;
  duration(name: string, milliseconds: number, attributes: Readonly<Record<string, string>>): void;
}

export async function confirmOrder(telemetry: Telemetry, orderId: string): Promise<void> {
  await telemetry.span("order.confirm", { "order.id": orderId }, async () => {
    // domain use case remains independent from one SDK/exporter
  });
}

The adapter test must propagate W3C trace context through Hono requests and the event envelope, export a span, flush during shutdown, and prove no unsupported Node or V8 API is reached. If that probe fails, keep structured correlation and use a compatible collector/export route rather than claiming observability is complete.

RED means rate, errors, and duration. Measure at the customer boundary first, then add dependency and saturation signals needed to explain the outcome.

sum(rate(parcelflow_order_terminal_seconds_bucket{
  route="POST /orders",
  request_class="valid",
  outcome="accepted",
  le="2"
}[5m]))
/
sum(rate(parcelflow_order_terminal_seconds_count{
  route="POST /orders",
  request_class="valid"
}[5m]))

The histogram observes elapsed time from accepted request to terminal outcome for every valid submission, including failures and deadline expiry. Its le="2" accepted bucket divided by the valid-submission count therefore measures the stated latency-and-success indicator rather than merely counting HTTP 2xx responses. An example objective is “99.5% of valid order submissions reach an accepted terminal response within two seconds over 28 days.” The course must label the target as an example, not an industry standard. Evaluate the same ratio over the 28-day compliance window, alert on sustained multi-window error-budget burn, and preserve the exact query, bucket boundaries, exclusions, and deployment revision in incident evidence.

One endpoint cannot answer both “is the process alive?” and “should it receive new traffic?” Readiness turns false before shutdown, in-flight work drains, telemetry flushes, and only then does the process exit.

let ready = true;
const server = Bun.serve({
  port: 3000,
  routes: {
    "/health/live": () => Response.json({ live: true }),
    "/health/ready": () => Response.json({ ready }, { status: ready ? 200 : 503 }),
  },
  fetch: app.fetch,
});

async function stop(signal: string): Promise<void> {
  ready = false;
  console.info(JSON.stringify({ event: "shutdown_started", signal }));
  await server.stop(false);
  await telemetry.flush();
  process.exit(0);
}

process.once("SIGTERM", () => void stop("SIGTERM"));
process.once("SIGINT", () => void stop("SIGINT"));

The deployed probe sends traffic during termination and verifies that new requests leave the draining task, accepted work completes or is replayable, and the trace exporter flushes within the deadline.

Tests must observe outputs from real boundaries instead of asserting that instrumentation functions were called.

bun test tests/observability/evidence-schema.test.ts
bun test tests/observability/context-propagation.test.ts
bun run scripts/fault-drill.ts --fault payment-timeout
bun run scripts/fault-drill.ts --fault telemetry-export-blocked
bun run scripts/probe-recovery.ts --require-confirmation-slo --require-queue-drain

The run fails on missing correlation, forbidden high-cardinality labels, secret leakage, silent exporter loss, absent alert delivery, action without authority, or recovery declared before customer and pipeline probes pass.

More telemetry increases diagnosis power but also cost, privacy risk, and operator noise. Sampling reduces trace volume but can erase rare failures; aggregation controls metric cost but can hide a tenant-specific incident. Retain all errors and selected slow traces, limit labels to bounded dimensions, keep customer identifiers out of shared metric labels, and use authorized tenant-scoped queries when detail is necessary.

Decision rule: add a signal only when its owner, question, cardinality, retention, denial behavior, failure evidence, and supported action are explicit; close an incident only after the same customer probe and evidence-path control show the expected effect.

Telemetry APIs and runtime compatibility evolve, so stale assumptions can cause silent evidence loss during an incident. These primary specifications and maintained vendor references define the contracts that the adapter probes must verify.

• OpenTelemetry specification
• OpenTelemetry semantic conventions
• W3C Trace Context
• Bun Node.js compatibility status
• Google SRE Workbook: alerting on SLOs

Observability becomes dashboard theatre when evidence never supports an owned action or measured effect. Preserve these responsibility boundaries when tools or deployment shape change.

• Metrics aggregate, logs explain local facts, traces connect causality, and audit records accountable actions.
• Observability is complete only when evidence supports a bounded decision and follow-up measurement.
• Domain code depends on a telemetry port; pinned Bun compatibility tests validate the OpenTelemetry adapter.
• Readiness, graceful drain, replayability, and exporter flush are part of customer reliability.

An operating loop is incomplete if a signal, decision, action, or recovery probe cannot be joined, because the operator cannot prove what changed. Use this review before accepting the day's incident evidence.

• [ ] RED metrics and the example customer SLO have bounded labels, owners, windows, and exclusions.
• [ ] Request, trace, order, event, policy, and deployment IDs correlate without leaking secrets or foreign tenant data.
• [ ] OpenTelemetry adapter compatibility is tested under the pinned Bun runtime.
• [ ] Liveness, readiness, graceful stop, in-flight drain, and telemetry flush are independently proven.
• [ ] Payment-timeout and telemetry-export drills include positive controls and measured recovery.
• [ ] Incident decisions, actions, and follow-up evidence remain distinct from sampled telemetry.