Observability as an Operating Loop
Turn ParcelFlow behavior into correlated evidence, an accountable decision, and a measured recovery.
The enterprise problem and today’s slice
Enterprise problem: ParcelFlow can authenticate and isolate tenants, yet an operator still cannot safely answer whether checkout is failing, which boundary caused it, who changed the system, or whether a restart actually restored the customer outcome.
Whole-course context: The incoming system has tenant-safe human and workload authority plus durable sync and event flows; this day makes those flows diagnosable before the production deployment and final release proof.
Today’s slice: Define logs, metrics, traces, and audit as different evidence products; correlate them with stable IDs; add request-rate, error, duration, service-level objective, health, readiness, and graceful-stop behavior; and keep OpenTelemetry behind a tested adapter.
End-of-day evidence: One injected payment timeout yields a tenant-safe trace, bounded structured logs, RED metrics, a service-level-objective alert, an operator decision, a graceful restart, and a passing customer recovery probe.
Still unsolved: Cloud resources, immutable container release, canary promotion, rollback, full adversarial E2E, export, revocation, and teardown remain for the deployment and capstone days.
The smallest complete operating model
The general rule is behavior → trustworthy evidence → verified action. Metrics, logs, and traces are useful only when they help a responsible actor choose a bounded action and measure its effect.
A simple example is a readiness failure that triggers a graceful restart and then passes. The realistic example is a payment timeout correlated across an order trace and event replay. The failure case is a green dashboard with silently dropped spans or a restart that never improves confirmation success. The reusable decision rule is to reject any signal that has no owner, decision it can support, failure observation, or follow-up measurement.
Give each signal one job
Logs record discrete bounded facts, metrics aggregate numeric behavior, traces connect causal work across boundaries, and audit records who attempted or authorized consequential actions. They complement rather than replace one another.
| Evidence | Best question | ParcelFlow example | Misuse to avoid |
|---|---|---|---|
| Metrics | How often, how slow, how many? | Confirmation request rate, error ratio, duration histogram, outbox lag | Customer/order IDs as labels that explode cardinality |
| Logs | What bounded fact occurred here? | Payment deadline exceeded with error class and trace ID | Full token, secret, card, or unbounded request body |
| Traces | Which causal path and dependency consumed time? | Orders → Payment → outbox → fulfillment consumer | Sampling without retained error or high-latency paths |
| Audit | Who attempted or approved what under which policy? | Operator authorized adapter restart | Treating telemetry sampling as decision-grade history |
Use a shared envelope to keep correlation DRY without forcing every signal into the same backend:
type EvidenceContext = Readonly<{
requestId: string;
traceId: string;
tenantRef: string;
service: "orders" | "inventory" | "fulfillment";
deploymentDigest: string;
}>;
export function logFailure(ctx: EvidenceContext, error: unknown): void {
const errorClass = error instanceof Error ? error.name : "UnknownFailure";
console.error(JSON.stringify({
severity: "error",
event: "order_confirmation_failed",
errorClass,
...ctx,
}));
}
Instrument through a replaceable OpenTelemetry adapter
Instrumentation libraries and Bun's Node compatibility evolve, so business code should depend on a small telemetry port. OpenTelemetry defines vendor-neutral APIs and semantic conventions; the selected JavaScript SDK/exporter combination still requires a pinned-version compatibility test under Bun.
export interface Telemetry {
span<T>(name: string, attributes: Readonly<Record<string, string>>, run: () => Promise<T>): Promise<T>;
count(name: string, value: number, attributes: Readonly<Record<string, string>>): void;
duration(name: string, milliseconds: number, attributes: Readonly<Record<string, string>>): void;
}
export async function confirmOrder(telemetry: Telemetry, orderId: string): Promise<void> {
await telemetry.span("order.confirm", { "order.id": orderId }, async () => {
// domain use case remains independent from one SDK/exporter
});
}
The adapter test must propagate W3C trace context through Hono requests and the event envelope, export a span, flush during shutdown, and prove no unsupported Node or V8 API is reached. If that probe fails, keep structured correlation and use a compatible collector/export route rather than claiming observability is complete.
Define RED metrics and a customer SLO
RED means rate, errors, and duration. Measure at the customer boundary first, then add dependency and saturation signals needed to explain the outcome.
sum(rate(parcelflow_order_terminal_seconds_bucket{
route="POST /orders",
request_class="valid",
outcome="accepted",
le="2"
}[5m]))
/
sum(rate(parcelflow_order_terminal_seconds_count{
route="POST /orders",
request_class="valid"
}[5m]))
The histogram observes elapsed time from accepted request to terminal outcome for every valid submission, including failures and deadline expiry. Its le="2" accepted bucket divided by the valid-submission count therefore measures the stated latency-and-success indicator rather than merely counting HTTP 2xx responses. An example objective is “99.5% of valid order submissions reach an accepted terminal response within two seconds over 28 days.” The course must label the target as an example, not an industry standard. Evaluate the same ratio over the 28-day compliance window, alert on sustained multi-window error-budget burn, and preserve the exact query, bucket boundaries, exclusions, and deployment revision in incident evidence.
Separate liveness, readiness, and graceful stop
One endpoint cannot answer both “is the process alive?” and “should it receive new traffic?” Readiness turns false before shutdown, in-flight work drains, telemetry flushes, and only then does the process exit.
let ready = true;
const server = Bun.serve({
port: 3000,
routes: {
"/health/live": () => Response.json({ live: true }),
"/health/ready": () => Response.json({ ready }, { status: ready ? 200 : 503 }),
},
fetch: app.fetch,
});
async function stop(signal: string): Promise<void> {
ready = false;
console.info(JSON.stringify({ event: "shutdown_started", signal }));
await server.stop(false);
await telemetry.flush();
process.exit(0);
}
process.once("SIGTERM", () => void stop("SIGTERM"));
process.once("SIGINT", () => void stop("SIGINT"));
The deployed probe sends traffic during termination and verifies that new requests leave the draining task, accepted work completes or is replayable, and the trace exporter flushes within the deadline.
Run the incident and telemetry-loss drills
Tests must observe outputs from real boundaries instead of asserting that instrumentation functions were called.
bun test tests/observability/evidence-schema.test.ts
bun test tests/observability/context-propagation.test.ts
bun run scripts/fault-drill.ts --fault payment-timeout
bun run scripts/fault-drill.ts --fault telemetry-export-blocked
bun run scripts/probe-recovery.ts --require-confirmation-slo --require-queue-drain
The run fails on missing correlation, forbidden high-cardinality labels, secret leakage, silent exporter loss, absent alert delivery, action without authority, or recovery declared before customer and pipeline probes pass.
Failure modes, trade-offs, and decision rule
More telemetry increases diagnosis power but also cost, privacy risk, and operator noise. Sampling reduces trace volume but can erase rare failures; aggregation controls metric cost but can hide a tenant-specific incident. Retain all errors and selected slow traces, limit labels to bounded dimensions, keep customer identifiers out of shared metric labels, and use authorized tenant-scoped queries when detail is necessary.
Decision rule: add a signal only when its owner, question, cardinality, retention, denial behavior, failure evidence, and supported action are explicit; close an incident only after the same customer probe and evidence-path control show the expected effect.
Primary sources
Telemetry APIs and runtime compatibility evolve, so stale assumptions can cause silent evidence loss during an incident. These primary specifications and maintained vendor references define the contracts that the adapter probes must verify.
Key takeaways
Observability becomes dashboard theatre when evidence never supports an owned action or measured effect. Preserve these responsibility boundaries when tools or deployment shape change.
- Metrics aggregate, logs explain local facts, traces connect causality, and audit records accountable actions.
- Observability is complete only when evidence supports a bounded decision and follow-up measurement.
- Domain code depends on a telemetry port; pinned Bun compatibility tests validate the OpenTelemetry adapter.
- Readiness, graceful drain, replayability, and exporter flush are part of customer reliability.
Checklist
An operating loop is incomplete if a signal, decision, action, or recovery probe cannot be joined, because the operator cannot prove what changed. Use this review before accepting the day's incident evidence.
- [ ] RED metrics and the example customer SLO have bounded labels, owners, windows, and exclusions.
- [ ] Request, trace, order, event, policy, and deployment IDs correlate without leaking secrets or foreign tenant data.
- [ ] OpenTelemetry adapter compatibility is tested under the pinned Bun runtime.
- [ ] Liveness, readiness, graceful stop, in-flight drain, and telemetry flush are independently proven.
- [ ] Payment-timeout and telemetry-export drills include positive controls and measured recovery.
- [ ] Incident decisions, actions, and follow-up evidence remain distinct from sampled telemetry.