Resilience Under Partial Failure

Enterprise problem: ParcelFlow customers can reach Order while Inventory, its network path, or an event consumer is slow or unavailable; careless retries can then create duplicate fake authorizations, leaked reservations, and cascading overload.

Whole-course context: The incoming evidence is an Order outbox plus idempotent Fulfillment and Notification consumers that survive crash, duplicate, and ordering failures; today bounds what happens while dependencies are unhealthy.

Today’s slice: We allocate deadlines, retry only safe transient work, add jittered backoff, circuit breaking, bulkheads, backpressure, load shedding, compensation, and fault-injection tests.

End-of-day evidence: Deterministic tests prove one order, one fake authorization, no leaked reservation, bounded concurrency, explicit shedding, and recovery after dependency health returns.

Still unsolved: Production identity, tenant-to-service authority, observability operating loops, and deployment automation remain deferred.

Retrying every error can turn overload into collapse and repeat non-idempotent effects, so only classified transient failures on idempotent operations receive attempts within the remaining deadline. RFC 9110 explains why idempotent methods can be retried after communication failure and warns against automatically retrying non-idempotent requests unless their semantics are known to be idempotent. RFC 9110, idempotent methods.

General rule: retry when the failure is transient, the operation is idempotent, and enough budget remains. Simple example: retry a status query after connection reset, but not a fresh fake authorization. ParcelFlow example: reservation POST is made application-idempotent by (tenant_id, idempotency_key), so a retry returns the same reservation. Failure mode: three layers each retry three times, producing 27 calls while the customer waits. Decision rule: one layer owns retries and records every attempt against the original deadline.

Bun's fetch API supports AbortSignal.timeout() for request bounds; timeout is a control, not evidence that the remote side rolled back. Bun fetch timeouts.

const transient = new Set(["ECONNRESET", "ETIMEDOUT", "INVENTORY_503"]);

export async function withRetry<T>(operation: (signal: AbortSignal) => Promise<T>, deadlineMs: number): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    const remaining = deadlineMs - Date.now();
    if (remaining <= 25) throw new Error("INVENTORY_DEADLINE_EXCEEDED");
    try {
      return await operation(AbortSignal.timeout(remaining));
    } catch (error) {
      const code = error instanceof Error ? error.message : "UNKNOWN";
      if (!transient.has(code) || attempt >= 2) throw error;
      const capMs = Math.min(25 * 2 ** attempt, remaining - 10);
      const jitterMs = Math.floor(Math.random() * Math.max(1, capMs));
      await Bun.sleep(jitterMs);
    }
  }
}

Exponential backoff increases delay after repeated failure; jitter randomizes that delay so concurrent clients do not synchronize their next attempt. Cap both delay and attempt count, and never sleep past the total budget.

Retries cannot repair a dependency that remains unhealthy, so containment must reject quickly and preserve capacity for recovery and unrelated operations.

General rule: a circuit breaker limits calls to a failing dependency; a bulkhead limits how much local concurrency that dependency can consume; backpressure bounds queued demand; load shedding rejects excess rather than hiding an unbounded queue. Simple example: allow 20 Inventory calls and queue 40, then return 503 with a retry hint. ParcelFlow example: after a rolling failure threshold, the breaker opens; later a small number of half-open probes decide whether to close. Failure mode: one global semaphore lets noisy tenant traffic monopolize all slots. Decision rule: cap globally for dependency safety and fairly per tenant, while reserving probe/recovery capacity.

export class Bulkhead {
  #active = 0;
  constructor(private readonly maxActive: number) {}
  async run<T>(work: () => Promise<T>): Promise<T> {
    if (this.#active >= this.maxActive) throw new Error("DEPENDENCY_OVERLOADED");
    this.#active++;
    try { return await work(); } finally { this.#active--; }
  }
}

inventory_resilience:
  total_deadline_ms: 1200
  max_attempts: 3
  bulkhead_active: 20
  queue_limit: 40
  per_tenant_active: 4
  circuit_open_after_consecutive_failures: 8
  half_open_probes: 2

Bun documents that server.stop() stops accepting new connections and, by default, allows in-flight requests to complete. That lifecycle behaviour supports draining, but the application still needs admission limits and deadlines. Bun server lifecycle.

A timeout leaves the caller uncertain about whether Inventory committed, so compensation must query authority before it sends an inverse command.

General rule: reconcile first, compensate second, and make both steps idempotent. Simple example: query reservation by original key; release only if it exists and Order cannot continue. ParcelFlow example: if res-801 exists but fake Payment and order commit never happened, the recovery worker either completes the same order or releases res-801 with release:ord-801. Failure mode: blindly reserving again leaks stock, while blindly releasing may undo a successful order. Decision rule: drive recovery from authoritative state plus a durable recovery-case state machine.

CREATE UNIQUE INDEX one_open_recovery_case
  ON recovery_cases (tenant_id, order_id)
  WHERE resolved_at IS NULL;

CREATE UNIQUE INDEX fake_authorization_once
  ON fake_authorizations (tenant_id, order_id);

There is no real charge in this course: fake Payment writes only a deterministic local authorization record. The production invariant being rehearsed is still strict—no duplicate charge-like effect and no reservation leak.

Resilience code often stays untested until an incident, so deterministic fault injection must exercise timeouts, resets, overload, half-open recovery, and compensation state.

import { expect, test } from "bun:test";

test("transient reset retries one reservation key and fake-authorizes once", async () => {
  const rig = new ParcelFlowRig().inventoryFailsOnce("ECONNRESET");
  const result = await rig.placeOrder({ tenantId: "tn-a", orderId: "ord-801", idempotencyKey: "idem-0801" });
  expect(result.state).toBe("accepted");
  expect(rig.reservations("ord-801")).toHaveLength(1);
  expect(rig.fakeAuthorizations("ord-801")).toHaveLength(1);
});

test("bulkhead sheds excess and never exceeds its cap", async () => {
  const rig = new ParcelFlowRig().inventoryBlocks();
  const results = await Promise.allSettled(Array.from({ length: 12 }, (_, i) => rig.placeOrder({ tenantId: "tn-a", orderId: `ord-${i}` })));
  expect(rig.maxObservedInventoryConcurrency()).toBe(4);
  expect(results.filter((result) => result.status === "rejected").length).toBeGreaterThan(0);
});

test("uncertain reservation reconciles then releases without leak", async () => {
  const rig = new ParcelFlowRig().timeoutAfterReservationCommit();
  await expect(rig.placeOrder({ tenantId: "tn-a", orderId: "ord-802", idempotencyKey: "idem-0802" })).rejects.toThrow();
  await rig.reconcile("ord-802");
  expect(rig.activeReservations("ord-802")).toHaveLength(0);
  expect(rig.realCharges()).toHaveLength(0);
});

bun test tests/resilience/retry.test.ts tests/resilience/bulkhead.test.ts tests/resilience/recovery.test.ts
# expected terminal evidence: 3 pass, max_concurrency=4, shed_count>0, active_reservation_leaks=0, real_charges=0, run_id=res-0808

Availability mechanisms can become failure multipliers when applied without semantics, so every control needs a bounded purpose and observable invariant.

• One end-to-end deadline contains all attempts and waits.
• Retry only transient, application-idempotent work with the original key and jittered backoff.
• Circuit breakers reduce futile calls; bulkheads, backpressure, and shedding protect capacity.
• Timeout means unknown, not rolled back; query authoritative state before compensation.
• Recovery must preserve one order, one reservation outcome, one fake authorization, and zero real charges.
• Fault injection must prove both failure containment and return to healthy service.

Healthy happy paths do not prove resilience, so release only with repeatable partial-failure and recovery evidence.

• [ ] One layer owns retry policy and the original deadline.
• [ ] Non-transient and non-idempotent operations are not automatically retried.
• [ ] Backoff is exponential, jittered, capped, and budget-aware.
• [ ] Circuit open, half-open, and closed transitions are tested.
• [ ] Global and per-tenant concurrency plus queue depth are bounded.
• [ ] Excess work receives an explicit load-shed response.
• [ ] Reconciliation queries Inventory before completing or compensating.
• [ ] Tests prove no duplicate fake charge, no real charge, and no reservation leak.