Resilience Under Partial Failure

Enterprise problem: ParcelFlow customers can reach Order while Inventory, its network path, or an event consumer is slow or unavailable; careless retries can then create duplicate fake authorizations, leaked reservations, and cascading overload.

Whole-course context: The incoming evidence is an Order outbox plus idempotent Fulfillment and Notification consumers that survive crash, duplicate, and ordering failures; today bounds what happens while dependencies are unhealthy.

Today’s slice: We allocate deadlines, retry only safe transient work, add jittered backoff, circuit breaking, bulkheads, backpressure, load shedding, compensation, and fault-injection tests.

End-of-day evidence: Deterministic tests prove one order, one fake authorization, no leaked reservation, bounded concurrency, explicit shedding, and recovery after dependency health returns.

Still unsolved: Production identity, tenant-to-service authority, observability operating loops, and deployment automation remain deferred.

Partial failure creates uncertainty rather than a clean total outage, so customers need bounded decisions and recovery states that never invite unsafe blind resubmission.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D08-UC-01	Tenant order operator	Place an order while a dependency is intermittently slow	One order reaches accepted within its budget, with one reservation and one explicitly fake authorization	Overloaded or open-circuit paths return a stable retry-later response; no duplicate fake charge and no reservation leak occur
D08-UC-02	Operations responder	Restore safe processing after Inventory degradation	Circuit moves through recovery probe to closed, queued work drains within limits, and reconciliation closes uncertain orders	Fault injection proves bounded concurrency and shedding; compensation releases an orphaned reservation or records manual recovery without touching another tenant

Retries can improve availability or magnify damage depending on operation semantics, so stories bind automation to idempotency, budgets, and visible terminal evidence.

Story ID	Use case IDs	User story	Observable acceptance conditions
D08-US-01	D08-UC-01	As a tenant order operator, I want a bounded accepted, retry-later, or recovering state, so that a slow dependency does not duplicate my order or conceal an uncertain outcome	Same idempotency key produces one order, reservation, and fake authorization; exhausted budget returns a stable code and support trace
D08-US-02	D08-UC-02	As an operations responder, I want automated containment and explicit compensation, so that one unhealthy dependency cannot consume all capacity or leak tenant resources	Semaphore cap, queue limit, circuit transitions, shed count, compensation result, unaffected-tenant probe, and immutable recovery-run ID are observed

Layered retries can spend more time than the customer allowed and multiply load, so one end-to-end budget governs every attempt, wait, and recovery action.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D08-FLOW-01	D08-UC-01, D08-UC-02	Happy	Operator selects Place order during one transient Inventory connection failure	1. Order validates tenant and idempotency key, then allocates a total deadline.<br>2. Bulkhead admits the call and the closed circuit permits Inventory.<br>3. First idempotent reservation attempt fails before a response with a classified transient error.<br>4. Order waits bounded exponential backoff plus jitter and retries with the same key and remaining deadline.<br>5. Inventory returns the original-or-new reservation, fake Payment authorizes once, and Order commits accepted plus outbox.	Actor, tenant, order, scope, precondition, expected and observed result, environment, timestamp, trace ID, attempt count, remaining budget, one reservation, one fake authorization, recovery-run ID
D08-FLOW-02	D08-UC-01, D08-UC-02	Recovery	Responder selects Run reconciliation after Inventory timeouts open the circuit	1. Circuit rejects new reservation calls quickly while a bounded queue sheds excess with DEPENDENCY_OVERLOADED.<br>2. Existing uncertain order is queried by idempotency key after the dependency recovers.<br>3. If reservation exists and order can continue, fake Payment and order commit once; otherwise compensation sends idempotent release.<br>4. A limited half-open probe succeeds and closes the circuit.<br>5. Product shows recovered or compensated, while another tenant's positive-control order remains isolated.	Circuit transitions, queue depth, shed count, original key, authoritative reservation status, release result if needed, zero real charges, one-or-zero fake authorization as specified, trace ID, immutable reconciliation-run ID

When every request can occupy every downstream slot, one slow dependency starves unrelated work, so resilience controls sit at the Order→Inventory boundary and expose their state to operators.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D08-UC-01	POST /v1/orders	Order Hono API, deadline-aware Inventory client, retry classifier, circuit breaker, Inventory API, fake Payment	Order database for order/fake authorization/recovery state; Inventory database for reservation	Stable timeout/overload code, attempt ledger, circuit state, one owned effect per idempotency key
D08-UC-02	Reconciliation action and health probe	Recovery worker, Inventory status/release endpoints, bulkhead, breaker, fault injector	Recovery ledger in Order store; Inventory remains authoritative for reservation status	Reconciliation run, compensation result, queue/shed metrics, half-open probe, unaffected-tenant evidence

Recovery without durable intent can repeat compensation or forget uncertain work after restart, so Order records recovery state while Inventory remains the only authority that confirms or releases reservations.

Generated-application database: Required in this slice — ParcelFlow Order and Inventory application stores persist tenant-scoped idempotency, reservation, fake authorization, and reconciliation state needed to prevent duplicate or leaked effects.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
Order	Order database, Order service	order_id	reservation_id opaque Inventory reference	tenant_id	(tenant_id, idempotency_key) unique; accepted requires one reservation and one fake authorization	Created pending; accepted, failed, or recovering; retained for support then tenant-erased	D08-UC-01, D08-UC-02
FakeAuthorization	Order database, fake Payment adapter	authorization_id	order_id local FK	tenant_id	(tenant_id, order_id) unique; no external payment or real charge permitted	Created at most once after reservation; voided on failed order; deleted with order after retention	D08-UC-01, D08-UC-02
Reservation	Inventory database, Inventory service	reservation_id	order_id opaque Order reference	tenant_id	(tenant_id, order_id, sku) unique; reserve and release commands idempotent	Reserved then consumed, expired, or released; audit retained then deleted	D08-UC-01, D08-UC-02
DependencyAttempt	Order database, resilience client	attempt_id	order_id local FK; reservation_id optional opaque reference	tenant_id	Attempt number, error class, start, duration, and remaining budget immutable	Appended per attempt; payload-free diagnostics expire before order evidence	D08-UC-01
RecoveryCase	Order database, recovery worker	recovery_case_id	order_id local FK; reconciliation_run_id immutable evidence reference	tenant_id	One open case per order; compare-and-swap state; compensation idempotency key required	Opened on uncertainty; resolved accepted/compensated/manual; retained through audit then archived	D08-UC-02
CircuitSnapshot	Operational state store, Order runtime	dependency_key	None — dependency policy is the root record	None — breaker protects shared dependency capacity and carries no tenant payload	State transition timestamp and threshold config recorded; not business truth	Ephemeral runtime state checkpointed for diagnosis; expires after operational retention	D08-UC-02

Retrying every error can turn overload into collapse and repeat non-idempotent effects, so only classified transient failures on idempotent operations receive attempts within the remaining deadline. RFC 9110 explains why idempotent methods can be retried after communication failure and warns against automatically retrying non-idempotent requests unless their semantics are known to be idempotent. RFC 9110, idempotent methods.

General rule: retry when the failure is transient, the operation is idempotent, and enough budget remains. Simple example: retry a status query after connection reset, but not a fresh fake authorization. ParcelFlow example: reservation POST is made application-idempotent by (tenant_id, idempotency_key), so a retry returns the same reservation. Failure mode: three layers each retry three times, producing 27 calls while the customer waits. Decision rule: one layer owns retries and records every attempt against the original deadline.

Bun's fetch API supports AbortSignal.timeout() for request bounds; timeout is a control, not evidence that the remote side rolled back. Bun fetch timeouts.

const transient = new Set(["ECONNRESET", "ETIMEDOUT", "INVENTORY_503"]);

export async function withRetry<T>(operation: (signal: AbortSignal) => Promise<T>, deadlineMs: number): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    const remaining = deadlineMs - Date.now();
    if (remaining <= 25) throw new Error("INVENTORY_DEADLINE_EXCEEDED");
    try {
      return await operation(AbortSignal.timeout(remaining));
    } catch (error) {
      const code = error instanceof Error ? error.message : "UNKNOWN";
      if (!transient.has(code) || attempt >= 2) throw error;
      const capMs = Math.min(25 * 2 ** attempt, remaining - 10);
      const jitterMs = Math.floor(Math.random() * Math.max(1, capMs));
      await Bun.sleep(jitterMs);
    }
  }
}

Exponential backoff increases delay after repeated failure; jitter randomizes that delay so concurrent clients do not synchronize their next attempt. Cap both delay and attempt count, and never sleep past the total budget.

Retries cannot repair a dependency that remains unhealthy, so containment must reject quickly and preserve capacity for recovery and unrelated operations.

General rule: a circuit breaker limits calls to a failing dependency; a bulkhead limits how much local concurrency that dependency can consume; backpressure bounds queued demand; load shedding rejects excess rather than hiding an unbounded queue. Simple example: allow 20 Inventory calls and queue 40, then return 503 with a retry hint. ParcelFlow example: after a rolling failure threshold, the breaker opens; later a small number of half-open probes decide whether to close. Failure mode: one global semaphore lets noisy tenant traffic monopolize all slots. Decision rule: cap globally for dependency safety and fairly per tenant, while reserving probe/recovery capacity.

export class Bulkhead {
  #active = 0;
  constructor(private readonly maxActive: number) {}
  async run<T>(work: () => Promise<T>): Promise<T> {
    if (this.#active >= this.maxActive) throw new Error("DEPENDENCY_OVERLOADED");
    this.#active++;
    try { return await work(); } finally { this.#active--; }
  }
}

inventory_resilience:
  total_deadline_ms: 1200
  max_attempts: 3
  bulkhead_active: 20
  queue_limit: 40
  per_tenant_active: 4
  circuit_open_after_consecutive_failures: 8
  half_open_probes: 2

Bun documents that server.stop() stops accepting new connections and, by default, allows in-flight requests to complete. That lifecycle behaviour supports draining, but the application still needs admission limits and deadlines. Bun server lifecycle.

A timeout leaves the caller uncertain about whether Inventory committed, so compensation must query authority before it sends an inverse command.

General rule: reconcile first, compensate second, and make both steps idempotent. Simple example: query reservation by original key; release only if it exists and Order cannot continue. ParcelFlow example: if res-801 exists but fake Payment and order commit never happened, the recovery worker either completes the same order or releases res-801 with release:ord-801. Failure mode: blindly reserving again leaks stock, while blindly releasing may undo a successful order. Decision rule: drive recovery from authoritative state plus a durable recovery-case state machine.

CREATE UNIQUE INDEX one_open_recovery_case
  ON recovery_cases (tenant_id, order_id)
  WHERE resolved_at IS NULL;

CREATE UNIQUE INDEX fake_authorization_once
  ON fake_authorizations (tenant_id, order_id);

There is no real charge in this course: fake Payment writes only a deterministic local authorization record. The production invariant being rehearsed is still strict—no duplicate charge-like effect and no reservation leak.

Resilience code often stays untested until an incident, so deterministic fault injection must exercise timeouts, resets, overload, half-open recovery, and compensation state.

import { expect, test } from "bun:test";

test("transient reset retries one reservation key and fake-authorizes once", async () => {
  const rig = new ParcelFlowRig().inventoryFailsOnce("ECONNRESET");
  const result = await rig.placeOrder({ tenantId: "tn-a", orderId: "ord-801", idempotencyKey: "idem-0801" });
  expect(result.state).toBe("accepted");
  expect(rig.reservations("ord-801")).toHaveLength(1);
  expect(rig.fakeAuthorizations("ord-801")).toHaveLength(1);
});

test("bulkhead sheds excess and never exceeds its cap", async () => {
  const rig = new ParcelFlowRig().inventoryBlocks();
  const results = await Promise.allSettled(Array.from({ length: 12 }, (_, i) => rig.placeOrder({ tenantId: "tn-a", orderId: `ord-${i}` })));
  expect(rig.maxObservedInventoryConcurrency()).toBe(4);
  expect(results.filter((result) => result.status === "rejected").length).toBeGreaterThan(0);
});

test("uncertain reservation reconciles then releases without leak", async () => {
  const rig = new ParcelFlowRig().timeoutAfterReservationCommit();
  await expect(rig.placeOrder({ tenantId: "tn-a", orderId: "ord-802", idempotencyKey: "idem-0802" })).rejects.toThrow();
  await rig.reconcile("ord-802");
  expect(rig.activeReservations("ord-802")).toHaveLength(0);
  expect(rig.realCharges()).toHaveLength(0);
});

bun test tests/resilience/retry.test.ts tests/resilience/bulkhead.test.ts tests/resilience/recovery.test.ts
# expected terminal evidence: 3 pass, max_concurrency=4, shed_count>0, active_reservation_leaks=0, real_charges=0, run_id=res-0808

Availability mechanisms can become failure multipliers when applied without semantics, so every control needs a bounded purpose and observable invariant.

• One end-to-end deadline contains all attempts and waits.
• Retry only transient, application-idempotent work with the original key and jittered backoff.
• Circuit breakers reduce futile calls; bulkheads, backpressure, and shedding protect capacity.
• Timeout means unknown, not rolled back; query authoritative state before compensation.
• Recovery must preserve one order, one reservation outcome, one fake authorization, and zero real charges.
• Fault injection must prove both failure containment and return to healthy service.

Healthy happy paths do not prove resilience, so release only with repeatable partial-failure and recovery evidence.

• [ ] One layer owns retry policy and the original deadline.
• [ ] Non-transient and non-idempotent operations are not automatically retried.
• [ ] Backoff is exponential, jittered, capped, and budget-aware.
• [ ] Circuit open, half-open, and closed transitions are tested.
• [ ] Global and per-tenant concurrency plus queue depth are bounded.
• [ ] Excess work receives an explicit load-shed response.
• [ ] Reconciliation queries Inventory before completing or compensating.
• [ ] Tests prove no duplicate fake charge, no real charge, and no reservation leak.