Full-System E2E Proof and Retirement

Enterprise problem: Green unit tests and a successful deployment cannot prove that an order survives duplicate requests, stock rejection, payment timeout, event replay, tenant attack, canary failure, or retirement without lost data, duplicate effects, residual access, or incomplete evidence.

Whole-course context: The incoming system includes Bun services, explicit ports and dependencies, tenant-owned PostgreSQL state, sync contracts, outbox events, idempotent consumers, resilience controls, identity boundaries, operational evidence, and an immutable Pulumi deployment; this final day tests their interactions as one released product.

Today’s slice: Run a deterministic production-shaped bun:test orchestrator and CI matrix across positive, denied, failure, recovery, rollback, export, revocation, ingress-disable, and ephemeral teardown paths, then decide whether Bun and the extracted services remain justified.

End-of-day evidence: A digest-bound release dossier joins every required scenario to actor, tenant, resource, precondition, expected and observed result, environment, time, source, trace, event, artifact, deployment, policy, and immutable run identifiers.

Still unsolved: Real payment-card processing, organization-specific compliance certification, permanent production deletion, and irreversible cloud teardown remain owner-governed operations outside this public course.

A final proof must cover both continued service and safe exit. These cases test a release owner accepting the system and a tenant owner leaving it without residual authority.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D12-UC-01	Accountable release owner	Decide whether the exact ParcelFlow release can safely serve customers	The complete E2E matrix passes against pinned source, schema, policy, infrastructure, and image digests; canary v2 is either promoted or observably rolled back	Any unexpected allow, duplicate effect, lost event, broken evidence join, failed positive control, or destructive rollback blocks release with a reproducible counterexample
D12-UC-02	Tenant administrator	Export records and end ParcelFlow access without affecting another tenant	Tenant export is complete and checksummed; human and workload grants are revoked; ingress is disabled; residual probes deny access; only the disposable test stack is torn down	Missing export item, surviving token/route/queue consumer, cross-tenant impact, deletion-before-export, or unapproved retained resource blocks retirement

“All tests passed” hides which accountable decision those tests support, so each story names the terminal customer evidence and the conditions that invalidate it.

Story ID	Use case IDs	User story	Observable acceptance conditions
D12-US-01	D12-UC-01	As a release owner, I want one deterministic suite to exercise the normal product surfaces and failure boundaries, so that approval binds the exact system customers will receive	Ten required scenarios run through HTTP, PostgreSQL, queue, identity, telemetry, ALB/ECS, and release controls; every result joins to pinned digests and no test bypasses normal enforcement
D12-US-02	D12-UC-02	As a tenant administrator, I want export and access termination proven before disposable infrastructure is removed, so that custody transfers safely and no residual credential or route survives	Export count/checksum is accepted, entry points are disabled, grants and secrets are revoked, residual probes deny access, unaffected tenant control passes, and teardown targets only the named ephemeral stack

The final flow starts with customer-visible behavior and ends with a release or retirement verdict, not merely a process exit code. Recovery and denial remain first-class evidence.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D12-FLOW-01	D12-UC-01	Happy	Release owner starts the candidate E2E workflow	1. Resolve immutable source, schema, policy, image, and infrastructure tuple.<br>2. Provision isolated tenants and fake-provider fixtures.<br>3. Run confirmed, duplicate, out-of-stock, timeout, event-ordering, tenant-denial, outbox-replay, evidence-chain, and canary cases through normal surfaces.<br>4. Verify positive controls and durable state.<br>5. Seal observations.<br>6. Approve only the exact tuple.	Ten scenario results, all required evidence fields, test and deployment logs, signed or digest-bound dossier, actor approval, environment, timestamps, and candidate verdict
D12-FLOW-02	D12-UC-01	Recovery	v2 canary fails confirmation while v1 remains healthy	1. Detect failed v2 customer probe.<br>2. Block promotion.<br>3. Restore ALB routing to v1 digest.<br>4. Drain v2.<br>5. Replay durable work.<br>6. Verify order/data/message state and v1 customer recovery.<br>7. Seal rollback transition.	Failed v2 trace, unaffected v1 control, before/after weights, digests, rollback update, drain result, preserved RDS rows, queue state, recovery probe, and release verdict rolled-back
D12-FLOW-03	D12-UC-02	Denied	Tenant administrator requests retirement before export is accepted	1. Lifecycle service checks prerequisites.<br>2. Missing custody receipt blocks destructive steps.<br>3. Ingress and grants remain governed rather than partially deleted.<br>4. Export completes and administrator accepts checksum.<br>5. Lifecycle resumes from recorded checkpoint.	Denied transition, missing prerequisite, no deletion, completed export receipt, resumed checkpoint, actor, tenant, policy, timestamp, and lifecycle run ID
D12-FLOW-04	D12-UC-02	Recovery	Accepted export permits retirement of the ephemeral tenant and stack	1. Quiesce writes and drain work.<br>2. Disable public ingress.<br>3. Revoke human sessions, workload grants, and secret versions.<br>4. Stop consumers and prove queue scope.<br>5. Run residual denial and unaffected-tenant positive controls.<br>6. Destroy only the named disposable stack.<br>7. Preserve minimal tombstone and custody evidence.	Export checksum, disabled route, revocation IDs, residual denials, unaffected control, exact Pulumi stack target, destroy update ID, residual-resource inventory, and signed retirement receipt

The capstone architecture preserves every owner introduced by prior days. The E2E harness coordinates observations but receives no special authority to bypass APIs, row policy, task roles, release controls, or lifecycle prerequisites.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D12-UC-01	CI E2E workflow and release approval surface	Bun test orchestrator, fixture manager, Orders, Inventory, Payment simulator, outbox publisher, SQS, Fulfillment, identity/policy services, evidence collector, ECS/ALB release controller, rollback controller	Orders and Inventory PostgreSQL own domain state; SQS owns in-flight events; identity stores own grants; telemetry/audit stores own evidence; release store owns candidate verdict	Scenario timeout, unexpected allow, duplicate side effect, unreplayed event, evidence mismatch, digest drift, canary failure, rollback failure, or positive-control failure
D12-UC-02	Tenant export and retirement action	Export service, lifecycle state machine, ingress controller, session/workload/secret revokers, queue drainer, residual probe runner, Pulumi destroy workflow	Domain services own exported rows; export store owns custody receipt; identity/security stores own revocations; ALB/ECS own disabled runtime; Pulumi backend owns disposable stack state; tombstone store owns minimal result	Missing export, active writer, surviving route/token/consumer, foreign-tenant regression, wrong stack target, residual resource, or deletion-before-prerequisite denial

Final evidence becomes untrustworthy if a mutable test report can point at changed artifacts or if a retirement tombstone contains the customer payload it claims was removed. Bind observations to immutable inputs and keep custody, revocation, teardown, and minimal residual evidence separate.

Generated-application database: Not created in this slice — the capstone verifies and retires the existing ParcelFlow databases; new durable state belongs only to release, test-evidence, export, revocation, and lifecycle control stores.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
CandidateTuple	Release store owned by release governance	candidate_digest	Opaque source, lockfile, schema, policy, image, Pulumi stack, and provider-version references	None — candidate describes shared executable and infrastructure inputs	Every input immutable and resolved; any change creates another candidate	Create before testing, approve or reject once, retain through rollback/audit, expire per release policy	D12-UC-01
ScenarioObservation	Append-only evidence store owned by verification service	scenario_observation_id	Local candidate_digest; opaque actor, tenant, resource, trace, event, order, deployment, and source fixture references	Fixture tenant_id	Scenario, precondition, expected, observed, environment, timestamp, positive control, and immutable IDs mandatory; secrets redacted	Append during run, seal into dossier, retain by evidence policy, expire sensitive fixtures separately	D12-UC-01, D12-UC-02
ReleaseDossier	Release evidence store owned by accountable release owner	dossier_digest	Local candidate and observation-manifest references; opaque approval reference	None — dossier spans tenant-scoped fixtures without transferring tenant authority	Complete required matrix, no unexpected allow, exact digests, independent approval, and terminal verdict required	Seal after run; invalidate on changed input; retain through release and rollback history	D12-UC-01
ExportReceipt	Export store owned by tenant data service	export_receipt_id	Opaque tenant, requester, destination custody, schema, and lifecycle references	tenant_id	Record counts, schema version, checksum, encryption, destination, acceptance, and time immutable	Stage, deliver, accept; securely delete staging copy; destination copy transfers to customer custody	D12-UC-02
RevocationReceipt	Security evidence store owned by revocation service	revocation_receipt_id	Opaque session, workload grant, secret version, route, consumer, and policy references	tenant_id or explicit environment scope	Exact authority, requested/observed disable time, denial probe, and unaffected control required	Append after each revocation; retain minimal metadata; credential value destroyed	D12-UC-02
LifecycleRun	Lifecycle store owned by retirement orchestrator	lifecycle_run_id	Local export and revocation references; opaque Pulumi stack/update and residual-probe references	tenant_id plus named disposable environment	State machine enforces export acceptance before destructive action and exact stack identity before destroy	Quiesce, export, disable, revoke, probe, destroy disposable target, complete or block; retain checkpoint history	D12-UC-02
RetirementTombstone	Minimal evidence store owned by governance	retirement_tombstone_id	Hash-only references to retired tenant/environment and lifecycle run	Tenant identifier hash	Contains actions, owners, policy, times, results, authorized residuals, and proof IDs; never customer payload or secret	Retain under minimal evidence policy; expire after obligations end; cannot recreate deleted records	D12-UC-02

The general rule is claim → adversarial observation → accountable verdict. A release claim without a falsifier is optimism; an observation without an exact candidate cannot be reproduced; a verdict without owner and consequence cannot govern production.

The simple example sends one valid order and checks CONFIRMED. The realistic example injects payment timeout, restarts the outbox publisher, attacks tenant scope, and fails a canary while durable state remains. The failure case is a test that calls an internal function or deletes a resource directly, bypassing the path customers use. The reusable decision rule is to test through normal surfaces with least-privilege fixtures and reject any verdict that cannot join observation to immutable input and owner.

Each scenario has one decisive terminal state and one unaffected positive control. Running them in a fixed order makes lifecycle dependencies explicit, while isolated fixture IDs prevent one result from contaminating another.

Scenario	Trigger and expected terminal state	Required evidence
Confirmed order	Valid tenant submits in-stock order; CONFIRMED	One order, reservation, payment authorization, event chain, notification, and joined trace
Idempotent duplicate	Same idempotency key submitted twice; same order returned	One order row, one charge, one reservation, one terminal notification
Out of stock	Requested quantity unavailable; REJECTED_OUT_OF_STOCK	No payment call, no negative stock, rejection event, customer response
Payment timeout compensation	Fake provider times out; bounded retries end in documented compensation	Retry count/deadline, reservation released or explicit review state, no duplicate charge
Duplicate/out-of-order events	Consumer receives repeats and later version first	Inbox/dedup evidence, no repeated effect, explicit stale-event handling
Tenant denial	Tenant B requests tenant A order	403, no foreign query result, tenant A control succeeds
Outbox replay	Publisher stops after commit and before send, then restarts	Durable outbox row, eventual event, one effect, replay/trace identifiers
Evidence chain	One confirmed order queried across all evidence owners	Request, trace, order, event, policy, artifact, deployment, decision, and run IDs join
Canary rollback	v2 confirmation probe fails	v2 blocked, v1 routing restored, data/messages preserved, v1 recovery passes
Export and retirement	Accepted export followed by revoke, disable, residual probe, disposable teardown	Checksum/custody, revocation receipts, denials, unaffected tenant, exact destroy target, tombstone

An E2E orchestrator should coordinate fixtures and assertions, not reproduce service logic. It talks through HTTP and observable stores with bounded polling, records artifacts even on failure, and never imports domain handlers.

import { beforeAll, describe, expect, test } from "bun:test";
import { mkdir } from "node:fs/promises";

type Result = { status: number; body: unknown };
const required = (name: string): string => {
  const value = process.env[name];
  if (!value) throw new Error(`missing ${name}`);
  return value;
};
const BASE_URL = required("BASE_URL");
const CONTROL_URL = process.env.CONTROL_URL ?? BASE_URL;
const TENANT_A_TOKEN = required("TENANT_A_TOKEN");
const TENANT_B_TOKEN = required("TENANT_B_TOKEN");
const CONTROL_TOKEN = required("TEST_CONTROL_TOKEN");
const CANDIDATE_DIGEST = required("CANDIDATE_DIGEST");
const RUN_ID = process.env.RUN_ID ?? crypto.randomUUID();

function record(value: unknown): Record<string, unknown> {
  if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("expected object");
  return value as Record<string, unknown>;
}
function stringField(value: unknown, key: string): string {
  const field = record(value)[key];
  if (typeof field !== "string") throw new Error(`missing string ${key}`);
  return field;
}
function numberField(value: unknown, key: string): number {
  const field = record(value)[key];
  if (typeof field !== "number") throw new Error(`missing number ${key}`);
  return field;
}
async function call(base: string, path: string, token: string, init: { method?: string; body?: unknown; headers?: Record<string, string> } = {}): Promise<Result> {
  const response = await fetch(`${base}${path}`, {
    method: init.method ?? (init.body === undefined ? "GET" : "POST"),
    headers: { authorization: `Bearer ${token}`, "content-type": "application/json", ...init.headers },
    body: init.body === undefined ? undefined : JSON.stringify(init.body),
  });
  const body = response.status === 204 ? null : await response.json().catch(() => null);
  return { status: response.status, body };
}
const api = (path: string, token: string, init?: Parameters<typeof call>[3]) => call(BASE_URL, path, token, init);
const control = (path: string, body?: unknown) => call(CONTROL_URL, `/test-control${path}`, CONTROL_TOKEN, { body });
const unique = (name: string) => `${name}-${RUN_ID}`;

async function eventually(load: () => Promise<Result>, accept: (value: Result) => boolean, label: string): Promise<Result> {
  const deadline = Date.now() + 30_000;
  let last: Result = { status: 0, body: null };
  while (Date.now() < deadline) {
    last = await load();
    if (accept(last)) return last;
    await Bun.sleep(100);
  }
  throw new Error(`${label} did not converge; last=${JSON.stringify(last)}`);
}
async function seed(sku: string, available: number): Promise<void> {
  expect((await control("/inventory/seed", { tenantId: "tenant-a", sku, available })).status).toBe(200);
}
async function submit(key: string, sku: string, quantity = 1, headers?: Record<string, string>): Promise<Result> {
  return api("/v1/orders", TENANT_A_TOKEN, { body: { sku, quantity }, headers: { "idempotency-key": key, ...headers } });
}
async function confirmed(orderId: string): Promise<Result> {
  return eventually(
    () => api(`/v1/orders/${orderId}`, TENANT_A_TOKEN),
    ({ status, body }) => status === 200 && record(body).status === "CONFIRMED",
    `order ${orderId}`,
  );
}
const effects = (orderId: string) => control(`/orders/${orderId}/effects`);

async function run(command: string[], cwd?: string): Promise<string> {
  const process = Bun.spawn(command, { cwd, env: process.env, stdout: "pipe", stderr: "pipe" });
  const [stdout, stderr, exitCode] = await Promise.all([
    new Response(process.stdout).text(),
    new Response(process.stderr).text(),
    process.exited,
  ]);
  if (exitCode !== 0) throw new Error(`${command.join(" ")} failed: ${stderr}`);
  return stdout.trim();
}

describe("ParcelFlow release candidate", () => {
  beforeAll(async () => {
    expect((await api("/health/ready", TENANT_A_TOKEN)).status).toBe(200);
    await mkdir("evidence", { recursive: true });
  });

  test("confirmed order reaches one complete effect chain", async () => {
    const sku = unique("confirmed");
    await seed(sku, 2);
    const first = await submit(unique("idem-confirmed"), sku);
    expect([201, 202]).toContain(first.status);
    const orderId = stringField(first.body, "orderId");
    await confirmed(orderId);
    expect(record((await effects(orderId)).body)).toMatchObject({ authorizations: 1, reservations: 1, notifications: 1 });
  });

  test("same idempotency key returns one order and one effect set", async () => {
    const sku = unique("duplicate");
    const key = unique("idem-duplicate");
    await seed(sku, 2);
    const first = await submit(key, sku);
    const duplicate = await submit(key, sku);
    const orderId = stringField(first.body, "orderId");
    expect(stringField(duplicate.body, "orderId")).toBe(orderId);
    await confirmed(orderId);
    expect(record((await effects(orderId)).body)).toMatchObject({ authorizations: 1, reservations: 1, notifications: 1 });
  });

  test("out of stock rejects before payment and never makes stock negative", async () => {
    const sku = unique("empty");
    await seed(sku, 0);
    const denied = await submit(unique("idem-empty"), sku);
    expect(denied.status).toBe(409);
    expect(record(denied.body)).toMatchObject({ error: { code: "OUT_OF_STOCK" } });
    const state = await control(`/inventory/${sku}`);
    expect(record(state.body)).toMatchObject({ available: 0, paymentCalls: 0 });
  });

  test("payment timeout reaches compensated without a leaked reservation", async () => {
    const sku = unique("timeout");
    const key = unique("idem-timeout");
    await seed(sku, 1);
    await control("/faults", { kind: "payment-timeout", idempotencyKey: key });
    const started = await submit(key, sku);
    expect([202, 504]).toContain(started.status);
    const orderId = stringField(started.body, "orderId");
    await eventually(() => api(`/v1/orders/${orderId}`, TENANT_A_TOKEN), ({ body }) => record(body).status === "COMPENSATED", "compensation");
    expect(record((await effects(orderId)).body)).toMatchObject({ activeReservations: 0, duplicateAuthorizations: 0 });
  });

  test("duplicate and out-of-order events create one effect", async () => {
    const exercise = await control("/events/exercise-ordering", {
      tenantId: "tenant-a",
      aggregateId: unique("aggregate"),
      deliveries: [{ eventId: unique("evt-3"), sequence: 3 }, { eventId: unique("evt-2"), sequence: 2 }, { eventId: unique("evt-3"), sequence: 3 }],
    });
    expect(exercise.status).toBe(200);
    expect(record(exercise.body)).toMatchObject({ duplicateEffects: 0, finalSequence: 3, quarantinedGapRecovered: true });
  });

  test("cross-tenant access is denied with a positive control", async () => {
    const sku = unique("tenant");
    await seed(sku, 1);
    const created = await submit(unique("idem-tenant"), sku);
    const orderId = stringField(created.body, "orderId");
    const own = await api(`/v1/orders/${orderId}`, TENANT_A_TOKEN);
    const foreign = await api(`/v1/orders/${orderId}`, TENANT_B_TOKEN);
    expect(own.status).toBe(200);
    expect(foreign.status).toBe(403);
    expect(numberField((await control(`/orders/${orderId}/foreign-row-count`)).body, "count")).toBe(0);
  });

  test("outbox replay completes exactly once", async () => {
    const sku = unique("replay");
    const key = unique("idem-replay");
    await seed(sku, 1);
    await control("/publisher/pause-after-commit", { idempotencyKey: key });
    const created = await submit(key, sku);
    const orderId = stringField(created.body, "orderId");
    expect((await control("/publisher/restart")).status).toBe(200);
    await confirmed(orderId);
    expect(record((await effects(orderId)).body)).toMatchObject({ outboxPublications: 1, notifications: 1 });
  });

  test("evidence chain joins the immutable candidate", async () => {
    const sku = unique("evidence");
    await seed(sku, 1);
    const created = await submit(unique("idem-evidence"), sku);
    const orderId = stringField(created.body, "orderId");
    await confirmed(orderId);
    const chain = await api(`/v1/evidence/orders/${orderId}`, TENANT_A_TOKEN);
    expect(chain.status).toBe(200);
    expect(record(chain.body)).toMatchObject({ candidateDigest: CANDIDATE_DIGEST, orderId });
    for (const field of ["requestId", "traceId", "eventId", "policyRevision", "artifactDigest", "deploymentRevision", "runId"]) {
      expect(typeof record(chain.body)[field], field).toBe("string");
    }
  });

  test("failed canary rolls routing back while stable state survives", async () => {
    const cwd = required("PULUMI_WORKDIR");
    const stack = required("PULUMI_STACK");
    const sku = unique("canary");
    await seed(sku, 1);
    await run(["pulumi", "config", "set", "canaryWeight", "5", "--stack", stack], cwd);
    await run(["pulumi", "up", "--yes", "--stack", stack], cwd);
    const failed = await submit(unique("idem-canary"), sku, 1, { "x-release-track": "canary" });
    expect(failed.status).toBeGreaterThanOrEqual(500);
    await run(["pulumi", "config", "set", "canaryWeight", "0", "--stack", stack], cwd);
    await run(["pulumi", "up", "--yes", "--stack", stack], cwd);
    const controlOrder = await api("/v1/orders?limit=1", TENANT_A_TOKEN);
    expect(controlOrder.status).toBe(200);
    expect(record((await control("/queue/state")).body)).toMatchObject({ lostMessages: 0 });
  });

  test("export, revoke, disable, prove isolation, then optionally destroy disposable stack", async () => {
    expect((await api("/v1/tenants/tenant-a/quiesce", TENANT_A_TOKEN, { body: {} })).status).toBe(202);
    await eventually(() => api("/v1/tenants/tenant-a/drain", TENANT_A_TOKEN), ({ body }) => record(body).activeWriters === 0 && record(body).queueDepth === 0, "tenant drain");
    const exported = await fetch(`${BASE_URL}/v1/tenants/tenant-a/export`, { headers: { authorization: `Bearer ${TENANT_A_TOKEN}` } });
    expect(exported.status).toBe(200);
    const bytes = new Uint8Array(await exported.arrayBuffer());
    const checksum = [...new Uint8Array(await crypto.subtle.digest("SHA-256", bytes))].map((byte) => byte.toString(16).padStart(2, "0")).join("");
    await Bun.write(`evidence/tenant-a-${RUN_ID}.ndjson`, bytes);
    expect((await api("/v1/tenants/tenant-a/export/accept", TENANT_A_TOKEN, { body: { checksum } })).status).toBe(200);
    expect((await api("/v1/tenants/tenant-a/retire", TENANT_A_TOKEN, { body: { checksum, revokeSessions: true, revokeWorkloads: true, revokeSecrets: true, disableIngress: true } })).status).toBe(202);
    expect([401, 403, 410]).toContain((await api("/v1/orders", TENANT_A_TOKEN)).status);
    expect((await api("/v1/orders?limit=1", TENANT_B_TOKEN)).status).toBe(200);

    if (process.env.DESTROY_EPHEMERAL === "true") {
      const cwd = required("PULUMI_WORKDIR");
      const stack = required("PULUMI_STACK");
      expect(stack).toMatch(/^parcelflow\/e2e-[a-zA-Z0-9-]+$/);
      expect(await run(["pulumi", "stack", "output", "environmentMarker", "--stack", stack], cwd)).toBe("disposable-e2e");
      await Bun.write("evidence/stack-before-destroy.json", await run(["pulumi", "stack", "export", "--stack", stack], cwd));
      await run(["pulumi", "preview", "--destroy", "--diff", "--stack", stack], cwd);
      await run(["pulumi", "destroy", "--yes", "--stack", stack], cwd);
      await Bun.write("evidence/stack-after-destroy.json", await run(["pulumi", "stack", "export", "--stack", stack], cwd));
    }
  });
});

Save this single fence as tests/e2e/parcelflow-release.test.ts. It implements all ten matrix rows without importing domain handlers or an absent harness. The authenticated /test-control surface exists only in the disposable E2E deployment, injects faults and reads evidence but cannot create domain outcomes directly. Polling uses explicit deadlines and includes last observed state in failures; fixed sleeps are forbidden because they hide whether the system converged. Run this file without --concurrent so the final lifecycle case remains last.

A screenshot or pass count cannot reproduce a distributed result. Every probe emits one machine-readable observation that is sealed with the candidate manifest.

export type ScenarioObservation = Readonly<{
  observationId: string;
  candidateDigest: string;
  scenario: string;
  actor: string;
  tenantId: string | null;
  resource: string;
  scope: readonly string[];
  precondition: string;
  expected: string;
  observed: string;
  environment: string;
  timestamp: string;
  sourceFixtureId: string;
  requestId: string | null;
  traceId: string | null;
  artifactDigest: string;
  deploymentRevision: string;
  policyRevision: string;
  runId: string;
}>;

The evidence validator rejects missing fields, secret-shaped values, changed candidate digests, an unexpected allow, a negative result without a positive control, or any scenario that bypasses public/internal authenticated product entry points.

TDD is strongest for domain rules and adapter contracts, where a failing focused test gives rapid design feedback. It is insufficient alone for network, database, queue, identity, telemetry, deployment, and lifecycle interactions. Balance fast tests with fewer high-value full-system paths.

name: parcelflow-release-proof
on:
  workflow_dispatch:
    inputs:
      stable_image_digest:
        description: Immutable stable image URI and digest
        required: true
      canary_image_digest:
        description: Immutable intentionally failing canary image URI and digest
        required: true
permissions:
  contents: read
  id-token: write
jobs:
  focused-tests:
    strategy:
      fail-fast: false
      matrix:
        suite: [unit, adapter-contract, integration, security]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
      - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
        with:
          bun-version: "1.3.0"
      - run: bun install --frozen-lockfile
      - run: bun run test:${{ matrix.suite }}
      - if: always()
        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
        with:
          name: evidence-${{ matrix.suite }}
          path: evidence/

  deployed-e2e:
    needs: focused-tests
    runs-on: ubuntu-latest
    env:
      PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
      PULUMI_STACK: parcelflow/e2e-${{ github.run_id }}
      PULUMI_WORKDIR: infra
      CANDIDATE_DIGEST: ${{ github.sha }}
      RUN_ID: ${{ github.run_id }}
      DESTROY_EPHEMERAL: "true"
      TENANT_A_TOKEN: ${{ secrets.E2E_TENANT_A_TOKEN }}
      TENANT_B_TOKEN: ${{ secrets.E2E_TENANT_B_TOKEN }}
      TEST_CONTROL_TOKEN: ${{ secrets.E2E_CONTROL_TOKEN }}
    steps:
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
      - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
        with:
          bun-version: "1.3.0"
      - uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708 # v5
        with:
          role-to-assume: ${{ secrets.E2E_AWS_ROLE_ARN }}
          aws-region: eu-west-2
      - run: |
          curl --fail --show-error --location --output pulumi.tar.gz https://github.com/pulumi/pulumi/releases/download/v3.227.0/pulumi-v3.227.0-linux-x64.tar.gz
          echo "${{ vars.PULUMI_3_227_0_LINUX_X64_SHA256 }}  pulumi.tar.gz" | sha256sum --check
          tar --extract --gzip --file pulumi.tar.gz
          echo "$PWD/pulumi" >> "$GITHUB_PATH"
      - run: bun install --frozen-lockfile
      - run: |
          pulumi -C infra stack init "$PULUMI_STACK"
          pulumi -C infra config set stableImageDigest "${{ inputs.stable_image_digest }}"
          pulumi -C infra config set canaryImageDigest "${{ inputs.canary_image_digest }}"
          pulumi -C infra config set certificateArn "${{ secrets.E2E_CERTIFICATE_ARN }}"
          pulumi -C infra config set hostedZoneId "${{ vars.E2E_HOSTED_ZONE_ID }}"
          pulumi -C infra config set publicHostname "e2e-${GITHUB_RUN_ID}.${{ vars.E2E_DNS_ZONE_NAME }}"
          pulumi -C infra config set canaryWeight 0
          pulumi -C infra config set protectData false
          pulumi -C infra config set enableTestControl true
          pulumi -C infra preview --diff
          pulumi -C infra up --yes
          CLUSTER=$(pulumi -C infra stack output clusterArn)
          MIGRATION_TASK=$(pulumi -C infra stack output migrationTaskDefinitionArn)
          NETWORK=$(pulumi -C infra stack output runtimeNetworkJson)
          RUN_TASK=$(aws ecs run-task --cluster "$CLUSTER" --launch-type FARGATE --task-definition "$MIGRATION_TASK" --network-configuration "$NETWORK" --query 'tasks[0].taskArn' --output text)
          aws ecs wait tasks-stopped --cluster "$CLUSTER" --tasks "$RUN_TASK"
          test "$(aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$RUN_TASK" --query 'tasks[0].containers[0].exitCode' --output text)" = "0"
          echo "BASE_URL=$(pulumi -C infra stack output albUrl)" >> "$GITHUB_ENV"
          echo "CONTROL_URL=$(pulumi -C infra stack output albUrl)" >> "$GITHUB_ENV"
      - run: bun test tests/e2e/parcelflow-release.test.ts --timeout 1200000
      - if: always()
        run: |
          mkdir -p evidence
          pulumi -C infra stack export --stack "$PULUMI_STACK" > evidence/final-stack-state.json
          pulumi -C infra destroy --yes --stack "$PULUMI_STACK"
      - if: always()
        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
        with:
          name: release-evidence-${{ github.run_id }}
          path: evidence/

The intended balance is many pure domain tests, one conformance suite per adapter, real PostgreSQL/SQS-compatible integration tests, targeted authorization/fault tests, and the ten-scenario deployed E2E. Action revisions, Bun, Pulumi, images, and the Pulumi archive checksum are immutable inputs. The AWS role is obtained through GitHub OIDC rather than a stored cloud key. Do not mock the behavior under review, and do not demand E2E for a pure deterministic function whose contract is already completely observed at unit level.

Code rollback and customer-data rollback are different operations. The capstone deploys an intentionally failing v2 fixture, sends bounded canary traffic, observes the failed confirmation probe, and restores v1 routing without restoring RDS or discarding SQS messages.

bun test tests/e2e/parcelflow-release.test.ts \
  --test-name-pattern "failed canary rolls routing back" \
  --timeout 1200000
pulumi -C "$PULUMI_WORKDIR" stack output release --stack "$PULUMI_STACK"

The supplied test implementation changes the Pulumi weight, applies the update, targets v2 deterministically through the canary-only header rule, observes failure, restores weight zero, applies again, and queries stable customer and queue state. Rollback fails if ordinary traffic still reaches v2, v2 does not drain, a schema is backward-incompatible, an accepted order disappears, a message is lost, or the v1 positive control does not recover within its objective.

Retirement is an ordered state machine because deleting infrastructure before export or authority revocation destroys proof and may leave access elsewhere. The course tears down only a disposable test stack; real production deletion remains an explicit owner gate.

DESTROY_EPHEMERAL=true bun test tests/e2e/parcelflow-release.test.ts \
  --test-name-pattern "export, revoke, disable" \
  --timeout 1200000

The supplied lifecycle test first quiesces tenant writes and drains scoped work, then exports and accepts the checksum, then revokes and disables access. It proves the retired tenant is denied while tenant B remains healthy before environment cleanup. Stack destruction is a separate final E2E-environment action: the implementation validates the exact parcelflow/e2e-* stack and the Pulumi disposable-e2e marker, exports before/after state, previews destroy, and refuses protected stacks. The tombstone stores hashes and proof identifiers, never exported customer payload.

Bun is a poor choice when a required dependency depends on unsupported Node/V8 behavior, the team cannot operate its runtime, or the workload is dominated by a Python-native scientific/ML library with no safe service boundary. A compatibility failure is evidence to use Node.js or Python, not a challenge to hide with a polyfill.

Microservices are a poor choice when one team releases the whole system together, modules share one transactional invariant, network failure adds no useful isolation, independent scaling is unmeasured, or operational cost exceeds the bounded customer benefit. In those conditions, merge Inventory or Fulfillment back behind the same ports and preserve the modular monolith. The architecture succeeds when boundaries can change without changing customer contracts or corrupting owned state.

Decision rule: retain Bun and each service boundary only while pinned compatibility, team ownership, measured scaling/failure evidence, customer outcomes, and operating cost justify them; otherwise choose the simpler runtime or deployment shape and rerun the same acceptance suite.

E2E tests provide broad confidence but are slower and more failure-prone than focused tests. Parallel execution reduces time but can create fixture interference; retries reduce noise but can hide races. Use unique tenant/run IDs, deterministic fake providers, explicit eventual deadlines, no shared mutable fixtures, and zero automatic retry for unexpected authorization allows or duplicate financial effects.

The release verdict is approved, blocked, or rolled-back; retirement is blocked, retired-with-authorized-residuals, or retired-and-disposable-stack-destroyed. No generic “green” or “done” state is accepted. One unexpected allow, duplicate charge, lost committed event, unjoined artifact, destructive rollback, missing export, surviving route, or wrong-stack target falsifies the relevant claim.

Test, queue, identity, database, and teardown behavior changes over time, so a release dossier cannot rest on undocumented assumptions. These primary sources anchor the mechanics that the final normal-path and adversarial probes must re-verify.

• Bun test runner
• Bun test configuration and coverage
• OpenID Connect Core 1.0
• PostgreSQL transaction isolation
• Amazon SQS at-least-once delivery
• Pulumi destroy command

A capstone can still mislead when it counts passing components instead of customer outcomes, boundary denials, and safe exit. Preserve these conclusions as the reusable decision model for any future runtime or deployment shape.

• The final product is the customer journey plus its denial, failure, recovery, release, and exit evidence—not a set of independently green services.
• bun:test coordinates normal product surfaces; it does not bypass services or duplicate domain logic.
• TDD drives domain and adapter design, while deployed E2E proves cross-boundary behavior that focused tests cannot observe.
• Roll back executable routing without blindly rolling back customer data or durable messages.
• Bun and microservices remain choices, not course dogma; the same acceptance contract must survive a simpler replacement.

The final verdict is invalid if any required scenario, immutable join, positive control, or lifecycle prerequisite is absent. Use this list as the release-and-retirement gate, not as a substitute for reading the produced evidence.

• [ ] Candidate source, lockfile, schema, policy, images, infrastructure, provider, and deployment revisions are immutable and joined.
• [ ] Confirmed, duplicate, out-of-stock, payment-timeout, duplicate/out-of-order, tenant-denial, outbox-replay, evidence-chain, canary-rollback, and retirement scenarios pass or produce blocking counterexamples.
• [ ] Every negative path includes an unaffected authorized positive control.
• [ ] No scenario bypasses normal HTTP, identity, row-policy, queue, release, or lifecycle enforcement.
• [ ] Evidence contains actor, resource, scope, precondition, expected, observed, environment, timestamp, and immutable source/run/trace/artifact identifiers.
• [ ] Export is accepted before destructive steps; ingress, sessions, workload grants, secrets, and consumers are disabled or revoked before residual probes.
• [ ] Destroy validation targets only the named disposable stack; production deletion remains owner-gated.
• [ ] Final verdict explicitly approves, blocks, rolls back, or retires the exact observed tuple.