Full-System E2E Proof and Retirement
Prove ParcelFlow as one customer system: release, attack, fail, recover, roll back, export, revoke, disable, and retire.
The enterprise problem and today’s slice
Enterprise problem: Green unit tests and a successful deployment cannot prove that an order survives duplicate requests, stock rejection, payment timeout, event replay, tenant attack, canary failure, or retirement without lost data, duplicate effects, residual access, or incomplete evidence.
Whole-course context: The incoming system includes Bun services, explicit ports and dependencies, tenant-owned PostgreSQL state, sync contracts, outbox events, idempotent consumers, resilience controls, identity boundaries, operational evidence, and an immutable Pulumi deployment; this final day tests their interactions as one released product.
Today’s slice: Run a deterministic production-shaped bun:test orchestrator and CI matrix across positive, denied, failure, recovery, rollback, export, revocation, ingress-disable, and ephemeral teardown paths, then decide whether Bun and the extracted services remain justified.
End-of-day evidence: A digest-bound release dossier joins every required scenario to actor, tenant, resource, precondition, expected and observed result, environment, time, source, trace, event, artifact, deployment, policy, and immutable run identifiers.
Still unsolved: Real payment-card processing, organization-specific compliance certification, permanent production deletion, and irreversible cloud teardown remain owner-governed operations outside this public course.
The smallest complete proof model
The general rule is claim → adversarial observation → accountable verdict. A release claim without a falsifier is optimism; an observation without an exact candidate cannot be reproduced; a verdict without owner and consequence cannot govern production.
The simple example sends one valid order and checks CONFIRMED. The realistic example injects payment timeout, restarts the outbox publisher, attacks tenant scope, and fails a canary while durable state remains. The failure case is a test that calls an internal function or deletes a resource directly, bypassing the path customers use. The reusable decision rule is to test through normal surfaces with least-privilege fixtures and reject any verdict that cannot join observation to immutable input and owner.
Define the complete scenario matrix
Each scenario has one decisive terminal state and one unaffected positive control. Running them in a fixed order makes lifecycle dependencies explicit, while isolated fixture IDs prevent one result from contaminating another.
| Scenario | Trigger and expected terminal state | Required evidence |
|---|---|---|
| Confirmed order | Valid tenant submits in-stock order; CONFIRMED | One order, reservation, payment authorization, event chain, notification, and joined trace |
| Idempotent duplicate | Same idempotency key submitted twice; same order returned | One order row, one charge, one reservation, one terminal notification |
| Out of stock | Requested quantity unavailable; REJECTED_OUT_OF_STOCK | No payment call, no negative stock, rejection event, customer response |
| Payment timeout compensation | Fake provider times out; bounded retries end in documented compensation | Retry count/deadline, reservation released or explicit review state, no duplicate charge |
| Duplicate/out-of-order events | Consumer receives repeats and later version first | Inbox/dedup evidence, no repeated effect, explicit stale-event handling |
| Tenant denial | Tenant B requests tenant A order | 403, no foreign query result, tenant A control succeeds |
| Outbox replay | Publisher stops after commit and before send, then restarts | Durable outbox row, eventual event, one effect, replay/trace identifiers |
| Evidence chain | One confirmed order queried across all evidence owners | Request, trace, order, event, policy, artifact, deployment, decision, and run IDs join |
| Canary rollback | v2 confirmation probe fails | v2 blocked, v1 routing restored, data/messages preserved, v1 recovery passes |
| Export and retirement | Accepted export followed by revoke, disable, residual probe, disposable teardown | Checksum/custody, revocation receipts, denials, unaffected tenant, exact destroy target, tombstone |
Build the Bun test orchestrator
An E2E orchestrator should coordinate fixtures and assertions, not reproduce service logic. It talks through HTTP and observable stores with bounded polling, records artifacts even on failure, and never imports domain handlers.
import { beforeAll, describe, expect, test } from "bun:test";
import { mkdir } from "node:fs/promises";
type Result = { status: number; body: unknown };
const required = (name: string): string => {
const value = process.env[name];
if (!value) throw new Error(`missing ${name}`);
return value;
};
const BASE_URL = required("BASE_URL");
const CONTROL_URL = process.env.CONTROL_URL ?? BASE_URL;
const TENANT_A_TOKEN = required("TENANT_A_TOKEN");
const TENANT_B_TOKEN = required("TENANT_B_TOKEN");
const CONTROL_TOKEN = required("TEST_CONTROL_TOKEN");
const CANDIDATE_DIGEST = required("CANDIDATE_DIGEST");
const RUN_ID = process.env.RUN_ID ?? crypto.randomUUID();
function record(value: unknown): Record<string, unknown> {
if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("expected object");
return value as Record<string, unknown>;
}
function stringField(value: unknown, key: string): string {
const field = record(value)[key];
if (typeof field !== "string") throw new Error(`missing string ${key}`);
return field;
}
function numberField(value: unknown, key: string): number {
const field = record(value)[key];
if (typeof field !== "number") throw new Error(`missing number ${key}`);
return field;
}
async function call(base: string, path: string, token: string, init: { method?: string; body?: unknown; headers?: Record<string, string> } = {}): Promise<Result> {
const response = await fetch(`${base}${path}`, {
method: init.method ?? (init.body === undefined ? "GET" : "POST"),
headers: { authorization: `Bearer ${token}`, "content-type": "application/json", ...init.headers },
body: init.body === undefined ? undefined : JSON.stringify(init.body),
});
const body = response.status === 204 ? null : await response.json().catch(() => null);
return { status: response.status, body };
}
const api = (path: string, token: string, init?: Parameters<typeof call>[3]) => call(BASE_URL, path, token, init);
const control = (path: string, body?: unknown) => call(CONTROL_URL, `/test-control${path}`, CONTROL_TOKEN, { body });
const unique = (name: string) => `${name}-${RUN_ID}`;
async function eventually(load: () => Promise<Result>, accept: (value: Result) => boolean, label: string): Promise<Result> {
const deadline = Date.now() + 30_000;
let last: Result = { status: 0, body: null };
while (Date.now() < deadline) {
last = await load();
if (accept(last)) return last;
await Bun.sleep(100);
}
throw new Error(`${label} did not converge; last=${JSON.stringify(last)}`);
}
async function seed(sku: string, available: number): Promise<void> {
expect((await control("/inventory/seed", { tenantId: "tenant-a", sku, available })).status).toBe(200);
}
async function submit(key: string, sku: string, quantity = 1, headers?: Record<string, string>): Promise<Result> {
return api("/v1/orders", TENANT_A_TOKEN, { body: { sku, quantity }, headers: { "idempotency-key": key, ...headers } });
}
async function confirmed(orderId: string): Promise<Result> {
return eventually(
() => api(`/v1/orders/${orderId}`, TENANT_A_TOKEN),
({ status, body }) => status === 200 && record(body).status === "CONFIRMED",
`order ${orderId}`,
);
}
const effects = (orderId: string) => control(`/orders/${orderId}/effects`);
async function run(command: string[], cwd?: string): Promise<string> {
const process = Bun.spawn(command, { cwd, env: process.env, stdout: "pipe", stderr: "pipe" });
const [stdout, stderr, exitCode] = await Promise.all([
new Response(process.stdout).text(),
new Response(process.stderr).text(),
process.exited,
]);
if (exitCode !== 0) throw new Error(`${command.join(" ")} failed: ${stderr}`);
return stdout.trim();
}
describe("ParcelFlow release candidate", () => {
beforeAll(async () => {
expect((await api("/health/ready", TENANT_A_TOKEN)).status).toBe(200);
await mkdir("evidence", { recursive: true });
});
test("confirmed order reaches one complete effect chain", async () => {
const sku = unique("confirmed");
await seed(sku, 2);
const first = await submit(unique("idem-confirmed"), sku);
expect([201, 202]).toContain(first.status);
const orderId = stringField(first.body, "orderId");
await confirmed(orderId);
expect(record((await effects(orderId)).body)).toMatchObject({ authorizations: 1, reservations: 1, notifications: 1 });
});
test("same idempotency key returns one order and one effect set", async () => {
const sku = unique("duplicate");
const key = unique("idem-duplicate");
await seed(sku, 2);
const first = await submit(key, sku);
const duplicate = await submit(key, sku);
const orderId = stringField(first.body, "orderId");
expect(stringField(duplicate.body, "orderId")).toBe(orderId);
await confirmed(orderId);
expect(record((await effects(orderId)).body)).toMatchObject({ authorizations: 1, reservations: 1, notifications: 1 });
});
test("out of stock rejects before payment and never makes stock negative", async () => {
const sku = unique("empty");
await seed(sku, 0);
const denied = await submit(unique("idem-empty"), sku);
expect(denied.status).toBe(409);
expect(record(denied.body)).toMatchObject({ error: { code: "OUT_OF_STOCK" } });
const state = await control(`/inventory/${sku}`);
expect(record(state.body)).toMatchObject({ available: 0, paymentCalls: 0 });
});
test("payment timeout reaches compensated without a leaked reservation", async () => {
const sku = unique("timeout");
const key = unique("idem-timeout");
await seed(sku, 1);
await control("/faults", { kind: "payment-timeout", idempotencyKey: key });
const started = await submit(key, sku);
expect([202, 504]).toContain(started.status);
const orderId = stringField(started.body, "orderId");
await eventually(() => api(`/v1/orders/${orderId}`, TENANT_A_TOKEN), ({ body }) => record(body).status === "COMPENSATED", "compensation");
expect(record((await effects(orderId)).body)).toMatchObject({ activeReservations: 0, duplicateAuthorizations: 0 });
});
test("duplicate and out-of-order events create one effect", async () => {
const exercise = await control("/events/exercise-ordering", {
tenantId: "tenant-a",
aggregateId: unique("aggregate"),
deliveries: [{ eventId: unique("evt-3"), sequence: 3 }, { eventId: unique("evt-2"), sequence: 2 }, { eventId: unique("evt-3"), sequence: 3 }],
});
expect(exercise.status).toBe(200);
expect(record(exercise.body)).toMatchObject({ duplicateEffects: 0, finalSequence: 3, quarantinedGapRecovered: true });
});
test("cross-tenant access is denied with a positive control", async () => {
const sku = unique("tenant");
await seed(sku, 1);
const created = await submit(unique("idem-tenant"), sku);
const orderId = stringField(created.body, "orderId");
const own = await api(`/v1/orders/${orderId}`, TENANT_A_TOKEN);
const foreign = await api(`/v1/orders/${orderId}`, TENANT_B_TOKEN);
expect(own.status).toBe(200);
expect(foreign.status).toBe(403);
expect(numberField((await control(`/orders/${orderId}/foreign-row-count`)).body, "count")).toBe(0);
});
test("outbox replay completes exactly once", async () => {
const sku = unique("replay");
const key = unique("idem-replay");
await seed(sku, 1);
await control("/publisher/pause-after-commit", { idempotencyKey: key });
const created = await submit(key, sku);
const orderId = stringField(created.body, "orderId");
expect((await control("/publisher/restart")).status).toBe(200);
await confirmed(orderId);
expect(record((await effects(orderId)).body)).toMatchObject({ outboxPublications: 1, notifications: 1 });
});
test("evidence chain joins the immutable candidate", async () => {
const sku = unique("evidence");
await seed(sku, 1);
const created = await submit(unique("idem-evidence"), sku);
const orderId = stringField(created.body, "orderId");
await confirmed(orderId);
const chain = await api(`/v1/evidence/orders/${orderId}`, TENANT_A_TOKEN);
expect(chain.status).toBe(200);
expect(record(chain.body)).toMatchObject({ candidateDigest: CANDIDATE_DIGEST, orderId });
for (const field of ["requestId", "traceId", "eventId", "policyRevision", "artifactDigest", "deploymentRevision", "runId"]) {
expect(typeof record(chain.body)[field], field).toBe("string");
}
});
test("failed canary rolls routing back while stable state survives", async () => {
const cwd = required("PULUMI_WORKDIR");
const stack = required("PULUMI_STACK");
const sku = unique("canary");
await seed(sku, 1);
await run(["pulumi", "config", "set", "canaryWeight", "5", "--stack", stack], cwd);
await run(["pulumi", "up", "--yes", "--stack", stack], cwd);
const failed = await submit(unique("idem-canary"), sku, 1, { "x-release-track": "canary" });
expect(failed.status).toBeGreaterThanOrEqual(500);
await run(["pulumi", "config", "set", "canaryWeight", "0", "--stack", stack], cwd);
await run(["pulumi", "up", "--yes", "--stack", stack], cwd);
const controlOrder = await api("/v1/orders?limit=1", TENANT_A_TOKEN);
expect(controlOrder.status).toBe(200);
expect(record((await control("/queue/state")).body)).toMatchObject({ lostMessages: 0 });
});
test("export, revoke, disable, prove isolation, then optionally destroy disposable stack", async () => {
expect((await api("/v1/tenants/tenant-a/quiesce", TENANT_A_TOKEN, { body: {} })).status).toBe(202);
await eventually(() => api("/v1/tenants/tenant-a/drain", TENANT_A_TOKEN), ({ body }) => record(body).activeWriters === 0 && record(body).queueDepth === 0, "tenant drain");
const exported = await fetch(`${BASE_URL}/v1/tenants/tenant-a/export`, { headers: { authorization: `Bearer ${TENANT_A_TOKEN}` } });
expect(exported.status).toBe(200);
const bytes = new Uint8Array(await exported.arrayBuffer());
const checksum = [...new Uint8Array(await crypto.subtle.digest("SHA-256", bytes))].map((byte) => byte.toString(16).padStart(2, "0")).join("");
await Bun.write(`evidence/tenant-a-${RUN_ID}.ndjson`, bytes);
expect((await api("/v1/tenants/tenant-a/export/accept", TENANT_A_TOKEN, { body: { checksum } })).status).toBe(200);
expect((await api("/v1/tenants/tenant-a/retire", TENANT_A_TOKEN, { body: { checksum, revokeSessions: true, revokeWorkloads: true, revokeSecrets: true, disableIngress: true } })).status).toBe(202);
expect([401, 403, 410]).toContain((await api("/v1/orders", TENANT_A_TOKEN)).status);
expect((await api("/v1/orders?limit=1", TENANT_B_TOKEN)).status).toBe(200);
if (process.env.DESTROY_EPHEMERAL === "true") {
const cwd = required("PULUMI_WORKDIR");
const stack = required("PULUMI_STACK");
expect(stack).toMatch(/^parcelflow\/e2e-[a-zA-Z0-9-]+$/);
expect(await run(["pulumi", "stack", "output", "environmentMarker", "--stack", stack], cwd)).toBe("disposable-e2e");
await Bun.write("evidence/stack-before-destroy.json", await run(["pulumi", "stack", "export", "--stack", stack], cwd));
await run(["pulumi", "preview", "--destroy", "--diff", "--stack", stack], cwd);
await run(["pulumi", "destroy", "--yes", "--stack", stack], cwd);
await Bun.write("evidence/stack-after-destroy.json", await run(["pulumi", "stack", "export", "--stack", stack], cwd));
}
});
});
Save this single fence as tests/e2e/parcelflow-release.test.ts. It implements all ten matrix rows without importing domain handlers or an absent harness. The authenticated /test-control surface exists only in the disposable E2E deployment, injects faults and reads evidence but cannot create domain outcomes directly. Polling uses explicit deadlines and includes last observed state in failures; fixed sleeps are forbidden because they hide whether the system converged. Run this file without --concurrent so the final lifecycle case remains last.
Preserve a complete evidence envelope
A screenshot or pass count cannot reproduce a distributed result. Every probe emits one machine-readable observation that is sealed with the candidate manifest.
export type ScenarioObservation = Readonly<{
observationId: string;
candidateDigest: string;
scenario: string;
actor: string;
tenantId: string | null;
resource: string;
scope: readonly string[];
precondition: string;
expected: string;
observed: string;
environment: string;
timestamp: string;
sourceFixtureId: string;
requestId: string | null;
traceId: string | null;
artifactDigest: string;
deploymentRevision: string;
policyRevision: string;
runId: string;
}>;
The evidence validator rejects missing fields, secret-shaped values, changed candidate digests, an unexpected allow, a negative result without a positive control, or any scenario that bypasses public/internal authenticated product entry points.
Run a layered CI matrix without replacing E2E
TDD is strongest for domain rules and adapter contracts, where a failing focused test gives rapid design feedback. It is insufficient alone for network, database, queue, identity, telemetry, deployment, and lifecycle interactions. Balance fast tests with fewer high-value full-system paths.
name: parcelflow-release-proof
on:
workflow_dispatch:
inputs:
stable_image_digest:
description: Immutable stable image URI and digest
required: true
canary_image_digest:
description: Immutable intentionally failing canary image URI and digest
required: true
permissions:
contents: read
id-token: write
jobs:
focused-tests:
strategy:
fail-fast: false
matrix:
suite: [unit, adapter-contract, integration, security]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: "1.3.0"
- run: bun install --frozen-lockfile
- run: bun run test:${{ matrix.suite }}
- if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: evidence-${{ matrix.suite }}
path: evidence/
deployed-e2e:
needs: focused-tests
runs-on: ubuntu-latest
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
PULUMI_STACK: parcelflow/e2e-${{ github.run_id }}
PULUMI_WORKDIR: infra
CANDIDATE_DIGEST: ${{ github.sha }}
RUN_ID: ${{ github.run_id }}
DESTROY_EPHEMERAL: "true"
TENANT_A_TOKEN: ${{ secrets.E2E_TENANT_A_TOKEN }}
TENANT_B_TOKEN: ${{ secrets.E2E_TENANT_B_TOKEN }}
TEST_CONTROL_TOKEN: ${{ secrets.E2E_CONTROL_TOKEN }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: "1.3.0"
- uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708 # v5
with:
role-to-assume: ${{ secrets.E2E_AWS_ROLE_ARN }}
aws-region: eu-west-2
- run: |
curl --fail --show-error --location --output pulumi.tar.gz https://github.com/pulumi/pulumi/releases/download/v3.227.0/pulumi-v3.227.0-linux-x64.tar.gz
echo "${{ vars.PULUMI_3_227_0_LINUX_X64_SHA256 }} pulumi.tar.gz" | sha256sum --check
tar --extract --gzip --file pulumi.tar.gz
echo "$PWD/pulumi" >> "$GITHUB_PATH"
- run: bun install --frozen-lockfile
- run: |
pulumi -C infra stack init "$PULUMI_STACK"
pulumi -C infra config set stableImageDigest "${{ inputs.stable_image_digest }}"
pulumi -C infra config set canaryImageDigest "${{ inputs.canary_image_digest }}"
pulumi -C infra config set certificateArn "${{ secrets.E2E_CERTIFICATE_ARN }}"
pulumi -C infra config set hostedZoneId "${{ vars.E2E_HOSTED_ZONE_ID }}"
pulumi -C infra config set publicHostname "e2e-${GITHUB_RUN_ID}.${{ vars.E2E_DNS_ZONE_NAME }}"
pulumi -C infra config set canaryWeight 0
pulumi -C infra config set protectData false
pulumi -C infra config set enableTestControl true
pulumi -C infra preview --diff
pulumi -C infra up --yes
CLUSTER=$(pulumi -C infra stack output clusterArn)
MIGRATION_TASK=$(pulumi -C infra stack output migrationTaskDefinitionArn)
NETWORK=$(pulumi -C infra stack output runtimeNetworkJson)
RUN_TASK=$(aws ecs run-task --cluster "$CLUSTER" --launch-type FARGATE --task-definition "$MIGRATION_TASK" --network-configuration "$NETWORK" --query 'tasks[0].taskArn' --output text)
aws ecs wait tasks-stopped --cluster "$CLUSTER" --tasks "$RUN_TASK"
test "$(aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$RUN_TASK" --query 'tasks[0].containers[0].exitCode' --output text)" = "0"
echo "BASE_URL=$(pulumi -C infra stack output albUrl)" >> "$GITHUB_ENV"
echo "CONTROL_URL=$(pulumi -C infra stack output albUrl)" >> "$GITHUB_ENV"
- run: bun test tests/e2e/parcelflow-release.test.ts --timeout 1200000
- if: always()
run: |
mkdir -p evidence
pulumi -C infra stack export --stack "$PULUMI_STACK" > evidence/final-stack-state.json
pulumi -C infra destroy --yes --stack "$PULUMI_STACK"
- if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: release-evidence-${{ github.run_id }}
path: evidence/
The intended balance is many pure domain tests, one conformance suite per adapter, real PostgreSQL/SQS-compatible integration tests, targeted authorization/fault tests, and the ten-scenario deployed E2E. Action revisions, Bun, Pulumi, images, and the Pulumi archive checksum are immutable inputs. The AWS role is obtained through GitHub OIDC rather than a stored cloud key. Do not mock the behavior under review, and do not demand E2E for a pure deterministic function whose contract is already completely observed at unit level.
Prove canary rollback preserves state
Code rollback and customer-data rollback are different operations. The capstone deploys an intentionally failing v2 fixture, sends bounded canary traffic, observes the failed confirmation probe, and restores v1 routing without restoring RDS or discarding SQS messages.
bun test tests/e2e/parcelflow-release.test.ts \
--test-name-pattern "failed canary rolls routing back" \
--timeout 1200000
pulumi -C "$PULUMI_WORKDIR" stack output release --stack "$PULUMI_STACK"
The supplied test implementation changes the Pulumi weight, applies the update, targets v2 deterministically through the canary-only header rule, observes failure, restores weight zero, applies again, and queries stable customer and queue state. Rollback fails if ordinary traffic still reaches v2, v2 does not drain, a schema is backward-incompatible, an accepted order disappears, a message is lost, or the v1 positive control does not recover within its objective.
Export, revoke, disable ingress, and retire safely
Retirement is an ordered state machine because deleting infrastructure before export or authority revocation destroys proof and may leave access elsewhere. The course tears down only a disposable test stack; real production deletion remains an explicit owner gate.
DESTROY_EPHEMERAL=true bun test tests/e2e/parcelflow-release.test.ts \
--test-name-pattern "export, revoke, disable" \
--timeout 1200000
The supplied lifecycle test first quiesces tenant writes and drains scoped work, then exports and accepts the checksum, then revokes and disables access. It proves the retired tenant is denied while tenant B remains healthy before environment cleanup. Stack destruction is a separate final E2E-environment action: the implementation validates the exact parcelflow/e2e-* stack and the Pulumi disposable-e2e marker, exports before/after state, previews destroy, and refuses protected stacks. The tombstone stores hashes and proof identifiers, never exported customer payload.
Decide when not to use Bun or microservices
Bun is a poor choice when a required dependency depends on unsupported Node/V8 behavior, the team cannot operate its runtime, or the workload is dominated by a Python-native scientific/ML library with no safe service boundary. A compatibility failure is evidence to use Node.js or Python, not a challenge to hide with a polyfill.
Microservices are a poor choice when one team releases the whole system together, modules share one transactional invariant, network failure adds no useful isolation, independent scaling is unmeasured, or operational cost exceeds the bounded customer benefit. In those conditions, merge Inventory or Fulfillment back behind the same ports and preserve the modular monolith. The architecture succeeds when boundaries can change without changing customer contracts or corrupting owned state.
Decision rule: retain Bun and each service boundary only while pinned compatibility, team ownership, measured scaling/failure evidence, customer outcomes, and operating cost justify them; otherwise choose the simpler runtime or deployment shape and rerun the same acceptance suite.
Failure modes, trade-offs, and final verdict
E2E tests provide broad confidence but are slower and more failure-prone than focused tests. Parallel execution reduces time but can create fixture interference; retries reduce noise but can hide races. Use unique tenant/run IDs, deterministic fake providers, explicit eventual deadlines, no shared mutable fixtures, and zero automatic retry for unexpected authorization allows or duplicate financial effects.
The release verdict is approved, blocked, or rolled-back; retirement is blocked, retired-with-authorized-residuals, or retired-and-disposable-stack-destroyed. No generic “green” or “done” state is accepted. One unexpected allow, duplicate charge, lost committed event, unjoined artifact, destructive rollback, missing export, surviving route, or wrong-stack target falsifies the relevant claim.
Primary sources
Test, queue, identity, database, and teardown behavior changes over time, so a release dossier cannot rest on undocumented assumptions. These primary sources anchor the mechanics that the final normal-path and adversarial probes must re-verify.
Key takeaways
A capstone can still mislead when it counts passing components instead of customer outcomes, boundary denials, and safe exit. Preserve these conclusions as the reusable decision model for any future runtime or deployment shape.
- The final product is the customer journey plus its denial, failure, recovery, release, and exit evidence—not a set of independently green services.
bun:testcoordinates normal product surfaces; it does not bypass services or duplicate domain logic.- TDD drives domain and adapter design, while deployed E2E proves cross-boundary behavior that focused tests cannot observe.
- Roll back executable routing without blindly rolling back customer data or durable messages.
- Bun and microservices remain choices, not course dogma; the same acceptance contract must survive a simpler replacement.
Checklist
The final verdict is invalid if any required scenario, immutable join, positive control, or lifecycle prerequisite is absent. Use this list as the release-and-retirement gate, not as a substitute for reading the produced evidence.
- [ ] Candidate source, lockfile, schema, policy, images, infrastructure, provider, and deployment revisions are immutable and joined.
- [ ] Confirmed, duplicate, out-of-stock, payment-timeout, duplicate/out-of-order, tenant-denial, outbox-replay, evidence-chain, canary-rollback, and retirement scenarios pass or produce blocking counterexamples.
- [ ] Every negative path includes an unaffected authorized positive control.
- [ ] No scenario bypasses normal HTTP, identity, row-policy, queue, release, or lifecycle enforcement.
- [ ] Evidence contains actor, resource, scope, precondition, expected, observed, environment, timestamp, and immutable source/run/trace/artifact identifiers.
- [ ] Export is accepted before destructive steps; ingress, sessions, workload grants, secrets, and consumers are disabled or revoked before residual probes.
- [ ] Destroy validation targets only the named disposable stack; production deletion remains owner-gated.
- [ ] Final verdict explicitly approves, blocks, rolls back, or retires the exact observed tuple.