Prove the Product Through Failure and Recovery

Enterprise problem: A collection of healthy microservices is not a trustworthy product. Customers need the onboarding outcome, operators need a recovery path, and reviewers need correlated evidence across asynchronous boundaries.

Whole-course context: The preceding days built the control plane, run orchestration, policy and connector brokers, release controller, evidence projector, event delivery, and multi-environment infrastructure.

Today’s slice: Exercise one Northstar supplier onboarding from approved blueprint to evidence while injecting duplicate events, broker outage, connector timeout, cross-tenant access, capability revocation, and failed deployment canary.

End-of-day evidence: Every injected failure has a bounded terminal state, an owner, and an evidence chain; unaffected positive controls still pass.

Still unsolved: Product-specific UI depth, commercial billing, and every provider connector are intentionally outside this reference foundation.

The customer outcome is a reliable, reviewable implementation of 10 failure recovery and evidence capstone. This day introduces the mechanism before policy detail and evidence review; it does not repeat the same customer stories in prose, tables, and diagrams.

Control-plane service owns desired state; runtime workers own execution; the policy/release boundary owns privileged effects. Compute: isolated service or sandbox tasks. Storage: PostgreSQL owns durable state, Redis is a bounded cache, object storage keeps artifacts/evidence, and queues are delivery buffers rather than authority.

Implement the topic as an owned state transition with a named controller, durable record, bounded worker action, and observable terminal evidence. Verify a denied or failed path before calling the mechanism complete.

Run the cross-service product smoke scenario.

assert.equal(connectorBusinessEffects, 1);
assert.equal(crossTenantDecision.allowed, false);
assert.equal(revokedCapabilityDecision.allowed, false);
assert.equal(failedCanaryRecoveredDigest, previousArtifactDigest);
assert.equal(correlatedEvidenceRecords, 4);

These assertions are product invariants, not implementation details. Each service remains responsible for one decision while the scenario proves their contracts compose.

Customer outcome

The success criterion begins with a valuable customer job.

Do not call infrastructure health a completed onboarding.

Controlled workflow

Intent, execution, and output have separate immutable identities.

A run can execute only an approved digest with explicit budgets.

Governed effect

External authority is checked immediately before the effect.

Deny changed, expired, revoked, or cross-tenant claims without fallback.

Reliable distributed path

Delivery failure and duplicate delivery have explicit states.

Correlate retries, DLQ records, and compensations to the originating business action.

Release and evidence loop

Runtime and delivery decisions become one inspectable product history.

A release claim passes only when positive controls, negative probes, recovery evidence, and customer-visible state agree.

1. Start local dependencies with Compose and run the product smoke scenario.
2. Repeat the same event and confirm one business effect.
3. attempt a cross-tenant read and connector execution.
4. Revoke a capability between issue and execution.
5. Reject a production canary and verify rollback.
6. Query evidence by tenant and correlation ID; verify no secret material appears.

The product claim is falsified by a duplicate supplier effect, cross-tenant disclosure, post-revocation success, unrecovered failed canary, missing causal link, or secret in evidence. Passing evidence is a single correlated bundle containing blueprint digest, run and artifact IDs, policy decisions, connector receipt, idempotency claim, promotion or rollback event, and customer-visible evidence records.