Observability, Incidents, and Governance

Enterprise problem: One healthy launch does not show that a customer can control a fleet through failures, cost pressure, legal obligations, restoration, and retirement; hidden routes, grants, or backups can survive after an app appears deleted.

Whole-course context: Day 5 produced digest-bound v1/v2 releases, compatible migration and canary evidence, a rollback trace, and a corrected production deployment; today operates those artifacts across their remaining lifecycle.

Today’s slice: Customer and HelixWorks administrators correlate cross-plane telemetry and audit, enforce SLOs, quotas, and cost controls, rehearse an incident and restore, govern residency and retention, then export and retire one app completely.

End-of-day evidence: An assurance dossier joins incident, break-glass, restore, export, decommission, grant-revocation, deletion, backup-expiry, and residual-access observations through immutable IDs.

Still unsolved: Only organization-specific risk acceptance and ongoing reassessment remain; the course claims the tested archetype envelope, not universal correctness.

The customer outcome is a reliable, reviewable implementation of 06 observability incidents and governance. This day introduces the mechanism before policy detail and evidence review; it does not repeat the same customer stories in prose, tables, and diagrams.

Control-plane service owns desired state; runtime workers own execution; the policy/release boundary owns privileged effects. Compute: isolated service or sandbox tasks. Storage: PostgreSQL owns durable state, Redis is a bounded cache, object storage keeps artifacts/evidence, and queues are delivery buffers rather than authority.

Implement the topic as an owned state transition with a named controller, durable record, bounded worker action, and observable terminal evidence. Verify a denied or failed path before calling the mechanism complete.

Operational controls become misleading when a dashboard hides ownership or a delete button hides residual resources, because the customer cannot tell whether service recovered, access ended, or retained copies remain. The sections below reveal one Workboard operating system from its lifecycle position through a minimal loop, authority boundaries, SRP/IaC contracts, and observed incident-to-retirement evidence.

The generated application is a lifecycle, not a code-generation event. Day 6 consumes the accepted release and closes the operating loop through telemetry, incident response, restoration, export, retirement, retention, and observed residual-access proof.

The terminal state is never a generic “done.” It is an observed operating, restored-with-limitation, retired-with-policy-retained-records, or retired-and-expired decision that names every remaining owner and obligation.

It locates Day 6 in the complete create-to-retire lifecycle and makes recovery, export, retirement, retention, and deletion proof part of the product outcome.

Do not call an application complete while its operating state, recovery obligations, or terminal custody remain unnamed.

The smallest complete operating model turns an observed condition into a governed action and then measures whether the action worked. For Workboard, a signal without an accountable decision is noise, while an action without a resulting observation is an unproved mutation.

Telemetry explains runtime behavior; audit proves who attempted or authorized a consequential action. They join through controlled references, but neither replaces the other. The final proof may be restored service, denied break-glass, an accepted export, a deletion tombstone, or a legal-hold decision.

It supplies the smallest loop that can detect a Workboard condition, make an accountable change, and prove the new state.

Require every operational signal to support a named decision, and require every decision to end in a measured positive or negative result.

Centralizing all operational data into one privileged store can turn observability into a new authorization bypass. Each plane keeps its own identity, audit authority, retention, and enforcement point; correlation carries opaque references and approved fields, never a transferable permission.

Telemetry is sampled operational evidence; audit is decision-grade history. Neither alone proves access ended, so grant revocation always pairs the owning-plane event with an observed denial and an unaffected positive control.

The provider adapters remain capability-measured rather than assumed equivalent. Daytona audit logs can contribute provider-plane events when Daytona is selected; Cloudflare runtime and deployment observations contribute only their documented hosted-runtime signals. Neither adapter is promoted into Workboard audit authority, customer approval, backup ownership, or compliance conclusion.

It expands observation into provider control-plane, hosted-runtime, generated-application, and external-source boundaries with distinct identities, retention, telemetry, and audit ownership.

Correlate planes through approved identifiers; never centralize their authority or infer equivalent audit, network, secret, or preview semantics across providers.

A fleet super-service that can read telemetry, grant emergency access, restore data, and delete evidence violates least privilege and makes incident review circular. SRP assigns one durable state machine and authoritative store to each service, while shared identifiers, evidence envelopes, and policy schemas remain DRY in a versioned contract package. Declarative IaC pins the operating controls and separates reviewed policy from environment parameters.

The full model matters only when one Workboard workflow exercises healthy operation, refused access, incident containment, isolated restore, customer acceptance, export, retirement, retention or purge, and residual denial. Every branch below terminates in immutable evidence rather than a dashboard status.

A fleet dashboard that merges every event into one authority can let support staff cross customer boundaries and can hide which owner must respond. HelixWorks correlates provider control plane, hosted runtime, and generated-application signals while preserving separate authorization, audit stores, retention, and customer ownership.

Shared join fields include customer organization, app, environment, release ID, artifact digest, deployment, policy version, trace ID, and timestamp. Generated-app tenant and end-user fields remain access-controlled and are not promoted into broad metric labels. A customer admin sees its fleet; HelixWorks staff see only operational scope granted by policy or a recorded support grant.

Metrics alone cannot explain one request, while debug logs alone cannot prove an access decision after they expire. Telemetry provides operational signals—traces, metrics, and redacted logs—whereas audit records are durable, decision-grade events about access, policy, deployment, support, export, and deletion.

Propagate trace context across the gateway, app, authorization service, data store, queue, connector broker, and source proxy. Attach release digest and environment to traces. Keep high-cardinality tenant, user, resource, prompt, and content values out of global metric labels; put authorized investigation detail in protected traces or audit events. Test redaction with synthetic secrets and personal data markers before telemetry export.

Provider and app audit streams stay distinct but correlatable. Each append-only event includes event ID, actor, resource, scope, decision, policy version, trace ID, source plane, environment, occurrence time, observation time, and integrity metadata. Writers cannot rewrite history; retention changes, reads, exports, support access, and verification failures are themselves audited.

A platform-wide average can look healthy while one customer journey fails or one generated app consumes the fleet, so operators need per-service objectives and enforceable resource boundaries. A service-level indicator (SLI) is a measured user outcome; a service-level objective (SLO) is its target over a window.

An improvised emergency response can destroy evidence or turn provider support into standing customer-data access. The incident plan assigns command, operations, security, communications, customer liaison, and evidence roles before a page occurs, and break-glass remains narrow, expiring, approved, and audited.

A break-glass request names ticket, incident, human actor, customer, plane, resource, actions, reason, approvers, start, expiry, and recording requirements. It uses strong authentication and just-in-time credentials, pages the customer where policy requires, and cannot bypass audit. Control-plane access, runtime shell access, and generated-app data access are separate grants. Closeout revokes each credential and proves denial; a ticket marked closed is not evidence of revocation.

A backup job can be green while its contents are incomplete, cross-region policy is wrong, or restoration cannot meet the customer’s recovery objective. Recovery point objective (RPO) bounds acceptable data loss; recovery time objective (RTO) bounds acceptable restoration time, and both are defined per state class.

Inventory generated-app database, object storage, configuration needed for recovery, encryption-key dependencies, audit data, and evidence indexes. Preview data, production app data, source-system data, and the immutable production artifact have different owners and backup needs. Do not copy Salesforce or Snowflake source data into backups unless the customer contract and minimization policy explicitly require it.

Encrypt backups with separately controlled keys, restrict and audit access, validate residency, make retention and legal holds explicit, and protect catalogs from deletion with the workload. A restore creates a new isolated recovery environment first. Verify checksums, schema and migration state, tenant counts, equal-local-ID isolation fixtures, object references, application authorization, app-scoped connector state, and audit continuity before switching traffic. Workboard has no Day 4 revenue connector grant, so its restore must positively prove those separate grants remain unchanged. Artifact rollback and database restore remain separate approvals.

Deleting a project record does not prove that deployments, app data, backups, connector grants, or exported copies disappeared. HelixWorks maintains a lifecycle inventory for every data class and executes customer-approved policy at each owner boundary.

An operations course that stops after recovery leaves the hardest lifecycle claim untested, so one executable workflow carries Workboard from incident detection through verified restoration and final retirement. The lab ends only when residual probes show no controllable access and backup/retention obligations have reached their declared terminal state.

Execute the workflow with test tenants Alpha and Beta:

1. Inject a bounded Workboard failure that burns the mutation SLO and plants a synthetic integrity marker. Declare the incident, preserve trace/audit IDs, stop risky release expansion, and communicate scope.
2. Request one runtime break-glass grant. Confirm it cannot access HelixWorks organization policy or generated-app tenant data outside the approved command set; record every command; expire and revoke it.
3. Restore the latest eligible production backup into an isolated recovery environment. Measure RPO/RTO, verify checksum and schema, exercise Alpha/Beta equal-local-ID isolation, application roles, queues, objects, and audit continuity, then make an explicit traffic decision.
4. After service acceptance, begin retirement: announce the cutoff, block new Workboard memberships and sharing grants, freeze writes, drain jobs, and create the final customer export with manifest, checksum, encryption, and receipt.
5. Disable custom and platform routes, remove DNS/TLS bindings, scale workloads to zero, revoke Workboard runtime identity, secrets, webhooks, API keys, sessions, preview links, application grants, and project collaborators as scoped by the retirement order. Do not emit a connector revocation for sf-pipeline or snow-bookings, which belong to app-revenue.
6. Delete production and preview app data, objects, caches, indexes, queues, artifacts after their approved evidence window, and provider project records; write separate tombstones because project deletion is not transitive proof.
7. Advance simulated policy time or use an approved accelerated test class to observe backup, audit, and retained-copy expiry. Preserve only hold-authorized records and prove the hold’s scope and access controls.
8. Probe former Workboard URLs, custom domain, APIs, object URLs, queues, workload credentials, app sessions, preview links, project APIs, backup catalog, and search/telemetry indexes for denial or policy-authorized tombstone access. Separately execute positive controls through the connector broker to prove sf-pipeline and snow-bookings still serve app-revenue.
9. In a separate revenue-retirement reference flow, quiesce app-revenue, revoke only its workload and delegated connector grants, purge derived caches, and prove broker denial while Salesforce and Snowflake source data and source-native audits remain owned and retained by those sources.

Scattered dashboards cannot support a defensible customer decision, so the final dossier joins claims to immutable observations without merging authorization domains. It reports successes, failures, residual custody, holds, exceptions, and the tested envelope.

Each evidence row contains actor, resource, scope, precondition, expected, observed, environment, observedAt, and an immutable traceId, runId, artifactDigest, eventId, or receipt ID. The dossier includes:

• fleet inventory with app, owner, region, release digest, SLO, quota, budget, and lifecycle state;
• correlated but separately authorized provider, runtime, connector, and generated-app telemetry/audit;
• alert, incident declaration, containment, revocation, communication, and break-glass command evidence;
• backup identity, checksum, key dependency, restore run, RPO/RTO, and tenant/application verification;
• final export manifest, checksum, encryption and custody receipt;
• disabled route, DNS/TLS, workload and deployment observations;
• Workboard workload, session, link, membership, API-key and support-grant revocation observations, plus separate revenue-retirement connector revocation observations;
• data/object/index/queue/project deletion tombstones, legal-hold exceptions, and retention/backup expiry events;
• residual-access probe matrix showing every former path denied or intentionally retained under policy.

The final customer decision states operating, restored-with-limitation, retired-with-policy-retained-records, or retired-and-expired. It never says “deleted” when a backup or legal hold remains.

One restored Workboard instance is not platform reliability evidence, so governance aggregates repeated fleet outcomes while preserving per-app accountability. Shared invariants run against Workboard, the revenue dashboard, and the public intake app; domain-specific SLOs and data policy remain with each application owner.

The model registry records model/revision, intended use, evaluations, data terms, region, and deprecation. The tool and connector registries record schema, owner, permissions, side effects, approvals, rate limits, kill switches, and exit plans. Policies and exceptions are versioned, independently approved according to risk, time-bounded, and linked to deployments and evidence. Humans remain accountable; an agent supplies work and observations but never accepts residual risk.

Fleet statistics can support a platform claim only for the measured sample, period, workload, and controls. Passing all three archetypes exercises the intended envelope; it does not establish arbitrary application correctness.

Operations and deletion controls copied from product marketing can overstate recovery or erasure, so use primary specifications, government guidance, and official provider documentation. The original sources were accessed on 2026-07-28; portable-provider references were checked on 2026-08-06.

• OpenTelemetry Specification — trace, metric, log, resource, and context models for cross-service telemetry.
• NIST SP 800-34 Rev. 1 — contingency planning, recovery strategies, testing, and plan maintenance.
• NIST SP 800-88 Rev. 2 — media sanitization and evidence appropriate to the selected sanitization method.
• NIST SP 800-53 Rev. 5 — primary control catalog covering audit, contingency, access, incident, and system integrity controls.
• NIST AI RMF 1.0 — govern, map, measure, and manage functions for AI risk.
• Daytona audit logs — official organization audit fields, actor/action/target/outcome semantics, and programmatic access reference.
• Daytona sandboxes — official sandbox lifecycle, isolation, automated expiry, and deletion-state reference.
• Cloudflare Sandbox SDK API — official isolated lifecycle, execution, storage, backup, port, and tunnel capability index.
• Cloudflare Sandbox tunnels — official tunnel lifecycle and quick-versus-named exposure guidance.
• Cloudflare Wrangler configuration — official environment, binding, limit, route, and observability configuration reference.

A healthy dashboard can hide unauthorized access, untested recovery, or undeleted state, so operating a generated-app platform means controlling the entire fleet lifecycle, not celebrating a release. Recovery, governance, and retirement are credible only when every boundary produces scoped evidence and retained data is named honestly.

• Correlate provider, runtime, connector, and app signals without merging their permissions or retention.
• Use customer-journey SLOs, hard security invariants, quotas, and cost budgets together.
• Make break-glass narrow, expiring, recorded, independently revoked, and customer-visible where required.
• Restore into isolation and prove tenant, authorization, data, and audit integrity before traffic.
• Finish retirement with export, decommission, grant revocation, deletion tombstones, retention/backup expiry, and residual-access probes.
• Provision operations through pinned DRY IaC modules and keep each SRP owner’s policy, state machine, and evidence distinct.
• Normalize Daytona or Cloudflare observations through portable contracts without treating a provider event as a compliance conclusion.

Retirement is incomplete while any route, workload, grant, data copy, or backup remains unaccounted for, so every checked item must cite a current evidence row. A project-deleted event cannot satisfy the rest of this list.

• [ ] Provider, runtime, connector, and app telemetry/audit are correlatable but separately authorized and retained.
• [ ] Fleet SLOs, hard security invariants, quotas, costs, owners, alerts, and runbooks are active.
• [ ] Incident containment, communication, break-glass expiry, and independent revocation were observed.
• [ ] Restore met or measured RPO/RTO and passed checksum, schema, tenant, role, object, queue, and audit tests.
• [ ] Final export has scope, manifest, count, checksum, encryption, destination, approval, and custody receipt.
• [ ] Platform/custom routes, DNS/TLS bindings, workloads, jobs, and deployments are disabled.
• [ ] Workboard workload, secret, webhook, API-key, session, link, membership, project, and support grants are revoked; revenue connector grants remain active until the separately evidenced app-revenue retirement.
• [ ] Production/preview data, objects, caches, indexes, queues, artifacts, and project records have separate deletion tombstones.
• [ ] Backup, audit, evidence, and held-copy expiry or authorized retention is observed and named.
• [ ] Residual probes cover former URLs, APIs, objects, queues, identities, connector sources, catalogs, and indexes.
• [ ] Operations IaC drift, provider-adapter observations, destructive-plan approvals, and legal-hold vetoes are recorded.
• [ ] Compliance evidence pack covers architecture, release, operations, recovery, access exit, data lifecycle, exceptions, and falsifiers.
• [ ] Every evidence row carries actor, resource, scope, precondition, expected and observed result, environment, timestamp, and immutable identifier.

Close Northstar's lifecycle with the read-only evidence projection in project-evidence.ts. It consumes versioned facts from the platform services, suppresses duplicate delivery, and allowlists identifiers instead of copying arbitrary payloads into an assurance view.

public async handle(rawEvent: unknown): Promise<boolean> {
  const event = eventEnvelopeSchema.parse(rawEvent);
  const claimed = await this.inbox.claim(
    CONSUMER,
    `${event.tenantId}:${event.idempotencyKey}`,
  );
  if (!claimed) return false;

  await this.projection.append({
    tenantId: event.tenantId,
    eventId: event.eventId,
    eventType: event.eventType,
    occurredAt: event.occurredAt,
    correlationId: event.correlationId,
    causationId: event.causationId,
    aggregateId: event.aggregateId,
    runId: safeIdentifier(event.payload, 'runId'),
    decisionId: safeIdentifier(event.payload, 'decisionId'),
    artifactId: safeIdentifier(event.payload, 'artifactId'),
    releaseId: safeIdentifier(event.payload, 'releaseId'),
  });
  return true;
}

The event declares an immutable fact; the worker validates it, claims its tenant-scoped idempotency key, and appends a non-authoritative projection. Software state changes in the inbox and evidence store. Locally the test uses process memory; on AWS, SQS network traffic, ECS worker CPU/RAM, and database disk carry the same control relationship. A single row after duplicate delivery, correlated event IDs, and absence of secret references are observable evidence.

SRP makes projection the worker's only responsibility. DRY reuses versioned envelopes and the inbox port. IoC/DI swaps projection and inbox implementations at the composition root. MVC keeps this read model separate from commands. PubSub decouples service ownership from fleet evidence; IaC provisions the bus, queues, dead-letter queues, logs, encryption, and task identity.

pnpm vitest run workers/evidence-projector/src/evidence-projector.test.ts
pnpm smoke:product
pnpm verify

Deliver the same event twice, query with another tenant, inject a malformed envelope, and place an aws-sm:// reference in an unapproved payload field. Passing evidence means one row only, strict tenant isolation, schema rejection, and no secret reference in the projection. This projection supports investigation; it is not an authorization source and does not by itself prove backup restoration, legal deletion, SLO compliance, or incident response.