Observability, Incidents, and Enterprise Governance

Observability is the ability to understand a system's internal state from the signals it emits. Workboard needs two connected views: how the coding agent produced a change and how the deployed application behaves, because a clean application graph cannot explain a risky tool call and a complete agent trace cannot show a production tenant failure.

Instrument the agent runtime with one trace for each goal and a span—a timed unit of work—for every model call, tool call, approval, policy decision, patch, test run, retry, and stop condition. Record safe metadata such as model and tool versions, prompt-template version, input and output sizes, latency, token or cost totals, sandbox identity, gate result, and source commit. Do not make raw prompts or generated source universally searchable; apply classification, redaction, access controls, and retention before export.

Instrument the application separately. A Workboard request trace should cross the web edge, session check, authorization decision, API handler, tenant-scoped database call, queue or background worker, and response. Metrics count and aggregate behavior: request rate, errors, latency, saturation, authorization denials, todo-write success, queue age, and tenant-isolation probes. Logs explain discrete events with structured fields. Each signal has different strengths, so one should lead an operator to the others.

Join the planes using immutable release metadata: artifactDigest, commitSha, releaseId, environment, and policyVersion. This lets an operator move from a production error to the deployed digest, its release decision, and the agent run that created the source without pretending that telemetry alone proves causation.

Correlation means carrying stable identifiers across signals so an operator can navigate one event without guessing by timestamp. Traces show a request's path, metrics reveal population-level change, and logs provide carefully selected detail; useful observability combines all three while minimizing sensitive data.

Propagate a traceId across trusted service calls and attach the exact release digest and environment to access-controlled spans and logs. Cardinality is the number of distinct values a field can take; release IDs and digests accumulate over time, so they are not inherently low-cardinality metric labels. For metrics, use a bounded deployment cohort or service version only when it fits the cardinality budget. Tenant IDs, user IDs, todo text, prompt contents, and resource IDs can likewise explode metric cost or expose data, so keep them out of metric labels. When a tenant-specific investigation is authorized, use access-controlled traces or audit records instead of a global per-tenant metric label.

Use a redaction policy before telemetry leaves the process. Deny by default for passwords, tokens, cookies, authorization headers, todo titles and descriptions, prompt bodies, source files, model output, database parameters, and personal identifiers. Allow narrowly typed fields such as result code, duration, tool name, model version, policy version, digest, and pseudonymous principal reference. Hashing a predictable email address is pseudonymization, not guaranteed anonymization, so it still needs access and retention controls.

telemetryPolicy:
  allow:
    - traceId
    - releaseId
    - artifactDigest
    - operationName
    - resultCode
    - durationMs
    - policyVersion
  drop:
    - authorizationHeader
    - sessionCookie
    - todoText
    - promptBody
    - toolOutput

Sample routine successful traces to control cost, but retain errors, policy denials, canary traffic, and security probes at a higher rate. Keep sampling decisions consistent enough to preserve an end-to-end trace. Test redaction with planted synthetic secrets and fail the telemetry pipeline's release gate if those markers reach the collector.

An audit trail is a durable history of security- and governance-relevant decisions. Unlike debug logs, it must answer who acted, in which tenant, on what target, under which policy, with what result, and when—even if ordinary telemetry has expired.

Workboard uses the course artifact AuditEvent with { actor, tenantId, action, target, decision, policyVersion, traceId, occurredAt }. Add a unique event ID, schema version, source service, and integrity metadata in the concrete implementation. Emit events for sign-in outcomes, membership changes, sharing changes, authorization decisions, exports, administrative reads, API-key lifecycle, revocation, policy changes, agent approvals, release decisions, and emergency exceptions. Never store secret values or entire todo bodies merely to make the record detailed.

Protect the trail against silent editing and deletion. Restrict append permission to named workloads, separate readers from writers, encrypt transport and storage, define retention and legal holds, and replicate or export to a security-controlled account. Use tamper-evident techniques such as signed batches, hash chaining, immutable retention controls, or an independently protected digest ledger; then test verification and restoration. “Immutable storage” is only meaningful if privileged identities and retention changes are also audited.

Audit events should carry the operational traceId, while traces link back to the audit event ID. The trace explains execution detail; the audit trail establishes the decision record. Clock synchronization, sequence information, and explicit late-arrival handling make incident timelines defensible. Access to audit data is itself audited because these records reveal sensitive organizational behavior.

A service-level objective (SLO) is a measured reliability target over a stated window, while an error budget is the amount of allowed failure implied by that target. SLOs turn “Workboard seems healthy” into a shared threshold for release pace and incident response.

Choose user-centered service-level indicators (SLIs), the actual measurements behind an SLO. For Workboard, useful SLIs include successful authenticated page loads, successful tenant-authorized todo mutations, end-to-end latency, queue completion time, and revocation completion within its declared limit. Security invariants such as cross-tenant access denial are not tradable error budgets: one confirmed isolation breach pages immediately and stops promotion.

SLI	Example objective	Decision use
Authorized todo mutations	99.9% succeed over 28 days	Error budget governs routine release pace
Interactive latency	99% below 750 ms over 28 days	Canary comparison and capacity planning
Revocation workflow	99.9% of controllable access paths deny within 5 minutes	Escalate slow cache, session, or connection legs
Tenant-isolation probe	100% denied	Immediate security incident; no budget trade

An actionable alert names a user impact, contains a runbook, reaches an owner who can act, and avoids paging on harmless internal noise. Prefer multi-window burn-rate alerts, which detect rapid or sustained error-budget consumption, plus direct invariant alerts for security. Include release ID and digest, affected journey, trace links, current and baseline values, likely scope, and the first safe action. Deduplicate related symptoms and test that routing works during handoffs; an alert nobody receives is not a control.

Incident response is a prepared sequence for limiting harm and restoring safe service when reality violates an assumption. Workboard's runbook uses explicit stages—detect, contain, revoke, rollback or recover, investigate, and improve—so urgency does not erase accountability.

Detect and classify. Confirm the signal, declare severity, start an incident record, assign command and communications roles, and preserve timestamps, trace IDs, release digests, audit events, and affected tenant scope. A model-behavior anomaly, leaked secret, compromised dependency, tenant escape, and availability regression require different containment.

Contain. Stop canary expansion; disable the affected agent tool, model route, feature, integration, tenant path, or artifact; isolate suspicious workloads; and block known indicators. Prefer the smallest effective boundary, but protect users before preserving convenience. Record every emergency action.

Revoke. Invalidate compromised agent credentials, signing authority, sessions, refresh tokens, API keys, membership grants, caches, websocket connections, and background leases as applicable. Measure each leg rather than declaring revocation complete from one API response, and state any residual lifetime for self-contained tokens or offline data.

Rollback or recover. Route traffic to the last accepted digest when the transitional schema remains compatible. If state is corrupted, restore or repair through a rehearsed procedure, validate tenant and authorization invariants, and communicate user-visible limitations. Rollback removes faulty code from service; it cannot retract data already exposed or external actions already performed.

Investigate and improve. Build a timeline from protected evidence, identify contributing technical and organizational conditions, and distinguish facts from hypotheses. Convert the findings into owned changes: a new acceptance case, policy rule, evaluation, monitor, access boundary, supplier action, or runbook rehearsal. Verify each improvement and update the risk register; avoid reducing a system failure to one person's mistake.

Governance assigns decision rights and review duties to people rather than leaving safety to an autonomous system. Workboard uses risk tiers so the review burden grows with possible impact, while a RACI matrix makes clear who is Responsible, Accountable, Consulted, and Informed.

Change tier	Examples	Minimum treatment
Low	Documentation or isolated test refactor	Automated gates, peer review, evidence record
Medium	Model version, read-only agent tool, UI or dependency change	Evaluation comparison, security review as triggered, staged release
High	Write-capable tool, authorization policy, identity flow, tenant data path	Independent security and product approval, incident exercise, constrained canary
Critical	Signing root, audit retention, emergency bypass, destructive production tool	Executive risk owner plus security approval, dual control, explicit recovery proof

The model registry records provider, model and revision, intended uses, evaluation results, data-handling terms, geographic or regulatory constraints, and deprecation plan. The tool registry records schema, owner, permissions, sandbox, side effects, approval class, rate limits, and kill switch. Policy changes are versioned, reviewed, tested on allow and denial cases, and linked to deployments. Vendor review covers service dependencies, sub-processors, incident obligations, availability and exit plans, but vendor certification never replaces testing the integrated Workboard behavior.

Assign one accountable owner for each domain. A practical RACI has engineering Responsible for implementation and operational evidence; the product owner Accountable for user-value acceptance; security Responsible or Consulted for threat controls and incident response; platform Responsible for build, runtime, and telemetry; privacy and legal Consulted for data and license decisions; procurement Consulted for vendor terms; and an enterprise risk owner Accountable for high residual-risk acceptance. The coding agent is never Responsible or Accountable: it supplies work and evidence, but a person or organizational role owns the decision.

Exceptions use the same discipline as release policy: exact scope, reason, owner, compensating control, independent approval, expiry, and review date. Track repeated exceptions as a control-design signal. Suspend an exception immediately when its assumptions change, and do not let an “emergency” path become a permanent unobserved policy.

An assurance dossier is the reviewable package that connects Workboard's claims to controls and evidence. It supports a release, release-with-time-bounded-exception, or hold decision by making uncertainty visible; it is not a certificate that generated code can never fail.

Package the executable specification and acceptance matrix; versioned source, model, prompts, tools, policies, and dependencies; evaluation corpus and repeat-run trends; deterministic validation results; security and two-tenant denial tests; identity, sharing, and revocation evidence; the digest-keyed release manifest; SBOM, signature verification, and checked provenance predicate; canary thresholds; rollback rehearsal; SLOs and alert tests; audit-integrity verification; incident exercise; vendor reviews; and every active exception.

Each important claim maps to an EvidenceRecord with { criterionId, gate, result, artifactUri, commitSha, environment, observedAt }. Review freshness, independence, scope, and non-vacuity: a green test from different bytes, an expired scan, or an unchecked signed attestation is not supporting evidence. Record gaps as residual risks with likelihood, impact, affected tenants or data, compensating controls, owner, review date, and exit condition.

The final ReleaseDecision is { artifactDigest, gateResults, provenance, exceptions, approvers, decision }. Use one decision vocabulary throughout the lifecycle: release means go; release-with-time-bounded-exception means conditional go whose conditions are measurable, owned, expiring, and automatically enforced where possible; and hold means no-go, preserving the evidence, naming what must change, and preventing deployment. More evidence can increase confidence and narrow uncertainty; it cannot prove perfection, remove unknown failure modes, or transfer accountability to the agent.

These primary sources ground the telemetry, continuous verification, evaluation, and AI-risk practices used here. The access date matters for living specifications and program pages.

• OpenTelemetry Specification — specification version 1.59.0 as displayed on access; individual signal components carry their own stability levels. Accessed 2026-07-26.
• NIST AI 600-1, Generative AI Profile — final cross-sector profile published July 2024; use it with AI RMF 1.0 for generative-AI risk management. Accessed 2026-07-26.
• NIST SP 800-207, Zero Trust Architecture — final publication, August 2020; resource-focused access and explicit authentication and authorization rather than implicit network trust. Accessed 2026-07-26.
• NIST AI Test, Evaluation, Validation and Verification — living official program page on measurement and evaluation methods, not a numbered standard. Accessed 2026-07-26.

Operational assurance connects how Workboard was generated, what bytes were released, and how those bytes behave in production. The resulting evidence supports accountable risk decisions without promising that the application or agent is flawless.

• Observe the coding agent and Workboard application as separate planes joined by commit, release, digest, environment, and policy identifiers.
• Correlate traces, metrics, redacted logs, and audit events while keeping secrets and tenant content out of broad telemetry.
• Make the audit trail durable, access-controlled, tamper-evident, restorable, and itself audited.
• Define user-centered SLOs and error budgets, but treat tenant isolation and other security invariants as immediate incidents rather than tradable failure.
• Rehearse detect, contain, revoke, rollback or recover, investigate, and improve; measure every revocation and recovery leg.
• Govern models, tools, policies, vendors, and exceptions with risk tiers, named human owners, and clear RACI duties.
• Decide release, release-with-time-bounded-exception, or hold from a digest-keyed assurance dossier, evidence quality, exceptions, and explicit residual risk.

This checklist is the final operational and governance review for Workboard. Every checked item should point to a current artifact, named owner, or rehearsed action.

• [ ] Agent spans cover model, tool, approval, patch, gate, retry, cost, and stop decisions without exposing secrets.
• [ ] Application traces cross session, authorization, API, tenant-scoped data, queues, and responses and identify the deployed digest.
• [ ] Trace, metric, and log fields follow tested redaction, cardinality, retention, sampling, and access policies.
• [ ] AuditEvent records cover access, policy, revocation, agent approval, release, and exception decisions; integrity and restoration are tested.
• [ ] User-centered SLOs, error budgets, security invariants, burn alerts, paging routes, and runbooks have named owners.
• [ ] The incident exercise covered detect, contain, revoke, rollback or recover, investigate, improve, communications, and evidence preservation.
• [ ] Model, tool, policy, and vendor registries record versions, permissions, evaluation, ownership, data terms, kill switches, and exit plans.
• [ ] Risk tiers and the RACI assign accountable human decision-makers; the coding agent owns no approval.
• [ ] Every exception is scoped, justified, compensated, independently approved, expiring, and visible in the release decision.
• [ ] The assurance dossier maps each material claim to fresh, relevant evidence and records residual risks, owners, review dates, and exit conditions.
• [ ] The final digest-keyed ReleaseDecision is release, release-with-time-bounded-exception, or hold under the organization's stated risk tolerance.