Operate and Support with Observable Evidence

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: services/control_plane/domain.py

def retire(self, project: Project, actor: str) -> None:
        self._owner(project, actor)
        project.status = "retired"
        with self.repository.transaction():
            self.repository.save(project)
            self._audit(project, actor, "project.retired")
        self.runtime.delete(project.organization_id, project.project_id)

    def delete(self, project: Project, actor: str) -> None:
        self._owner(project, actor)
        if project.status != "retired":
            raise ValueError("retire project before deletion")

Code to reality

Declared intent

Enforce the product lifecycle so published runtime state is retired before durable deletion.

Interpreter

The injected Forge model applies ownership and state-machine rules independently of HTTP or storage technology.

Software effect

Retirement removes the hosted runtime and records status before deletion can tombstone control-plane state.

Hardware effect

Runtime containers and storage can be released in order while retained evidence remains physically durable.

Observable evidence

A denied early delete, successful retirement, absent runtime, tombstoned project, and retained audit event prove order.

The source tables below remain the detailed contract. Begin with these customer paths:

• D14-UC-01
  • Person: Support engineer
  • Job: Diagnose a failed Research Brief, Service Desk, or Field Inspection request
  • Observable result: Engineer identifies the failing boundary and restores the customer path using correlated, redacted evidence
• D14-UC-02
  • Person: Security and privacy reviewer
  • Job: Ensure operational evidence reveals no unauthorised customer content
  • Observable result: Logs, metrics, and traces retain identifiers and outcomes needed for support without raw prompt, connector secret, or domain record

Now turn each customer job into a story with a result that an engineer can check:

• D14-US-01
  • Story: As a support engineer, I want one request trace linked to release and infrastructure observations, so that I can identify the owning boundary before…
  • Observable acceptance: Case records symptom, actor, resource, scope, expected and observed result, environment, time, trace, release, cluster, and run IDs
• D14-US-02
  • Story: As a privacy reviewer, I want telemetry fields validated and redacted, so that support evidence cannot become a copy of customer data
  • Observable acceptance: Synthetic secret and domain-record canaries are absent from accepted telemetry, unsafe events are quarantined, and safe correlation fields remain

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D14-FLOW-01
  • Trigger: Customer opens a support case with a failed request ID
  • Responsible systems: Support Portal, Evidence Correlator, telemetry pipeline, generated-app policy probes, Kubernetes and GitOps observers, AWS inventory observers
  • Authoritative state: Case Store for diagnosis; each control plane remains authoritative for its live state
  • Owned record: SupportCase
  • Observable evidence: Case with actor, resource, scope, precondition, expected and observed results, environment, timestamps, trace, release, cluster, run, recovery, and…
  • Failure signal: Missing correlation, stale API observation, green GitOps with failed customer probe, or action taken by non-owner
• D14-FLOW-02
  • Trigger: Telemetry pipeline receives a synthetic secret or raw domain field
  • Responsible systems: Instrumentation SDK, redaction processor, schema registry, quarantine store, privacy review workflow
  • Authoritative state: Schema Registry for allowed fields; Quarantine Store for denied events
  • Owned record: EvidenceBundle
  • Observable evidence: Denial reason, quarantined event ID, corrected schema revision, negative canary result, safe trace correlation, environment, time, and run ID
  • Failure signal: Secret canary accepted, tenant field leaked, correlation destroyed, or quarantine bypassed

Enterprise problem: A Zheta Forge application can show green deployment and GitOps status while customers receive slow, denied, or incorrect responses, so support teams need evidence that crosses boundaries without exposing tenant data. Whole-course context: The incoming evidence is a release with known artifact and local rollback records; today defines the larger operating evidence required around it. Today’s slice: We inspect the implemented tenant-scoped evidence store and distinguish that local proof from future traces, metrics, Kubernetes, Argo CD, cluster-API, and AWS observations. End-of-day evidence: The runnable proof is a durable tenant-filtered event plus local health and lifecycle results; the complete support bundle remains the production target. Still unsolved: Customer export, application retirement, durable deletion, and live AWS operational proof remain deferred.

Operations fails when dashboards report component health without answering whether a customer job completed. These use cases start from a customer symptom and constrain support access.

More telemetry can increase risk without improving decisions, so every signal needs an operational question and a data boundary. These stories make diagnosis and evidence safety observable.

A green Argo CD application—meaning the GitOps controller reports its Git-declared Kubernetes resources reconciled—does not prove the Kubernetes API, AWS capacity, connector, or customer response is healthy now. The flows diagnose bottom-up only after capturing the customer-visible symptom.

An operator who jumps directly to Kubernetes can miss a provider outage or generated-app authorization defect and may mutate the wrong owner’s state. The evidence service correlates observations while leaving each plane authoritative for its own records.

Observability becomes an uncontrolled data lake when retention and field ownership are implicit. The model stores operational evidence and domain data separately, with redaction before durable ingestion.

Generated-application database: Not created in this runnable slice — the evidence store retains provider lifecycle events only; generated-app domain records and authorization outcomes remain owned by a future application database.

The same symptom can originate in hardware, cluster API, workload, delivery, or application policy, so read each boundary without mutating it first. The current repository has no live AWS or end-to-end telemetry fixture; its concrete evidence boundary is the tenant-filtered SQL event store. Read the exact implementation in services/shared/persistence.py.

def events(self, organization_id: str) -> list[dict[str, object]]:
    return [json.loads(row[0]) for row in self.database.all("SELECT payload FROM evidence_events WHERE organization_id=? ORDER BY observed_at, evidence_id", (organization_id,))]

Operational truth is a chain of scoped observations, not one universal green badge.

• Start with customer evidence, then diagnose substrate to service without skipping owners.
• GitOps health, Kubernetes readiness, and AWS existence prove different boundaries.
• Redacted correlation IDs connect evidence without copying customer domain data.

A support case is actionable only when evidence names its source, freshness, and owner.

• [ ] Customer symptom and unaffected control are reproducible
• [ ] Every observation is read-only until the failing owner is identified
• [ ] Telemetry schema forbids secrets and raw domain records
• [ ] Recovery contains environment, run, release, trace, and timestamp evidence