Change, Redeploy, and Roll Back Safely

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: services/control_plane/domain.py

def retire(self, project: Project, actor: str) -> None:
        self._owner(project, actor)
        project.status = "retired"
        with self.repository.transaction():
            self.repository.save(project)
            self._audit(project, actor, "project.retired")
        self.runtime.delete(project.organization_id, project.project_id)

    def delete(self, project: Project, actor: str) -> None:
        self._owner(project, actor)
        if project.status != "retired":
            raise ValueError("retire project before deletion")

Code to reality

Declared intent

Enforce the product lifecycle so published runtime state is retired before durable deletion.

Interpreter

The injected Forge model applies ownership and state-machine rules independently of HTTP or storage technology.

Software effect

Retirement removes the hosted runtime and records status before deletion can tombstone control-plane state.

Hardware effect

Runtime containers and storage can be released in order while retained evidence remains physically durable.

Observable evidence

A denied early delete, successful retirement, absent runtime, tombstoned project, and retained audit event prove order.

The source tables below remain the detailed contract. Begin with these customer paths:

• D13-UC-01
  • Person: Application maintainer
  • Job: Deploy an approved change to one generated application
  • Observable result: New Research Brief, Service Desk, or Field Inspection revision serves bounded traffic and meets its acceptance probes
• D13-UC-02
  • Person: Incident commander
  • Job: Restore service after a defective revision
  • Observable result: Route returns to a known-good compatible artifact and required data repair is explicit

Now turn each customer job into a story with a result that an engineer can check:

• D13-US-01
  • Story: As an application maintainer, I want a new immutable revision exposed to bounded traffic, so that failures are detected before all users are affected
  • Observable acceptance: Baseline, candidate digest, traffic percentage, compatibility result, metrics, environment, timestamps, run, and traces are recorded
• D13-US-02
  • Story: As an incident commander, I want recovery to choose rollback or forward repair explicitly, so that restoring code does not hide unsafe data state
  • Observable acceptance: Decision record names route, artifact, configuration, schema, data side effects, owner, observed recovery, and unaffected control

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D13-FLOW-01
  • Trigger: Maintainer selects Deploy approved change
  • Responsible systems: Release Orchestrator, compatibility checker, GitOps/runtime deployment controller, Route Controller, telemetry evaluator
  • Authoritative state: Release Store for intent; Git for workload declaration; runtime API for live deployment
  • Owned record: DeploymentRevision
  • Observable evidence: Actor, application, digests, policy, compatibility result, traffic split, expected and observed signals, environment, timestamp, run, and trace IDs
  • Failure signal: Candidate not ready, controller drift, threshold breach, or schema incompatibility
• D13-FLOW-02
  • Trigger: Canary breaches its error threshold
  • Responsible systems: Incident workflow, Route Controller, GitOps/runtime controller, Migration Service, generated-app data owner
  • Authoritative state: Incident Store for decision; generated-app database for domain state
  • Owned record: MigrationLedger
  • Observable evidence: Rollback decision, restored route digest, schema/data status, affected denial or recovery trace, unaffected positive trace, environment, time, and…
  • Failure signal: Route restored but errors persist, incompatible down-migration, repair incomplete, or unaffected control fails

Enterprise problem: Once customers depend on a Zheta Forge application, a prompt, connector, schema, or code change can break requests or corrupt data, and “redeploy the old commit” may not reverse database effects. Whole-course context: The incoming evidence is a local release with a known content-addressed artifact; today selects a recorded release for redeployment. Today’s slice: We implement owner-only artifact rollback in the local runtime and distinguish it from unimplemented configuration rollback, canary routing, schema compatibility, and data repair. End-of-day evidence: A known release restores its exact stored artifact and publishes release.rolled_back; an unknown release is denied and the project remains readable. Still unsolved: Canary and GitOps recovery, schema/data repair, long-term observability, support operations, and retirement remain deferred.

A redeployment is unsafe when the platform cannot say which changes are reversible and which require repair. These use cases cover controlled change and recovery from a bad release.

“Roll back” is ambiguous unless the customer can see whether bytes, configuration, and data all returned to a safe state. These stories require evidence for each dimension.

Replacing every pod at once converts one bad revision into a full outage, so deployment must expose a bounded comparison before promotion. Recovery begins from customer impact and follows the owner of the divergent state.

Rollback fails when multiple controllers write the same state or when runtime recovery is expected to undo generated-app data. One mutating owner per resource keeps diagnosis and repair directional.

A prior container image is not a recovery plan when schema or business records have changed. The model preserves compatibility and repair decisions alongside release identity.

Generated-application database: Not created in this runnable slice — rollback changes the stored hosted-runtime payload only; app-owned schema, migration ledger, compatibility checks, and business records remain production requirements.

Imperative pod edits may briefly hide symptoms and then be overwritten by GitOps, so production recovery must change the owning declaration. The current product model implements a smaller local rollback: the owner selects an existing release and the runtime redeploys its stored artifact. It does not route canary traffic, revert Git, or prove Argo CD recovery. Read the exact method in services/control_plane/domain.py.

def rollback(self, project: Project, actor: str, release_id: str) -> dict[str, str]:
    self._owner(project, actor)
    release = next((item for item in project.releases if item["release_id"] == release_id), None)
    if release is None:
        raise ValueError("release not found")
    self.runtime.deploy(project.organization_id, project.project_id, {"mode": "published", **release, "source": self.artifacts.get(project.organization_id, release["artifact_id"])})
    self._audit(project, actor, "release.rolled_back")
    return release

Rollback is a coordinated business recovery decision, not merely an older image tag.

• Bounded traffic limits blast radius and creates comparative evidence.
• Artifact, configuration, schema, and data reversibility are separate questions.
• Repair the desired state through its single owner, then verify the customer path.

A change is safe only when promotion and recovery are both rehearsed.

• [ ] Prior and candidate artifacts are immutable and identifiable
• [ ] Compatibility is checked before traffic changes
• [ ] Threshold breach stops promotion automatically
• [ ] Recovery proves affected and unaffected customer paths