15

Export, Retire, and Delete Completely

End the customer lifecycle with a verifiable export and an ordered deletion across every owning plane.

Run it in the public monorepo

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: services/control_plane/domain.py

def retire(self, project: Project, actor: str) -> None:
        self._owner(project, actor)
        project.status = "retired"
        with self.repository.transaction():
            self.repository.save(project)
            self._audit(project, actor, "project.retired")
        self.runtime.delete(project.organization_id, project.project_id)

    def delete(self, project: Project, actor: str) -> None:
        self._owner(project, actor)
        if project.status != "retired":
            raise ValueError("retire project before deletion")

Code to reality

Declared intent
Enforce the product lifecycle so published runtime state is retired before durable deletion.
Interpreter
The injected Forge model applies ownership and state-machine rules independently of HTTP or storage technology.
Software effect
Retirement removes the hosted runtime and records status before deletion can tombstone control-plane state.
Hardware effect
Runtime containers and storage can be released in order while retained evidence remains physically durable.
Observable evidence
A denied early delete, successful retirement, absent runtime, tombstoned project, and retained audit event prove order.

Start with the people and the result they need

The source tables below remain the detailed contract. Begin with these customer paths:

  • D15-UC-01
    • Person: Organization owner
    • Job: Export one application’s source, configuration, and customer-owned domain data
    • Observable result: Owner receives a versioned, checksummed package with schema and scope manifest for Research Brief, Service Desk, or Field Inspection
  • D15-UC-02
    • Person: Data protection administrator
    • Job: Retire and delete an application after accepted export and retention checks
    • Observable result: Domains, sessions, grants, connectors, workloads, secrets, data, and scheduled backups become unavailable in dependency order

Turn each customer job into a testable story

Now turn each customer job into a story with a result that an engineer can check:

  • D15-US-01
    • Story: As an organization owner, I want a portable export with a manifest, so that I can retain or migrate my application without depending on Zheta Forge
    • Observable acceptance: Package names schema, records, assets, source, configuration exclusions, checksum, actor, scope, environment, timestamp, and export run ID
  • D15-US-02
    • Story: As a data protection administrator, I want deletion orchestrated across every owner, so that no active authority or customer data survives unnoticed
    • Observable acceptance: Receipt records each owner’s completion, retention exception, backup expiry, denied probe, unaffected control, environment, time, run, and traces

Add real state and observable proof

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

  • D15-FLOW-01
    • Trigger: Owner selects Export application
    • Responsible systems: Authorization Service, Export Orchestrator, source/config owners, Artifact Service, generated-app Export Adapter
    • Authoritative state: Export Store for manifest; source and generated-app stores remain authoritative for exported content
    • Owned record: ExportManifest
    • Observable evidence: Actor, application, record counts, exclusions, schema, checksum, expected and observed scope, environment, timestamp, export artifact, run, and…
    • Failure signal: Record-count mismatch, secret included, checksum failure, expired download, or owner not authorised
  • D15-FLOW-02
    • Trigger: Administrator selects Retire after accepted export
    • Responsible systems: Retirement Orchestrator, Grant and Connector Services, Route Controller, Runtime Deployer, Secret Manager adapter, generated-app Erasure Worker,…
    • Authoritative state: Retirement Ledger for orchestration; each boundary store for deletion state
    • Owned record: RetirementRun
    • Observable evidence: Owner acceptance, revocation IDs, route and workload absence, erasure counts, retention exceptions, backup expiry, denied and positive traces,…
    • Failure signal: Missing acknowledgement, new writes after freeze, live route, active grant, remaining data, or backup without expiry

The enterprise problem and today’s slice

Enterprise problem: A customer can stop using a Zheta Forge application while artifacts, grants, runtime secrets, backups, connector authority, and generated-app records continue to exist, creating cost, privacy, and security risk. Whole-course context: The incoming evidence is a locally released project with provider collaboration and audit records; today closes the lifecycle that is implemented. Today’s slice: We export provider project metadata, retire and tombstone its hosted-runtime row, require retirement before project deletion, and preserve audit evidence; connector revoke, artifact erasure, backups, and generated-app data remain future owner jobs. End-of-day evidence: The domain test proves wrong-order denial, tenant-scoped project absence, runtime retirement, and retained project.deleted evidence; a production erasure receipt does not yet exist. Still unsolved: Multi-owner erasure and the internal design principles and infrastructure implementation used to reproduce it are taught next.

Customer use cases

Deletion is not a single database statement because different owners retain different resources and obligations. These use cases separate a usable export from complete, ordered retirement.

Use case IDActorCustomer jobSuccess outcomeDenial or recovery evidence
D15-UC-01Organization ownerExport one application’s source, configuration, and customer-owned domain dataOwner receives a versioned, checksummed package with schema and scope manifest for Research Brief, Service Desk, or Field InspectionExport is denied to an unauthorised actor; missing records or checksum mismatch block retirement
D15-UC-02Data protection administratorRetire and delete an application after accepted export and retention checksDomains, sessions, grants, connectors, workloads, secrets, data, and scheduled backups become unavailable in dependency orderPost-delete probes deny access; incomplete owner acknowledgement pauses deletion; unaffected application remains healthy

Actor-centred user stories

A successful “delete” response is weak evidence when asynchronous resources and backups remain. These stories require customer acceptance and boundary-by-boundary completion.

Story IDUse case IDsUser storyObservable acceptance conditions
D15-US-01D15-UC-01As an organization owner, I want a portable export with a manifest, so that I can retain or migrate my application without depending on Zheta ForgePackage names schema, records, assets, source, configuration exclusions, checksum, actor, scope, environment, timestamp, and export run ID
D15-US-02D15-UC-02As a data protection administrator, I want deletion orchestrated across every owner, so that no active authority or customer data survives unnoticedReceipt records each owner’s completion, retention exception, backup expiry, denied probe, unaffected control, environment, time, run, and traces

End-to-end product flows

Deleting runtime pods before exporting data can destroy the customer’s only usable copy, while deleting data before revoking access can race active writes. The flows impose prerequisites and pause safely when proof is incomplete.

Flow IDUse case IDsPathTriggerNumbered stepsTerminal evidence
D15-FLOW-01D15-UC-01HappyOwner selects Export application1. Verify owner and export scope. 2. Create a consistent read snapshot. 3. Package source, allowed configuration, assets, schema, and domain records. 4. Exclude secrets and provider-owned internals. 5. Sign manifest and checksum. 6. Record owner download and acceptance.Actor, application, record counts, exclusions, schema, checksum, expected and observed scope, environment, timestamp, export artifact, run, and trace IDs
D15-FLOW-02D15-UC-02RecoveryAdministrator selects Retire after accepted export1. Freeze releases and writes. 2. Revoke shares, sessions, connector grants, and workload identities. 3. Remove routes. 4. Delete runtime resources and secrets. 5. Delete generated-app records and assets. 6. Schedule backup expiry. 7. Probe deleted and unaffected applications.Owner acceptance, revocation IDs, route and workload absence, erasure counts, retention exceptions, backup expiry, denied and positive traces, environment, time, and run ID

The owner asks for two connected outcomes: a usable copy and proof the hosted application no longer remains active. Retirement cannot start until the export is accepted or explicitly waived under policy.

System design derived from the flows

A central deletion service cannot directly own every record without breaking service boundaries and retention controls. It orchestrates idempotent lifecycle jobs—operations safe to repeat without broadening the result—whose actual owners acknowledge completion.

Use case IDEntry pointResponsible servicesAuthoritative storeFailure evidence
D15-UC-01Control-plane Export actionAuthorization Service, Export Orchestrator, source/config owners, Artifact Service, generated-app Export AdapterExport Store for manifest; source and generated-app stores remain authoritative for exported contentRecord-count mismatch, secret included, checksum failure, expired download, or owner not authorised
D15-UC-02Control-plane Retirement actionRetirement Orchestrator, Grant and Connector Services, Route Controller, Runtime Deployer, Secret Manager adapter, generated-app Erasure Worker, backup ownerRetirement Ledger for orchestration; each boundary store for deletion stateMissing acknowledgement, new writes after freeze, live route, active grant, remaining data, or backup without expiry

The control plane coordinates but does not seize ownership: runtime owners remove deployment resources, and generated-app owners export and erase domain data. Cross-plane references are opaque; deletion authority is separately granted, narrowly scoped, audited, and revoked after the run.

Data model and ownership

Retirement cannot be reconstructed from transient job logs, so the platform needs a durable ledger without retaining the customer content it just deleted. Records describe proofs, exceptions, and deadlines rather than copying payloads.

Generated-application database: Not created in this runnable slice — export covers provider project metadata, retirement tombstones hosted-runtime state, and deletion tombstones the provider project; customer-domain export and erasure require a future app-owned store.

Record or entityStore and ownerPrimary keyForeign key or opaque referenceTenant keyMaterial constraintLifecycle and deletionUse case IDs
ExportManifestExport Store, owned by Export Orchestratorexport_idOpaque application_id, source revision, asset, and generated-app snapshot refsorganization_idManifest is immutable, excludes secrets, records schema, counts, checksum, and expiryGenerate, sign, download, accept, expire payload; retain minimal receipt by policyD15-UC-01, D15-UC-02
RetirementRunControl-plane Retirement Ledger, owned by Lifecycle Orchestratorretirement_run_idOpaque export_id and per-owner job IDsorganization_idState machine cannot pass export acceptance, write freeze, or revocation without required proofRequest, pause, resume idempotently, complete, retain immutable proof, delete metadata by policyD15-UC-02
AppErasureJobGenerated-application database, owned by Erasure Workererasure_job_idOpaque retirement_run_idapp_tenant_idDeletes only exact tenant/application scope; records counts and tombstone without payloadQueue after freeze, execute, verify, retain tombstone, purge by legal scheduleD15-UC-02
BackupExpiryBackup catalogue, owned by Backup Servicebackup_expiry_idOpaque retirement_run_id and backup set IDsenvironment and tenant scopeEvery retained copy has legal basis and immutable expiry; restore is blocked after erasure cutoffSchedule, verify expiry, retain non-content proof, purge with catalogue policyD15-UC-02

The ledger advances only when each owner supplies its proof or a justified retention exception. A denied request to the retired app plus a healthy unrelated app distinguishes complete retirement from a platform outage.

Execute deletion as a guarded state machine

Broad recursive deletion is unsafe because an empty or wrong identifier can expand the blast radius. The current implementation enforces a smaller state machine: owner-only export, retire before delete, tenant-scoped runtime tombstone, and retained deletion evidence. It does not yet implement a signed export manifest, dry-run plan, per-owner erasure jobs, or AWS deletion. Read the exact lifecycle in services/control_plane/domain.py.

def retire(self, project: Project, actor: str) -> None:
    self._owner(project, actor)
    project.status = "retired"
    self.repository.save(project)
    self.runtime.delete(project.organization_id, project.project_id)
    self._audit(project, actor, "project.retired")

def delete(self, project: Project, actor: str) -> None:
    self._owner(project, actor)
    if project.status != "retired":
        raise ValueError("retire project before deletion")
Effect fieldWhat happens
Declared intentLet only the owner retire one tenant-scoped project and prevent deletion before retirement
InterpreterPython executes the Forge state machine; repository, runtime, and event adapters perform their owned effects
Software effectStatus becomes retired, runtime row is tombstoned, then project deletion can tombstone control state while audit evidence remains
Hardware effectLocal service CPU, network, and SQLite pages are used or released logically; no Kubernetes namespace, EKS cluster, S3 object, or AWS account is deleted
EvidenceEarly-delete denial, absent tenant/runtime project after the sequence, retained project.deleted evidence, and an unrelated project positive control

Key takeaways

Retirement is a multi-owner lifecycle with prerequisites, idempotency, and falsifiable completion evidence.

  • Export must be usable and accepted before destructive steps begin.
  • Revoke authority and freeze writes before erasing data.
  • Retention exceptions need an owner, basis, expiry, and blocked restore policy.

Checklist

An application is retired only when active access and unjustified retained data are both gone.

  • [ ] Export manifest includes schema, counts, checksum, exclusions, and expiry
  • [ ] Target resolution rejects empty, ambiguous, or wrong-owner identifiers
  • [ ] Every plane acknowledges its own deletion or justified retention exception
  • [ ] Post-delete denial and unaffected positive control are recorded