Export, Retire, and Delete Completely
End the customer lifecycle with a verifiable export and an ordered deletion across every owning plane.
Run it in the public monorepo
This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.
Source: services/control_plane/domain.py
def retire(self, project: Project, actor: str) -> None:
self._owner(project, actor)
project.status = "retired"
with self.repository.transaction():
self.repository.save(project)
self._audit(project, actor, "project.retired")
self.runtime.delete(project.organization_id, project.project_id)
def delete(self, project: Project, actor: str) -> None:
self._owner(project, actor)
if project.status != "retired":
raise ValueError("retire project before deletion")
Code to reality
- Declared intent
- Enforce the product lifecycle so published runtime state is retired before durable deletion.
- Interpreter
- The injected Forge model applies ownership and state-machine rules independently of HTTP or storage technology.
- Software effect
- Retirement removes the hosted runtime and records status before deletion can tombstone control-plane state.
- Hardware effect
- Runtime containers and storage can be released in order while retained evidence remains physically durable.
- Observable evidence
- A denied early delete, successful retirement, absent runtime, tombstoned project, and retained audit event prove order.
Start with the people and the result they need
The source tables below remain the detailed contract. Begin with these customer paths:
- D15-UC-01
- Person: Organization owner
- Job: Export one application’s source, configuration, and customer-owned domain data
- Observable result: Owner receives a versioned, checksummed package with schema and scope manifest for Research Brief, Service Desk, or Field Inspection
- D15-UC-02
- Person: Data protection administrator
- Job: Retire and delete an application after accepted export and retention checks
- Observable result: Domains, sessions, grants, connectors, workloads, secrets, data, and scheduled backups become unavailable in dependency order
Turn each customer job into a testable story
Now turn each customer job into a story with a result that an engineer can check:
- D15-US-01
- Story: As an organization owner, I want a portable export with a manifest, so that I can retain or migrate my application without depending on Zheta Forge
- Observable acceptance: Package names schema, records, assets, source, configuration exclusions, checksum, actor, scope, environment, timestamp, and export run ID
- D15-US-02
- Story: As a data protection administrator, I want deletion orchestrated across every owner, so that no active authority or customer data survives unnoticed
- Observable acceptance: Receipt records each owner’s completion, retention exception, backup expiry, denied probe, unaffected control, environment, time, run, and traces
Add real state and observable proof
Finally trace each story through the system that owns its state and the evidence that proves the outcome:
- D15-FLOW-01
- Trigger: Owner selects Export application
- Responsible systems: Authorization Service, Export Orchestrator, source/config owners, Artifact Service, generated-app Export Adapter
- Authoritative state: Export Store for manifest; source and generated-app stores remain authoritative for exported content
- Owned record: ExportManifest
- Observable evidence: Actor, application, record counts, exclusions, schema, checksum, expected and observed scope, environment, timestamp, export artifact, run, and…
- Failure signal: Record-count mismatch, secret included, checksum failure, expired download, or owner not authorised
- D15-FLOW-02
- Trigger: Administrator selects Retire after accepted export
- Responsible systems: Retirement Orchestrator, Grant and Connector Services, Route Controller, Runtime Deployer, Secret Manager adapter, generated-app Erasure Worker,…
- Authoritative state: Retirement Ledger for orchestration; each boundary store for deletion state
- Owned record: RetirementRun
- Observable evidence: Owner acceptance, revocation IDs, route and workload absence, erasure counts, retention exceptions, backup expiry, denied and positive traces,…
- Failure signal: Missing acknowledgement, new writes after freeze, live route, active grant, remaining data, or backup without expiry
The enterprise problem and today’s slice
Enterprise problem: A customer can stop using a Zheta Forge application while artifacts, grants, runtime secrets, backups, connector authority, and generated-app records continue to exist, creating cost, privacy, and security risk. Whole-course context: The incoming evidence is a locally released project with provider collaboration and audit records; today closes the lifecycle that is implemented. Today’s slice: We export provider project metadata, retire and tombstone its hosted-runtime row, require retirement before project deletion, and preserve audit evidence; connector revoke, artifact erasure, backups, and generated-app data remain future owner jobs. End-of-day evidence: The domain test proves wrong-order denial, tenant-scoped project absence, runtime retirement, and retained project.deleted evidence; a production erasure receipt does not yet exist. Still unsolved: Multi-owner erasure and the internal design principles and infrastructure implementation used to reproduce it are taught next.
Customer use cases
Deletion is not a single database statement because different owners retain different resources and obligations. These use cases separate a usable export from complete, ordered retirement.
| Use case ID | Actor | Customer job | Success outcome | Denial or recovery evidence |
|---|---|---|---|---|
| D15-UC-01 | Organization owner | Export one application’s source, configuration, and customer-owned domain data | Owner receives a versioned, checksummed package with schema and scope manifest for Research Brief, Service Desk, or Field Inspection | Export is denied to an unauthorised actor; missing records or checksum mismatch block retirement |
| D15-UC-02 | Data protection administrator | Retire and delete an application after accepted export and retention checks | Domains, sessions, grants, connectors, workloads, secrets, data, and scheduled backups become unavailable in dependency order | Post-delete probes deny access; incomplete owner acknowledgement pauses deletion; unaffected application remains healthy |
Actor-centred user stories
A successful “delete” response is weak evidence when asynchronous resources and backups remain. These stories require customer acceptance and boundary-by-boundary completion.
| Story ID | Use case IDs | User story | Observable acceptance conditions |
|---|---|---|---|
| D15-US-01 | D15-UC-01 | As an organization owner, I want a portable export with a manifest, so that I can retain or migrate my application without depending on Zheta Forge | Package names schema, records, assets, source, configuration exclusions, checksum, actor, scope, environment, timestamp, and export run ID |
| D15-US-02 | D15-UC-02 | As a data protection administrator, I want deletion orchestrated across every owner, so that no active authority or customer data survives unnoticed | Receipt records each owner’s completion, retention exception, backup expiry, denied probe, unaffected control, environment, time, run, and traces |
End-to-end product flows
Deleting runtime pods before exporting data can destroy the customer’s only usable copy, while deleting data before revoking access can race active writes. The flows impose prerequisites and pause safely when proof is incomplete.
| Flow ID | Use case IDs | Path | Trigger | Numbered steps | Terminal evidence |
|---|---|---|---|---|---|
| D15-FLOW-01 | D15-UC-01 | Happy | Owner selects Export application | 1. Verify owner and export scope. 2. Create a consistent read snapshot. 3. Package source, allowed configuration, assets, schema, and domain records. 4. Exclude secrets and provider-owned internals. 5. Sign manifest and checksum. 6. Record owner download and acceptance. | Actor, application, record counts, exclusions, schema, checksum, expected and observed scope, environment, timestamp, export artifact, run, and trace IDs |
| D15-FLOW-02 | D15-UC-02 | Recovery | Administrator selects Retire after accepted export | 1. Freeze releases and writes. 2. Revoke shares, sessions, connector grants, and workload identities. 3. Remove routes. 4. Delete runtime resources and secrets. 5. Delete generated-app records and assets. 6. Schedule backup expiry. 7. Probe deleted and unaffected applications. | Owner acceptance, revocation IDs, route and workload absence, erasure counts, retention exceptions, backup expiry, denied and positive traces, environment, time, and run ID |
The owner asks for two connected outcomes: a usable copy and proof the hosted application no longer remains active. Retirement cannot start until the export is accepted or explicitly waived under policy.
System design derived from the flows
A central deletion service cannot directly own every record without breaking service boundaries and retention controls. It orchestrates idempotent lifecycle jobs—operations safe to repeat without broadening the result—whose actual owners acknowledge completion.
| Use case ID | Entry point | Responsible services | Authoritative store | Failure evidence |
|---|---|---|---|---|
| D15-UC-01 | Control-plane Export action | Authorization Service, Export Orchestrator, source/config owners, Artifact Service, generated-app Export Adapter | Export Store for manifest; source and generated-app stores remain authoritative for exported content | Record-count mismatch, secret included, checksum failure, expired download, or owner not authorised |
| D15-UC-02 | Control-plane Retirement action | Retirement Orchestrator, Grant and Connector Services, Route Controller, Runtime Deployer, Secret Manager adapter, generated-app Erasure Worker, backup owner | Retirement Ledger for orchestration; each boundary store for deletion state | Missing acknowledgement, new writes after freeze, live route, active grant, remaining data, or backup without expiry |
The control plane coordinates but does not seize ownership: runtime owners remove deployment resources, and generated-app owners export and erase domain data. Cross-plane references are opaque; deletion authority is separately granted, narrowly scoped, audited, and revoked after the run.
Data model and ownership
Retirement cannot be reconstructed from transient job logs, so the platform needs a durable ledger without retaining the customer content it just deleted. Records describe proofs, exceptions, and deadlines rather than copying payloads.
Generated-application database: Not created in this runnable slice — export covers provider project metadata, retirement tombstones hosted-runtime state, and deletion tombstones the provider project; customer-domain export and erasure require a future app-owned store.
| Record or entity | Store and owner | Primary key | Foreign key or opaque reference | Tenant key | Material constraint | Lifecycle and deletion | Use case IDs |
|---|---|---|---|---|---|---|---|
| ExportManifest | Export Store, owned by Export Orchestrator | export_id | Opaque application_id, source revision, asset, and generated-app snapshot refs | organization_id | Manifest is immutable, excludes secrets, records schema, counts, checksum, and expiry | Generate, sign, download, accept, expire payload; retain minimal receipt by policy | D15-UC-01, D15-UC-02 |
| RetirementRun | Control-plane Retirement Ledger, owned by Lifecycle Orchestrator | retirement_run_id | Opaque export_id and per-owner job IDs | organization_id | State machine cannot pass export acceptance, write freeze, or revocation without required proof | Request, pause, resume idempotently, complete, retain immutable proof, delete metadata by policy | D15-UC-02 |
| AppErasureJob | Generated-application database, owned by Erasure Worker | erasure_job_id | Opaque retirement_run_id | app_tenant_id | Deletes only exact tenant/application scope; records counts and tombstone without payload | Queue after freeze, execute, verify, retain tombstone, purge by legal schedule | D15-UC-02 |
| BackupExpiry | Backup catalogue, owned by Backup Service | backup_expiry_id | Opaque retirement_run_id and backup set IDs | environment and tenant scope | Every retained copy has legal basis and immutable expiry; restore is blocked after erasure cutoff | Schedule, verify expiry, retain non-content proof, purge with catalogue policy | D15-UC-02 |
The ledger advances only when each owner supplies its proof or a justified retention exception. A denied request to the retired app plus a healthy unrelated app distinguishes complete retirement from a platform outage.
Execute deletion as a guarded state machine
Broad recursive deletion is unsafe because an empty or wrong identifier can expand the blast radius. The current implementation enforces a smaller state machine: owner-only export, retire before delete, tenant-scoped runtime tombstone, and retained deletion evidence. It does not yet implement a signed export manifest, dry-run plan, per-owner erasure jobs, or AWS deletion. Read the exact lifecycle in services/control_plane/domain.py.
def retire(self, project: Project, actor: str) -> None:
self._owner(project, actor)
project.status = "retired"
self.repository.save(project)
self.runtime.delete(project.organization_id, project.project_id)
self._audit(project, actor, "project.retired")
def delete(self, project: Project, actor: str) -> None:
self._owner(project, actor)
if project.status != "retired":
raise ValueError("retire project before deletion")
| Effect field | What happens |
|---|---|
| Declared intent | Let only the owner retire one tenant-scoped project and prevent deletion before retirement |
| Interpreter | Python executes the Forge state machine; repository, runtime, and event adapters perform their owned effects |
| Software effect | Status becomes retired, runtime row is tombstoned, then project deletion can tombstone control state while audit evidence remains |
| Hardware effect | Local service CPU, network, and SQLite pages are used or released logically; no Kubernetes namespace, EKS cluster, S3 object, or AWS account is deleted |
| Evidence | Early-delete denial, absent tenant/runtime project after the sequence, retained project.deleted evidence, and an unrelated project positive control |
Key takeaways
Retirement is a multi-owner lifecycle with prerequisites, idempotency, and falsifiable completion evidence.
- Export must be usable and accepted before destructive steps begin.
- Revoke authority and freeze writes before erasing data.
- Retention exceptions need an owner, basis, expiry, and blocked restore policy.
Checklist
An application is retired only when active access and unjustified retained data are both gone.
- [ ] Export manifest includes schema, counts, checksum, exclusions, and expiry
- [ ] Target resolution rejects empty, ambiguous, or wrong-owner identifiers
- [ ] Every plane acknowledges its own deletion or justified retention exception
- [ ] Post-delete denial and unaffected positive control are recorded