05

Generate an Application Through an Auditable Job

Convert one immutable specification into one reproducible artifact while making retries, model calls, and policy decisions observable.

Run it in the public monorepo

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: services/control_plane/domain.py

def create(self, project_id: str, organization_id: str, name: str, actor: str, archetype: str) -> Project:
        project = Project(project_id, organization_id, name, actor, blueprint={"name": name, "archetype": archetype})
        project.collaborators.add(actor)
        with self.repository.transaction():
            self.repository.create(project)
            self._audit(project, actor, "project.created")
        return project

Code to reality

Declared intent
Create one tenant-owned project and its audit evidence as a single product operation.
Interpreter
Python executes the Forge domain model after the controller supplies authenticated command values.
Software effect
The repository durably owns the project while the event port publishes project.created evidence.
Hardware effect
The service process uses CPU and writes database pages to its persistent volume or managed database.
Observable evidence
A tenant-scoped GET plus a project.created evidence record proves both state and accountability.

Start with the people and the result they need

The source tables below remain the detailed contract. Begin with these customer paths:

  • D05-UC-01
    • Person: Application creator
    • Job: Generate a runnable application from an approved revision
    • Observable result: Run reaches succeeded with one immutable source artifact digest
  • D05-UC-02
    • Person: Platform operator
    • Job: Retry interrupted generation without duplicate work or spend
    • Observable result: Same idempotency key returns the original run; approved retry resumes from recorded state

Turn each customer job into a testable story

Now turn each customer job into a story with a result that an engineer can check:

  • D05-US-01
    • Story: As an application creator, I want generation to name its exact inputs and output digest, so that I can review what was produced
    • Observable acceptance: Run shows revision hash, generator version, model/tool policy, artifact digest, and terminal status
  • D05-US-02
    • Story: As a platform operator, I want idempotent retries, so that network failures do not duplicate artifacts or charges
    • Observable acceptance: Repeated key/input returns one run ID; conflicting input receives denial and audit trace

Add real state and observable proof

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

  • D05-FLOW-01
    • Trigger: Creator selects an approved revision and chooses Generate
    • Responsible systems: Application Catalog, Generation Orchestrator, Artifact Builder, policy validator
    • Authoritative state: Control-plane generation tables and S3-compatible artifact store
    • Owned record: GenerationRun
    • Observable evidence: Actor, revision hash, run ID, queue message ID, versions, digest, policy result, environment, timestamp, trace ID
    • Failure signal: Rejected revision, worker failure, policy violation, digest mismatch
  • D05-FLOW-02
    • Trigger: Worker stops after artifact upload but before acknowledgement
    • Responsible systems: Generation Orchestrator, queue adapter, Audit Evidence
    • Authoritative state: Idempotency reservation and generation-run state
    • Owned record: SourceArtifact
    • Observable evidence: Delivery count, same artifact digest, one run, recovered status, conflicting retry denial, positive-control read
    • Failure signal: Key conflict, lease timeout, poison message, or duplicate terminal transition

The enterprise problem and today’s slice

Enterprise problem: Synchronous generation hides partial work, duplicated model charges, and changing inputs, so a timeout can leave customers unable to tell whether code exists or is safe to retry.

Whole-course context: The incoming artifact is a tenant-scoped project with an embedded archetype blueprint; today turns that stable input into content-addressed source.

Today’s slice: Execute the current synchronous deterministic generator through an injected port, persist its source and digest, then define the durable asynchronous run state still needed for worker-loss recovery.

End-of-day evidence: The API returns a sha256: artifact ID, restored project state references that artifact once, generated archetype tests reject unsafe output, and artifact.generated is published; no generation-run or retry receipt exists yet.

Still unsolved: The artifact is not previewed, connected to enterprise systems, published, or deployed to AWS.

Customer use cases

Generation is a durable workflow rather than a long HTTP request. These use cases produce one artifact and contain a retry or policy failure without duplicating output.

Use case IDActorCustomer jobSuccess outcomeDenial or recovery evidence
D05-UC-01Application creatorGenerate a runnable application from an approved revisionRun reaches succeeded with one immutable source artifact digestInvalid or unsafe output reaches a terminal failed state with diagnostic evidence
D05-UC-02Platform operatorRetry interrupted generation without duplicate work or spendSame idempotency key returns the original run; approved retry resumes from recorded stateConflicting key/input pair is denied while the first run remains authoritative

Actor-centred user stories

An artifact without provenance cannot be reviewed or reproduced. These stories bind output to inputs and make retries deterministic.

Story IDUse case IDsUser storyObservable acceptance conditions
D05-US-01D05-UC-01As an application creator, I want generation to name its exact inputs and output digest, so that I can review what was producedRun shows revision hash, generator version, model/tool policy, artifact digest, and terminal status
D05-US-02D05-UC-02As a platform operator, I want idempotent retries, so that network failures do not duplicate artifacts or chargesRepeated key/input returns one run ID; conflicting input receives denial and audit trace

End-to-end product flows

The customer starts generation in Forge Studio and receives a durable run identifier immediately. Workers may fail, but the run state machine preserves what may happen next.

Flow IDUse case IDsPathTriggerNumbered stepsTerminal evidence
D05-FLOW-01D05-UC-01, D05-UC-02HappyCreator selects an approved revision and chooses Generate1. Authorize revision.; 2. Reserve idempotency key.; 3. Append run and queue message.; 4. Worker generates and validates source.; 5. Store immutable artifact.; 6. Mark run succeeded.Actor, revision hash, run ID, queue message ID, versions, digest, policy result, environment, timestamp, trace ID
D05-FLOW-02D05-UC-01, D05-UC-02RecoveryWorker stops after artifact upload but before acknowledgement1. Queue redelivers.; 2. Worker loads run state.; 3. Digest match prevents duplicate upload.; 4. Run finalizes.; 5. Conflicting retry is denied.Delivery count, same artifact digest, one run, recovered status, conflicting retry denial, positive-control read

Generation remains a customer action even though execution becomes asynchronous; the run ID is the immediate product response.

System design derived from the flows

A queue does not own workflow truth; it only transports work. Generation Orchestrator owns the run state machine, while Artifact Builder owns deterministic packaging and object storage owns immutable bytes.

Use case IDEntry pointResponsible servicesAuthoritative storeFailure evidence
D05-UC-01POST /applications/{id}/generation-runsApplication Catalog, Generation Orchestrator, Artifact Builder, policy validatorControl-plane generation tables and S3-compatible artifact storeRejected revision, worker failure, policy violation, digest mismatch
D05-UC-02Retry with idempotency keyGeneration Orchestrator, queue adapter, Audit EvidenceIdempotency reservation and generation-run stateKey conflict, lease timeout, poison message, or duplicate terminal transition

The orchestrator decides allowed transitions; the queue delivers; the builder packages output and records a content digest instead of mutating application metadata directly.

Data model and ownership

Durable retries require a stored state machine even though no generated application is running. The artifact is hosted-runtime input, not generated-app domain data.

Generated-application database: Not created in this slice — generation state belongs to the provider control plane and source artifacts belong to the hosted-runtime artifact store.

Record or entityStore and ownerPrimary keyForeign key or opaque referenceTenant keyMaterial constraintLifecycle and deletionUse case IDs
GenerationRunControl-plane PostgreSQL, owned by Generation Orchestratorgeneration_run_idrevision_id local read reference and opaque artifact digestorganization_idUnique idempotency key plus revision hash; terminal states immutableQueued, running, succeeded/failed/cancelled; retained with app; deleted by lifecycle jobD05-UC-01, D05-UC-02
SourceArtifactS3-compatible object store, owned by Artifact Buildersha256 digestOpaque generation_run_id metadataorganization_idImmutable and content-addressed; policy-approved before release useCreated once, retained while referenced, exported or garbage-collected after retentionD05-UC-01, D05-UC-02

The run record explains the lifecycle, the digest identifies bytes, and combined evidence distinguishes successful generation from a recovered or denied retry.

Run and interrupt one generation

Happy-path-only tests miss the moment between artifact creation and durable workflow state. The current runnable slice is synchronous: it deterministically generates one source artifact, stores it, saves its reference, and publishes evidence. It does not yet implement the queue, worker-loss injection, or retry state machine described as the production target. Read the exact operation in services/control_plane/domain.py.

def generate(self, project: Project, actor: str) -> dict[str, str]:
    self._authorize(project, actor)
    generated = self.generator.generate(project.blueprint)
    self.artifacts.put(project.organization_id, generated["artifact_id"], generated["source"])
    artifact = {"artifact_id": generated["artifact_id"]}
    if artifact not in project.artifacts:
        project.artifacts.append(artifact)
        self.repository.save(project)
    self._audit(project, actor, "artifact.generated")
    return artifact
Declared intentInterpreterSoftware effectHardware effectEvidence
Generate once for an authorized collaborator and retain a content-addressed artifact referencePython executes the Forge model; injected generator, artifact-store, repository, and event adapters perform each boundary actionWrites source to local SQLite, adds one artifact reference idempotently, saves the project, and publishes artifact.generatedGenerator and control-plane CPU produce and hash source; SQLite volumes receive artifact, project, and evidence dataPOST /projects/{id}/generate, returned sha256: artifact ID, restored project state, and tests/test_domain.py; retry-after-worker-loss remains future work

Decision rules

Use durable asynchronous jobs when work outlives an HTTP timeout or must survive worker loss. The current synchronous implementation teaches deterministic artifact identity and adapter boundaries only; do not claim retry or recovery until a durable run record, broker, idempotency key, and worker-loss test exist.