Draw the Three Planes and Split the Microservices

Enterprise problem: If one service creates applications, generates code, hosts previews, and authorizes every eventual app user, one bug or leaked credential can cross the whole product. Teams also cannot tell which service may change which data.

Whole-course context: Day 01 bounded HelixWorks as an application-building platform, and Day 02 supplied a local Kubernetes substrate. Today steps back to a smaller Docker Compose lab so you can see the product boundaries before Kubernetes adds more moving parts.

Today’s slice: You will send one support-application request through the provider control plane and hosted runtime, then inspect its independently progressing audit path. You will also mark the generated-application plane honestly: this revision generates HTML, but it does not implement the generated app's own end-user authentication or domain database.

End-of-day evidence: Successful create, generate, and preview responses; two authorization denials; and a stopped-broker/stopped-subscriber recovery trace tied to revision ae970246d39778a1d50354136cbab869223da109.

Still unsolved: A browser-accessible preview URL, generated-app users and roles, support-ticket data, enterprise connectors, publication, and Kubernetes delivery remain later slices.

The governing rule is simple: split a boundary when authority or authoritative state changes, not merely when the code uses a different verb. By the end, you will be able to predict which component must accept, reject, persist, or recover each step.

An architecture lesson is useless if a beginner cannot reproduce its evidence. Pinning the source revision makes every excerpt and expected response below refer to the same code, while a Compose project name gives this lab its own container, network, and volume names.

You need Git, Docker with the Compose plugin, Python 3, and curl. Ports 8080 and 8088 must be free. The lab creates no cloud resources.

git clone https://github.com/ZGTR/helixworks-kubernetes-lab.git helixworks-day03
cd helixworks-day03
git checkout --detach ae970246d39778a1d50354136cbab869223da109

export LAB_PROJECT=helixworks-day03
test "$(git rev-parse HEAD)" = "ae970246d39778a1d50354136cbab869223da109"
docker compose -p "$LAB_PROJECT" config --quiet

The detached checkout prevents a later branch update from silently changing the exercise. LAB_PROJECT is a shell variable used by every Compose command below; if you open a new terminal, export it again before continuing.

Calling every component a “microservice” hides the real risk: a service can be small yet still hold authority that belongs somewhere else. A plane is a boundary with its own actors, policy, and authoritative state—the state that wins when copies disagree.

1. The provider control plane is the part of HelixWorks that accepts the customer's intent. It owns the Acme organization reference, project, blueprint, generated artifact, collaborators, and provider audit trail.
2. The hosted runtime is where HelixWorks records a preview or published deployment. It owns the runtime copy and its lifecycle, not the customer's project policy.
3. The generated-application plane is the application HelixWorks produced. In a complete support app, this plane would own support agents, sessions, tickets, and rules such as “an agent may read tickets assigned to their queue.”

The third plane is not another name for the runtime. The runtime hosts an artifact; the generated application decides what its own users may do with app-owned data. This pinned lab reaches the runtime with generated HTML, but the HTML is only a form and button. There is no support user, session, ticket store, app role, or app-data authorization check yet.

Splitting on nouns such as “generator” or “runtime” can still produce the wrong design, because names do not prove ownership. For any new responsibility, ask the following questions in order; a changed answer is evidence that you crossed a boundary.

A diagram can claim clean boundaries while the code quietly bypasses them. The pinned source tree exposes one public provider entry point, two synchronous capability services, and an asynchronous evidence path:

In services/control_plane/domain.py, the provider model depends on small ports—interfaces describing a required capability—rather than importing HTTP or database details:

class Generator(Protocol):
    def generate(self, blueprint: dict[str, object]) -> dict[str, str]: ...


class Runtime(Protocol):
    def deploy(self, organization_id: str, app_id: str,
               payload: dict[str, object]) -> dict[str, object]: ...

This is dependency inversion: the policy code names what it needs, and the outer composition code supplies how to reach it. services/control_plane/app.py supplies HTTP clients for the two ports:

class GeneratorClient:
    def generate(self, blueprint):
        return request_json("POST", f"{self.url}/generate", blueprint)

class RuntimeClient:
    def deploy(self, organization_id, app_id, payload):
        return request_json("PUT", f"{self.url}/tenants/{organization_id}/apps/{app_id}", payload)

The separation has a concrete effect. services/generator/app.py deterministically turns the workflow blueprint into HTML and a content hash. services/runtime/app.py stores that artifact as a tenant-and-app-scoped preview payload. Neither service decides whether the requesting person is an Acme collaborator; Forge._authorize makes that provider decision before either port is called.

Starting containers does not prove the application path works, and the static web container is not the generated preview. First prove all six processes exist, then probe the public control-plane endpoint and the four internal service endpoints separately.

compose.yaml wires the control plane to generator, runtime, broker, and evidence by Compose service name:

control-plane:
  environment:
    GENERATOR_URL: "http://generator:8080"
    RUNTIME_URL: "http://runtime:8080"
    EVIDENCE_URL: "http://evidence:8080"
    BROKER_TOPIC: "http://broker:8080"
  volumes: ["control-data:/data"]
  ports: ["8080:8080"]

Build and start the isolated lab:

docker compose -p "$LAB_PROJECT" up --build -d
docker compose -p "$LAB_PROJECT" ps --services --status running

Expected observation: broker, control-plane, evidence, generator, runtime, and web are listed. Order may differ.

Probe the two host-visible endpoints:

curl --fail --silent --show-error http://localhost:8080/healthz
curl --fail --silent --show-error http://localhost:8088/ >/dev/null

The first command returns {"service": "control-plane"}; the second exits successfully. Now ask the control-plane container to probe the internal-only services on their Compose network:

docker compose -p "$LAB_PROJECT" exec -T control-plane python - <<'PY'
import json
import urllib.request

for service in ("broker", "evidence", "generator", "runtime"):
    with urllib.request.urlopen(f"http://{service}:8080/healthz", timeout=5) as response:
        print(service, response.status, json.load(response))
PY

Expected observation: four lines with status 200, each naming the service probed. This proves reachability; it does not yet prove that ownership rules or persistence work.

A component list says nothing about how a customer's request crosses boundaries. This trace keeps one scenario—Acme's “Support operations” workflow—and checks the output of every hop before moving on.

Mint a short-lived bearer token, a signed credential representing the provider actor owner@acme.test in organization acme:

TOKEN="$(python3 scripts/mint-local-token.py)"
AUTH=(-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json')

Create the provider project:

curl --fail --silent --show-error "${AUTH[@]}" \
  --data '{"project_id":"support","organization_id":"acme","name":"Support operations","archetype":"workflow"}' \
  http://localhost:8080/projects | python3 -m json.tool

Expected observation: HTTP success with project_id: "support", owner: "owner@acme.test", status: "draft", a workflow blueprint, and no artifacts. ForgeController.command first rejects a body whose organization differs from the signed token, then Forge.create stores the project and adds its owner as the first collaborator.

Generate an immutable artifact:

curl --fail --silent --show-error "${AUTH[@]}" \
  --data '{}' http://localhost:8080/projects/support/generate | python3 -m json.tool

Expected observation: an artifact_id beginning with sha256:. The provider authorizes the collaborator, sends only the blueprint to generator, stores the returned HTML under the organization and content hash, then adds the artifact reference to the project. The generated HTML is <main><h1>Support operations</h1><form><button>Submit request</button></form></main>; it contains presentation, not support-ticket behavior or end-user authorization.

Deploy that artifact as a preview record:

curl --fail --silent --show-error "${AUTH[@]}" \
  --data '{}' http://localhost:8080/projects/support/preview | python3 -m json.tool

Expected observation: a JSON object containing organization_id: "acme", app_id: "support", mode: "preview", the same artifact_id, and the generated source. The runtime has persisted a hosted deployment payload. This revision does not return a browser preview URL, so the honest milestone is “runtime accepted and stored the preview,” not “an end user used the app.”

The synchronous path is now visible:

A positive trace cannot reveal whether a boundary actually protects anything. Run two negative controls: a valid provider identity that is not a collaborator, and a direct generator call that lacks the internal service credential.

First mint a token for another person in the same Acme organization. Same-tenant identity is not enough; the project collaborator set is a separate authorization rule.

INTRUDER_TOKEN="$(ACTOR=intruder@acme.test python3 scripts/mint-local-token.py)"
curl --silent --show-error --include \
  -H "Authorization: Bearer $INTRUDER_TOKEN" -H 'Content-Type: application/json' \
  --data '{}' http://localhost:8080/projects/support/generate

Expected observation: HTTP 401 and {"error": "actor is not a collaborator"}. No new artifact is added because the control plane rejects the call before invoking generator.

Next call generator from inside the Compose network without X-Service-Token. A provider bearer token is deliberately not accepted at this internal interface.

docker compose -p "$LAB_PROJECT" exec -T control-plane python - <<'PY'
import urllib.error
import urllib.request

request = urllib.request.Request(
    "http://generator:8080/generate",
    data=b'{"name":"Bypass","archetype":"workflow"}',
    headers={"Content-Type": "application/json"},
    method="POST",
)
try:
    urllib.request.urlopen(request, timeout=5)
except urllib.error.HTTPError as error:
    print(error.code, error.read().decode())
PY

Expected observation: 401 {"error": "service authentication required"}. services/shared/auth.py defines these as distinct checks: verify_bearer validates the public actor and organization, while require_service_token validates service-to-service calls.

These denials prove provider collaborator policy and internal service authentication. They do not prove generated-app authorization, because no generated-app identity boundary exists in this revision.

An audit subscriber that must be online during every customer request would turn an evidence outage into a product outage. This experiment stops both transport and subscriber, proves the control plane durably records pending events, then restarts one stage at a time.

An outbox is a database table written by the business service before a background relay publishes its rows. In services/control_plane/outbox.py, publish inserts a content-hashed message, while relay marks it published only after the broker accepts it. services/evidence/app.py acknowledges a broker delivery only after its own store accepts the event.

Stop broker and evidence, then create another support workflow and carry it through preview. Generator and runtime remain available.

docker compose -p "$LAB_PROJECT" stop broker evidence
TOKEN="$(python3 scripts/mint-local-token.py)"
AUTH=(-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json')

curl --fail --silent --show-error "${AUTH[@]}" \
  --data '{"project_id":"support-recovery","organization_id":"acme","name":"Support recovery","archetype":"workflow"}' \
  http://localhost:8080/projects >/dev/null
curl --fail --silent --show-error "${AUTH[@]}" \
  --data '{}' http://localhost:8080/projects/support-recovery/generate >/dev/null
curl --fail --silent --show-error "${AUTH[@]}" \
  --data '{}' http://localhost:8080/projects/support-recovery/preview >/dev/null
sleep 3

All three HTTP commands exit successfully even though audit transport is down. Inspect only this project's pending outbox rows:

docker compose -p "$LAB_PROJECT" exec -T control-plane python - <<'PY'
import sqlite3

rows = sqlite3.connect("/data/control.db").execute(
    "SELECT published_at, attempts, last_error FROM event_outbox "
    "WHERE payload LIKE '%support-recovery%' ORDER BY rowid"
).fetchall()
print(rows)
PY

Expected observation: three rows, one each for project.created, artifact.generated, and preview.deployed. Their published_at values are None; retry counts are at least one and last_error describes the unavailable broker. The provider request and its audit intent survived in the same control-plane volume.

Restart only the broker and wait for the outbox relay:

docker compose -p "$LAB_PROJECT" start broker
sleep 3
docker compose -p "$LAB_PROJECT" exec -T control-plane python - <<'PY'
import sqlite3

row = sqlite3.connect("/data/control.db").execute(
    "SELECT COUNT(*), SUM(published_at IS NOT NULL) FROM event_outbox "
    "WHERE payload LIKE '%support-recovery%'"
).fetchone()
print(row)
PY

Expected observation: (3, 3). The broker accepted all three messages although evidence is still stopped. Confirm that its evidence subscription has three unacknowledged deliveries:

docker compose -p "$LAB_PROJECT" exec -T broker python - <<'PY'
import sqlite3

row = sqlite3.connect("/data/broker.db").execute(
    "SELECT COUNT(*) FROM topic_deliveries AS d "
    "JOIN topic_messages AS m ON m.message_id=d.message_id "
    "WHERE d.subscriber='evidence' AND d.acknowledged_at IS NULL "
    "AND m.payload LIKE '%support-recovery%'"
).fetchone()
print(row)
PY

Expected observation: (3,). Finally restart evidence and wait until it owns all three events:

docker compose -p "$LAB_PROJECT" start evidence
for attempt in $(seq 1 15); do
  RECOVERED="$(curl --fail --silent --show-error -H "Authorization: Bearer $TOKEN" \
    http://localhost:8080/evidence | python3 -c \
    'import json,sys; print(sum(e["project_id"] == "support-recovery" for e in json.load(sys.stdin)["events"]))')"
  [ "$RECOVERED" = 3 ] && break
  sleep 1
done
printf 'support-recovery evidence events=%s\n' "$RECOVERED"

Expected observation: support-recovery evidence events=3. The stages progressed independently: provider transaction to outbox, outbox to broker, and broker to evidence store. A duplicate delivery is safe because the evidence ID is content-derived and unique.

Recovery becomes guesswork when two services both believe their copy is authoritative. The local lab uses four Compose volumes and five logical tables; the table below says who may change each record and what another plane is allowed to keep.

An end-to-end command can still support a claim larger than the implementation. Keep the conclusion bounded to observations made in this lesson.

This lab proves:

• a signed provider actor can create one tenant-scoped support project;
• only a project collaborator can request generation and preview;
• only a service credential can call generator directly;
• generator returns deterministic, content-addressed HTML;
• runtime durably records the tenant/app preview payload;
• control-plane work can finish while broker and evidence are unavailable;
• outbox, broker delivery, and evidence storage catch up in order after recovery.

This lab does not prove:

• that the generated support app has end users, login, sessions, roles, tickets, or ticket authorization;
• that web on port 8088 serves the generated preview;
• that a preview has a public URL, TLS, isolation from another runtime, or production availability;
• that Docker Compose provides Kubernetes, Argo CD, multi-machine resilience, or cloud durability.

That distinction is the three-plane model doing useful work: it tells you which missing proof belongs to which future boundary instead of calling the whole platform “done.”

A lesson is complete only when another person can falsify its claims. Check the observed result beside each command, then remove only the isolated Compose project and its four lab volumes.

• [ ] git rev-parse HEAD prints ae970246d39778a1d50354136cbab869223da109.
• [ ] Compose lists six running services before the failure experiment.
• [ ] Create returns the Acme-owned support project.
• [ ] Generate returns a sha256: artifact ID.
• [ ] Preview returns mode: "preview" and the same artifact ID.
• [ ] A valid same-tenant non-collaborator receives 401 before generation.
• [ ] Generator rejects a direct call without the internal service token.
• [ ] Three support-recovery outbox rows remain pending while broker is stopped.
• [ ] The broker holds three unacknowledged evidence deliveries while evidence is stopped.
• [ ] Evidence reaches three events after its process restarts.
• [ ] You can name the owning plane for project, runtime deployment, audit event, and future support ticket.

Cleanup is intentionally scoped by the same project name used at startup:

docker compose -p "$LAB_PROJECT" down --volumes
docker compose -p "$LAB_PROJECT" ps --all

The first command removes this lab's containers, network, and four volumes. It does not prune unrelated Docker resources or delete the cloned source directory. The second command should show no resources for helixworks-day03.