Change, Redeploy, and Roll Back Safely
Make product change routine by promoting immutable revisions and recovering through the owner of each desired state.
System map · Day 13
Whole-system design
Five stable layers. Today's work is expanded and linked; the rest stays in context.
Product and authority
Covered — People and product entry points · Identity and policy · HelixWorks control plane · Generated application plane
Delivery and desired state
Covered — CI and immutable artifacts
Git desired state
Design target · not proved
Git ownership is the production recovery target; the current local redeploy flow does not commit the selected release to Git.
Argo CD reconciliation
Design target · not proved
Argo CD reconciliation is the production rollback target; this lesson proves only the current provider-owned local rollback seam.
Cloud and orchestration
Covered — Terraform and AWS APIs · Accounts, VPC, DNS, and private paths · Kubernetes or EKS control plane
Compute and traffic
Covered — Worker compute · Platform service workloadsAhead — Ambient mesh data plane
Generated app workloads
Source-backed today
Replaces running application instances with the selected immutable artifact revision.
Storage and evidence
Covered — Product data and artifactsAhead — Infrastructure state · Cluster desired and live state
Evidence and observability
Source-backed today
Compares deployment state and customer probes before and after rollback to prove recovery.
The enterprise problem and today’s slice
Enterprise problem: Once customers depend on a HelixWorks application, a prompt, connector, schema, or code change can break requests or corrupt data, and “redeploy the old commit” may not reverse database effects. Whole-course context: The incoming evidence is a local release with a known content-addressed artifact; today selects a recorded release for redeployment. Today’s slice: We implement owner-only artifact rollback in the local runtime and distinguish it from unimplemented configuration rollback, canary routing, schema compatibility, and data repair. End-of-day evidence: A known release restores its exact stored artifact and publishes release.rolled_back; an unknown release is denied and the project remains readable. Still unsolved: Canary and GitOps recovery, schema/data repair, long-term observability, support operations, and retirement remain deferred.
Customer outcome and implementation focus
The customer outcome is a reviewable change, redeploy, and roll back safely change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.
Components in focus
Change API, deployment controller, and release resolver; Kubernetes pods/ReplicaSets; release state database, image registry, and audit storage; cache: not involved for rollout truth.
This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.
Redeploy a revision and roll back through the owner
Reconcile through the owning controller
Imperative pod edits may briefly hide symptoms and then be overwritten by GitOps, so production recovery must change the owning declaration. The current product model implements a smaller local rollback: the owner selects an existing release and the runtime redeploys its stored artifact. It does not route canary traffic, revert Git, or prove Argo CD recovery. Read the exact method in services/control_plane/domain.py.
def rollback(self, project: Project, actor: str, release_id: str) -> dict[str, str]:
self._owner(project, actor)
release = next((item for item in project.releases if item["release_id"] == release_id), None)
if release is None:
raise ValueError("release not found")
self.runtime.deploy(project.organization_id, project.project_id, {"mode": "published", **release, "source": self.artifacts.get(project.organization_id, release["artifact_id"])})
self._audit(project, actor, "release.rolled_back")
return release
Key takeaways
Rollback is a coordinated business recovery decision, not merely an older image tag.
- Bounded traffic limits blast radius and creates comparative evidence.
- Artifact, configuration, schema, and data reversibility are separate questions.
- Repair the desired state through its single owner, then verify the customer path.
Checklist
A change is safe only when promotion and recovery are both rehearsed.
- [ ] Prior and candidate artifacts are immutable and identifiable
- [ ] Compatibility is checked before traffic changes
- [ ] Threshold breach stops promotion automatically
- [ ] Recovery proves affected and unaffected customer paths