Infrastructure as Code: Desired State and Evidence
Treat infrastructure changes as reviewable intent plus controlled execution and evidence, not as a brand name or a folder of scripts.
The enterprise problem and today’s slice
Enterprise problem: Ziba’s GKE environment can be recreated manually, but undocumented console choices and one-off commands make the next recovery slow, unauditable, and likely to produce different networking or memory capacity. Whole-course context: The previous cloud slice traced Ziba from Google Cloud networking to a running pod; today consumes that resource inventory and turns its important decisions into Infrastructure as Code (IaC), a delivery discipline for managing infrastructure through versioned machine-readable definitions. Today’s slice: We define the IaC loop—declare, review, execute, observe, and reconcile—without treating any one tool as the discipline itself. End-of-day evidence: A reviewer can connect a versioned change to an approved diff, immutable run, observed cloud and cluster effects, and a negative policy result. Still unsolved: Terraform’s state mechanics, Argo CD’s Kubernetes reconciliation, secrets implementation, and production approval remain separate slices.
Customer use cases
Code alone does not make infrastructure reproducible, so Ziba needs a lifecycle that controls who can propose, approve, execute, and verify a change. These use cases cover normal evolution and recovery from drift.
| Use case ID | Actor | Customer job | Success outcome | Denial or recovery evidence |
|---|---|---|---|---|
| D63-UC-01 | Ziba platform engineer | Increase GKE node capacity through a reviewed definition | Approved intent creates the expected cloud resources and Ziba remains reachable on the recorded revision | Policy or approval blocks an unsafe public network or undersized node before execution, with unchanged production proof |
| D63-UC-02 | Ziba incident commander | Detect and recover an out-of-band infrastructure change | Drift is attributed, reviewed, reconciled to the chosen desired state, and customer health is measured | Unauthorized reconciliation is denied; observed drift and unaffected positive control remain recorded |
Actor-centred user stories
IaC can automate mistakes faster unless the customer outcome and failure evidence are explicit. These stories make review, execution authority, and post-change measurement independently observable.
| Story ID | Use case IDs | User story | Observable acceptance conditions |
|---|---|---|---|
| D63-US-01 | D63-UC-01 | As a Ziba platform engineer, I want infrastructure intent reviewed as a machine-generated diff, so that capacity changes are understandable before cloud APIs allocate resources | Commit, proposed diff, policy result, approver, execution identity, provider resource IDs, HTTP probe, environment, timestamp, and run ID are captured |
| D63-US-02 | D63-UC-02 | As a Ziba incident commander, I want live infrastructure compared with approved intent, so that I can choose to revert drift or adopt it without silently overwriting evidence | Drift report names actor/resource/precondition, unauthorized apply is denied, approved recovery converges, and Ziba’s positive-control request remains healthy |
End-to-end product flows
An infrastructure definition is inert until an authorised engine translates it into provider API calls, so the execution boundary must be visible. Both flows end with observed infrastructure and customer evidence rather than “pipeline passed.”
| Flow ID | Use case IDs | Path | Trigger | Numbered steps | Terminal evidence |
|---|---|---|---|---|---|
| D63-FLOW-01 | D63-UC-01 | Happy | Engineer opens a capacity-change review | 1. Edit the versioned desired state. 2. Validate syntax and organisational policy. 3. Generate a proposed resource diff. 4. Review blast radius and approve the immutable revision. 5. Execute under a bounded machine identity. 6. Observe provider and Kubernetes state. 7. Probe Ziba and archive evidence. | Actor, resource, scope, precondition, expected/observed diff, environment, timestamp, commit, approval, run ID, cloud resource IDs, and HTTP result |
| D63-FLOW-02 | D63-UC-02 | Recovery | Drift detector finds a manually changed node-pool setting | 1. Read live state without mutation. 2. Attribute the difference from audit logs. 3. Decide whether intent or reality is correct. 4. Update code if adopting, or approve reconciliation if reverting. 5. Deny execution by an unapproved identity. 6. Reconcile and re-measure Ziba. | Drift artifact, decision record, denied principal, approved run ID, final observed state, and unaffected customer request trace |
System design derived from the flows
IaC is a discipline, not a single product: Terraform, OpenTofu, cloud-native deployment services, Pulumi, and configuration tools can implement portions of the loop with different state and execution models. Google describes IaC as defining cloud resources in human-readable files that can be versioned, reused, and shared; Terraform is one supported implementation, not the definition of IaC (Google Cloud IaC overview).
| Use case ID | Entry point | Responsible services | Authoritative store | Failure evidence |
|---|---|---|---|---|
| D63-UC-01 | Pull request for GKE capacity | Git host, validator, policy engine, chosen IaC engine, approval system, Google Cloud APIs, evidence store | Git for reviewed intent; chosen engine/backend for execution metadata; Google Cloud APIs for live resources | Invalid definition, policy denial, unapproved diff, provider API error, partial execution, or failed Ziba probe |
| D63-UC-02 | Drift alert and recovery review | Live-state reader, cloud audit logs, diff engine, approval system, executor, Kubernetes probes | Provider APIs for reality, Git for chosen intent, audit/evidence store for attribution and decision | Unattributed drift, stale comparison, denied executor, unreconciled resource, or customer regression |
Data model and ownership
Desired state, execution metadata, and live provider state answer different questions, so none should silently overwrite the others. This ownership model preserves a reviewable chain from proposal to physical effect.
Generated-application database: Not created in this slice — durable Git revisions, approvals, execution records, provider resources, audit logs, and probes are sufficient for infrastructure delivery evidence.
| Record or entity | Store and owner | Primary key | Foreign key or opaque reference | Tenant key | Material constraint | Lifecycle and deletion | Use case IDs |
|---|---|---|---|---|---|---|---|
| InfrastructureIntent | Git repository, owned by Ziba platform engineering | Commit SHA plus path | Opaque environment and module references | Environment/account ID | Reviewed commit is immutable; sensitive values are references, not plaintext | Proposed on branch, merged after approval, retained in history, retired by reviewed deletion | D63-UC-01, D63-UC-02 |
| ChangeProposal | CI and approval store, owned by release governance | Run ID plus proposed commit SHA | Opaque intent commit and prior evidence IDs | Environment/account ID | Diff must be generated from the exact commit and execution context later approved | Created per proposal, invalidated when inputs change, retained with approval/audit policy | D63-UC-01 |
| LiveResourceObservation | Provider inventory and evidence store, owned by platform operations | Provider resource ID plus observation time | Opaque intent address and execution run ID | Cloud project or account ID | Observation records source, freshness, and read identity; it is evidence, not automatically desired state | Refreshed by read-only discovery, retained through incident/audit window, expires by policy | D63-UC-01, D63-UC-02 |
| DriftDecision | Incident/change store, owned by incident commander and approver | Decision ID | Opaque change proposal, audit event, and affected resource IDs | Environment/account ID | Must explicitly choose revert, adopt, or defer with owner and deadline | Created on material drift, closed after proof, retained immutably | D63-UC-02 |
Expand the IaC loop one block at a time
Calling configuration “the source of truth” hides the fact that hardware and cloud APIs can diverge, so a complete loop needs both desired and observed state. Expand the smallest model into five independently reviewable stages.
| Stage | General rule | Ziba example | Failure mode | Reusable decision rule |
|---|---|---|---|---|
| Desired state | Declare outcomes and constraints, not undocumented clicks | Regional GKE cluster, bounded network, node pool shape | Secret or mutable artifact committed | Store references to secrets and pin versions/digests |
| Compare | Produce a reviewable change before mutation | One node-pool size change, no VPC replacement | Stale live-state read understates the diff | Refresh or declare freshness, then invalidate approval if inputs change |
| Approve | Separate authorship from production authority | Platform reviewer accepts cost and availability effect | Self-approved broad change | Require policy plus accountable approval proportional to blast radius |
| Execute | Use bounded, auditable machine authority | CI identity calls Google Cloud API | Long-lived owner credential leaks | Prefer short-lived identity with environment/resource scope |
| Observe | Verify infrastructure and customer outcome | Node joins, pod schedules, blog responds | Provider reports success but Ziba is down | Pair provider state with cluster and customer probes |
Declarative versus imperative is not absolute
Teams often frame declarative files and imperative commands as enemies, which obscures that a safe system uses both at different layers. Desired state describes the target; an engine still performs ordered imperative API operations, and diagnostics remain intentionally read-only commands.
environment: production
gke:
region: europe-west2
mode: standard
node_pool:
machine_type: e2-standard-4
min_nodes: 1
max_nodes: 4
policy:
public_control_plane: false
workload_identity: required
The example is a conceptual contract, not a Terraform schema. A Terraform module, Pulumi program, Google Cloud Infrastructure Manager deployment, or other reviewed engine could consume an equivalent intent. Tool choice changes syntax, state, providers, and operations; it does not remove the need for review, authority, evidence, drift decisions, and deletion ownership.
Code versus actual effects
A diff that says “update in place” may still restart software or replace a VM, so review must map abstract declarations to actual effects. The same IaC discipline applies whether the executor targets GCP, AWS, an on-premises API, or Kubernetes.
| Code or workflow action | Control-plane effect | Software effect | Hardware or facility effect | Deployment/customer effect |
|---|---|---|---|---|
Change machine_type | Desired resource shape changes; engine may propose replacement | Kubelet/container runtime leave old node and start on new node after drain | Provider allocates different vCPU/DRAM-backed VM; on premises this could require physical capacity | Pods reschedule; disruption depends on replicas and budgets |
Change max_nodes | Autoscaling policy ceiling changes | Autoscaler may later request more nodes | No hardware is allocated merely by raising the ceiling | Future Pending pods may gain capacity; current OOM limit is unchanged |
| Add a firewall rule | Network policy object changes | Packets may be admitted or denied by provider data plane | NICs and network devices enforce updated policy; no new server is implied | Ziba may become reachable or exposed beyond intent |
| Approve a plan | Approval record binds identity to immutable inputs | No workload changes yet | No resources allocated yet | Creates authority to execute, not evidence of deployment |
| Execute apply | Provider APIs create/update/delete resources | Agents/controllers converge software around changed resources | VMs, disks, addresses, and routing may be allocated or released | Customer path can improve, degrade, or remain unchanged until verified |
| Delete definition | Proposed desired object disappears | Engine may stop or remove dependent software | Apply may destroy real paid capacity; Git deletion alone does not | Service can be retired only after export, traffic removal, and evidence |
A tool-selection framework
Choosing a fashionable tool before naming the ownership and failure model creates avoidable lock-in. Select an implementation by the APIs it controls, how it represents state, how it previews change, and how the team operates failures.
| Question | Why it matters |
|---|---|
| Which provider and on-prem APIs must be managed? | Provider coverage and maturity determine what can be expressed safely |
| Where is desired state stored and reviewed? | Determines provenance, access, rollback narrative, and change collaboration |
| Does the tool keep state or continuously query reality? | Determines locking, recovery, import, drift, and concurrency procedures |
| Can it preview create/update/replace/delete accurately? | Determines whether blast radius is reviewable before mutation |
| How are credentials supplied? | Determines compromise scope and auditability |
| How are partial failures resumed? | Determines whether operators can recover without duplicate or orphaned resources |
| Who owns day-two upgrades and decommissioning? | IaC is lifecycle management, not only creation |
Decision rule: first design the IaC operating contract, then choose tools that satisfy it. Terraform is a strong multi-provider implementation when explicit plan/apply and stateful resource mapping fit; Argo CD is a complementary reconciler for Kubernetes application state, not a replacement for creating every underlying network and cluster.
Run a safe evidence rehearsal
IaC cannot be proven safely by reading code alone, so rehearse a bounded non-production change and its failure path. The following pseudocode expresses the workflow independently of a particular engine.
iac fmt --check
iac validate
iac policy-check --environment=staging
iac plan --out=change.plan
iac show change.plan
iac apply change.plan
iac observe --environment=staging
curl --fail --show-error "${ZIBA_STAGING_URL}/healthz"
Replace iac with the chosen implementation and its exact commands. Save the proposed diff before apply, bind approval to its checksum, reject changed inputs, and archive both provider observations and the customer probe.
After the chosen engine executes, these concrete read-only commands are runnable against the Ziba GKE environment and capture the provider, Kubernetes, and customer views independently:
gcloud container clusters describe ziba-prod \
--region=europe-west2 \
--format='json(name,location,currentMasterVersion,nodePools)'
kubectl get nodes -o custom-columns=NAME:.metadata.name,MEMORY:.status.allocatable.memory
kubectl -n blog get deployment ziba -o jsonpath='{.metadata.generation}{" "}{.status.observedGeneration}{"\n"}'
curl --fail --show-error "${ZIBA_URL}/healthz"
Key takeaways
Infrastructure as Code is an operating discipline that connects versioned intent to controlled execution and falsifiable evidence. A configuration file or successful provider response covers only part of that loop.
- IaC is broader than Terraform; Terraform is one implementation with a particular language, provider, plan, and state model.
- Desired state, execution metadata, and live reality have separate owners.
- Declarative intent still causes imperative provider operations and physical allocation.
- Drift is a decision input: revert it, adopt it in reviewed intent, or explicitly defer it.
- Infrastructure success must be paired with Ziba’s customer-visible health.
Checklist
An IaC change is complete only when its proposed and actual effects can be reconstructed. Use this checklist before granting any production executor authority.
- [ ] Versioned desired state contains no plaintext secrets and pins material dependencies
- [ ] Diff is generated from fresh live observations and the exact reviewed commit
- [ ] Policy and human approval are bound to immutable inputs
- [ ] Executor uses short-lived, least-privilege authority
- [ ] Evidence includes provider IDs, Kubernetes observations, customer probe, negative path, time, and run ID
- [ ] Drift, rollback, import/adoption, and retirement owners are documented