63

Infrastructure as Code: Desired State and Evidence

Treat infrastructure changes as reviewable intent plus controlled execution and evidence, not as a brand name or a folder of scripts.

The enterprise problem and today’s slice

Enterprise problem: Ziba’s GKE environment can be recreated manually, but undocumented console choices and one-off commands make the next recovery slow, unauditable, and likely to produce different networking or memory capacity. Whole-course context: The previous cloud slice traced Ziba from Google Cloud networking to a running pod; today consumes that resource inventory and turns its important decisions into Infrastructure as Code (IaC), a delivery discipline for managing infrastructure through versioned machine-readable definitions. Today’s slice: We define the IaC loop—declare, review, execute, observe, and reconcile—without treating any one tool as the discipline itself. End-of-day evidence: A reviewer can connect a versioned change to an approved diff, immutable run, observed cloud and cluster effects, and a negative policy result. Still unsolved: Terraform’s state mechanics, Argo CD’s Kubernetes reconciliation, secrets implementation, and production approval remain separate slices.

Customer use cases

Code alone does not make infrastructure reproducible, so Ziba needs a lifecycle that controls who can propose, approve, execute, and verify a change. These use cases cover normal evolution and recovery from drift.

Use case IDActorCustomer jobSuccess outcomeDenial or recovery evidence
D63-UC-01Ziba platform engineerIncrease GKE node capacity through a reviewed definitionApproved intent creates the expected cloud resources and Ziba remains reachable on the recorded revisionPolicy or approval blocks an unsafe public network or undersized node before execution, with unchanged production proof
D63-UC-02Ziba incident commanderDetect and recover an out-of-band infrastructure changeDrift is attributed, reviewed, reconciled to the chosen desired state, and customer health is measuredUnauthorized reconciliation is denied; observed drift and unaffected positive control remain recorded

Actor-centred user stories

IaC can automate mistakes faster unless the customer outcome and failure evidence are explicit. These stories make review, execution authority, and post-change measurement independently observable.

Story IDUse case IDsUser storyObservable acceptance conditions
D63-US-01D63-UC-01As a Ziba platform engineer, I want infrastructure intent reviewed as a machine-generated diff, so that capacity changes are understandable before cloud APIs allocate resourcesCommit, proposed diff, policy result, approver, execution identity, provider resource IDs, HTTP probe, environment, timestamp, and run ID are captured
D63-US-02D63-UC-02As a Ziba incident commander, I want live infrastructure compared with approved intent, so that I can choose to revert drift or adopt it without silently overwriting evidenceDrift report names actor/resource/precondition, unauthorized apply is denied, approved recovery converges, and Ziba’s positive-control request remains healthy

End-to-end product flows

An infrastructure definition is inert until an authorised engine translates it into provider API calls, so the execution boundary must be visible. Both flows end with observed infrastructure and customer evidence rather than “pipeline passed.”

Flow IDUse case IDsPathTriggerNumbered stepsTerminal evidence
D63-FLOW-01D63-UC-01HappyEngineer opens a capacity-change review1. Edit the versioned desired state.
2. Validate syntax and organisational policy.
3. Generate a proposed resource diff.
4. Review blast radius and approve the immutable revision.
5. Execute under a bounded machine identity.
6. Observe provider and Kubernetes state.
7. Probe Ziba and archive evidence.
Actor, resource, scope, precondition, expected/observed diff, environment, timestamp, commit, approval, run ID, cloud resource IDs, and HTTP result
D63-FLOW-02D63-UC-02RecoveryDrift detector finds a manually changed node-pool setting1. Read live state without mutation.
2. Attribute the difference from audit logs.
3. Decide whether intent or reality is correct.
4. Update code if adopting, or approve reconciliation if reverting.
5. Deny execution by an unapproved identity.
6. Reconcile and re-measure Ziba.
Drift artifact, decision record, denied principal, approved run ID, final observed state, and unaffected customer request trace

System design derived from the flows

IaC is a discipline, not a single product: Terraform, OpenTofu, cloud-native deployment services, Pulumi, and configuration tools can implement portions of the loop with different state and execution models. Google describes IaC as defining cloud resources in human-readable files that can be versioned, reused, and shared; Terraform is one supported implementation, not the definition of IaC (Google Cloud IaC overview).

Use case IDEntry pointResponsible servicesAuthoritative storeFailure evidence
D63-UC-01Pull request for GKE capacityGit host, validator, policy engine, chosen IaC engine, approval system, Google Cloud APIs, evidence storeGit for reviewed intent; chosen engine/backend for execution metadata; Google Cloud APIs for live resourcesInvalid definition, policy denial, unapproved diff, provider API error, partial execution, or failed Ziba probe
D63-UC-02Drift alert and recovery reviewLive-state reader, cloud audit logs, diff engine, approval system, executor, Kubernetes probesProvider APIs for reality, Git for chosen intent, audit/evidence store for attribution and decisionUnattributed drift, stale comparison, denied executor, unreconciled resource, or customer regression

Data model and ownership

Desired state, execution metadata, and live provider state answer different questions, so none should silently overwrite the others. This ownership model preserves a reviewable chain from proposal to physical effect.

Generated-application database: Not created in this slice — durable Git revisions, approvals, execution records, provider resources, audit logs, and probes are sufficient for infrastructure delivery evidence.

Record or entityStore and ownerPrimary keyForeign key or opaque referenceTenant keyMaterial constraintLifecycle and deletionUse case IDs
InfrastructureIntentGit repository, owned by Ziba platform engineeringCommit SHA plus pathOpaque environment and module referencesEnvironment/account IDReviewed commit is immutable; sensitive values are references, not plaintextProposed on branch, merged after approval, retained in history, retired by reviewed deletionD63-UC-01, D63-UC-02
ChangeProposalCI and approval store, owned by release governanceRun ID plus proposed commit SHAOpaque intent commit and prior evidence IDsEnvironment/account IDDiff must be generated from the exact commit and execution context later approvedCreated per proposal, invalidated when inputs change, retained with approval/audit policyD63-UC-01
LiveResourceObservationProvider inventory and evidence store, owned by platform operationsProvider resource ID plus observation timeOpaque intent address and execution run IDCloud project or account IDObservation records source, freshness, and read identity; it is evidence, not automatically desired stateRefreshed by read-only discovery, retained through incident/audit window, expires by policyD63-UC-01, D63-UC-02
DriftDecisionIncident/change store, owned by incident commander and approverDecision IDOpaque change proposal, audit event, and affected resource IDsEnvironment/account IDMust explicitly choose revert, adopt, or defer with owner and deadlineCreated on material drift, closed after proof, retained immutablyD63-UC-02

Expand the IaC loop one block at a time

Calling configuration “the source of truth” hides the fact that hardware and cloud APIs can diverge, so a complete loop needs both desired and observed state. Expand the smallest model into five independently reviewable stages.

StageGeneral ruleZiba exampleFailure modeReusable decision rule
Desired stateDeclare outcomes and constraints, not undocumented clicksRegional GKE cluster, bounded network, node pool shapeSecret or mutable artifact committedStore references to secrets and pin versions/digests
CompareProduce a reviewable change before mutationOne node-pool size change, no VPC replacementStale live-state read understates the diffRefresh or declare freshness, then invalidate approval if inputs change
ApproveSeparate authorship from production authorityPlatform reviewer accepts cost and availability effectSelf-approved broad changeRequire policy plus accountable approval proportional to blast radius
ExecuteUse bounded, auditable machine authorityCI identity calls Google Cloud APILong-lived owner credential leaksPrefer short-lived identity with environment/resource scope
ObserveVerify infrastructure and customer outcomeNode joins, pod schedules, blog respondsProvider reports success but Ziba is downPair provider state with cluster and customer probes

Declarative versus imperative is not absolute

Teams often frame declarative files and imperative commands as enemies, which obscures that a safe system uses both at different layers. Desired state describes the target; an engine still performs ordered imperative API operations, and diagnostics remain intentionally read-only commands.

environment: production
gke:
  region: europe-west2
  mode: standard
  node_pool:
    machine_type: e2-standard-4
    min_nodes: 1
    max_nodes: 4
policy:
  public_control_plane: false
  workload_identity: required

The example is a conceptual contract, not a Terraform schema. A Terraform module, Pulumi program, Google Cloud Infrastructure Manager deployment, or other reviewed engine could consume an equivalent intent. Tool choice changes syntax, state, providers, and operations; it does not remove the need for review, authority, evidence, drift decisions, and deletion ownership.

Code versus actual effects

A diff that says “update in place” may still restart software or replace a VM, so review must map abstract declarations to actual effects. The same IaC discipline applies whether the executor targets GCP, AWS, an on-premises API, or Kubernetes.

Code or workflow actionControl-plane effectSoftware effectHardware or facility effectDeployment/customer effect
Change machine_typeDesired resource shape changes; engine may propose replacementKubelet/container runtime leave old node and start on new node after drainProvider allocates different vCPU/DRAM-backed VM; on premises this could require physical capacityPods reschedule; disruption depends on replicas and budgets
Change max_nodesAutoscaling policy ceiling changesAutoscaler may later request more nodesNo hardware is allocated merely by raising the ceilingFuture Pending pods may gain capacity; current OOM limit is unchanged
Add a firewall ruleNetwork policy object changesPackets may be admitted or denied by provider data planeNICs and network devices enforce updated policy; no new server is impliedZiba may become reachable or exposed beyond intent
Approve a planApproval record binds identity to immutable inputsNo workload changes yetNo resources allocated yetCreates authority to execute, not evidence of deployment
Execute applyProvider APIs create/update/delete resourcesAgents/controllers converge software around changed resourcesVMs, disks, addresses, and routing may be allocated or releasedCustomer path can improve, degrade, or remain unchanged until verified
Delete definitionProposed desired object disappearsEngine may stop or remove dependent softwareApply may destroy real paid capacity; Git deletion alone does notService can be retired only after export, traffic removal, and evidence

A tool-selection framework

Choosing a fashionable tool before naming the ownership and failure model creates avoidable lock-in. Select an implementation by the APIs it controls, how it represents state, how it previews change, and how the team operates failures.

QuestionWhy it matters
Which provider and on-prem APIs must be managed?Provider coverage and maturity determine what can be expressed safely
Where is desired state stored and reviewed?Determines provenance, access, rollback narrative, and change collaboration
Does the tool keep state or continuously query reality?Determines locking, recovery, import, drift, and concurrency procedures
Can it preview create/update/replace/delete accurately?Determines whether blast radius is reviewable before mutation
How are credentials supplied?Determines compromise scope and auditability
How are partial failures resumed?Determines whether operators can recover without duplicate or orphaned resources
Who owns day-two upgrades and decommissioning?IaC is lifecycle management, not only creation

Decision rule: first design the IaC operating contract, then choose tools that satisfy it. Terraform is a strong multi-provider implementation when explicit plan/apply and stateful resource mapping fit; Argo CD is a complementary reconciler for Kubernetes application state, not a replacement for creating every underlying network and cluster.

Run a safe evidence rehearsal

IaC cannot be proven safely by reading code alone, so rehearse a bounded non-production change and its failure path. The following pseudocode expresses the workflow independently of a particular engine.

iac fmt --check
iac validate
iac policy-check --environment=staging
iac plan --out=change.plan
iac show change.plan
iac apply change.plan
iac observe --environment=staging
curl --fail --show-error "${ZIBA_STAGING_URL}/healthz"

Replace iac with the chosen implementation and its exact commands. Save the proposed diff before apply, bind approval to its checksum, reject changed inputs, and archive both provider observations and the customer probe.

After the chosen engine executes, these concrete read-only commands are runnable against the Ziba GKE environment and capture the provider, Kubernetes, and customer views independently:

gcloud container clusters describe ziba-prod \
  --region=europe-west2 \
  --format='json(name,location,currentMasterVersion,nodePools)'
kubectl get nodes -o custom-columns=NAME:.metadata.name,MEMORY:.status.allocatable.memory
kubectl -n blog get deployment ziba -o jsonpath='{.metadata.generation}{" "}{.status.observedGeneration}{"\n"}'
curl --fail --show-error "${ZIBA_URL}/healthz"

Key takeaways

Infrastructure as Code is an operating discipline that connects versioned intent to controlled execution and falsifiable evidence. A configuration file or successful provider response covers only part of that loop.

  • IaC is broader than Terraform; Terraform is one implementation with a particular language, provider, plan, and state model.
  • Desired state, execution metadata, and live reality have separate owners.
  • Declarative intent still causes imperative provider operations and physical allocation.
  • Drift is a decision input: revert it, adopt it in reviewed intent, or explicitly defer it.
  • Infrastructure success must be paired with Ziba’s customer-visible health.

Checklist

An IaC change is complete only when its proposed and actual effects can be reconstructed. Use this checklist before granting any production executor authority.

  • [ ] Versioned desired state contains no plaintext secrets and pins material dependencies
  • [ ] Diff is generated from fresh live observations and the exact reviewed commit
  • [ ] Policy and human approval are bound to immutable inputs
  • [ ] Executor uses short-lived, least-privilege authority
  • [ ] Evidence includes provider IDs, Kubernetes observations, customer probe, negative path, time, and run ID
  • [ ] Drift, rollback, import/adoption, and retirement owners are documented