Bootstrap Kubernetes and Argo CD as Code

Enterprise problem: A platform team needs Argo CD before Argo CD can reconcile anything, but an undocumented sequence of local commands creates an unrepeatable cluster, hides ownership, and leaves recovery dependent on one operator's terminal history.

Whole-course context: This course starts with Docker and an empty directory, establishes the Kubernetes and Argo CD bootstrap boundary today, then lets Argo CD pull application and platform desired state from reviewed Git on Days 02 and 03.

Today’s slice: An operator uses one OpenTofu state to create a local Kind cluster and a second state to install a pinned Argo CD Helm release, while every mutation remains declarative and every inspection command remains read-only.

End-of-day evidence: Two successful OpenTofu applies, two committed dependency lock files, a ready Kind node, an argocd namespace, healthy Argo CD workloads, and state listings prove exactly which reconciler owns each bootstrap resource.

Still unsolved: No application repository is registered, no AppProject or ApplicationSet exists, no workload has been deployed, and production identity, secrets, promotion, recovery, observability, and upgrades remain deferred.

Teams often treat “install Argo CD” as one task even though cluster creation and in-cluster bootstrapping fail at different times and require different recovery paths. These use cases make the two customer outcomes and their evidence explicit.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D01-UC-01	Platform engineer	Create a reproducible Kubernetes cluster from an empty workstation directory	A named Kind cluster and kubeconfig are created by the pinned Kind provider, and the cluster state lists only cluster-lifecycle resources	A failed plan or apply leaves provider diagnostics and state evidence; retry or destroy concerns only the cluster state
D01-UC-02	GitOps platform operator	Install Argo CD declaratively after the Kubernetes API is reachable	The namespace and pinned argo-cd chart release become ready through the bootstrap state, with no manual resource creation	A failed Helm release remains attributable to the bootstrap plan; the operator can inspect events, correct code, and re-apply without changing cluster ownership

A tool checklist cannot prove that operators know what they own or how to recover it. These stories define visible acceptance conditions for both halves of the bootstrap.

Story ID	Use case IDs	User story	Observable acceptance conditions
D01-US-01	D01-UC-01	As a platform engineer, I want the local cluster created from pinned IaC so that every teammate can reproduce and remove it safely.	tofu state list in infra/cluster shows one Kind cluster; kubectl get nodes through the generated kubeconfig reports Ready; .terraform.lock.hcl is versioned.
D01-US-02	D01-UC-02	As a GitOps platform operator, I want Argo CD installed from a separate pinned plan so that chart changes never pretend to own the cluster.	tofu state list in infra/bootstrap shows the namespace and Helm release; Argo CD pods become ready; no imperative create command appears in the runbook.

The safe path has a hard readiness boundary: providers that call the Kubernetes API are configured only after that API exists. The recovery path preserves the same ordering instead of repairing resources by hand.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D01-FLOW-01	D01-UC-01, D01-UC-02	Happy	Operator runs the cluster apply, then the bootstrap apply	1. Docker reports a running engine.<br>2. OpenTofu initializes the cluster root and selects the locked Kind provider.<br>3. The Kind resource creates a control-plane container and writes kubeconfig.<br>4. Read-only Kubernetes inspection proves the node is ready.<br>5. OpenTofu initializes the bootstrap root and configures Kubernetes and Helm providers from that existing kubeconfig.<br>6. The namespace and pinned Argo CD chart become ready.	Separate state listings, lock files, a ready node, ready Argo CD pods, and Helm release metadata bind each result to its owner.
D01-FLOW-02	D01-UC-01, D01-UC-02	Recovery	Bootstrap apply fails or the environment must be removed	1. Inspect the failed plan, release status, pods, and events without mutating them.<br>2. Correct provider, chart values, or workstation prerequisites in Git.<br>3. Re-run the bootstrap plan and apply.<br>4. For teardown, destroy bootstrap while the API still exists.<br>5. Destroy the Kind cluster only after the bootstrap state is empty.	The corrected commit and successful plan replace terminal folklore; both states finish empty in dependency order during teardown.

Combining both use cases in one state creates a provider-bootstrap dependency: planning in-cluster resources may require an API endpoint that the same plan has not created yet. The design gives each entry point one state owner and an explicit filesystem handoff.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D01-UC-01	make cluster-apply	OpenTofu CLI, tehcyx/kind provider, Docker engine, Kind node container	infra/cluster OpenTofu state owns cluster lifecycle; generated kubeconfig is a non-secret local handoff	OpenTofu diagnostics, Docker container status, Kind provider state, and read-only node inspection
D01-UC-02	make bootstrap-apply	OpenTofu CLI, hashicorp/kubernetes provider, hashicorp/helm provider, Kubernetes API, official Argo CD Helm chart	infra/bootstrap OpenTofu state owns the argocd namespace and Helm release; Kubernetes stores their live objects	OpenTofu diagnostics, Helm release status, Kubernetes workload status, events, and bootstrap state

Bootstrap data is operational metadata, not application-domain data. Treating kubeconfig, IaC state, Helm release records, and future application records as one store would blur access and deletion responsibilities.

Generated-application database: Not created in this slice — Day 01 creates only a local Kubernetes control plane and Argo CD bootstrap resources; future application data remains outside both OpenTofu states.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
KindCluster	Cluster OpenTofu state owned by platform engineering	Resource address kind_cluster.gitops	Docker container and generated kubeconfig are provider-managed references	Local environment name gitops-lab	Cluster lifecycle appears only in cluster state	Destroyed last, after every API-dependent bootstrap resource is removed	D01-UC-01
KubeconfigHandoff	Local ignored file owned by the workstation operator	Path .local/kubeconfig	References cluster endpoint, CA, and client credentials	Local environment name gitops-lab	Never committed; file permissions and workstation access protect credentials	Recreated with cluster apply and removed with local environment cleanup	D01-UC-01, D01-UC-02
ArgoNamespace	Bootstrap OpenTofu state and Kubernetes API owned by GitOps platform engineering	Resource address kubernetes_namespace_v1.argocd	Uses KubeconfigHandoff only for provider authentication	Kubernetes cluster gitops-lab	Namespace ownership remains separate from future workload namespaces	Destroyed before cluster; retained while the Argo CD release exists	D01-UC-02
ArgoCDRelease	Bootstrap OpenTofu state and Helm release storage owned by GitOps platform engineering	Resource address helm_release.argocd	References namespace and chart argo-cd version 9.5.17	Kubernetes cluster gitops-lab	Chart version and values are explicit; no second reconciler mutates release objects	Upgraded through reviewed bootstrap code and destroyed before namespace and cluster	D01-UC-02

IaC succeeds only when each live resource has one desired-state writer. If OpenTofu and Argo CD both reconcile the same object, the systems can fight forever and neither state is trustworthy.

Use this contract throughout the course:

Layer	Mutating owner	Inputs	What it must not own
Local cluster lifecycle	Cluster OpenTofu state	Reviewed HCL, locked Kind provider, Docker	Argo CD release or application workloads
Argo CD installation	Bootstrap OpenTofu state	Existing kubeconfig, locked providers, pinned chart and values	Kind cluster lifecycle or Day 02 applications
Application delivery	Argo CD from Day 02 onward	Reviewed Git manifests and policies	Cluster creation or its own bootstrap release
Inspection	kubectl, OpenTofu show/state commands	Live API and state	Any mutation

This is inversion of control at the delivery boundary: CI will validate and publish Git changes, while Argo CD pulls them with in-cluster credentials. CI does not push manifests into Kubernetes and does not hold a production kubeconfig.

Bootstrap failure is often a missing prerequisite disguised as an infrastructure bug. Check tools before planning so Docker availability, OpenTofu syntax, and Kubernetes inspection fail early.

Install Docker, OpenTofu, and kubectl through your operating system's trusted package path, then run only version and status checks:

docker version
tofu version
kubectl version --client
git --version

Docker must report a reachable engine, not only an installed client. OpenTofu creates the cluster through the provider, so do not run a separate Kind create command.

A single root is tempting because it removes one command, but it couples provider configuration to a not-yet-existing API. Two roots apply SRP to state: cluster owns the API endpoint, while bootstrap consumes that endpoint and owns in-cluster installation.

Create this layout and keep application delivery outside both state roots:

Commit both .terraform.lock.hcl files after initialization. Ignore state, provider caches, plan files, and the credential-bearing kubeconfig:

**/.terraform/*
**/*.tfstate
**/*.tfstate.*
**/*.tfplan
.local/*
!.local/.gitkeep

The .local path is an explicit local interface between roots, not a shared state backend. In a team environment, each state belongs in a separately permissioned remote backend with locking and encryption.

gitops-platform/
├── .gitignore
├── Makefile
├── infra/
│   ├── cluster/
│   │   ├── .terraform.lock.hcl
│   │   ├── versions.tf
│   │   └── main.tf
│   └── bootstrap/
│       ├── .terraform.lock.hcl
│       ├── versions.tf
│       ├── providers.tf
│       ├── main.tf
│       └── argocd-values.yaml
├── platform/
│   └── README.md
└── .local/
    └── kubeconfig

Cluster creation is replace-oriented because the Kind provider does not update an existing cluster in place. This consequence is acceptable for a disposable lab and makes the production substitution boundary visible.

Put the following in infra/cluster/versions.tf:

terraform {
  required_version = ">= 1.9.0"
  required_providers {
    kind = {
      source  = "tehcyx/kind"
      version = "0.11.0"
    }
  }
}

Put the cluster resource in infra/cluster/main.tf:

provider "kind" {}

locals {
  kubeconfig_path = abspath("${path.module}/../../.local/kubeconfig")
}

resource "kind_cluster" "gitops" {
  name            = "gitops-lab"
  wait_for_ready  = true
  kubeconfig_path = local.kubeconfig_path

  kind_config {
    kind        = "Cluster"
    api_version = "kind.x-k8s.io/v1alpha4"

    node {
      role = "control-plane"
    }
  }
}

Initialize, review, and apply from that root:

mkdir -p .local
tofu -chdir=infra/cluster init
tofu -chdir=infra/cluster fmt -check
tofu -chdir=infra/cluster validate
tofu -chdir=infra/cluster plan -out=cluster.tfplan
tofu -chdir=infra/cluster apply cluster.tfplan

Review the generated lock file before committing it. The lock records the selected provider build and checksums, while versions.tf records the allowed selection.

A successful apply proves provider completion, not that the operator is using the intended kubeconfig or that the node stayed healthy. Read-only queries connect state evidence to the live Kubernetes API.

export KUBECONFIG="$PWD/.local/kubeconfig"
kubectl config current-context
kubectl get nodes -o wide
kubectl get --raw=/readyz
tofu -chdir=infra/cluster state list
tofu -chdir=infra/cluster show

Expected state contains kind_cluster.gitops; expected node status is Ready; /readyz returns ok. These commands inspect only—OpenTofu remains the sole mutating owner of bootstrap infrastructure.

The Kubernetes and Helm providers must connect during planning and applying, so configuring them beside the cluster resource creates a temporal dependency the graph cannot safely solve. The separate root starts only after the kubeconfig contract exists.

Put the provider constraints in infra/bootstrap/versions.tf:

terraform {
  required_version = ">= 1.9.0"
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "3.2.1"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "3.2.0"
    }
  }
}

Configure both providers from the same explicit handoff in infra/bootstrap/providers.tf:

locals {
  kubeconfig_path = abspath("${path.module}/../../.local/kubeconfig")
}

provider "kubernetes" {
  config_path    = local.kubeconfig_path
  config_context = "kind-gitops-lab"
}

provider "helm" {
  kubernetes = {
    config_path    = local.kubeconfig_path
    config_context = "kind-gitops-lab"
  }
}

The file is ignored because it contains client credentials. In production, use short-lived workload identity or an exec credential plugin rather than copying a long-lived administrator kubeconfig into CI.

An unpinned chart can change CRDs, permissions, defaults, and controller images between identical applies. Pin chart 9.5.17, review its changelog before upgrades, and keep its values in a separately reviewable file.

Put the namespace and release in infra/bootstrap/main.tf:

resource "kubernetes_namespace_v1" "argocd" {
  metadata {
    name = "argocd"

    labels = {
      "app.kubernetes.io/part-of"    = "argocd"
      "app.kubernetes.io/managed-by" = "opentofu"
    }
  }
}

resource "helm_release" "argocd" {
  name       = "argocd"
  namespace  = kubernetes_namespace_v1.argocd.metadata[0].name
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  version    = "9.5.17"

  atomic          = true
  cleanup_on_fail = true
  timeout         = 600
  wait            = true
  wait_for_jobs   = true

  values = [file("${path.module}/argocd-values.yaml")]
}

Keep the first infra/bootstrap/argocd-values.yaml deliberately small:

global:
  logging:
    format: json

configs:
  params:
    server.insecure: false

server:
  service:
    type: ClusterIP

This local release is not a production topology. Production values must add reviewed SSO, RBAC, network policy, ingress and TLS, resource requests and limits, high availability, metrics, notifications, and external secret references based on the organization's threat model.

Manual recovery commands create live drift that the next plan may undo or misinterpret. Fix configuration in Git, re-plan, and let only the bootstrap state mutate its namespace and Helm release.

tofu -chdir=infra/bootstrap init
tofu -chdir=infra/bootstrap fmt -check
tofu -chdir=infra/bootstrap validate
tofu -chdir=infra/bootstrap plan -out=bootstrap.tfplan
tofu -chdir=infra/bootstrap apply bootstrap.tfplan

Then collect read-only proof:

export KUBECONFIG="$PWD/.local/kubeconfig"
kubectl get namespace argocd
kubectl -n argocd get pods
kubectl -n argocd get deployments,statefulsets
kubectl -n argocd get events --sort-by=.metadata.creationTimestamp
kubectl -n argocd get secrets -l owner=helm,status=deployed
tofu -chdir=infra/bootstrap state list
tofu -chdir=infra/bootstrap show

Do not print secret contents or the initial administrator password into logs. A deployed Helm metadata Secret proves the release exists without exposing its data.

Operators under pressure will bypass a runbook if the safe path is harder than the unsafe one. Small Make targets keep ordering visible while preserving normal OpenTofu review and state semantics.

.PHONY: cluster-init cluster-plan cluster-apply bootstrap-init bootstrap-plan bootstrap-apply verify destroy

cluster-init:
	tofu -chdir=infra/cluster init

cluster-plan:
	tofu -chdir=infra/cluster plan -out=cluster.tfplan

cluster-apply:
	tofu -chdir=infra/cluster apply cluster.tfplan

bootstrap-init:
	tofu -chdir=infra/bootstrap init

bootstrap-plan:
	tofu -chdir=infra/bootstrap plan -out=bootstrap.tfplan

bootstrap-apply:
	tofu -chdir=infra/bootstrap apply bootstrap.tfplan

verify:
	KUBECONFIG=.local/kubeconfig kubectl get nodes
	KUBECONFIG=.local/kubeconfig kubectl -n argocd get pods

destroy:
	tofu -chdir=infra/bootstrap destroy
	tofu -chdir=infra/cluster destroy

The targets do not hide planning. A human or CI job still reviews the saved plan before applying it, and policy should prevent an unreviewed apply to shared environments.

Best-practice labels become empty slogans unless they change file ownership and runtime authority. Here each principle removes a specific failure mode from the bootstrap.

• DRY: one kubeconfig path local defines the handoff in each root, one values file defines the release configuration, and later environments call shared modules rather than copying full provider stacks.
• SRP: cluster lifecycle, Argo CD bootstrap, platform declarations, and application declarations have different roots and owners; changing an app never plans the cluster.
• IoC: once bootstrapped, Argo CD pulls reviewed desired state and reconciles it from inside the cluster; delivery CI validates Git but does not push Kubernetes objects.
• One reconciler: OpenTofu owns only cluster and bootstrap resources. Argo CD owns later application and platform workloads. Read-only kubectl proves state but never repairs it.

No-clickops is stronger than “we rarely use the UI.” The Argo CD UI is an observational surface; durable desired-state changes enter through reviewed code, and emergency actions must be reconciled back to Git with evidence.

Destroying the cluster first removes the API that the bootstrap providers need to uninstall the chart and namespace. The result is stale bootstrap state and a recovery exercise that teaches the wrong habit.

Run the reverse of creation:

tofu -chdir=infra/bootstrap plan -destroy -out=bootstrap-destroy.tfplan
tofu -chdir=infra/bootstrap apply bootstrap-destroy.tfplan
tofu -chdir=infra/bootstrap state list

tofu -chdir=infra/cluster plan -destroy -out=cluster-destroy.tfplan
tofu -chdir=infra/cluster apply cluster-destroy.tfplan
tofu -chdir=infra/cluster state list

Both final state listings should be empty. Remove ignored local plans and kubeconfig after confirming no process still depends on them; never commit those files.

Kind is a development implementation of the cluster contract, not a production recommendation. A production platform swaps the cluster root for an EKS, GKE, or AKS module while preserving the downstream bootstrap interface and reconciler boundary.

The production cluster module should output only what bootstrap requires:

output "cluster_endpoint" {
  value     = module.cluster.endpoint
  sensitive = true
}

output "cluster_ca_certificate" {
  value     = module.cluster.cluster_ca_certificate
  sensitive = true
}

output "cluster_name" {
  value = module.cluster.name
}

Prefer provider authentication through short-lived cloud identity in the bootstrap pipeline. Separate remote state backends, encryption, locking, least-privilege roles, network access controls, policy checks, and protected approvals replace the local filesystem handoff without merging responsibilities.

Pins make builds repeatable only until maintainers publish a security fix that the platform fails to adopt. Treat version updates as small reviewed changes with changelog, plan, compatibility, and rollback evidence.

This lesson pins the publication-date examples:

Dependency	Pin	Review action before change
OpenTofu	Team-supported >= 1.9.0 constraint	Pin the executable in CI and test state compatibility
Kind provider	tehcyx/kind 0.11.0	Read provider release notes and recreate the disposable lab
Kubernetes provider	hashicorp/kubernetes 3.2.1	Check Kubernetes client compatibility and plan namespace changes
Helm provider	hashicorp/helm 3.2.0	Check provider schema and release-upgrade behavior
Argo CD chart	argo-cd 9.5.17	Read official chart changelog, rendered diff, CRD notes, and Argo CD upgrade guide

Commit versions.tf, both generated .terraform.lock.hcl files, chart values, and reviewed manifests. Never commit state, saved plans, kubeconfig, tokens, passwords, private keys, or secret values.

A green pod list can coexist with uncommitted locks, mixed ownership, or a hidden imperative step. Finish only when code, state, live status, and repository hygiene agree.

• [ ] Docker was the only local substrate created outside OpenTofu.
• [ ] Cluster state contains the Kind cluster and no Argo CD resources.
• [ ] Bootstrap state contains the namespace and pinned Helm release and no cluster-lifecycle resource.
• [ ] Both .terraform.lock.hcl files are committed and their checksum changes were reviewed.
• [ ] The chart is exactly 9.5.17; both API providers use the declared versions.
• [ ] Every mutating action was an OpenTofu apply against reviewed code.
• [ ] kubectl was used only for current-context, readiness, listing, and event inspection.
• [ ] No kubeconfig, state, plan, password, token, or secret value is staged.
• [ ] Teardown succeeds bootstrap first, cluster second.
• [ ] The production module seam can replace Kind without giving Argo CD ownership of its own cluster or release.

Examples age, and bootstrap code controls privileged components. Verify provider schemas, chart release notes, and Argo CD behavior against primary documentation before adapting this lab to a shared cluster.

• Argo CD declarative setup
• Argo CD cluster bootstrapping
• Official Argo Helm chart repository
• Kind provider registry documentation
• Kubernetes provider registry documentation
• Helm provider registry documentation

Day 01 ends with a controller ready to observe Git, but deliberately no application declarations. Day 02 will introduce the repository contract, Kustomize overlays, AppProject, and ApplicationSet so Argo CD—not CI or a browser—becomes the application reconciler.