68

Ziba Kubernetes Platform Decision Capstone

Choose and prove Ziba's on-premises, AWS, or Google Cloud operating model from hardware through Git, Terraform, Argo CD, autoscaling, and recovery.

The enterprise problem and today’s slice

Enterprise problem: Ziba’s OOM-killed blog can be moved to a larger server, an on-premises cluster, Amazon EKS, or Google Kubernetes Engine (GKE), but a provider choice made before ownership, recovery, latency, compliance, and scaling evidence merely relocates the outage. Whole-course context: The incoming artifacts now include memory diagnosis, workload manifests, observability, GitOps, KEDA replica policy, and Karpenter node evidence; today assembles them into one reviewable operating model. Today’s slice: We compare on-premises and managed cloud, AWS and Google Cloud, Infrastructure as Code (IaC) with Terraform, Argo CD delivery, pod and node autoscalers, and the physical effects of each declaration. End-of-day evidence: A decision record, cost/failure model, Git revision, Terraform plan, deployment proof, OOM recovery drill, denied path, and provider-specific resource IDs support a go/no-go. Still unsolved: Contract negotiation, exact regional prices, data-classification approval, and migration scheduling require Ziba’s owners.

The thesis is that Kubernetes portability ends at the API boundary: the same workload YAML can describe pods, but hardware supply, identity, networks, storage, upgrades, cost, and failure recovery remain platform-specific. Begin with need, operating model, and evidence—then expand each box.

Customer use cases

A platform decision is useful only if it improves Ziba’s customer outcome and remains operable during failure. These cases cover routine publishing and an evidence-driven recovery that can falsify the chosen model.

Use case IDActorCustomer jobSuccess outcomeDenial or recovery evidence
D68-UC-01Ziba product ownerPublish and serve the blog through a supportable platform with predictable performance and costA Git-reviewed release reaches healthy pods, survives representative load, meets latency and error objectives, and produces attributable infrastructure costA policy-breaking release is denied before production while the prior version remains available
D68-UC-02Ziba accountable SRERecover from an OOM, failed region/zone or node, and bad deployment within agreed objectivesDiagnosis identifies the failed layer; rollback or capacity action restores service within recovery objectives and preserves evidenceA drill exposes any unowned control, missing backup, unschedulable pod, unrecoverable state, or provider dependency before go-live

Actor-centred user stories

“Managed Kubernetes” does not mean managed application reliability, so each actor must know which layers the provider owns and which Ziba still operates. Acceptance joins product evidence to infrastructure and recovery evidence.

Story IDUse case IDsUser storyObservable acceptance conditions
D68-US-01D68-UC-01As Ziba’s product owner, I want a platform choice derived from workload and governance evidence, so that we buy the least operational complexity that still meets customer objectivesThe decision record compares on-premises, EKS and GKE; names assumptions and falsifiers; links Git, Terraform, Argo, Kubernetes and provider IDs; and records load, reliability and cost observations
D68-US-02D68-UC-02As Ziba’s accountable SRE, I want every failure mapped to an owned control and tested recovery, so that “the cloud manages it” never hides an unowned outage pathOOM, Pending capacity, node loss, bad Git revision, state loss and access denial drills record actor, resource, scope, precondition, expected, observed, environment, timestamp and immutable run ID

End-to-end product flows

The release path crosses source, infrastructure, cluster reconciliation, runtime, and customer traffic, so success at one controller is not production proof. The two flows make a normal change and a failed change terminate in customer-visible evidence.

Flow IDUse case IDsPathTriggerNumbered stepsTerminal evidence
D68-FLOW-01D68-UC-01HappyProduct owner approves a tested blog release1. Engineer updates application and platform declarations in reviewed Git paths.
2. CI builds and identifies an immutable image.
3. Terraform plan changes only approved cloud or on-premises infrastructure.
4. Authorised automation applies infrastructure and records state.
5. Argo CD detects the approved manifest revision and reconciles Kubernetes objects.
6. Scheduler places pods; workload autoscaling changes replicas; node automation supplies machines if required.
7. Synthetic and load tests verify customer objectives and cost attribution.
Actor, approvals, Git SHA, image digest, Terraform plan/apply ID, state version, Argo sync ID, pod/node/provider IDs, SLO result, cost labels, environment and timestamps
D68-FLOW-02D68-UC-02RecoveryNew release OOMs or cannot schedule during the canary1. Monitoring detects customer and runtime symptoms.
2. SRE distinguishes cgroup OOM, node pressure, Pending capacity, or release defect.
3. Argo restores the last approved manifest revision or platform operator applies the bounded capacity fix.
4. Kubernetes recreates pods and node automation supplies capacity if justified.
5. Customer probe and positive control pass.
6. Evidence is retained and the unsafe revision is blocked.
Alert, failed-layer diagnosis, previous/current Git SHAs, rollback or capacity action ID, pod termination/scheduling events, node/provider IDs, recovery time, customer probe, unaffected control and immutable incident ID

System design derived from the flows

Portability claims fail when declarative tools are drawn as one controller, because Terraform, Argo CD, Kubernetes, autoscalers, and providers own different state. The architecture separates desired infrastructure, desired workloads, runtime reconciliation, and the hardware underneath.

Use case IDEntry pointResponsible servicesAuthoritative storeFailure evidence
D68-UC-01Reviewed application/platform Git changeCI registry, Terraform runner and provider, Argo CD, Kubernetes controllers, workload autoscaler, node supplier, cloud or on-premises infrastructureGit for reviewed intent; Terraform backend for resource bindings; Kubernetes API for live desired/status; registry for image digest; provider inventory for machines/network/storageRejected review, failed plan/apply, Argo OutOfSync/Degraded, image pull error, Pending pod, failed node supply, SLO or cost-budget breach
D68-UC-02Incident alert and recovery runbookObservability, incident workflow, Argo rollback/revert path, Kubernetes scheduler/kubelet, KEDA/HPA/VPA, Cluster Autoscaler/Karpenter/provider automation, backup systemMetrics/logs/traces/events and immutable evidence store; Git revision history; Terraform state versions; backup catalogue; provider and Kubernetes auditOOMKilled, MemoryPressure, failed scheduling, node/zone loss, drift, stale state lock, failed restore, denied authority, missed recovery-time or recovery-point objective

Terraform is an IaC tool: declarative configuration describes infrastructure and state maps configuration addresses to real remote objects (HashiCorp Terraform introduction, Terraform state). Argo CD instead compares Git-hosted Kubernetes manifests with live cluster resources; automatic sync can reconcile differences without giving CI direct deployment credentials (Argo CD automated sync). Neither replaces the other.

Data model and ownership

Recovery becomes dangerous when Git, Terraform state, Kubernetes state, and provider inventory are treated as interchangeable sources of truth. Each record owns a different binding, and cross-plane identifiers remain references rather than inherited authority.

Generated-application database: Not created in this slice — Ziba’s blog content store is pre-existing; this capstone adds durable platform intent, resource bindings, deployment history, audit, backup, and evidence without inventing a new application database.

Record or entityStore and ownerPrimary keyForeign key or opaque referenceTenant keyMaterial constraintLifecycle and deletionUse case IDs
PlatformDecisionRevisionArchitecture repository, owned by Ziba accountable ownerdecision ID plus revisionOpaque evidence bundle and risk-approval IDsZiba organisation and environmentAssumptions, alternatives, owner map, falsifiers, costs, RTO and RPO must be explicit and approvedRevised on material evidence change; superseded revisions retained; exported and retained on platform retirementD68-UC-01, D68-UC-02
InfrastructureBindingTerraform remote backend, owned by platform engineeringworkspace plus resource addressOpaque AWS/GCP/on-premises resource IDCloud account/project or data-centre clusterOne reviewed configuration address maps to the intended remote object; state locking, encryption and access policy requiredVersioned on apply; resources destroyed only through approved plan; state retained/exported and secrets excludedD68-UC-01, D68-UC-02
DeploymentRevisionGit, registry and Argo application history, owned by delivery platformGit SHA plus image digestOpaque Argo sync and Kubernetes workload UIDsZiba environment and namespaceProduction references immutable image digest and reviewed manifests; rollback target remains deployableCreated per release, promoted/reverted through Git, retained for audit and rollback, expired by policyD68-UC-01, D68-UC-02
RuntimeEvidenceBundleObservability and evidence store, owned by SREimmutable run or incident IDOpaque Git, Terraform, Argo, pod, node and provider resource IDsZiba environmentPositive and negative observations share scope, precondition, expected result, observed result and time windowWritten per release/drill, immutable for retention, exported before platform retirement, expires by policyD68-UC-01, D68-UC-02
BackupAndRestorePointProvider-independent backup catalogue, owned by data owner and SREbackup IDOpaque source store, encryption key and restore-run IDsZiba data classification and environmentRPO, encryption, integrity and restore test must meet policy; backup is not valid until restoredCreated by schedule, immutable where required, restore-tested, expired and crypto-erased by retention policyD68-UC-02

Choose the ownership boundary before the provider

Provider comparison is meaningless until Ziba decides which undifferentiated work it wants to own. “On-premises” and “cloud” describe location and commercial boundary, while “managed” describes which control-plane or node operations a provider assumes.

LayerSelf-managed on premisesAmazon EKS standardGKE StandardGKE Autopilot
Physical host, power, cooling, failed partsZiba owns or contracts allAWSGoogleGoogle
Kubernetes control plane and etcd operationsZibaAWS manages control planeGoogle manages control planeGoogle manages control plane
Worker node shape, OS lifecycle and capacityZibaZiba chooses EC2/node approach; AWS options automate portionsZiba controls node pools and scalingGoogle manages node infrastructure and scaling within Autopilot constraints
Kubernetes workload specs and app reliabilityZibaZibaZibaZiba, within platform constraints
Identity, network, data, backup and cost policyZiba integrates local systemsShared responsibility with AWS servicesShared responsibility with Google servicesShared responsibility with more infrastructure defaults managed by Google
Strong reason to chooseData gravity, disconnected operation, specialised hardware, or mandatory local controlAWS integration and need for Kubernetes/EC2 flexibilityGoogle Cloud integration with direct node controlMinimise node operations for workloads compatible with constraints
Avoid whenTeam cannot staff hardware, etcd, upgrades, security, capacity and disaster recoveryKubernetes/EC2 flexibility adds more operating surface than Ziba needsManual node control has no demonstrated requirementPrivileged or specialised configuration conflicts with Autopilot constraints

AWS documents that EKS manages the Kubernetes control plane, while nodes in standard configurations remain EC2 capacity in the customer account (What is Amazon EKS, EKS control plane). Google documents Autopilot as a GKE mode where it manages node infrastructure, scaling, security defaults, and other configuration; Standard provides direct node control (GKE modes, GKE Autopilot).

The decision rule is to start with the most managed option that satisfies measured workload, security, network, data, and recovery constraints. Move toward lower-level control only when a named requirement and test falsify the managed option.

IaC, Terraform, GitOps, and Argo CD are different layers

Tool overlap causes destructive drift when two controllers believe they own the same object. IaC is the practice of versioning and reconciling infrastructure declarations; Terraform is one implementation, GitOps is an operating model centred on version-controlled desired state and reconciliation, and Argo CD reconciles Kubernetes application resources.

ObjectRecommended ownerWhy
VPC/VPC network, subnets, firewall/security groups, cluster, IAM/service accounts, Terraform backendTerraform or one approved infrastructure reconcilerThese are provider infrastructure with cross-resource dependencies and remote IDs
Namespace policy, Deployment, Service, KEDA object, Argo Application child resourcesArgo CD from application/platform GitThese are Kubernetes desired-state objects suited to continuous diff and reconciliation
HPA replica count, Deployment status, NodeClaim, pod/node statusRuntime controllersThese are controller outputs or live status; committing them as desired state creates conflicts
SecretsDedicated secret-management delivery pathPlaintext Git and Terraform plan/state can expose values; store references or encrypted payloads according to policy

Argo CD automatic pruning is disabled unless enabled, and self-healing is a separate option; those safety choices must be explicit (Argo CD automated sync). A Terraform state file contains resource bindings and may contain sensitive values, so protect the backend with access control, encryption, locking/versioning where supported, and recovery tests.

One infrastructure declaration, two cloud effects

Terraform syntax can look uniform while provider resources produce different networks, control planes, VMs, disks, and identities. The following skeleton deliberately stops at the cluster boundary; production modules must add private access, least-privilege identity, logging, encryption, node strategy, backups, and organisation policy.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    google = {
      source  = "hashicorp/google"
      version = "~> 7.0"
    }
  }
}

resource "aws_eks_cluster" "ziba" {
  name     = "ziba-production"
  role_arn = aws_iam_role.eks_control_plane.arn
  vpc_config {
    subnet_ids              = aws_subnet.private[*].id
    endpoint_private_access = true
    endpoint_public_access  = false
  }
}

resource "google_container_cluster" "ziba" {
  name             = "ziba-production"
  location         = "europe-west2"
  enable_autopilot = true
  network          = google_compute_network.ziba.id
  subnetwork       = google_compute_subnetwork.private.id
}
Code blockControl-plane effectVM, network, storage, or hardware effect
aws_eks_clusterAWS creates and operates an EKS Kubernetes API/control plane linked to Ziba’s IAM and VPCNo worker EC2 instance is created by this resource; a separate managed-node, Karpenter, Auto Mode, or other compute decision supplies machines
private EKS endpoint flagsKubernetes API reachability is private through configured VPC pathsRequires actual routing, DNS and operator connectivity inside or connected to the VPC
google_container_cluster with AutopilotGoogle creates a GKE control plane and applies Autopilot operating constraintsGoogle provisions and manages node infrastructure in response to eligible workload requests; Ziba still pays according to the service model and owns workload declarations
provider version constraintsTerraform selects compatible provider plugin APIsDoes not pin Kubernetes, node image, machine type, or physical host; those need separate declarations/policies

The sample cannot be applied alone because its IAM, subnet and network dependencies are intentionally referenced but not defined. That is a safety boundary: use reviewed organisation modules rather than pasting an incomplete internet cluster into production.

One workload declaration, different physical outcomes

Kubernetes manifests preserve a useful workload contract across providers, but scheduling translates that contract into different supply mechanisms. Keep the image digest, requests, limits, probes, topology and disruption policy portable; isolate provider-specific storage, identity, ingress and node declarations.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ziba-blog
  namespace: ziba
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ziba-blog
  template:
    metadata:
      labels:
        app: ziba-blog
    spec:
      containers:
        - name: blog
          image: registry.example/ziba/blog@sha256:replace-with-approved-digest
          resources:
            requests:
              cpu: "500m"
              memory: 768Mi
            limits:
              memory: 1Gi
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 3000
            periodSeconds: 5
DeclarationKubernetes effectOn-premises hardware effectAWS effectGCP effect
replicas: 3Deployment/ReplicaSet desires three podsScheduler needs three placements on Ziba-owned serversPods use existing EKS nodes or cause approved node automation to launch EC2Pods use GKE Standard nodes or cause Autopilot/node automation to supply Compute Engine-backed capacity
request 768MiScheduler reserves 768 MiB of allocatable memory per replicaConsumes placement budget backed by installed DRAMConstrains eligible EC2 node capacityConstrains eligible GKE compute capacity and may affect Autopilot provisioning/billing
limit 1GiKubelet/runtime configures each container cgroup ceilingLinux kernel on Ziba server can OOM-kill the process above the ceilingKernel inside EC2 worker can OOM-kill it; EKS control-plane management does not prevent thisKernel on GKE worker can OOM-kill it; Autopilot does not repair an application leak
readiness probeEndpoint receives traffic only while readyProbe runs from kubelet on the serverProbe runs on EC2-backed nodeProbe runs on Compute Engine-backed node
image digestRuntime pulls exact immutable contentBytes occupy local registry cache/diskBytes traverse configured registry/network and occupy node storageBytes traverse configured registry/network and occupy node storage

The code is portable only if the image architecture, storage, identity, networking, admission and resource requirements are supported in every target. “It applied” is weaker than “the customer path and recovery drill passed.”

Fit each autoscaler to one question

Autoscalers form a dependency chain, not a menu of interchangeable brands, so enable each only for the variable it owns. Wrong-layer automation can multiply OOM failures or create machines that still cannot run the pod.

QuestionControllerChangesDoes not change
How many replicas match CPU, memory or custom metric demand?HPAWorkload scale subresourcePer-pod request/limit or machine supply
How many workers match queue/lag/event demand, including activation from zero?KEDA plus generated HPAExternal metric/activation and workload replicasNode count or per-process memory ceiling
What request/limit fits observed per-pod usage?VPA, under a reviewed modePod resource recommendations or declarationsReplica count for traffic or physical RAM installed
How many machines are needed from predefined groups?Cluster AutoscalerNode-group desired capacityContainer limit or application leak
Which AWS machine offering satisfies Pending pod constraints?KarpenterNodeClaim and EC2 node lifecycleQueue depth or desired pod replicas
Who supplies nodes in a highly managed mode?GKE Autopilot or EKS Auto Mode according to service contractProvider-managed compute lifecycle within supported controlsZiba’s application correctness, requests, data, SLOs or recovery ownership

Kubernetes documents that horizontal workload scaling and node autoscaling combine as load → replicas → Pending demand → nodes; node autoscalers use pod requests and constraints rather than post-start real usage (Kubernetes node autoscaling). Therefore Ziba must retain the OOM diagnosis even after adopting dynamic nodes.

Deploy through Argo CD and prove the result

An automated sync can rapidly reproduce a bad declaration, so Git review, immutable images, health checks, and rollback evidence matter more as automation increases. This Application delegates Kubernetes resources to Argo CD while leaving provider infrastructure to Terraform.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ziba-production
  namespace: argocd
spec:
  project: ziba
  source:
    repoURL: https://github.com/example/ziba-platform.git
    targetRevision: main
    path: environments/production
  destination:
    server: https://kubernetes.default.svc
    namespace: ziba
  syncPolicy:
    automated:
      enabled: true
      prune: false
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
FieldSoftware actionInfrastructure/hardware consequence
targetRevision: mainArgo compares the tracked Git revision with live Kubernetes objectsNo hardware change until changed manifests create pods, services, volumes, or scaling demand
selfHeal: trueArgo re-reconciles supported live drift toward GitA manual emergency edit may be reverted; use the approved pause/revert procedure
prune: falseAutomatic sync does not automatically delete resources absent from GitReduces accidental deletion risk but can leave obsolete pods/services/resources consuming capacity until reviewed cleanup
Deployment requests/replicas in the pathKubernetes controllers create and place podsMay consume existing server/VM capacity or trigger node automation
terraform plan -out=ziba.tfplan
terraform show ziba.tfplan
terraform apply ziba.tfplan
argocd app get ziba-production
argocd app wait ziba-production --health --sync --timeout 600
kubectl -n ziba get deploy,pods,events -o wide
kubectl get nodes -o wide
CommandRequired evidenceWhat it cannot prove alone
terraform planExact proposed infrastructure graph and reviewer approvalThat remote resources work or that the blog is healthy
terraform applyApply ID, resulting provider resource IDs and new state versionThat Argo deployed the intended image or pods serve users
argocd app waitGit revision reached Synced and Healthy under Argo’s health modelCustomer latency, data correctness, node headroom, backup restore or cost
kubectl ...Workload revision, pod states, events and node bindingsProvider billing lifecycle or a representative customer result
Synthetic/load/restore drillCustomer outcome, failure recovery, capacity and evidence bundleFuture provider behavior; repeat continuously and on material changes

Score the decision with falsifiers

A weighted matrix can launder assumptions into a precise-looking score, so every score needs evidence, an owner, and a condition that would reverse it. Use hard gates first, then score only surviving options.

CriterionHard gate or measureOn-premisesEKSGKE
Data residency and disconnected operationRequired jurisdiction and offline durationStrong if local controls pass; Ziba owns proofRegion/service contract and connectivity must passRegion/service contract and connectivity must pass
Team operating capacityOn-call can recover control plane, nodes, network, storage and backupsHighest staffing burdenControl plane offloaded; node/app duties remain by modeControl plane offloaded; Autopilot can offload more node duties
Workload constraintsPrivilege, kernel, hardware, storage and network testsMaximum controlBroad EC2 flexibilityStandard broad; Autopilot constrained
Elasticity latencyMeasured zero/low-capacity burst to healthy serviceRequires pre-purchased buffer or automationManaged groups, Karpenter, or Auto Mode testStandard autoscaling or Autopilot test
RecoveryOOM, node, zone/site, control-plane access, backup restoreZiba proves all layers and secondary-site strategyShared-responsibility runbooks and regional designShared-responsibility runbooks and regional design
CostThree-year total cost including people, idle buffer, egress, support and migrationCapital, facilities, hardware lifecycle, staffService, EC2, storage, network, support, staffService, compute, storage, network, support, staff
ExitExport data/config and rebuild elsewhere within objectiveHardware ownership does not guarantee application portabilityProvider integrations and data movement testedProvider integrations and data movement tested

Recommended default for a small blog team is the most managed eligible service, because control-plane and node toil usually dominate any benefit from owning servers. That recommendation is an inference, not a universal fact: choose on-premises when validated local-control or disconnected-operation requirements dominate, EKS when AWS integration or EC2/Karpenter flexibility is decisive, and GKE Autopilot when the workload fits its constraints and minimising node operations is decisive.

The final OOM and disaster drill

A platform is not production-ready until it survives the original failure and a platform failure without hiding behind autoscaling. Run the same falsifiable tests on each finalist so the decision compares evidence rather than demos.

  1. Replay representative publish and read traffic with the approved image digest.
  2. Force the worker above its test cgroup limit in a non-production environment; verify OOMKilled, alerting, diagnosis, and bounded recovery.
  3. Create legitimate backlog; verify KEDA/HPA pod scaling and queue-age recovery.
  4. Exhaust current placement capacity; verify Pending evidence and the chosen node supplier’s machine lifecycle.
  5. Terminate one worker node; verify disruption policy, replacement, availability and provider resource cleanup.
  6. Revert a bad Git revision; verify Argo returns to the exact approved image and manifests.
  7. Restore content/configuration from backup into an isolated target; measure recovery point and recovery time.
  8. Deny the Terraform, Argo, scaler, and node-controller identities one at a time; verify least privilege, alerting, and unaffected positive controls.
  9. Export manifests, Terraform state/configuration, evidence and data according to the retirement plan; prove the platform is not a one-way door.

Key takeaways

The capstone decision is an ownership choice backed by drills, not a Kubernetes brand comparison. Portable manifests are valuable, but the operational system extends from Git and controllers down to provider APIs, virtual machines, disks, networks, and physical hardware.

  • On-premises gives Ziba control and responsibility from facilities through etcd, nodes, and applications.
  • EKS and GKE manage Kubernetes control-plane work; node responsibility varies by service mode.
  • IaC is a practice, Terraform is an infrastructure reconciler, GitOps is an operating model, and Argo CD reconciles Kubernetes desired state.
  • KEDA/HPA change pod count, VPA changes or recommends pod resources, and node automation changes machine supply.
  • Karpenter is AWS-specific in this design; Cluster Autoscaler and managed provider modes cover different supply models.
  • An OOM remains a container/kernel event even when the provider manages the control plane or nodes.
  • The winning platform is the least complex option that passes hard constraints, customer SLOs, recovery drills, cost bounds, and exit tests.

Checklist

The decision is not approved until an independent reviewer can connect every declaration to real state and physical consequence. Record gaps as failed criteria rather than optimistic future work.

  • [ ] The decision record names objective, thesis, boundary, assumptions, alternatives, owners, trade-offs and falsifiers.
  • [ ] On-premises, EKS and GKE modes were compared at the hardware, control-plane, node, workload, data and support layers.
  • [ ] Terraform and Argo CD have non-overlapping ownership, protected state, least-privilege identities and reviewed recovery.
  • [ ] Workload YAML uses immutable image digests, truthful requests/limits, probes, topology and disruption controls.
  • [ ] KEDA/HPA/VPA and Cluster Autoscaler/Karpenter/managed node supply each own one explicit scaling question.
  • [ ] The original OOM, Pending capacity, node loss, bad release, denied authority and restore paths were tested.
  • [ ] Evidence links Git SHA, image digest, Terraform state/apply, Argo sync, pod, node and provider resource IDs.
  • [ ] Cost includes people, idle buffer, support, storage, network, egress, migration and retirement—not only VM price.
  • [ ] RTO, RPO, data residency, security, support and exit criteria pass hard gates.
  • [ ] Production approval is based on measured customer and recovery evidence, not a successful apply or green controller alone.