DaemonSet, Job & CronJob

This course is built around the public HelixWorks Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: terraform/main.tf

resource "kind_cluster" "helixworks" {
  name            = var.cluster_name
  node_image      = var.kind_node_image
  wait_for_ready  = true
  kubeconfig_path = local.kubeconfig_path

  kind_config {
    kind        = "Cluster"
    api_version = "kind.x-k8s.io/v1alpha4"

Code to reality

Declared intent

Create the local Kubernetes substrate from versioned infrastructure code.

Interpreter

Terraform builds a dependency graph and the Kind provider calls the Docker and Kind APIs.

Software effect

Terraform records a cluster resource and Kind writes a kubeconfig for its Kubernetes API.

Hardware effect

Docker starts containers that consume the laptop resources used as control-plane and worker nodes.

Observable evidence

terraform state show and kubectl get nodes must agree that the declared cluster is ready.

Deployments assume long-running, horizontally scalable services, but some workloads don't fit: an agent that must run on every node, a one-off batch task that should finish and stop, or a recurring scheduled job.

Solution: Pick the right controller for the lifecycle: DaemonSet (one pod per node), Job (run to completion), CronJob (scheduled).

Think of three different port jobs. A safety inspector stationed on every berth watches each dock all day, and the moment a brand-new berth opens the port posts one there too. A one-off cargo haul is dispatched, runs until the load is delivered, then stops for good. A scheduled nightly haul goes out automatically every night on a fixed timetable. These map to a DaemonSet that runs one pod per node, a Job that runs to completion, and a CronJob that creates Jobs on a schedule.

These are controllers that live in the control plane and decide which pods land on which nodes. DaemonSet works at the node layer, Job/CronJob at the pod layer.

Beyond Deployments, Kubernetes ships controllers for non-standard workloads: node agents, one-off batch tasks, and scheduled tasks.

   DaemonSet  -> one pod on EVERY node           (agents)
   Job        -> run to completion, then stop     (batch task)
   CronJob    -> Job on a schedule                (cron)

Where they sit in the controller family

Every controller manages pods, but each answers a different question: how many, where, and for how long?

   CONTROLLER     "how many / where"            "how long does a pod live"
   -----------    --------------------------     --------------------------
   Deployment     N replicas, scheduler picks    forever (restarted on exit)
   DaemonSet      exactly 1 per matching node     forever (restarted on exit)
   Job            N completions, anywhere         until task succeeds, then stop
   CronJob        spawns a Job each tick          each Job stops when done

Rule of thumb: long-running = Deployment/DaemonSet, runs-then-exits = Job/CronJob. The restart expectation is the key difference.

A DaemonSet guarantees a copy of a pod runs on every (matching) node, and automatically places one on any new node that joins the cluster.

Use for node-level agents: log collectors (fluentd), monitoring (node-exporter), CNI plugins, kube-proxy.

Graph legend — node-exporter as a real per-node DaemonSet:

A Job runs a pod until its task finishes successfully, then stops — unlike a Deployment, it is not meant to stay running.

Graph legend — a Job runs once to completion:

A CronJob creates a Job automatically on a recurring cron schedule — the Kubernetes equivalent of a crontab entry.

This manifest defines a CronJob that backs up at 02:00 every night; the extra fields control overlap and how much run history Kubernetes keeps around.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup
spec:
  schedule: "0 2 * * *"            # daily at 02:00
  concurrencyPolicy: Forbid        # don't start a new run if one is still going
  startingDeadlineSeconds: 120     # skip a missed run if >2 min late
  successfulJobsHistoryLimit: 3    # keep last 3 successful Jobs
  failedJobsHistoryLimit: 1        # keep last 1 failed Job
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: busybox
              command: ['sh','-c','echo backing up...']

concurrencyPolicy — what if a run overlaps?

If a Job is still running when the next tick fires, this policy decides what happens. Critical for long backups or jobs that must not double-run.

   Allow   (default): start the new run anyway -> two Jobs run at once
   Forbid           : skip the new run while the old one is still going
   Replace          : kill the old run, start the new one

Object chain: CronJob -> Job -> Pod

A CronJob doesn't run containers directly — it creates Jobs, which create Pods. History limits control how many old Jobs are kept around.

Graph legend — the CronJob to Job to Pod chain for nightly pg_dump:

Everyday commands for inspecting and triggering these controllers.

kubectl get daemonset -A
kubectl get jobs
kubectl get cronjobs                               # shows SCHEDULE, LAST SCHEDULE
kubectl logs job/pi                                # logs from a Job's pod
kubectl create job manual --from=cronjob/backup    # trigger a CronJob now
kubectl get pods --selector=job-name=pi            # find a Job's pods
kubectl delete job pi                              # also deletes its pods

Reading the STATUS / COMPLETIONS columns

The list output tells you batch progress at a glance.

   $ kubectl get jobs
   NAME   COMPLETIONS   DURATION   AGE
   pi     1/1           5s         1m      <- done: 1 of 1 succeeded
   etl    2/4           30s        30s     <- in progress: 2 of 4 done

   $ kubectl get pods
   pi-xxxxx   0/1   Completed   0   1m      <- batch pods end "Completed", not Running

A quick decision guide for picking the right controller for the job.

   Need it on every node?           -> DaemonSet
   One-off batch task?              -> Job
   Recurring scheduled task?        -> CronJob
   Long-running stateless service?  -> Deployment (Day 8)

A realistic pipeline: a DaemonSet log agent on every node, plus a CronJob that backs up a database every night and a way to trigger it on demand.

Graph legend — a real per-node log agent plus a nightly Postgres backup:

The mistakes that surprise people first.

   * restartPolicy: Always on a Job  -> rejected; Jobs need Never or OnFailure
   * Pod stuck Completed, not Running -> that's correct for Jobs; don't "fix" it
   * CronJob never fires              -> check timezone, schedule syntax, and
                                         startingDeadlineSeconds skipping runs
   * Overlapping CronJob runs         -> set concurrencyPolicy: Forbid/Replace
   * DaemonSet skips a node           -> node has a taint; add a toleration (Day 14)
   * Job retries forever             -> set backoffLimit + activeDeadlineSeconds

Each controller drives a different pod lifecycle: scheduled by a CronJob, one per node by a DaemonSet, or run-to-completion by a Job.

Graph legend — each controller's distinct pod lifecycle:

• DaemonSet = exactly one pod per (matching) node, auto-scales with nodes; no replicas field.
• Job runs to completion (completions, parallelism, backoffLimit); use restartPolicy: Never or OnFailure.
• CronJob spawns Jobs on a cron schedule; control overlap with concurrencyPolicy and clutter with history limits.
• Batch pods end as Completed, not Running — that's expected.

• [ ] Deployed a DaemonSet and saw one pod per node
• [ ] Scoped a DaemonSet to a subset of nodes with a nodeSelector
• [ ] Ran a Job to completion and read its logs
• [ ] Tuned a Job with completions + parallelism
• [ ] Created a CronJob and manually triggered it with --from
• [ ] Set a concurrencyPolicy and watched history limits prune old Jobs