Apache Airflow (Workflow Orchestration)

Real data work is rarely one script. A nightly job might extract yesterday's orders, wait for that to finish, clean the data, load it into a warehouse, and only then refresh a dashboard — and each step depends on the one before it. Do this with a pile of cron jobs and you quickly hit pain: cron can start a job at a time, but it cannot express "run step C only after A and B succeed," it cannot retry a flaky step, and when something breaks at 3 a.m. you have no picture of what ran and what didn't.

Apache Airflow is a workflow orchestrator: a tool for defining pipelines as code, scheduling them, running their steps in the right dependency order, retrying failures, and giving you a UI to see the whole thing. It was built at Airbnb and is now the default open-source choice for orchestrating batch data pipelines — jobs that process a chunk of data on a schedule, as opposed to continuous streams.

The central concept in Airflow is the DAG (Directed Acyclic Graph). Break the name down: it is a graph of steps, the edges are directed (they point from a step to the step that depends on it), and it is acyclic (no step can eventually depend on itself — no loops). A DAG is exactly the right shape for a pipeline, because a pipeline is a set of tasks with dependencies and no circular waiting.

Read the arrows as "must finish before": load waits for both clean and enrich, and report waits for load. Airflow reads this graph and runs each task only once every task pointing into it has succeeded.

The pieces that make up a DAG:

• Task — a single unit of work, one node in the graph.
• Operator — a template that defines what kind of work a task does. A PythonOperator runs a Python function, a BashOperator runs a shell command, a PostgresOperator runs SQL. Instantiating an operator creates a task.
• Dependencies — the edges, written in Python as a >> b ("a then b").

Here is the DAG above expressed as code:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

with DAG(
    dag_id="daily_orders",
    schedule="@daily",
    start_date=datetime(2026, 1, 1),
    catchup=False,
) as dag:
    extract = PythonOperator(task_id="extract", python_callable=pull_orders)
    clean   = PythonOperator(task_id="clean",   python_callable=clean_orders)
    load    = PythonOperator(task_id="load",    python_callable=load_warehouse)

    extract >> clean >> load

The DAG is code, which means pipelines are version-controlled, reviewed, and tested like any other software — a major reason Airflow won over point-and-click schedulers.

Defining a DAG is only half the story; something has to run it. Airflow's runtime has a handful of components, each with one job. Naming them makes it clear where your task actually executes and where the state lives.

• Scheduler — the brain. It reads the DAGs, works out which tasks are due and whose dependencies are met, and queues them. It is the component that turns "this DAG runs @daily" into actual task runs.
• Executor — the mechanism that decides where a queued task runs. It is a pluggable choice:
  • LocalExecutor — runs tasks as subprocesses on one machine. Fine for small deployments.
  • CeleryExecutor — distributes tasks to a pool of worker machines via a message queue. Scales horizontally.
  • KubernetesExecutor — launches each task as its own Kubernetes pod. Isolated and elastic.
• Workers — the processes that actually execute the task code.
• Metadata DB — a relational database (usually PostgreSQL) that stores the state of every DAG, task, and run: what succeeded, what failed, what is queued. It is Airflow's single source of truth.
• Web UI — the dashboard for viewing DAGs, inspecting runs, reading logs, and manually triggering or re-running tasks.

Airflow schedules DAGs on an interval — @daily, @hourly, or a cron expression. Each scheduled run is tied to a data interval, the window of time it is responsible for (for a @daily DAG, one specific day). This matters because Airflow can reason about past intervals.

• Backfill — running a DAG for historical intervals it has not run yet. If you add a new pipeline today but want it to process the last 30 days, you backfill those 30 runs.
• Catchup — the automatic version: when a DAG's start_date is in the past, Airflow will, by default, run every interval from then until now. This is powerful and dangerous — a start_date a year back can launch 365 runs at once. Setting catchup=False (as in the code above) tells Airflow to only run from now forward.

Four features make Airflow production-grade rather than just a scheduler:

• Retries — a task can be configured to retry N times with a delay before it is marked failed, so a transient network blip doesn't fail the whole pipeline.
• SLAs (Service Level Agreements) — you can declare that a task should finish within a time budget; if it runs late, Airflow records an SLA miss and can alert you.
• XComs (Cross-Communication) — a mechanism for one task to pass a small piece of data to a later task (a computed value, a file path). XComs are for small values only — they live in the metadata DB, not for shuttling large datasets.
• Idempotency — the discipline that a task, if run twice for the same interval, produces the same result without duplicating or corrupting data. Because Airflow retries and backfills, tasks will sometimes run more than once, so each task should overwrite its target for its interval rather than blindly append. Idempotent tasks are what make retries and backfills safe.

Airflow's sweet spot is batch orchestration: scheduled pipelines that move and transform data in chunks, with dependencies between steps, where latency of minutes to hours is fine. ETL into a warehouse, nightly report generation, ML training pipelines, and data-quality checks all fit well.

What Airflow is not for is streaming — processing an unbounded, continuous flow of events with sub-second latency. That is the job of tools like Kafka, Flink, or Spark Streaming. Airflow schedules and coordinates jobs; it is not a real-time event processor. Reaching for Airflow to handle a live event stream is using the wrong tool: it thinks in scheduled intervals, not in a never-ending flow.

• Airflow orchestrates batch data pipelines: scheduling, dependency ordering, retries, and visibility that plain cron cannot provide.
• The DAG (Directed Acyclic Graph) is the core abstraction — tasks as nodes, dependencies as directed edges, no cycles — and it is defined as version-controlled Python code.
• Operators are task templates (Python, Bash, SQL); instantiating one creates a task, and a >> b sets the dependency.
• The scheduler decides what runs when, the executor (Local/Celery/Kubernetes) decides where, workers run the code, the metadata DB holds all state, and the web UI shows it.
• Backfill runs historical intervals; catchup does it automatically — set catchup=False to avoid an accidental flood of past runs.
• Retries, SLAs, and XComs (small cross-task data) are production essentials; idempotent tasks are what make retries and backfills safe.
• Airflow is for batch orchestration, not streaming — use Kafka/Flink for continuous, low-latency event flows.

• [ ] I can explain why cron falls short for multi-step, dependency-heavy pipelines.
• [ ] I can define a DAG and say why "directed" and "acyclic" both matter.
• [ ] I can distinguish an operator, a task, and a dependency.
• [ ] I can name the scheduler, executor, workers, metadata DB, and web UI and each one's job.
• [ ] I can pick between Local, Celery, and Kubernetes executors for a given scale.
• [ ] I can explain backfill vs catchup and why catchup=False is often set.
• [ ] I can explain why tasks must be idempotent and what XComs are for.
• [ ] I can decide when Airflow fits versus when a streaming tool is the right choice.