Data Pipelines

Data is born in one place — a user clicks a button, a payment posts, a sensor reads a temperature — and it is needed somewhere else entirely: a dashboard, a machine-learning model, a monthly report. A data pipeline is the automated flow that carries data from those sources to those destinations, transforming it along the way so it arrives in a shape the destination can use. "Automated" is the operative word: a pipeline runs on a schedule or continuously, without a person copying files by hand, and it does the same reliable thing every time it runs.

The simplest mental model is a series of connected stages, each doing one job before handing off to the next — like an assembly line for data. Raw data enters at one end, and clean, organized, query-ready data comes out the other.

Read it left to right: collect the data, reshape it, keep it, and hand it to whoever consumes it. Every real pipeline — however elaborate — is a variation on this four-move backbone. The rest of the day zooms into each move, then covers the choices (ETL vs ELT, batch vs streaming) and the disciplines (idempotency, backfills, quality) that separate a pipeline that works from one that quietly corrupts your data.

A pipeline is best understood as a chain of stages, each with a distinct responsibility. Naming them precisely makes the whole thing tractable, because a problem always lives in one stage.

• Sources — where data originates: application databases, third-party APIs, event streams, log files, IoT sensors, spreadsheets. A pipeline usually pulls from several at once.
• Ingestion — the act of collecting data from those sources and landing it somewhere the pipeline controls. This is where you decide batch vs streaming (below) and how to detect what's new.
• Transformation — turning raw, messy source data into clean, consistent, useful data: filtering out junk, fixing types, joining datasets, deduplicating, aggregating, and enforcing a schema.
• Storage — where the data rests. A quick recap of the two homes (covered in the storage day): a data lake stores raw data cheaply in open files (S3 + Parquet), schema decided at read time; a data warehouse stores cleaned, structured tables optimized for fast analytical queries (BigQuery, Snowflake, Redshift).
• Serving / consumption — where the data finally earns its keep: BI dashboards, ML model training, reports, reverse-ETL back into product features, or an API another service reads.

The stages are decoupled on purpose. Because each hands off to the next through storage rather than a direct call, you can rerun transformation without re-ingesting, swap the serving layer without touching ingestion, and debug one stage in isolation — the same loose-coupling payoff that decoupled architectures buy everywhere else.

Ingestion is the first real decision, and it splits on a single question: how fresh must the data be? There are two shapes, and many pipelines use both.

Batch ingestion collects data in chunks on a schedule — every hour, every night — and processes each chunk as a unit. It is simpler, cheaper, and perfectly fine when the destination can tolerate data that is minutes or hours old (most reporting and analytics can). Streaming ingestion processes each record the moment it arrives, continuously, so the destination is never more than seconds behind. It is more complex and more expensive, and you reach for it only when freshness genuinely matters — fraud detection, live dashboards, real-time recommendations.

A special and very common ingestion technique deserves its own name: CDC (Change Data Capture). Instead of re-copying an entire source database every run — slow and wasteful — CDC watches the database's transaction log and streams out only the rows that changed (inserts, updates, deletes) as they happen. It gives you near-real-time replication of a production database into your lake or warehouse without hammering the source with repeated full scans, and it is the standard way to feed analytical storage from an operational database.

Rule of thumb: default to batch — it is simpler and cheaper — and reach for streaming (and CDC) only where seconds-fresh data has real business value. Freshness is a cost, not a free upgrade.

Once data is ingested, it must be transformed and stored — but in which order? The two classic orderings are named by their steps, and the choice has shifted decisively over the last decade.

ETL — Extract, Transform, Load. Pull data from the source (extract), reshape it in a separate processing step (transform), and then load the finished, clean result into the warehouse. Transformation happens before storage, on a dedicated processing tier. This was the norm when warehouse storage and compute were expensive: you cleaned data first so you only paid to store the polished result.

ELT — Extract, Load, Transform. Pull the raw data (extract), load it into the warehouse as-is (load), and then transform it inside the warehouse using its own compute. Storage comes before transformation. This became dominant with cloud warehouses, and the reason is economic: modern cloud warehouses (BigQuery, Snowflake) offer cheap storage and enormous on-demand compute, so it is now cheaper and more flexible to dump raw data in first and transform it later — and because the raw data is retained, you can re-transform it a new way tomorrow without re-extracting.

Rule of thumb: on a modern cloud warehouse, prefer ELT — load raw, transform in place — because cheap storage plus elastic compute makes keeping the raw data and re-transforming it later both affordable and flexible. Reach back for ETL when data must be cleaned or masked before it is allowed to land (for example, stripping sensitive fields).

The batch/streaming split we saw at ingestion runs through the whole pipeline, not just its first stage, and it shapes how the entire system is built and reasoned about. It is worth stating the two end-to-end styles plainly.

A batch pipeline processes bounded chunks of data on a schedule. Its mental model is "run a job over a finite dataset, produce an output, stop." It is easy to reason about (a run has a clear start and end), easy to retry (just run the chunk again), and easy to test. The cost is latency: the data is always at least one batch-interval stale.

A streaming pipeline processes an unbounded, never-ending flow of events one at a time (or in tiny micro-batches). Its mental model is "a program that runs forever, reacting to each event." It delivers low latency but is harder in every other way: there is no natural "end," so you deal with out-of-order events, late arrivals, windowing (grouping events by time), and the difficulty of reprocessing history when a bug ships.

Many mature systems run both — a batch pipeline for correct, complete historical data and a streaming pipeline for fresh-but-approximate recent data — and reconcile them. The practical guidance is the same as at ingestion: batch is the sensible default; add streaming only where the latency it buys is worth its complexity and cost.

A real pipeline is not one step but many steps with dependencies: you cannot transform data before it is ingested, and you cannot serve it before it is transformed. Something has to run each step in the right order, only after its inputs are ready, and retry the ones that fail. That something is an orchestrator, and the standard way it models a pipeline is as a DAG (Directed Acyclic Graph) — a graph of tasks where each arrow means "this must finish before that starts," and "acyclic" means there are no loops (the flow always moves forward).

The orchestrator reads the DAG and does the tedious coordination: it runs a task only once all its upstream tasks succeed, runs independent branches in parallel, retries a failed task, and alerts a human when something stays broken. The best-known tool for this is Airflow, which lets you define the DAG in code and then schedules and monitors it — but you need no specific tool to understand the idea: a pipeline is a DAG of tasks, and an orchestrator walks that DAG in dependency order. That is the whole concept; the tool is an implementation detail.

A pipeline that merely moves data is easy; a pipeline you can trust requires a handful of disciplines. Skipping them is how pipelines silently produce wrong numbers that nobody notices for weeks. Each discipline answers a specific failure.

• Idempotency — re-running a step must produce the same result as running it once, never double-counted data. Because runs will be retried after failures, every step must be safe to repeat; the usual technique is to make a run replace its output partition rather than append to it, so a retry overwrites instead of duplicating.
• Backfills — running the pipeline over past data, not just new data: you fixed a transformation bug, or you added a new column, and now you must recompute history to match. A well-built pipeline makes backfills routine (rerun the DAG for a past date range), which is only safe because the steps are idempotent.
• Schema evolution — source data changes shape over time: a field is added, renamed, or retyped. The pipeline must tolerate this without breaking — typically by adding fields additively and versioning the schema — so an upstream change doesn't silently drop data or crash the load.
• Data quality — automated checks that assert the data is correct, not just present: row counts are in range, no unexpected nulls in a required column, values fall in valid bounds, a daily total isn't wildly off yesterday's. These checks run inside the pipeline and fail the run when violated, so bad data is caught before it reaches a dashboard.
• Lineage — a map of where each dataset came from and what fed it: which sources and which transformations produced this table. When a number looks wrong, lineage lets you trace it back to the stage that broke it.
• Observability — monitoring the pipeline itself: is it running, how long is each stage taking, how much data flowed, did freshness slip? A pipeline that fails silently is worse than one that fails loudly, so you alert on missed runs and volume anomalies.

Rule of thumb: build idempotency in from the start — it is the foundation that makes retries, backfills, and recovery safe — and add automated data-quality checks so the pipeline fails loudly on bad data instead of quietly serving it.

Data pipelines are not new, but they moved from a niche concern to a central discipline over the last decade, and three forces explain the surge — each amplifying the others.

Data volume exploded. Products emit vastly more data than they used to — every click, view, and event is captured — and that flood is too large to move or reshape by hand. When data arrives continuously and in enormous quantity, automated pipelines stop being a convenience and become the only way to handle it at all.

The demand for analytics and ML surged. Companies now expect to make decisions from data (BI dashboards, experimentation) and to build products on it (recommendations, fraud detection, forecasting, LLM training). Every one of those needs data collected, cleaned, and delivered in the right shape — which is exactly what a pipeline does. ML in particular is hungry: a model is only as good as the pipeline feeding it clean, timely training data.

Cloud storage and compute got cheap and elastic. Cloud object storage made it affordable to keep everything, and cloud warehouses made it affordable to process everything on demand — which is precisely what made ELT (load raw, transform later) viable and lowered the bar to building pipelines at all. What once required a costly on-prem cluster is now a few managed services you rent by the query.

Put together: there is far more data, far more reason to use it, and far cheaper infrastructure to move and process it. Those three forces turned the data pipeline from plumbing a few specialists cared about into core infrastructure most serious products now run.

• A data pipeline is the automated flow that moves and transforms data from sources to destinations, built as a chain of decoupled stages: ingestion → transformation → storage → serving.
• Ingestion is batch (scheduled chunks, simpler, cheaper) or streaming (record-by-record, fresher, costlier); CDC streams only changed rows from a database's log for near-real-time replication without full re-scans.
• ETL transforms before loading (fits expensive storage); ELT loads raw then transforms inside the warehouse — and ELT rose because cheap cloud storage plus elastic compute make keeping and re-transforming raw data affordable and flexible.
• The batch/streaming split runs through the whole pipeline: batch is bounded, easy to retry, and higher-latency; streaming is unbounded, low-latency, and harder (ordering, windows, replay). Batch is the default; add streaming where freshness pays.
• An orchestrator runs the pipeline as a DAG — tasks with dependency edges, no loops — executing each step only after its inputs succeed, running branches in parallel, and retrying failures (Airflow is the classic tool; the DAG is the idea).
• Trustworthy pipelines require idempotency (safe retries), backfills (recompute history), schema evolution (tolerate source changes), data-quality checks (fail loudly on bad data), lineage (trace numbers back), and observability (catch silent failures).
• Pipelines got popular because data volume exploded, analytics and ML demand surged, and cloud storage/compute became cheap and elastic — three forces that turned pipelines into core infrastructure.

• [ ] I can define a data pipeline and name its four stages: ingestion, transformation, storage, serving.
• [ ] I can distinguish batch from streaming ingestion and explain what CDC does and why it beats repeated full scans.
• [ ] I can explain ETL vs ELT and give the economic reason ELT rose with cloud warehouses.
• [ ] I can contrast batch and streaming pipelines end-to-end and state when each is worth it.
• [ ] I can describe a pipeline as a DAG and explain what an orchestrator does with it.
• [ ] I can explain why idempotency is foundational and how it makes backfills and retries safe.
• [ ] I can list the trust disciplines — idempotency, backfills, schema evolution, data quality, lineage, observability — and what failure each prevents.
• [ ] I can name the three forces (data volume, analytics/ML demand, cheap elastic cloud) that made data pipelines core infrastructure.