The Data Lakehouse

To understand the lakehouse, follow the three-step evolution that produced it. Each step fixed a real weakness of the previous one and introduced a new trade-off.

The data warehouse came first: a structured, high-performance store for analytics (yesterday's day covered Snowflake as one). It uses schema-on-write — you define the table's structure up front and the warehouse enforces it as data loads. That gives fast, reliable queries with proper transactions, but it is expensive at large scale, and it struggles with raw or unstructured data like logs, images, and JSON that doesn't fit a fixed schema.

The data lake was the reaction: dump all your raw data — any format, structured or not — as plain files into cheap cloud object storage (S3, GCS). It uses schema-on-read — no structure is enforced on the way in; you impose structure only when you query. This is cheap and infinitely flexible, but it is also messy: with no table management there are no ACID transactions (a half-finished write leaves corrupt files), no reliable schema, and the whole thing tends to rot into a "data swamp" nobody trusts.

The lakehouse is the synthesis: keep the lake's cheap, open storage, but add a management layer on top that gives it the warehouse's guarantees — ACID transactions, reliable schemas, and fast queries. You get the lake's cost and flexibility and the warehouse's reliability, in one place, without maintaining two separate systems.

What turns a pile of files into a reliable table is an open table format — a metadata layer that sits over your data files and tracks which files make up a table, what its schema is, and a transaction log of every change. The data itself is usually stored as Parquet (an open, columnar file format) in object storage; the table format is the bookkeeping that makes those files behave like a real database table.

Three formats dominate, and they solve the same problem in similar ways:

• Delta Lake — created by Databricks; a transaction log (_delta_log) recording every commit.
• Apache Iceberg — an open standard with strong support across engines; designed for very large tables and safe schema/partition changes.
• Apache Hudi — focused on fast upserts and incremental processing (efficiently applying changes and reading only what changed).

What they all add on top of raw Parquet files:

• ACID transactions — a write either fully commits or doesn't; concurrent readers never see a half-written table.
• Schema evolution — you can add, rename, or change columns safely, and the format tracks the change.
• Time travel — because every change is logged, you can query the table as it existed at a past version or timestamp, and roll back mistakes.

Read the stack bottom-up: cheap object storage holds columnar Parquet files, the table format wraps them with a transaction log and schema, and any compatible query engine reads through that layer.

A defining property of the lakehouse is that storage and the query engines are fully decoupled. The data lives in open files in object storage, owned by no single vendor, and multiple engines can query the same tables:

• Spark — large-scale batch processing and transformation.
• Trino (formerly Presto) — fast interactive SQL across the lake.
• Flink — streaming.

Because the format is open, you are not locked into one engine or vendor — you can point the right tool at the same data. This is the lake's flexibility carried into a governed, transactional world.

Raw data is rarely clean enough to report on directly, so lakehouses organize data into quality tiers, conventionally named after medals. The medallion architecture is a flow from raw to refined, each layer a set of tables:

• Bronze — raw data landed exactly as it arrived, untouched. It is the durable record you can always reprocess from.
• Silver — cleaned and conformed: duplicates removed, types fixed, tables joined into a validated, queryable form.
• Gold — aggregated, business-level tables ready for dashboards and reports (revenue by region, daily active users).

Data flows left to right, gaining quality and structure at each hop. Keeping raw Bronze means you can always rebuild Silver and Gold if logic changes — you never lose the original.

The three ways of managing analytical data map cleanly onto the trade-offs:

The lakehouse's pitch is that you no longer have to choose: one open, cheap, transactional layer serves the raw-data flexibility of a lake and the reliability of a warehouse.

• The lakehouse is the synthesis of two predecessors: the warehouse (structured, reliable, expensive, schema-on-write) and the lake (cheap, flexible, messy, schema-on-read, no ACID).
• It keeps the lake's cheap open object storage but adds warehouse-grade table management on top.
• Open table formats — Delta Lake, Apache Iceberg, Apache Hudi — layer a transaction log, schema, and metadata over Parquet files, adding ACID transactions, schema evolution, and time travel.
• Storage and compute are decoupled: multiple open engines (Spark, Trino, Flink) query the same tables with no vendor lock-in.
• The medallion architecture organizes data into Bronze (raw), Silver (cleaned), and Gold (business-ready) tiers, gaining quality at each hop and preserving the raw layer for reprocessing.
• The lakehouse removes the old either/or between lake flexibility and warehouse reliability by unifying them in one layer.

• [ ] I can trace the evolution from warehouse to lake to lakehouse and name each one's trade-off.
• [ ] I can define schema-on-write versus schema-on-read.
• [ ] I can explain why a plain data lake lacks ACID and tends toward a "data swamp."
• [ ] I can name Delta Lake, Iceberg, and Hudi and what they add over raw Parquet.
• [ ] I can explain how a table format delivers ACID, schema evolution, and time travel.
• [ ] I can describe the Bronze/Silver/Gold medallion flow and why raw Bronze is kept.
• [ ] I can contrast a lakehouse table with a lake table and a warehouse table.