The five-layer lakehouse on AWS: storage, catalog, table format, processing, consumption

Most data architecture arguments are really arguments about a single trade-off: governance versus flexibility. For two decades the industry had two answers to that trade-off, and each one solved half the problem while creating the other. The lakehouse is an attempt to stop choosing. It is worth understanding why, because the pattern only makes sense against the two things it replaces.

The two old answers, and why both fall short

The first answer was the data warehouse. It is disciplined: a governed, structured store with a defined schema, strong consistency, and fast SQL for reporting and BI. Decades of investment have made warehouses excellent at the job they were built for. The cost of that discipline is rigidity. The schema has to be defined up front, the data has to be structured to fit it, and storage and compute are typically coupled, so scaling one means paying for the other. Warehouses also struggle with the semi-structured and unstructured data, logs, JSON, images, that now make up most of what an organisation generates.

The second answer was the first-generation data lake: a cheap, flexible object store where you land everything in its raw form and worry about structure later. It is the opposite trade-off. Storage is inexpensive, any data type is welcome, and you decouple from any single engine. The cost this time is governance. Without a metadata layer or transactional guarantees, a lake quietly degrades into a data swamp: files nobody can find, schemas nobody agrees on, and no way to update a record without rewriting a partition by hand. The data is all there, and none of it is trustworthy.

The lakehouse takes the strengths of both: the cheap, flexible, open storage of the lake, with the schema, transactions, and governance of the warehouse layered on top. It does this not by inventing a monolith but by separating concerns into distinct layers, each replaceable, each doing one thing well.

The lakehouse in one picture

On AWS the pattern resolves into five layers, stacked from the bytes on disk up to the people and models asking questions. Read it bottom-up: storage holds the data, the catalog describes it, the table format makes it behave, processing transforms it, and consumption queries it.

The discipline of the model is in the boundaries. Each layer talks to its neighbours through open, well-defined interfaces, so you can swap a query engine without touching storage, or add a new table format without re-cataloguing. Take them in turn.

Storage: Amazon S3

The foundation is Amazon S3. Everything else in the architecture is built on the simple fact that object storage is cheap, effectively bottomless, highly durable, and completely decoupled from compute. You pay to store bytes, and separately and only when you use it, you pay to process them. That separation is the single most important property of the whole pattern.

It means there is one copy of the data that many engines share. The same Parquet files in the same bucket are read by your SQL engine, your Spark jobs, and your machine-learning pipelines, with no copying, no syncing, and no second version drifting out of date. S3 also lets you tier data by access pattern, so cold history costs a fraction of hot, recent data without moving it out of the lake. Storage stops being a constraint and becomes a commodity.

Catalog: the AWS Glue Data Catalog

Raw files in a bucket are not a dataset; they are a pile. What turns the pile into something queryable is the AWS Glue Data Catalog, the metadata layer of the lakehouse. It records the schemas, table definitions, column types, and partition information for the data sitting in S3, and it exposes that metadata through one consistent interface.

The point is shared definition. Because Athena, Redshift, EMR, and Glue jobs all read the same catalog, a table means the same thing to every engine that touches it. Data becomes discoverable, a query engine can list what exists and what shape it is, and consistently defined, so there is no per-team disagreement about what a column holds. The catalog is also where you attach fine-grained governance, with services such as AWS Lake Formation building on it to control who can see which tables, columns, and rows. Without this layer you are back in the swamp, guessing at structure file by file.

The table format: Apache Iceberg

The catalog describes the data, but on its own it does not make a directory of files behave like a database table. That is the job of an open table format, and on AWS the leading choice is Apache Iceberg. Iceberg sits over the files and adds the warehouse behaviour that the first-generation lake lacked: ACID transactions, so concurrent writes do not corrupt a table; schema evolution, so you can add or rename columns safely; time travel to query earlier snapshots; and reliable row-level updates and deletes.

This is the layer that ends the swamp. It is the difference between a folder of Parquet that you hope is consistent and a table you can trust enabling transactions on. AWS services including Athena, Glue, EMR, and Redshift can read and write Iceberg tables natively, so the format is a property of the data, not a feature locked inside one engine.

The table format is brief here on purpose, because it earns a post of its own. A companion piece in this series goes deep on Iceberg: how its snapshots and manifests work, and why it has become the default table format for lakehouses on AWS.

Processing: AWS Glue and Amazon EMR

Data rarely arrives in the shape the business needs. The processing layer is where it is cleaned, joined, aggregated, and moved through the stages of the lakehouse, and on AWS that work is carried mainly by two services. AWS Glue is serverless Spark: you write ETL or ELT logic and Glue provisions, runs, and tears down the compute, so there is no cluster to manage and you pay for the job, not the idle. It is the default for most transformation pipelines.

Amazon EMR is the managed-cluster option for larger or more specialised workloads, where you want control over the cluster, a broader set of open-source frameworks such as Spark, Hive, or Presto, or custom tuning that serverless does not expose. The two are complementary: Glue for the common case and the convenience of serverless, EMR when scale or customisation justifies running the cluster yourself. Both read and write through the catalog and table format, so the data they produce is immediately governed and queryable by everything above them.

Consumption: many engines, one copy

The top layer is where the data earns its keep. The defining feature of the lakehouse is that several specialised engines all query the same one copy of governed data in S3, each chosen for the job in front of it:

• Amazon Athena, serverless SQL straight against S3. Point it at a table in the catalog and query, with no infrastructure to run. Ideal for ad-hoc analysis and exploration.
• Amazon Redshift, the data warehouse for high-concurrency BI and complex analytical SQL, able to query lake data in place as well as data loaded into the warehouse.
• Amazon QuickSight, the BI and dashboarding layer that turns those queries into reports and visuals for the business.
• Amazon SageMaker, the machine-learning platform, reading the same curated data to train and serve models.

Because every one of these reads the shared catalog and the same files, there is no extract-and-copy step that creates a second, stale version for each tool. The engine changes to suit the question; the data does not move. A companion post in this series covers one of the most common decisions inside this layer in detail: when to reach for Athena and when to reach for Redshift.

Why this beats either old model

Stand back and the advantages line up against the failings the lakehouse was built to fix. You keep one governed copy of the data, in open formats, in your own storage, so it is trustworthy and discoverable, which the old lake was not. You apply the right engine to each job, SQL, BI, warehouse, or ML, instead of forcing every workload through one tool, which the old warehouse demanded. You avoid lock-in, because open table formats and open file formats mean no vendor owns your data. And because storage and compute are separate, the two scale independently: store petabytes cheaply and spin compute up and down to match demand, paying for each only when you use it.

None of this comes from a single magic service. It comes from the layered pattern: S3 for storage, Glue Data Catalog for metadata, Iceberg for table behaviour, Glue and EMR for processing, and Athena, Redshift, QuickSight, and SageMaker for consumption. Get the layers and their boundaries right, and you have the governance of a warehouse and the flexibility of a lake at the same time, which is the whole reason the pattern exists.