Apache Iceberg on AWS: the table format that ends the data swamp

Files in a bucket were never enough

The first-generation data lake is a simple idea: land your data as Parquet or CSV files on Amazon S3, register the folders in a catalogue like the AWS Glue Data Catalog, and point a query engine at them. It is cheap, it is open, and it scales to petabytes without anyone provisioning a database. For a while it feels like you have outgrown the warehouse entirely.

Then the cracks show, and they all trace back to one fact: a folder of files is not a table. There are no transactions, so a writer that fails halfway leaves you with half-written files and no way to undo them. Concurrent writes are risky, because two jobs appending to the same prefix have no referee. Schema changes are painful, because adding or retyping a column can mean rewriting every file or breaking every reader. And there is no consistent snapshot, so a reader running while a writer is mid-flight can see a table that never actually existed, some new files, some old, no clean line between them.

None of these is fatal on day one. They accumulate. A lake that nobody can fully trust to be correct, that everyone works around with bespoke locking and overnight-only write windows, is what people mean by the data swamp. The storage was never the problem. The missing table semantics were.

What an open table format adds

An open table format fixes this without abandoning the files-on-S3 model that made the lake attractive. It adds a metadata layer that sits over the data files and records, authoritatively, exactly which files make up a table at a given point in time. The data is still Parquet on S3. What changes is that the engine no longer discovers a table by listing a folder; it reads a metadata pointer that names the precise set of files in the current version of the table.

That one indirection is what turns a folder of files into a real table. Adding data becomes a metadata operation that swaps in a new, valid file list atomically. Readers follow a single pointer, so they always see one coherent version. The format, not a fragile convention about folder layout, becomes the source of truth for what the table contains.

Apache Iceberg, specifically

Apache Iceberg is the open table format that has become the de facto standard for this on AWS. Its central promise is ACID transactions on data sitting in S3. Every change to a table, an append, an overwrite, a delete, produces a new immutable snapshot: a complete, consistent list of the files that constitute the table at that moment. A commit is atomic. It either fully succeeds and becomes the new current snapshot, or it does not happen at all. There is no partial state for a reader to stumble into.

This gives you snapshot isolation. A query reads the snapshot that was current when it started and sees that consistent view for its whole duration, even if a write commits a new snapshot underneath it. Reads and writes stop fighting. You can run a long analytical query and an ingestion job against the same table at the same time, and both behave correctly.

Schema evolution that does not rewrite the data

On a raw Parquet lake, changing the schema is a minefield because column identity is positional, tied to where the column sits in the file. Iceberg instead assigns every column a stable internal ID. That lets you add, drop, rename, reorder and widen columns as pure metadata operations, with no rewrite of the underlying data files and no surprise breakage for downstream consumers.

The practical effect: a rename is a rename, not a silent re-read of a different column. Dropping a column does not resurrect an unrelated one because positions shifted. Adding a column is instant and the old files simply read back NULL for it. Schema change goes from a scheduled, risky migration to a routine, safe operation, which is exactly what an evolving data platform needs.

Partition evolution and hidden partitioning

The most expensive mistake on a classic lake is choosing the wrong partition key, because you are stuck with it. The physical folder layout encodes the partitioning, queries are written to know that layout, and changing it means rewriting the whole table and every query that touches it. Get it wrong and you carry that wrong choice forever.

Iceberg breaks both halves of that trap. With hidden partitioning, the table records the partition values in its metadata, so queries filter on the real column (WHERE event_time >= ...) and Iceberg works out which files to prune. The query author never needs to know the physical layout, and there is no extra derived partition column to remember to filter on. And with partition evolution, you can change the partitioning of an existing table, say from daily to hourly, as a metadata change that applies to new data, while old data keeps its original layout and old queries keep working. You are no longer married to your first guess.

Time travel and rollback

Because every commit leaves an immutable snapshot behind, the table's history is queryable. Time travel lets you query the table as it stood at a past snapshot or timestamp, which is invaluable for audit, debugging, and reproducing the exact inputs a model or report ran on. Rollback lets you undo a bad write by pointing the table back at a known-good earlier snapshot, turning what would have been a recovery-from-backup incident into a one-line operation.

-- query the table as it was at a point in time (Spark SQL on Iceberg)
SELECT * FROM sales.orders
  FOR TIMESTAMP AS OF '2026-06-01 09:00:00';

-- or pin to a specific historical snapshot id
SELECT * FROM sales.orders
  FOR VERSION AS OF 3821550127947524752;

Time travel reads an old snapshot directly. Snapshots are retained until you expire them, so audit and reproducibility have a real, queryable basis rather than a hope that a backup exists.

How AWS supports Iceberg

Iceberg's value on AWS is that it is not tied to one engine. The same Iceberg table, defined once, is readable and writable by most of the analytics stack:

Creating a table is ordinary SQL. In Athena, the only thing that marks it as Iceberg is the table type:

-- Athena: an Iceberg table, catalogued in Glue, data in S3
CREATE TABLE sales.orders (
  id        bigint,
  status    string,
  amount    decimal(12,2),
  event_time timestamp)
PARTITIONED BY (day(event_time))
LOCATION 's3://my-lakehouse/orders/'
TBLPROPERTIES ('table_type' = 'ICEBERG');

One table definition, then Glue jobs, Athena, Redshift and EMR all read and write it. Open format, open catalogue, many engines, no copying data per tool.

Amazon S3 Tables takes this a step further: a purpose-built bucket type for Iceberg that exposes tables as a first-class S3 resource and runs maintenance for you in the background. It is the easiest way to adopt Iceberg without standing up your own maintenance pipeline, which leads to the one thing nobody should gloss over.

It is not zero-ops

Iceberg removes the worst lake problems, but it introduces housekeeping of its own, and being honest about it is the difference between a healthy lakehouse and a slow one. Two jobs matter most. Compaction merges the many small files that frequent or streaming writes produce into fewer, larger files, because thousands of tiny files make queries slow and metadata heavy. Snapshot expiry removes old snapshots and the data files they alone reference, because the same history that powers time travel will accumulate storage and metadata indefinitely if you never prune it.

On Amazon EMR or Glue you run these as scheduled maintenance. Amazon S3 Tables automates compaction and snapshot management for you, which is much of its appeal, though you still set retention policy and keep an eye on cost. Either way, treat maintenance as a designed-in part of the platform, not an afterthought. An Iceberg table that is never compacted or expired drifts back toward the very swamp it was meant to drain.

Strip it back and the proposition is straightforward. Iceberg keeps the open, cheap, S3-native storage that made the lake attractive, and adds the transactions, schema and partition evolution, and time travel that the lake always lacked, accessible from Glue, Athena, Redshift, EMR and S3 Tables alike. Budget for the housekeeping, automate it where S3 Tables lets you, and the data swamp stops being the place your lake inevitably ends up.