The medallion pattern on AWS: Bronze, Silver, Gold done right

A data lakehouse gives you the cheap, open storage of a lake with the structure and transactions of a warehouse. But storage and table formats alone don’t stop the chaos. Without an agreed way to organise data through the platform, you end up with the same dumping ground in a more expensive coat. The medallion pattern, popularised in the lakehouse world, is the organising principle that keeps it disciplined: data moves through Bronze, Silver, and Gold layers, getting cleaner and more useful at each step.

It is a simple idea that is easy to get wrong. This is how to do it right on AWS.

The pattern in one line

Three layers, refined progressively, with a quality gate between each:

• Bronze is raw: source data exactly as it arrived.
• Silver is cleaned and conformed: deduplicated, type-correct, validated, joined to reference data.
• Gold is business-ready: aggregates, marts, and features shaped for consumption.

The discipline isn’t the three names. It is that data only moves forward, and only after it passes the quality bar for the next layer. Each layer has a known purpose and a known level of trust, and nothing skips a step.

Bronze: the record of what arrived

Bronze lands source data as it arrived, and leaves it alone. It is immutable and append-only: you write new data, you never edit or overwrite history. That property is what makes Bronze valuable, because it is the faithful record of what each source actually sent you, on the day it sent it.

Resist the temptation to “just tidy up” on the way in. Don’t fix the odd type, drop a column, or dedupe at Bronze. The moment you do, you have lost raw fidelity and you can no longer reprocess from source if a downstream rule turns out to be wrong. Keep Bronze raw precisely so that every later layer is reproducible: if your Silver logic has a bug, you fix the logic and rebuild Silver from Bronze, with nothing lost.

Silver: the trusted, conformed layer

Silver is where the work happens. This is the layer most queries should hit. The transform from Bronze to Silver does the unglamorous, essential things:

• Clean malformed values and handle nulls deliberately.
• Deduplicate so each business entity appears once.
• Conform types, so a date is a date and a number is a number everywhere.
• Apply data-quality rules, rejecting or quarantining records that fail.
• Join reference data so codes become meaningful and entities are enriched.

The output is one conformed model: a clean, consistent, query-able representation of the business, regardless of how messy or numerous the sources were. Silver is the single version of the truth that analysts and downstream jobs build on.

Gold: shaped for consumption

Gold is the layer your dashboards and models actually consume. Where Silver is a clean, normalised picture of the whole business, Gold is purpose-built: business-level aggregates, dimensional marts, and feature tables, each shaped for a specific consumption pattern.

The goal at Gold is speed and intuitiveness. An analyst opening a Gold mart should find it fast to query and easy to understand, with the joins and aggregations already done. A machine-learning team should find features ready to train on. You will often have many Gold tables built from the same Silver layer, each tailored to its audience, which is fine: Gold is allowed to denormalise and duplicate in the name of consumption.

Implementing it on AWS

The pattern maps cleanly onto a handful of AWS services. The shape we use most often:

• Amazon S3 for storage, with a bucket or prefix per layer, for example s3://lake/bronze/, s3://lake/silver/, s3://lake/gold/. The physical separation makes the layer of any dataset obvious and makes access control per layer straightforward.
• Apache Iceberg as the table format, with an Iceberg table per layer. Iceberg gives you ACID transactions, schema evolution, and time travel on top of S3, which is what lets append-only Bronze and rebuildable Silver behave like proper tables rather than loose files.
• AWS Glue (Spark) for the transforms: one Glue job to land Bronze, one to build Silver, one to build Gold. Glue’s Data Catalog registers the Iceberg tables so engines like Amazon Athena and Amazon Redshift can query them.
• AWS Step Functions or Glue workflows for orchestration, sequencing the jobs and enforcing that Silver runs only after Bronze, and Gold only after Silver.

The Bronze-to-Silver transform in a Glue Spark job is unremarkable, which is the point:

# Glue (Spark) job: build a conformed Silver table from raw Bronze
bronze = spark.read.format("iceberg").load("glue_catalog.lake.bronze_orders")

silver = (
    bronze
      .dropDuplicates(["order_id"])              # deduplicate
      .withColumn("order_ts", to_timestamp("order_ts"))  # conform types
      .filter(col("order_id").isNotNull())       # validate
      .join(broadcast(customers), "customer_id", "left")  # enrich w/ reference
)

# Iceberg gives Silver ACID writes, schema evolution, and time travel on S3
silver.writeTo("glue_catalog.lake.silver_orders").append()

One conformed Silver table from raw Bronze. The same shape repeats for Gold, where the transform aggregates and reshapes Silver for a specific audience.

A point worth stressing: move incrementally between layers. Process only the new or changed data at each hop rather than rebuilding the world every run. Iceberg’s snapshots and Glue’s job bookmarks make incremental processing the default, which keeps both cost and runtime in proportion to how much data actually arrived.

Where the quality gates sit

The gates are the part teams most often skip, and they are what separate the medallion pattern from “three folders with nicer names.” There are two:

• Between Bronze and Silver: schema enforcement, validation, and deduplication. This gate decides what is allowed to become trusted. Records that fail schema or validation are rejected or quarantined rather than promoted.
• Between Silver and Gold: business rules and aggregation. This gate decides what is allowed to become business-facing, applying the logic and definitions the business has agreed on.

AWS Glue Data Quality can enforce these expectations declaratively. You define rules, completeness, uniqueness, accepted ranges, referential checks, and the gate evaluates each batch against them before it is promoted. A batch that fails the Bronze-to-Silver rules doesn’t silently become trusted data; it is flagged, and the pipeline can stop, quarantine, or alert. The gate is enforced by the platform, not by hoping the upstream job behaved.

Why it prevents the swamp

A data swamp is what you get when nobody can answer two questions about a dataset: where does this sit, and how much can I trust it? The medallion pattern answers both by construction.

• Every dataset has a known layer and a known quality level. “It’s in Silver” is a statement about trust, not just location.
• Lineage is clear. Every Gold table traces to Silver, every Silver table to Bronze, every Bronze table to a source. With Iceberg’s history and the Glue catalog, you can follow the chain and reproduce any output.
• Consumers query Silver or Gold, never raw files. The cleaned, conformed, governed layers are the interface; Bronze stays an internal record.

None of this is exotic. It is S3, Iceberg, Glue, and a couple of orchestration choices, applied with the discipline that data only moves forward and only through a gate. That discipline is the difference between a lakehouse that stays trustworthy as it grows and one that quietly turns back into a swamp.