Create a Silver Lakehouse on Amazon S3 with Iceberg

Transform raw batch and CDC healthcare data in Amazon S3 into trusted Silver tables using AWS Glue, Apache Iceberg, Athena, and MWAA.
AWS • Lakehouse • Intermediate • Healthcare

Architecture Diagram

Build this project

  • Source code
  • Setup files
  • Architecture walkthrough
  • Implementation videos
  • Execution guidance
  • Support
Get Project Access

Included in Data0to1 Project Library

What this project solves

Raw healthcare data is useful, but it is not ready for analytics as-is.

It may arrive from batch loads, CDC streams, or both. The same business entity may appear more than once, records may change over time, and deleted records from CDC data need to be handled correctly.

In this pipeline, you will create the Silver layer of an AWS lakehouse. You will read raw batch and CDC data from Amazon S3, process it using AWS Glue, write trusted Silver tables using Apache Iceberg, and validate the results with Athena.

What you’ll build

  • Set up Silver Apache Iceberg tables on Amazon S3
  • Read raw batch and raw CDC data from S3
  • Combine batch and CDC records by table type
  • Keep only the latest version of each record
  • Handle CDC delete records correctly
  • Build trusted Silver tables for downstream pipelines
  • Validate Silver tables using Athena

How you’ll build it

  • Read raw batch and CDC datasets from Amazon S3
  • Use AWS Glue to process and standardize healthcare records
  • Apply latest-record logic and CDC delete handling
  • Write trusted Silver tables using Apache Iceberg
  • Orchestrate the Silver pipeline using MWAA/Airflow and validate with Athena

Tools you’ll use

Amazon S3 • AWS Glue • Apache Iceberg • Apache Parquet • AWS Glue Catalog • Amazon Athena • Amazon MWAA • Apache Airflow

What you’ll walk away with

After completing this pipeline, you will be able to:

  1. Explain how raw data becomes trusted Silver lakehouse tables
  2. Process batch and CDC data into a single Silver layer
  3. Use Apache Iceberg tables on Amazon S3
  4. Handle latest-record selection and CDC deletes
  5. Orchestrate and validate lakehouse pipelines using MWAA and Athena

Ready to build this project?

Get access to the project code, setup files, architecture walkthrough, implementation videos, and support.

Get Project Access