Bronze and Silver Lakehouse with Azure Databricks

Transform raw payments data in ADLS Gen2 into Bronze and Silver Delta tables using Azure Databricks, Delta Lake, lineage, dedupe logic, and validation.
Azure • Lakehouse • Intermediate • Payments

Architecture Diagram

Build this project

  • Source code
  • Setup files
  • Architecture walkthrough
  • Implementation videos
  • Execution guidance
  • Support
Get Project Access

Included in Data0to1 Project Library

What this project solves

Raw data in a data lake is rarely ready for analytics.

Files may come from multiple systems, contain duplicates, use different formats, or require standardization before downstream teams can trust them.

In this pipeline, you will create the Bronze and Silver layers of an Azure lakehouse. You will read raw payments data from ADLS Gen2, use Azure Databricks and Delta Lake to create Bronze tables with lineage and audit information, transform the data into trusted Silver tables, remove duplicates using latest-record logic, and validate the results using Synapse Serverless SQL.

What you’ll build

  • Build Bronze lakehouse tables on ADLS Gen2
  • Add lineage and audit information during ingestion
  • Store metadata and control totals for pipeline monitoring
  • Standardize raw datasets into business-friendly structures
  • Remove duplicate records using latest-record logic
  • Create trusted Silver tables using Delta Lake
  • Track pipeline runs and data quality checks
  • Query Silver tables using Synapse Serverless SQL
  • Explore Delta Lake features such as Time Travel and Optimize

How you’ll build it

  • Read raw payments data from ADLS Gen2
  • Use Azure Databricks to create Bronze Delta tables
  • Add lineage, audit information, and control totals
  • Clean, standardize, and deduplicate data into Silver tables
  • Validate Silver outputs using Synapse Serverless SQL

Tools you’ll use

Azure Data Lake Storage Gen2 • Azure Databricks • Delta Lake • Apache Spark • Synapse Serverless SQL • Azure Key Vault • Python

What you’ll walk away with

After completing this pipeline, you will be able to:

  1. Explain how Bronze and Silver lakehouse layers work on Azure
  2. Create Delta Lake tables on ADLS Gen2 using Databricks
  3. Add lineage, audit tracking, and control totals to lakehouse pipelines
  4. Clean and deduplicate raw data using latest-record logic
  5. Validate Silver Delta tables using Synapse Serverless SQL

Ready to build this project?

Get access to the project code, setup files, architecture walkthrough, implementation videos, and support.

Get Project Access