Silver Lakehouse → Gold Serving Layer

Azure • Lakehouse • Intermediate • Payments

Architecture Diagram

Overview

In the previous pipeline, raw data was cleaned and standardized into trusted Silver tables.

But Silver tables are still not the final shape that reporting and analytics teams usually need. Business users need simpler tables that are easier to query, join, and use for dashboards or downstream reporting.

In this pipeline, you will build the Gold serving layer on Azure.

You will take Silver Delta tables from ADLS Gen2, create Gold dimension and fact tables using Databricks, publish reporting-friendly Parquet snapshots, and make the data available through Synapse.

What You Will Build

  • Build business-ready Gold tables from Silver data
  • Create dimension tables for customers, merchants, wallets, and devices
  • Create fact tables for balances, ledger entries, settlements, refunds, chargebacks, and disputes
  • Create reconciliation outputs to compare internal and bank settlement data
  • Store Gold tables as Delta tables on ADLS Gen2
  • Publish Gold data as Parquet snapshots for Synapse
  • Load Gold snapshots into Synapse tables
  • Track Gold pipeline runs using audit logs
  • Query Gold data using Synapse

Tech Stack

Azure Data Lake Storage Gen2 • Azure Databricks • Delta Lake • Apache Spark Synapse Serverless SQL • Synapse Dedicated SQL Pool • SQL • Python

Learning Outcomes

After completing this pipeline, you will be able to:

  1. Transform Silver tables into Gold tables
  2. Build dimension and fact tables for analytics use cases
  3. Create reconciliation outputs for settlement comparison
  4. Store Gold data as Delta tables on ADLS Gen2
  5. Publish reporting-friendly Parquet snapshots
  6. Load Gold snapshots into Synapse tables
  7. Track Gold layer execution using audit logs
  8. Understand how curated lakehouse data is served for reporting
View Courses & Pricing →