Gold Serving Layer with Databricks and Synapse

Transform Silver payments data into Gold dimensions, facts, reconciliation outputs, and Synapse-ready Parquet snapshots using Databricks.
Azure • Lakehouse • Intermediate • Payments

Architecture Diagram

Build this project

  • Source code
  • Setup files
  • Architecture walkthrough
  • Implementation videos
  • Execution guidance
  • Support
Get Project Access

Included in Data0to1 Project Library

What this project solves

Silver tables are clean and trusted, but reporting teams usually need data shaped into business-friendly structures.

For payments analytics, teams may need customer, merchant, wallet, and device dimensions, along with fact tables for balances, ledger entries, settlements, refunds, chargebacks, and disputes. They may also need reconciliation outputs to compare internal and bank settlement data.

In this pipeline, you will create the Gold serving layer on Azure. You will transform Silver Delta tables using Azure Databricks, create business-ready dimension and fact tables, publish Gold data as Delta tables and Parquet snapshots, and make the outputs available through Synapse for reporting and downstream analytics.

What you’ll build

  • Build business-ready Gold tables from Silver data
  • Create dimensions for customers, merchants, wallets, and devices
  • Create fact tables for balances, ledger entries, settlements, refunds, chargebacks, and disputes
  • Create reconciliation outputs for settlement comparison
  • Store Gold tables as Delta tables on ADLS Gen2
  • Publish reporting-friendly Parquet snapshots for Synapse
  • Track Gold pipeline runs using audit logs

How you’ll build it

  • Read Silver Delta tables from ADLS Gen2
  • Use Databricks and Spark to transform Silver data into Gold models
  • Build dimensions, facts, and reconciliation outputs
  • Publish Gold Delta tables and Parquet snapshots
  • Query or load the Gold outputs through Synapse and validate the results

Tools you’ll use

Azure Data Lake Storage Gen2 • Azure Databricks • Delta Lake • Apache Spark Synapse Serverless SQL • Synapse Dedicated SQL Pool • SQL • Python

What you’ll walk away with

After completing this pipeline, you will be able to:

  1. Explain how Silver data becomes a Gold serving layer
  2. Build dimension and fact tables using Azure Databricks
  3. Create reconciliation outputs for payments analytics
  4. Publish Gold Delta tables and Synapse-ready snapshots
  5. Validate and query curated Gold data using Synapse

Ready to build this project?

Get access to the project code, setup files, architecture walkthrough, implementation videos, and support.

Get Project Access