Foundations
PostgreSQL to S3 Data Lake with AWS Glue
Build a healthcare data lake ingestion pipeline on S3 using AWS Glue, Parquet, Glue Catalog, and Athena.
Ready-made end-to-end pipeline projects across AWS, Azure, and GCP — with source code, setup files, architecture walkthroughs, implementation videos & execution guidance
Use them for portfolio practice, upskilling, cloud stack comparison, and faster project prototyping.
20+ pipeline projects across AWS, Azure, and GCP.
Source code • Setup files • Architecture walkthroughs • Implementation videos • Execution guidance • Support
Browse the AWS, Azure, and GCP pipeline projects included in this library.
Filter by Cloud and Topic
Build a healthcare data lake ingestion pipeline on S3 using AWS Glue, Parquet, Glue Catalog, and Athena.
Capture live PostgreSQL changes with AWS DMS and stream log-based CDC records into an S3 data lake.
Convert raw batch and CDC data in S3 into ACID-compliant Silver Iceberg tables using Glue, Athena, and MWAA.
Build Gold dimension and fact tables from Silver healthcare data using AWS Glue, Apache Iceberg, Athena, and MWAA.
Create clinical, operations, finance, and executive analytics marts in Redshift using modular dbt workflows.
Automate Redshift exports to S3 as Parquet files for downstream applications, reports, and business feeds.
Build a low-latency patient vitals streaming pipeline using Kinesis, Lambda validation, and real-time alerts.
Build a watermark-driven MySQL to ADLS Gen2 ingestion pipeline using ADF, metadata tables, and Synapse validation.
Automate vendor file ingestion with ADF using schema validation, deduplication checks, and metadata logs.
Build a Medallion Lakehouse with Databricks, Delta tables, lineage tracking, dedupe logic, and Silver upserts.
Turn Silver Delta data into Gold dimensions, facts, reconciliation outputs, and Synapse-ready snapshots.
Create finance, operations, risk, merchant, and executive analytics marts in Synapse using modular dbt workflows.
Build a real-time Cosmos DB CDC pipeline to publish payment events via Event Hubs and archive them in ADLS Gen2.
Build a low-latency payment streaming pipeline with Event Hubs and Databricks Structured Streaming for live KPI monitoring.
Build a watermark-driven MySQL to BigQuery batch ingestion pipeline using Dataflow, staging tables, and Cloud Composer.
Automate vendor file ingestion into BigQuery with metadata logs, archive handling, and quarantine routing.
Orchestrate BigQuery ELT transformations with SQL stored procedures, refined tables, and Composer DAGs.
Model refined retail data into reusable dimensions, facts, KPIs, and business marts using dbt on BigQuery.
Automate BigQuery exports to Cloud Storage as CSV/Parquet files with audit logs and Composer DAGs.
Train, evaluate, and score a return-risk model in BigQuery ML using SQL and retail order and return data.
Build a low-latency retail event streaming pipeline with Pub/Sub, Dataflow validation, BigQuery writes, and DLQ routing.
You get access to AWS, Azure, and GCP project tracks in one package. The library includes 20+ pipeline projects with source code, setup files, architecture walkthroughs, implementation videos, and support.
Choose a single cloud track if you want to focus exclusively on AWS, Azure, or GCP. Choose the Ultimate Project Library for full access to all projects across AWS, Azure, and GCP. If you are unsure, the Ultimate Project Library offers the best long-term value for multi-cloud data engineers.
Yes. These projects are designed to run inside your own cloud account so you can practice real setup, execution, validation, and cleanup instead of only watching demos.
Yes, most of our projects are designed to fit within the cloud providers’ free tiers and credits. Provided you follow our guidance and cleanup steps, you can easily shut down services and keep your learning free.
No cloud experience is required. We explain all setup and implementation steps inside the projects. However, having a basic understanding of SQL, Python, and general Data Engineering concepts will help you get the most out of the library.
You can ask questions through the project discussion area. We are here to help you resolve any issues and successfully complete your projects.