PostgreSQL CDC to Amazon S3 with AWS DMS

Capture inserts, updates, and deletes from PostgreSQL and stream them into Amazon S3 using AWS DMS, CDC, and logical replication.
AWS • Foundations • Intermediate • CDC Ingestion

Architecture Diagram

Build this project

  • Source code
  • Setup files
  • Architecture walkthrough
  • Implementation videos
  • Execution guidance
  • Support
Get Project Access

Included in Data0to1 Project Library

What this project solves

Scheduled batch ingestion is useful, but many systems need database changes to arrive much faster.

In healthcare systems, new lab orders, lab results, medication updates, billing charges, and patient record changes may happen throughout the day. Instead of waiting for the next batch run, teams often capture inserts, updates, and deletes directly from the source database and move them into the data lake.

In this pipeline, you will build a PostgreSQL CDC pipeline using AWS DMS. You will configure logical replication, capture database changes, write CDC data into Amazon S3, and validate the captured changes using the AWS data lake layer.

What you’ll build

  • Configure PostgreSQL for Change Data Capture
  • Create AWS DMS source and target endpoints
  • Set up a DMS replication task for CDC
  • Capture inserts, updates, and deletes from PostgreSQL
  • Load CDC records into Amazon S3
  • Validate captured changes in the data lake
  • Simulate healthcare transactions and observe CDC behavior

How you’ll build it

  • Prepare PostgreSQL, Amazon S3, IAM, and AWS DMS resources
  • Configure DMS source and target endpoints
  • Enable PostgreSQL logical replication for CDC
  • Generate inserts, updates, and deletes in PostgreSQL
  • Capture the changes into Amazon S3 and validate the end-to-end flow

Tools you’ll use

PostgreSQL • AWS DMS • Amazon S3 • AWS IAM • Glue Crawler • Athena • PostgreSQL Logical Replication

What you’ll walk away with

After completing this pipeline, you will be able to:

  1. Explain & Implement how CDC ingestion works on AWS
  2. Configure AWS DMS endpoints and replication tasks
  3. Capture PostgreSQL inserts, updates, and deletes
  4. Load CDC records into Amazon S3
  5. Validate and troubleshoot CDC data in an AWS data lake

Ready to build this project?

Get access to the project code, setup files, architecture walkthrough, implementation videos, and support.

Get Project Access