Vendor Files → ADLS Gen2 (Batch File Ingestion)

Azure • Foundations • Beginner • Payments

Architecture Diagram

Overview

Vendor data often arrives as files rather than database records.

Banks, payment processors, settlement partners, fraud providers, and external vendors typically deliver data through CSV or similar file formats on a daily, weekly, or monthly schedule.

Before this data can be used by downstream pipelines, it needs to be collected, validated, archived, and stored in a central data lake.

In this pipeline, you will build a file ingestion process on Azure that receives vendor files, stores them in ADLS Gen2, tracks delivery metadata, and prepares the data for downstream processing and analytics.

What You Will Build

  • Receive vendor files into Azure landing folders
  • Validate incoming files against expected patterns
  • Archive original files for audit and traceability
  • Track file deliveries using metadata tables
  • Prevent duplicate file processing
  • Maintain vendor file manifests
  • Store parsed vendor data in ADLS Gen2
  • Query ingested data using Synapse Serverless SQL
  • Validate file counts and ingestion results

Tech Stack

Azure Data Lake Storage Gen2 • Azure Data Factory • Azure SQL Database • Synapse Serverless SQL • Azure Key Vault • SQL

Learning Outcomes

After completing this pipeline, you will be able to:

  1. Build file-based ingestion pipelines on Azure
  2. Receive vendor files into a cloud data lake
  3. Validate incoming files before processing
  4. Archive source files for audit purposes
  5. Track file deliveries using metadata tables
  6. Prevent duplicate file processing
  7. Maintain file manifests and ingestion history
  8. Query ingested data using Synapse Serverless SQL
View Courses & Pricing →