Airflow orchestrates the M5 pipeline in Docker

Apache Airflow runs locally via Docker Compose (LocalExecutor). The m5_pipeline DAG calls the existing Python validation code, then stages parallel Bronze load markers for sales, prices, and calendar.

DAG run: success UI: localhost:8088 Next: Databricks Bronze

m5_pipeline DAG

Defined in airflow/dags/m5_pipeline_dag.py. After validation, the three Bronze staging loads run in parallel.

start

↓

check_source_files

↓

validate_raw_files

↓

load_sales

load_prices

load_calendar

↓

validate_bronze

↓

end

Successful graph run (Airflow UI)

All eight tasks completed successfully (~11 seconds). Parallel loads fan out after validation, then rejoin at validate_bronze.

Airflow Graph view of m5_pipeline showing all tasks successful — Airflow Graph view — `m5_pipeline` after a successful manual trigger.

How to run Docker + Airflow

Install and start Docker Desktop.
From the project, start the stack:
```
cd airflow
docker compose up -d
```
Open the UI: http://localhost:8088
Login: airflow / airflow
Find DAG m5_pipeline → unpause → Trigger DAG.
Open the run → Graph tab to confirm all tasks are green.

# useful commands
docker compose ps
docker compose logs -f airflow-scheduler
docker compose down

What each task does

Task	Role
`check_source_files`	Confirm the three raw CSVs exist under `data/raw/`
`validate_raw_files`	Schema / size / row-count checks + write ingestion manifest
`load_sales` / `load_prices` / `load_calendar`	Parallel Bronze staging markers (Databricks write comes next)
`validate_bronze`	Confirm all three staging markers exist for the batch