Airflow successfully triggered the Databricks Medallion job

1) End-to-end architecture

flowchart LR
  A[data/raw CSVs] --> B[Airflow m5_pipeline]
  B --> C[check + validate]
  C --> D[DatabricksRunNowOperator]
  D --> E[Job walmart_m5_medallion]
  E --> F[Bronze]
  F --> G[Silver]
  G --> H[Gold]
  H --> I[record_databricks_success]
      

2) Airflow DAG diagram

Defined in airflow/dags/m5_pipeline_dag.py.

flowchart LR
  start --> check_source_files --> validate_raw_files
  validate_raw_files --> run_databricks_medallion
  run_databricks_medallion --> record_databricks_success --> end
      

3) Airflow UI proof — Graph view

Latest run: every task green, including Databricks trigger and success recorder.

4) Airflow UI proof — Grid view

Earlier runs were short local-only validations. The latest run includes the full Databricks Medallion job (~11–12 minutes).

5) Databricks Job DAG diagram

Job walmart_m5_medallion runs notebooks sequentially through the Medallion layers.

flowchart TB
  bronze_ingestion["01 bronze_ingestion"] --> silver_sales["02 silver_sales"]
  silver_sales --> silver_products["03 silver_products"]
  silver_products --> silver_calendar["04 silver_calendar"]
  silver_calendar --> gold_dimensions["05 gold_dimensions"]
  gold_dimensions --> gold_facts["06 gold_facts"]
  gold_facts --> gold_aggregations["07 gold_aggregations"]
      

6) Combined control plane

flowchart TB
  subgraph Airflow["Apache Airflow m5_pipeline"]
    A1[check_source_files] --> A2[validate_raw_files]
    A2 --> A3[run_databricks_medallion]
    A3 --> A4[record_databricks_success]
  end

  subgraph Databricks["Databricks Job walmart_m5_medallion"]
    B1[01 Bronze] --> B2[02 Silver sales]
    B2 --> B3[03 Products/stores/prices]
    B3 --> B4[04 Calendar]
    B4 --> B5[05 Gold dims]
    B5 --> B6[06 Gold facts]
    B6 --> B7[07 Gold aggs]
  end

  A3 -->|Run Now + wait| B1
  B7 -->|job success| A4
      

7) Success marker written locally

{
  "status": "databricks_medallion_success",
  "batch_id": "m5_20260909T123803Z_ac813310",
  "databricks_job_id": "218487573209118",
  "layers": ["bronze", "silver", "gold"],
  "notebooks": [
    "01_bronze_ingestion",
    "02_silver_sales",
    "03_silver_products",
    "04_silver_calendar",
    "05_gold_dimensions",
    "06_gold_facts",
    "07_gold_aggregations"
  ]
}

File: data/ingestion/manifests/databricks_run_latest.json

What’s next

• ✓Airflow ↔ Databricks Jobs automation proven
• ○Power BI dashboards on workspace.walmart_m5_gold
• ○Rotate Databricks PAT used during setup (token was shared in chat)