Architecture steps
7 / 7
Through Jobs automation
End-to-end FMCG demand platform using the Kaggle M5 Forecasting dataset. Raw files were validated in Python, orchestrated with Airflow, and transformed in Databricks into a Medallion lakehouse ready for analytics / Power BI.
Published report walmart_visualization — Executive, Store, Product, and Demand pages on Gold aggregates. Open in Power BI ↗
Analysis pages: 08 Executive · 09 Store · 10 Product · 11 Demand
3 CSVs in data/raw
Validate + manifest
m5_pipeline DAG
Raw Delta + metadata
Star schema + aggs
| Layer | Schema | Tables | Highlight |
|---|---|---|---|
| Bronze | workspace.walmart_m5_bronze |
3 | calendar 1,969 · sales 30,490 · prices 6,841,121 |
| Silver | workspace.walmart_m5_silver |
5 | sales long-form 59,181,090 rows |
| Gold | workspace.walmart_m5_gold |
12 | 3 dims · 2 facts · 7 aggregates |
Kaggle M5 CSVs staged locally under data/raw/.
Source detection, schema/size/row checks, batch manifests.
Open →Docker LocalExecutor DAG with parallel Bronze staging branch.
Open →Volume landing + Delta tables with ingestion metadata.
Open →Wide→long sales, products, stores, prices, calendar cleans.
Open →Star schema facts/dims and BI aggregate tables.
Open →Dashboard 1 validated vs gold_export with key demand/store/category findings.
Open →Dashboard 2 validated vs gold_export: state mix, store ranks, category share, interactions.
Open →Dashboard 3 validated: Top 15 FOODS_3, volatility scatter, dept prices.
Open →Dashboard 4 validated: rolling demand, events, dual-axis sales combo.
Open →Latest successful run documented in 07 — Airflow → Databricks Jobs (includes Graph + Grid screenshots and DAG diagrams).