calendar
OK
1,969 rows · bronze_calendar
Stage 1 Python code now acts as the quality gate before Airflow / Databricks. It confirms every raw file is present, large enough, schema-correct, and row-count complete, then emits a batch manifest for Bronze enrichment.
data/raw/_batch_id, _ingestion_timestamp, _source_file, _ingestion_date)src/
├── config.py # paths + expected contracts
├── ingestion/
│ ├── metadata.py # batch ids + manifest builder
│ └── run_ingestion.py # CLI entrypoint
├── validation/
│ └── raw_files.py # availability / schema / size / rows
├── transformations/ # placeholder for later Silver/Gold
└── utils/
├── logging_utils.py
└── paths.py
| Source | Rows | Size | Checks | Bronze target |
|---|---|---|---|---|
calendar.csv |
1,969 | 103,469 bytes | all passed | bronze_calendar |
sales_train_evaluation.csv |
30,490 | 121,736,518 bytes | all passed | bronze_sales |
sell_prices.csv |
6,841,121 | 203,395,785 bytes | all passed | bronze_sell_prices |
Sales-specific contract also verified d_1…d_1941
(1,941 day columns).
| Field | Purpose | Example |
|---|---|---|
_batch_id |
Tie all files in one ingestion run | m5_20260907T000027Z_34b8d78f |
_ingestion_timestamp |
UTC timestamp of the batch | 2026-09-07T00:00:27.596014+00:00 |
_ingestion_date |
Calendar date partition helper | 2026-09-07 |
_source_file |
Lineage back to raw CSV name | calendar.csv |
python src/ingestion/run_ingestion.py # outputs: # data/ingestion/manifests/<batch_id>.json # data/ingestion/manifests/latest.json # exit code 0 = ready for Airflow → Bronze
With a ready manifest, the project can move to
Apache Airflow: wire tasks for
check_source_files → validate_raw_files →
parallel Bronze loads for sales, prices, and calendar.