Report ยท Chapter 8

Python scripts

Twenty scripts, numbered in the order they run.

Source: src/ and databricks/ 01 through 20 4 October 2026
An isometric view of site records moving through documents and a database into dashboards.

Section 1Run order

Each script keeps its number. A later script reads the file written by the earlier one and does not overwrite it. 02_load_rfi_excel.py calls 01_raw_data_profiling.py by that filename, so those two names stay paired.

FileWrites
src/ingestion/01_raw_data_profiling.pydocs/data_profiling.md
src/ingestion/02_load_rfi_excel.pydata/processed/raw_rfi.csv
src/ingestion/03_raw_rfi_investigation.pydata/processed/profiling/
src/cleaning/04_clean_rfi.pyrfi_clean.csv and rfi_excluded.csv
src/validation/05_validate_rfi.pyvalidation/validation_results.csv
src/features/06_build_rfi_features.pyrfi_features.csv
src/database/07_load_rfi_duckdb.pyrfi.duckdb
src/database/08_preview_rfi_tables.pyPrints the five tables. Writes nothing.
databricks/09_classify_rfi_discipline.pyBaseline discipline classifier. Logged in MLflow.
databricks/10_compare_discipline_models.pyGTE embedding classifier. Accuracy 0.70, macro F1 0.50.
databricks/11_zero_shot_discipline.pyZero-shot LLM. Accuracy 0.69, macro F1 0.69.
databricks/12_register_discipline_model.pyRegistered digital_engineering.ml.rfi_discipline_classifier version 1.
databricks/13_classify_rfi_issue_type.pyIssue-type baseline. Accuracy 0.70, macro F1 0.34 on 105 RFIs.
databricks/14_compare_issue_type_models.pyGTE issue-type classifier. Accuracy 0.72, macro F1 0.47 on 105 RFIs.
databricks/15_zero_shot_issue_type.pyZero-shot issue type. Accuracy 0.41, macro F1 0.11 on 105 RFIs.
databricks/16_register_issue_type_model.pyRegistered digital_engineering.ml.rfi_issue_type_classifier version 1.
databricks/17_classify_document_impact.pyDocument-impact baseline. Accuracy 0.81, Yes recall 0.42 on 70 RFIs.
databricks/18_compare_document_impact_models.pyGTE document-impact classifier. Accuracy 0.84, Yes recall 0.68 on 70 RFIs.
databricks/19_zero_shot_document_impact.pyZero-shot document impact. Accuracy 0.44, Yes recall 1.00 on 70 RFIs.
databricks/20_register_document_impact_model.pyRegisters the GTE document-impact classifier in Unity Catalog.

Section 2Local pipeline

Scripts 01 through 19 have been run. The workbook stays in data/raw/rfi_log.xlsx. Raw dates stay as Excel serials. The clean table holds RFI-001 through RFI-107. The DuckDB file holds fact_rfi and the four dimensions. Unity Catalog holds digital_engineering.ml.rfi_discipline_classifier version 1 and digital_engineering.ml.rfi_issue_type_classifier version 1. Both aliases are champion. Each model expects a GTE embedding vector. The document-impact runs are logged in rfi_document_impact. The GTE classifier is the one to keep: Yes recall 0.68 with 5 false Yes calls.

08_preview_rfi_tables.py opens rfi.duckdb read-only and prints every table. Question, response, and notes are shortened in that printout.

Section 3Next script

20_register_document_impact_model.py registers the GTE document-impact classifier as digital_engineering.ml.rfi_document_impact_classifier and sets the alias champion. That model scored accuracy 0.84 and Yes recall 0.68 on 70 RFIs. Its input is the embedding vector from databricks-gte-large-en, built from the question. It does not take raw text, and it does not change the RFI tables.