Credit Risk Scoring

Predicts the probability that a loan applicant defaults, using real Lending Club loan data. Built as a data-science-fresher-plus project: end-to-end pipeline discipline (stratified CV, leakage-safe preprocessing, held-out test set) without over-tooling a first version.

Business question: Should we approve this loan applicant, and how confident are we?

---

Dataset

9,578 Lending Club loans (2007–2010 era), 14 columns, target not.fully.paid (1 = defaulted). Source: loan_data.csv, a widely-used public extract of Lending Club's original release.

Loaded into SQLite (data/credit.db, table loans) rather than read directly from CSV — this is the "SQL work" layer: segmentation queries run against the database before any modeling starts (see notebooks/eda_queries.sql).

Baseline (stated before any model)

Predicting the majority class (no default) gets 84.0% accuracy.

This number by itself is why accuracy is never used to evaluate models in this project — a model that always says "will repay" would score 84% and be completely useless. AUC and precision/recall are used instead.

EDA findings (from SQL segmentation, real numbers)

Segment	Default rate
small_business purpose	27.8% (highest)
educational purpose	20.1%
credit_card / major_purchase purpose	~11.5% (lowest)
FICO < 680	23.6%
FICO 720+	10.3%
Failed underwriting policy (credit_policy=0)	27.8%
Met underwriting policy (credit_policy=1)	13.2%

These aren't incidental — purpose, fico, and credit_policy are also the strongest features in the final model (see below), so the EDA and the model tell the same story from two different angles.

Methodology

1. Stratified 5-fold cross-validation, not a single train/test split — every reported number is a mean ± std across folds, computed on the training set only.
2. No leakage: the ColumnTransformer (imputer + scaler + encoder) is wrapped inside an sklearn.Pipeline and refit fresh on each CV training fold. It never sees validation-fold or test-set data during fitting.
3. Held-out test set (20%, stratified) is split off before any CV and touched exactly once, in evaluate.py, after the winning model is already chosen.
4. Class imbalance (84/16 split) handled with class_weight='balanced' rather than oversampling — simpler, and avoids synthetic-data artifacts for a v1.

Model comparison (5-fold stratified CV, training set)

Model	Mean AUC	Std
Logistic Regression (class-balanced)	0.677	0.006
Random Forest (depth=8, class-balanced)	0.658	0.005

Logistic Regression wins and is the more interpretable choice for a lending decision anyway — coefficients translate directly into "why" for a rejected applicant, which matters for adverse-action explanations in real lending.

Held-out test results

• AUC: 0.684

• Classification report (threshold = 0.5):

  	precision	recall	f1
  repaid	0.89	0.65	0.75
  defaulted	0.24	0.57	0.34

At the default 0.5 threshold the model only catches 57% of actual defaults — too low for a risk-averse lender. See the threshold sweep below.

Threshold tuning (cost-sensitive decision, not just a metric)

A missed default (false negative) costs more than an unnecessarily cautious rejection (false positive), so the deployed threshold is not the default 0.5:

Threshold	Catches this % of defaults	Flags this % of good applicants
0.50	57.0%	34.6%
0.35 (used in the app)	92.2%	71.1%
0.25	98.7%	90.2%
0.15	100.0%	99.1%

0.35 is the deployed threshold — it catches the large majority of defaults while not flagging almost every applicant as risky (which is what happens below 0.25). This tradeoff is a business decision, stated explicitly here rather than left as an invisible default.

Feature importance (top drivers, logistic regression coefficients)

Feature	Coefficient	Direction
purpose = small_business	+0.647	↑ risk
fico	−0.315	↓ risk (higher FICO = safer)
credit_policy (met underwriting)	−0.298	↓ risk
log_annual_inc	−0.263	↓ risk (higher income = safer)
purpose = credit_card	−0.409	↓ risk
installment	+0.233	↑ risk
inq_last_6mths	+0.228	↑ risk (more inquiries = riskier)

This matches the EDA exactly: small_business loans and low FICO scores are flagged by both the raw SQL segmentation and the trained model independently — that agreement is a good sign the model learned something real rather than noise.

Deployment

Streamlit app (app/streamlit_app.py), four tabs:

• Score an Applicant — plain-language inputs (annual income in $, years of credit history, loan amount + term) instead of raw model features; the app derives log_annual_inc, days_with_cr_line, and installment internally. Returns a risk score, a risk band (Low/Moderate/High/Very high), an approve/review recommendation at the 0.35 threshold, an expected-loss dollar estimate, and a per-applicant explanation — since the deployed model is linear, each feature's exact contribution to this applicant's score is coefficient x transformed_value, shown per applicant rather than only as one global chart. This is the "SHAP per-applicant explanation" item from "Scoped out for v1" below, partially closed: exact for this linear model, though it won't carry over automatically if the model is ever swapped for a non-linear one.
• Batch Scoring — upload a CSV of multiple applicants (template provided) and get risk scores, recommendations, and expected loss for all of them, downloadable as CSV.
• Ask a Question — a deterministic, rule-based Q&A box. It answers only from numbers the app actually computed (the last scored applicant's result, test_metrics.json / cv_results.json, or the README's own dataset facts) — never from a general-purpose language model — so every answer is traceable to a real number and it says so plainly when it doesn't have a grounded answer, rather than guessing.
• Model Details — performance (AUC, threshold, CV score), the ROC and feature-importance figures, dataset summary, and known limitations stated explicitly, including a fair-lending disclosure (no disparate-impact testing has been done on this model, and it should support human underwriting judgment, not replace it).

The app only loads the pickled pipeline — it never retrains, so there's no train/serve skew between what was evaluated and what's live.

pip install -r requirements.txt
python src/load_data.py      # CSV -> SQLite
python src/train.py          # CV comparison, fit + save final pipeline
python src/evaluate.py       # held-out test metrics, figures
streamlit run app/streamlit_app.py

Run tests: pytest tests/ -v (6 tests: pipeline load, valid probability output, unseen-category handling, missing-value handling, a risk-direction sanity check, and preprocessor construction).

Expected-loss business layer

The app converts a bare probability into a dollar figure:

expected_loss = P(default) × (1 − recovery_rate) × loan_amount

with a 40% assumed recovery rate on defaulted loans — a simplification (real recovery varies by collateral and collections process) but it turns "23% risk" into a number a loan officer can actually act on.

Scoped out for v1 (and why)

These are real improvements, deliberately deferred rather than skipped by accident:

• Optuna / GridSearchCV hyperparameter tuning — would likely add 1–3 points of AUC. Skipped for v1 to keep the model simple enough to fully explain; worth adding once the base pipeline is solid.
• MLflow experiment tracking — useful once there are more than 2 models and a few configs to compare; overkill for the current LR-vs-RF comparison, which fits in one table.
• SHAP per-applicant explanations — the current feature importance is global (same for every applicant). Per-applicant SHAP values would give a true adverse-action explanation ("Applicant X was flagged because of their DTI specifically"), which is the real next step but adds a nontrivial dependency and runtime cost.
• XGBoost / LightGBM — likely to beat both LR and RF here; not included in v1 to keep the model comparison table small and the reasoning behind each result auditable by hand.

Repo structure

credit-risk-scoring/
├── data/
│   ├── raw/loan_data.csv       # original CSV
│   ├── credit.db               # SQLite (loans table)
│   └── held_out_test.csv       # test split, written once by train.py
├── notebooks/
│   └── eda_queries.sql         # SQL segmentation queries (this README's numbers)
├── src/
│   ├── load_data.py            # CSV -> SQLite
│   ├── preprocess.py           # ColumnTransformer (leakage-safe)
│   ├── train.py                # baseline, CV comparison, fit + save
│   └── evaluate.py             # held-out metrics, figures
├── app/
│   └── streamlit_app.py        # loan officer UI
├── models/
│   ├── pipeline.pkl            # fitted Pipeline (preprocessor + classifier)
│   ├── cv_results.json
│   ├── test_metrics.json
│   └── figures/                # ROC curve, feature importance plot
├── tests/
│   └── test_pipeline.py        # 6 tests
├── requirements.txt
└── README.md

Skill demonstrated

End-to-end classification with SQL-driven segmentation, leakage-safe cross-validation, business-layer cost-sensitive threshold selection, and production-shaped deployment (pickled pipeline, no retrain-on-request) — built and reasoned about independently, with explicit documentation of what was scoped out and why.