ClearScore Credit Scoring System

A credit risk assessment demo built with FastAPI, Streamlit, and a calibrated LightGBM model. Scores loan applicants, produces a calibrated default probability, maps it to a risk tier, and explains every decision with SHAP feature attributions.

Trained on the Kaggle "Home Credit Default Risk" dataset (application, bureau, and previous-application tables), rebranded for this project.

Features

• Calibrated ML model: LightGBM with sigmoid (Platt) calibration
• Four-tier risk classification: Low / Medium / High / Very High, with an F1-validated decision threshold (see Model Validation)
• SHAP explainability: plain-English explanations citing the top decision factors
• FastAPI backend: /score endpoint, structured logging, env-configurable thresholds, artifact-fingerprint versioning
• Streamlit demo UI: interactive scoring form with a risk gauge and SHAP chart
• Test suite: test_credit_scoring.py — scenario/smoke tests plus a held-out model-quality regression gate (TestHeldOutModelQuality)

Project Structure

ClearScore/
├── app.py                      # FastAPI backend (/score, /health)
├── streamlit_demo.py           # Streamlit demo UI
├── explain.py                  # SHAP-based plain-English explanation generation
├── evaluate.py                 # Held-out evaluation of the saved model artifacts
├── metrics.json                # Output of evaluate.py — the numbers below
├── test_credit_scoring.py      # pytest suite (scenario tests + model-quality gate)
├── requirements.txt            # Pinned Python dependencies
├── .streamlit/config.toml      # UI theme
├── clearscore_lgbm.pkl         # Base LightGBM model (used for SHAP)
├── clearscore_calibrated.pkl   # Sigmoid-calibrated classifier (used for prediction)
├── clearscore_encoders.pkl     # Fitted LabelEncoders (one per categorical feature)
├── clearscore_X_cal.parquet    # Calibration/held-out feature set (152 columns)
├── eda-credit.ipynb            # Feature engineering, training, calibration notebook
└── README.md                   # This file

Installation

Prerequisites

• Python 3.10+
• pip or conda

Setup

cd ClearScore
python -m venv .venv
.venv\Scripts\Activate.ps1       # Windows
# source .venv/bin/activate      # macOS/Linux
pip install -r requirements.txt

Running the System

Streamlit demo

streamlit run streamlit_demo.py

Opens http://localhost:8501. Enter applicant details across three tabs (Personal, Financial, Additional Features) and click Score this applicant for a probability, risk tier, gauge, and SHAP explanation. Not every model feature is collected in the form — fields the UI doesn't expose use representative fixed defaults, disclosed in the app.

FastAPI backend

uvicorn app:app --port 8000

Starts the API at http://localhost:8000 (interactive docs at /docs).

Config is env-overridable: THRESHOLD_LOW, THRESHOLD_MEDIUM, THRESHOLD_HIGH, LGBM_MODEL_PATH, CALIBRATED_MODEL_PATH, ENCODERS_PATH, X_CAL_PATH, CORS_ALLOW_ORIGINS, LOG_LEVEL, HOST, PORT.

POST /score

Scores an applicant. Request body is the ~50-field ApplicantRequest schema defined in app.py (mirrors the Home Credit feature set) — see /docs for the full schema and an example payload, or the abridged example below.

Request (abridged — see /docs for all fields):

{
  "name_contract_type": "Cash loans",
  "code_gender": "M",
  "flag_own_car": "N",
  "flag_own_realty": "Y",
  "cnt_children": 0,
  "amt_income_total": 202500.0,
  "amt_credit": 406597.5,
  "amt_annuity": 24700.5,
  "amt_goods_price": 406597.5,
  "name_income_type": "Working",
  "name_education_type": "Secondary / secondary special",
  "name_family_status": "Single / not married",
  "name_housing_type": "House / apartment",
  "region_population_relative": 0.018850,
  "days_birth": -9461,
  "days_employed": -637.0,
  "days_registration": -3648.0,
  "days_id_publish": -2120.0,
  "flag_mobil": 1, "flag_emp_phone": 1, "flag_work_phone": 0,
  "flag_cont_mobile": 1, "flag_phone": 0, "flag_email": 0,
  "cnt_fam_members": 1.0,
  "region_rating_client": 2, "region_rating_client_w_city": 2,
  "weekday_appr_process_start": "WEDNESDAY", "hour_appr_process_start": 10,
  "reg_region_not_live_region": 0, "reg_region_not_work_region": 0,
  "live_region_not_work_region": 0, "reg_city_not_live_city": 0,
  "reg_city_not_work_city": 0, "live_city_not_work_city": 0
}

Response:

{
  "sk_id_curr": null,
  "default_probability": 0.0421,
  "risk_tier": "Low",
  "recommendation": "APPROVE",
  "explanation": "Based on the analysis, this applicant has a 4.21% estimated probability of default. ...",
  "model_version": "a1b2c3d4e5f6",
  "warnings": []
}

model_version is a short SHA-256 fingerprint of the loaded model artifacts, so a response can be traced back to the exact .pkl files that produced it. warnings lists any input fields that didn't match a category seen during training (the model falls back to a default encoding for those, which can reduce accuracy).

GET /health

Returns status, models_loaded, model_version, model_loaded_at, and uptime_seconds.

---

Risk Tiers

Default Probability	Risk Tier	Recommendation
< 0.08	Low	APPROVE
0.08 – 0.1636	Medium	REVIEW
0.1636 – 0.25	High	LIKELY REJECT
≥ 0.25	Very High	REJECT

The Medium tier exists so a human reviews borderline cases rather than the model deciding outright. Thresholds are env-overridable (see above) and defined in app.py.

Model Validation

Numbers below come from evaluate.py, run against a held-out set (clearscore_X_cal.parquet, n=61,503) that the LightGBM ensemble never trained on. Full output in metrics.json.

Metric	Raw model	Calibrated model
AUC (held-out)	0.7684	0.7684 (ranking unaffected by calibration)
Brier score	0.1581	0.0673
Expected calibration error	0.2637	0.0042

Calibration measurably works: it cuts Brier score by more than half and expected calibration error by ~98%. This directly re-validates the fix from commit 158f856 ("fixed the demo, model not calibrated"), which shipped uncalibrated probabilities by mistake — TestHeldOutModelQuality.test_calibration_improves_on_raw now guards against that regression happening silently again.

Threshold: the F1-optimal threshold derived from the held-out set is 0.1636 — matching the hardcoded THRESHOLD_MEDIUM to 4 decimal places. The existing threshold is therefore validated, not just asserted. At that threshold: precision 25.1%, recall 45.0%, F1 32.2% (held-out default rate is 8.1%, so this is a genuinely imbalanced classification problem — precision in the 20–30% range at this recall is expected, not a red flag).

Limitations

Discrimination is decent, not exceptional. AUC 0.7684 is a reasonable result for a single LightGBM model with fairly light feature engineering, but it's below what's achievable on this dataset — public leaderboard solutions for the Kaggle "Home Credit Default Risk" competition (heavy feature engineering across all tables, ensembling) reach roughly 0.79–0.80 AUC. There's real headroom here; see Future Work below.

Precision at the operating threshold is low. At 0.1636, precision is 25.1% — 3 of every 4 applicants flagged REVIEW/REJECT are false positives. This is inherent to an 8.1% base rate at this recall level, not a bug, but it's a real operational cost: a production deployment would route a lot of "actually fine" applicants to manual review.

Methodology caveat: the held-out rows were also used to fit the sigmoid calibration curve (a 2-parameter Platt-scaling fit), so the Brier/ECE numbers above are mildly optimistic rather than from a fully independent test set — AUC is clean out-of-sample since it doesn't depend on calibration. A future retrain should use a proper 3-way train/calibration/test split. Full methodology and caveats are in evaluate.py's docstring.

To reproduce: evaluate.py requires the raw Kaggle "Home Credit Default Risk" CSVs (not included in this repo — ~1.5GB, download separately) via --data-dir.

Testing

pytest test_credit_scoring.py -v

Two kinds of tests:

• Scenario/smoke tests (TestLowRiskApplicants, TestHighRiskApplicants, TestRiskTierClassification, TestPreprocessing, TestSHAPExplainability, TestEdgeCases, TestIntegration) — hand-crafted applicant profiles checked against the tier thresholds, verifying the pipeline runs correctly. These no longer swallow exceptions into pytest.skip() — a real failure now fails the run.
• TestHeldOutModelQuality — the actual model-quality gate. Loads metrics.json and asserts AUC/Brier/threshold haven't regressed past a tolerance band. Skipped automatically if metrics.json is missing.

Architecture

Applicant data
    -> FastAPI /score endpoint
    -> Pydantic input validation (incl. days_birth < 0, ext_source_* in [0,1])
    -> Feature preprocessing (categorical encoding, column alignment)
    -> LightGBM prediction
    -> Sigmoid probability calibration
    -> Risk tier assignment
    -> SHAP TreeExplainer -> plain-English explanation
    -> JSON response

Operational notes

• Latency: ~31ms mean / ~34ms p95 per /score call end-to-end (preprocessing + prediction + SHAP + explanation), measured locally on a warm model with a single request at a time — not a load-tested figure.
• Auth: none. This is a portfolio demo with no auth or rate-limiting on /score; don't point it at untrusted traffic without adding some.
• CORS: open by default (CORS_ALLOW_ORIGINS=*), overridable via env var.
• Logging: structured logs via Python logging, with a request ID and timing per request (see the X-Request-ID response header).
• Errors: /score never leaks internal exception text to the client — failures are logged server-side with a traceback and returned as a generic 500.

Troubleshooting

Streamlit app won't start / port in use

streamlit run streamlit_demo.py --server.port 8502

FastAPI port in use

uvicorn app:app --port 8001

Model loading error — ensure these files exist in the project root: clearscore_lgbm.pkl, clearscore_calibrated.pkl, clearscore_encoders.pkl, clearscore_X_cal.parquet (or set the corresponding *_PATH env vars).

Validation error when calling /score — days_birth must be negative; ext_source_1/2/3 must be between 0 and 1 if provided; see /docs for full field constraints.

License

Personal portfolio project — no license is granted for reuse. Contact the author if you'd like to use this code.