Skip to content

About

Graph-Bayesian neural architecture for credit-card default prediction with uncertainty. Combines a GAT over a k-NN customer graph, a pgmpy Bayesian network for latent risk factors, and a Bayesian neural network with Monte Carlo sampling.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

GraphBayesRisk: Graph-Bayesian Neural Architecture for Uncertainty-Aware Credit Risk

Paper replication, extended (V2). This repository replicates "Uncertainty-Aware Robust Financial Risk Discrimination via Bayesian Network and Bayesian Neural Modeling" on the UCI Default of Credit Card Clients dataset, then extends it with a Graph Attention Network (GAT) branch. The final model fuses graph embeddings H with Bayesian-network latent factors Z and classifies with a Bayesian Neural Network (BNN) that reports a risk score and an uncertainty estimate for every customer.


Table of Contents

  1. What this project does
  2. Background and motivation
  3. Architecture at a glance
  4. Dataset
  5. Repository structure
  6. Stage 0: Data preparation
  7. Stage 1: Graph construction and GAT
  8. Stage 2: Bayesian network
  9. Stage 3: Fusion and Bayesian neural network
  10. Stage 4: Evaluation
  11. Pipeline orchestration (main.py)
  12. Every generated file
  13. Results
  14. Hyperparameter reference
  15. Getting started
  16. Dependencies
  17. Project history
  18. Documentation map
  19. Known issues and limitations
  20. Suggested next steps
  21. Dashboard
  22. References
  23. License

1. What this project does

Given a credit-card client's demographics, credit limit, six months of repayment status, bill amounts and payment amounts, the pipeline:

  1. predicts whether the client will default next month, and
  2. says how uncertain it is about that prediction.

It gets there with three models that each contribute a different kind of information:

Model Library Contribution Output
Graph Attention Network PyTorch Geometric Learns from similar customers (neighbours in a k-NN graph) H, 32 dims per customer
Bayesian Network pgmpy Models how features depend on each other and gives interpretable posteriors Z, 27 dims per customer
Bayesian Neural Network PyTorch (custom BayesLinear) Classifies the fused vector [H ‖ Z] with a distribution over every weight Risk score and variance
flowchart LR
    A["Raw customer record<br/>23 features"] --> B["Cleaning, splits,<br/>two data views"]
    B --> C["k-NN graph<br/>+ GAT"]
    B --> D["Binned data<br/>+ Bayesian Network"]
    C -->|"H (32)"| E["Fusion<br/>H ‖ Z (59)"]
    D -->|"Z (27)"| E
    E --> F["Bayesian Neural Network"]
    F --> G["50 Monte Carlo passes"]
    G --> H["Risk score (mean)<br/>Uncertainty (variance)"]
Loading

2. Background and motivation

A conventional credit model returns one number, such as "70% chance of default". It does not say whether the model has seen similar customers before or is extrapolating. Financial decisions need that distinction: ambiguous cases should go to a human reviewer.

The source paper combines a Bayesian Network (structured, interpretable) with a Bayesian Neural Network (flexible, uncertainty-aware). V1 of this repo implemented that paper directly, fusing 30 raw features with 27 latent factors. V2 replaces the raw features with graph embeddings, because the credit data has no natural relational structure and a k-NN similarity graph lets the model learn from each customer's neighbourhood.

flowchart TB
    subgraph V1["V1: paper replication"]
        direction LR
        V1a["30 raw features"] --> V1c["Fusion 57-dim"]
        V1b["Bayesian Network Z (27)"] --> V1c
        V1c --> V1d["BNN"]
    end
    subgraph V2["V2: this repository"]
        direction LR
        V2a["k-NN graph + GAT, H (32)"] --> V2c["Fusion 59-dim"]
        V2b["Bayesian Network Z (27)"] --> V2c
        V2c --> V2d["BNN"]
    end
    V1 -->|"GNN branch replaces raw features"| V2
Loading

Core mathematics

Bayesian network (paper Eq. 1 and 2)

p(z | G) = ∏ p(zᵢ | Pa(zᵢ))
p(y | x) = Σ_z p(y | z, x) · p(z | G)

G is the learned DAG, z the latent posteriors and Pa(zᵢ) the parents of a node.

BNN loss (paper Eq. 5 and 6)

L_task = -log p(y | x)                    (binary cross-entropy, positives up-weighted)
L      = L_task + λ · KL( q(w) ‖ p(w) ) / N

q(w) is the learned Gaussian posterior over weights, p(w) = N(0, 0.1²) the prior and N the number of training samples. Dividing the KL term by N is the standard ELBO scaling, so λ = 1 is the true Bayesian objective. In this repo the positive class is also weighted by n_negatives / n_positives (about 3.5) to counter class imbalance.

Monte Carlo inference (paper Eq. 7)

ŷ   = (1/T) Σₜ ŷₜ                         risk score
Var = (1/T) Σₜ (ŷₜ − ŷ)²                  uncertainty

with T = 50 stochastic forward passes.


3. Architecture at a glance

flowchart TB
    RAW[("data/raw<br/>UCI csv")]

    subgraph S0["Stage 0 · data_prep.py"]
        CLEAN["Clean<br/>(drop NaN, duplicates,<br/>clip negative PAY_AMT)"]
        SPLIT["Stratified split<br/>70 / 15 / 15, seed 42"]
        DISC["Discrete view<br/>4 quantile bins"]
        CONT["Continuous view<br/>log, one-hot, scale"]
        KNN["k-NN graph (k=5)<br/>edge_index + masks"]
        CLEAN --> SPLIT --> DISC
        SPLIT --> CONT --> KNN
    end

    subgraph S1["Stage 1 · gnn_model.py"]
        GAT["GAT 30 → 64 → 32"]
        H[/"H · graph_embeddings_*.csv"/]
        GAT --> H
    end

    subgraph S2["Stage 2 · bayes_network.py"]
        HC["Hill-climb + BIC"]
        CPD["BDeu CPDs"]
        VE["Variable elimination<br/>5 factor groups"]
        Z[/"Z · latent_z_*.csv"/]
        HC --> CPD --> VE --> Z
    end

    subgraph S3["Stage 3 · bnn_model.py"]
        FUSE["Fuse H ‖ Z → 59"]
        BNN["BayesLinear<br/>59 → 64 → 32 → 1"]
        FUSE --> BNN
    end

    subgraph S4["Stage 4 · evaluate.py"]
        MC["MC inference T=50"]
        PLOTS["Loss, t-SNE,<br/>uncertainty plots"]
        EXP["λ sweep +<br/>ablation"]
        MC --> PLOTS
        MC --> EXP
    end

    RAW --> CLEAN
    KNN --> GAT
    DISC --> HC
    H --> FUSE
    Z --> FUSE
    BNN --> MC
Loading

4. Dataset

Property Value
Name Default of Credit Card Clients
Source UCI Machine Learning Repository
Original paper Yeh & Lien (2009), Expert Systems with Applications
Raw rows 30,000
Rows after cleaning 29,999 (only the one row with a non-numeric value is dropped)
Features 23 raw (25 columns in the file, including ID and Y)
Target Y, default payment next month (1 = default)
Class balance about 77.9% non-default and 22.1% default in the cleaned data
Region and period Taiwan, April to September 2005

The raw file has two header rows. Row 1 holds generic labels (X1, X2, …) and row 2 holds the real names. The loader reads it with header=1.

Feature descriptions

Feature Description Type
LIMIT_BAL Credit limit in NT dollars (individual plus family credit) Continuous
SEX 1 = male, 2 = female Categorical
EDUCATION 1 = graduate school, 2 = university, 3 = high school, 4 = others (values 0, 5, 6 also occur) Categorical
MARRIAGE 1 = married, 2 = single, 3 = others (value 0 also occurs) Categorical
AGE Age in years Continuous
PAY_0, PAY_2 … PAY_6 Repayment status, September (PAY_0) back to April (PAY_6). -1 = paid duly, 1 = one month late, … 9 = nine or more months late. The dataset has no PAY_1. Ordinal
BILL_AMT1 … BILL_AMT6 Bill statement amounts, September back to April Continuous
PAY_AMT1 … PAY_AMT6 Previous payment amounts, September back to April Continuous
Y Default next month Binary target
flowchart LR
    subgraph demo["demo (4)"]
        d1["SEX"] --- d2["EDUCATION"] --- d3["MARRIAGE"] --- d4["AGE"]
    end
    subgraph credit["credit (1)"]
        c1["LIMIT_BAL"]
    end
    subgraph pay["pay_status (6)"]
        p1["PAY_0"] --- p2["PAY_2"] --- p3["PAY_3"] --- p4["PAY_4"] --- p5["PAY_5"] --- p6["PAY_6"]
    end
    subgraph bill["bill (6)"]
        b1["BILL_AMT1 … 6"]
    end
    subgraph pa["pay_amt (6)"]
        a1["PAY_AMT1 … 6"]
    end
    demo --> T(("Y"))
    credit --> T
    pay --> T
    bill --> T
    pa --> T
Loading

5. Repository structure

GraphBayesRisk/
├── README.md                    this file
├── LICENSE                      Apache 2.0
├── pyproject.toml               uv project metadata
├── requirements.txt             pip dependency list (includes torch_geometric)
├── uv.lock                      locked dependency versions
├── .python-version              3.11
├── .gitignore
├── replication_plan.md          original V1 plan
│
├── src/
│   ├── main.py                  runs every stage in order
│   ├── data_prep.py             cleaning, splits, two data views, k-NN graph
│   ├── gnn_model.py             GAT and GCN models, training, embedding export
│   ├── bayes_network.py         structure learning, CPDs, latent factor extraction
│   ├── bnn_model.py             BayesLinear, BNN, FusedDataset, training loop
│   └── evaluate.py              MC inference, metrics, plots, λ sweep, ablation
│
├── data/
│   ├── raw/                     default_of_credit_card_clients.csv
│   └── processed/               splits, graph arrays, H and Z files
│
├── models/saved_models/         bayesian_network.pkl, gat_risk_model.pt
├── outputs/
│   ├── evaluation_metrics.csv   final test metrics
│   ├── ablation_results.csv     H only vs Z only vs fused
│   ├── bnn_training_curves.csv  per-epoch losses
│   ├── models/                  bnn_model.pth, bnn_model_v2.pth
│   └── figures/                 four PNG plots
│
├── docs/                        Abstract, METHODOLOGY, slide content, flowchart image
├── phase/                       plan_v2.md and Phase1 to Phase4 notes
├── papers/                      source paper PDFs
└── notebooks/                   three empty placeholder notebooks

Module dependencies

flowchart LR
    main["main.py"] -. "subprocess" .-> dp["data_prep.py"]
    main -. "subprocess" .-> gnn["gnn_model.py"]
    main -. "subprocess" .-> bn["bayes_network.py"]
    main -. "subprocess" .-> bnn["bnn_model.py"]
    main -. "subprocess" .-> ev["evaluate.py"]
    ev -->|"import BNN, FusedDataset, train_model"| bnn
Loading

main.py launches each script as a separate process. The only Python import between modules is evaluate.py importing from bnn_model.py; everything else communicates through files.


6. Stage 0: Data preparation

File: src/data_prep.py

flowchart TB
    A["load_data<br/>header=1, keep ID,<br/>coerce to numeric"] --> B["clean_data<br/>dropna, drop_duplicates,<br/>drop ID, clip PAY_AMT ≥ 0,<br/>cast ints"]
    B --> C["train_test_split<br/>70 / 30 stratified"]
    C --> D["split the 30%<br/>50 / 50 stratified"]
    D --> E{"same row indices<br/>for both views"}
    E --> F["prepare_bn_data<br/>4 quantile bins<br/>(edges from train only)"]
    E --> G["prepare_bnn_data<br/>signed log1p, one-hot,<br/>StandardScaler"]
    F --> H[("train / val / test<br/>_discrete.csv")]
    G --> I[("train / val / test<br/>_continuous.csv")]
    I --> J["build_knn_graph<br/>k=5"]
    J --> K[("edge_index.npy<br/>node_features.npy<br/>node_labels.npy<br/>3 masks")]
Loading

Functions

Function What it does
load_data Reads the CSV with header=1, renames default payment next month to Y, coerces everything to numbers (bad cells become NaN). ID is kept until cleaning
clean_data Drops NaN rows and exact duplicate records (with ID still present, so different customers with identical features are not merged), then drops ID, clips PAY_AMT1–PAY_AMT6 at zero, casts Y, SEX, EDUCATION, MARRIAGE, AGE and the PAY columns to integers
prepare_bn_data Cuts AGE, LIMIT_BAL, BILL_AMT1–6 and PAY_AMT1–6 into 4 quantile bins (labels 0 to 3). Bin edges are computed on the training rows only and applied to all splits with pd.cut
prepare_bnn_data Applies `sign(x) · log1p(
build_knn_graph Concatenates train, val and test features and builds the graph described in Stage 1
main Runs the steps above and writes all files to data/processed/

Key decisions

  • One split for both views. The stratified split (seed 42) is computed once on row indices, so row i of a discrete file and row i of the matching continuous file are the same customer.
  • No scaler leakage. StandardScaler is fit on the training split only, then applied to validation and test. One-hot columns and Y are not scaled.
  • Train-only statistics. Both the scaler and the BN bin edges come from the training split, so no validation or test statistics reach the models.
  • Split sizes: train 20,999, validation 4,500, test 4,500 (29,999 total).

7. Stage 1: Graph construction and GAT

Files: src/data_prep.py (graph) and src/gnn_model.py (model)

Graph construction

Credit data has no relationships between customers, so one is created from feature similarity.

flowchart LR
    A["Scaled continuous features<br/>train + val + test<br/>29,999 × 30"] --> B["NearestNeighbors<br/>k+1 = 6, Euclidean"]
    B --> C["Drop self match<br/>(column 0)"]
    C --> D["Directed edges<br/>i → each of 5 neighbours"]
    D --> E["Add reverse edges<br/>(undirected)"]
    E --> F["np.unique<br/>remove duplicates"]
    F --> G["edge_index<br/>shape 2 × 223,740"]
Loading
Property Value
Nodes 29,999 (one per customer)
Node features 30 scaled continuous features (no label)
Edge entries in edge_index 223,740 (both directions stored, so 111,870 undirected pairs)
Average degree about 7.46
Setting Transductive: all nodes (train, val, test) are in one graph, and boolean masks pick which nodes count for loss and evaluation

The graph uses only features, never labels, so the test labels cannot reach the GAT through the edges.

GAT model

flowchart LR
    X["x<br/>N × 30"] --> L1["GATConv<br/>30 → 64<br/>heads=1, dropout 0.5"]
    L1 --> A1["ELU"]
    A1 --> DR["Dropout 0.5"]
    DR --> L2["GATConv<br/>64 → 32<br/>heads=1, concat=False"]
    L2 --> A2["ELU"]
    A2 --> HH(["H<br/>N × 32"])
    A2 --> CL["Linear 32 → 1"]
    CL --> SG["Sigmoid"]
    SG --> O(["proxy risk"])
Loading

The proxy head exists only to give the GAT a supervised signal. The product of this stage is H, the 32-dim layer output.

gnn_model.py also defines GCNRiskModel (two GCNConv layers, ReLU) as a baseline. The pipeline does not use it.

Training

sequenceDiagram
    participant M as GATRiskModel
    participant D as Graph data
    participant O as Adam (lr 0.005, wd 1e-4)
    loop up to 100 epochs
        M->>D: forward on the whole graph (train mode)
        M->>O: class-weighted BCE on train_mask nodes only, backward, step
        M->>D: forward again (eval mode, no grad)
        M-->>M: val loss and val AUC on val_mask nodes
        alt val loss improved
            M-->>M: save "best" state, reset patience
        else no improvement
            M-->>M: patience += 1 (stop at 15)
        end
    end
    M->>M: load saved best state (cloned tensors)
Loading
Function Purpose
load_graph_data Loads the NPY files and builds a PyG Data object with masks
compute_metrics Accuracy, precision, recall, F1, ROC-AUC at a 0.5 threshold
train_gnn Masked-loss training with early stopping on validation loss. The loss is BCE with positives weighted by n_neg / n_pos of the training nodes (about 3.5). The best state is deep-copied, so the restored model really is the best epoch
extract_and_save_embeddings Runs the trained GAT once, slices H by mask, writes three CSVs and prints proxy-head test metrics
main Seeds 42, picks CUDA if available, trains, saves models/saved_models/gat_risk_model.pt and the embeddings

8. Stage 2: Bayesian network

File: src/bayes_network.py

flowchart TB
    A[("train_discrete.csv")] --> B["HillClimbSearch<br/>score: BIC, max_iter 100"]
    K["ExpertKnowledge<br/>required: PAY chain<br/>forbidden: Y → anything"] --> B
    B --> C["DAG (45 edges)"]
    C --> D["DiscreteBayesianNetwork"]
    A --> E["BayesianEstimator<br/>BDeu, ESS 10"]
    D --> E
    E --> F["CPDs + check_model"]
    F --> G[("bayesian_network.pkl")]
    F --> H["extract_latent_features<br/>per row, 5 groups"]
    H --> I[("latent_z_train / val / test.csv")]
Loading

Structure learning constraints

Constraint Edges Reason
Required PAY_6 → PAY_5 → PAY_4 → PAY_3 → PAY_2 → PAY_0 Repayment status follows time
Forbidden Y → every other column The label is an outcome, so it can only be a child

Learned structure

The search found 45 edges. Highlights from the saved model:

flowchart LR
    PAY6["PAY_6"] --> PAY5["PAY_5"] --> PAY4["PAY_4"] --> PAY3["PAY_3"] --> PAY2["PAY_2"] --> PAY0["PAY_0"]
    PAY0 --> Y(("Y"))
    LIM["LIMIT_BAL"] --> Y
    PAY2 --> B1["BILL_AMT1"]
    PAY5 --> B4["BILL_AMT4"]
    B5["BILL_AMT5"] --> PAY6
    PAY2 --> A1["PAY_AMT1"]
    PAY3 --> A2["PAY_AMT2"]
    PAY4 --> A3["PAY_AMT3"]
    PAY5 --> A4["PAY_AMT4"]
    A5["PAY_AMT5"] --> PAY6
    AGE["AGE"] --> EDU["EDUCATION"]
    AGE --> MAR["MARRIAGE"]
    AGE --> SEX["SEX"]
    LIM --> AGE
    LIM --> EDU
    LIM --> SEX
Loading

Only PAY_0 and LIMIT_BAL are parents of Y. PAY_2 to PAY_5 each lead to the matching PAY_AMT, and the bill amounts and payment amounts form chains among themselves. The structure depends on the bin edges, so it differs slightly from earlier runs (the BN was learned on train-only bins).

Latent factor extraction

For each customer and each of the five factor groups, the code asks the network for the posterior of the group's target variable, given the evidence in its Markov blanket.

sequenceDiagram
    participant R as Customer row
    participant G as Factor group
    participant VE as VariableElimination
    R->>G: pick target (e.g. PAY_0)
    G->>G: evidence = Markov blanket of target, minus the group's own variables, minus Y
    G->>VE: query(target, evidence)
    alt query succeeds
        VE-->>G: posterior over target states
    else query fails
        G->>VE: query(target) with no evidence (prior)
        alt prior succeeds
            VE-->>G: prior over target states
        else prior fails
            G-->>G: uniform distribution
        end
    end
    G-->>R: columns z_group_0 … z_group_k
Loading
Factor group Target variable Variables in the group States Z columns
demo AGE SEX, EDUCATION, MARRIAGE, AGE 4 4
credit LIMIT_BAL LIMIT_BAL 4 4
pay_status PAY_0 PAY_0, PAY_2 … PAY_6 11 11
bill BILL_AMT1 BILL_AMT1–6 4 4
pay_amt PAY_AMT1 PAY_AMT1–6 4 4
Total 27

Excluding Y and the group's own variables from the evidence means a customer's own label and own target value never leak into Z.


9. Stage 3: Fusion and Bayesian neural network

File: src/bnn_model.py

Fusion

flowchart LR
    H["graph_embeddings_*.csv<br/>GNN_0 … GNN_31"] --> C["np.concatenate<br/>axis=1"]
    Z["latent_z_*.csv<br/>z_* (27 columns)"] --> C
    C --> X(["X · 59 dims"])
    Y["*_continuous.csv<br/>column Y"] --> L(["y labels"])
    X --> DS["FusedDataset"]
    L --> DS
Loading

FusedDataset has three modes used by the ablation:

Mode Input Dims
both [H ‖ Z] 59
graph_only H 32
latent_only Z 27

Only the Y column of the continuous CSV is read. The 30 raw continuous features are not part of the BNN input in V2.

The Bayesian layer

flowchart TB
    subgraph params["Learned parameters per weight and bias"]
        mu["μ (mean)"]
        rho["ρ (init -3.0)"]
    end
    rho --> sig["σ = log(1 + exp(ρ))"]
    eps["ε ~ N(0, 1)"] --> w
    mu --> w["w = μ + σ · ε"]
    sig --> w
    w --> lin["F.linear(x, w, b)"]
    mu --> kl
    sig --> kl["KL( N(μ, σ²) ‖ N(0, 0.1²) )<br/>closed form, summed"]
    kl --> store["self.kl_divergence"]
Loading

BayesLinear keeps weight_mu, weight_rho, bias_mu and bias_rho. Means start uniform in ±1/√in_features. The KL term is recomputed on every forward pass and read back with BNN.get_kl() (the sum over the three layers).

Network

flowchart LR
    I["Input<br/>59"] --> F1["BayesLinear 59 → 64"] --> R1["ReLU"] --> F2["BayesLinear 64 → 32"] --> R2["ReLU"] --> F3["BayesLinear 32 → 1"] --> S["Sigmoid"] --> P(["ŷ ∈ (0, 1)"])
Loading

Trainable parameters: (59·64 + 64) + (64·32 + 32) + (32·1 + 1) = 5,953 weights and biases, each with a mean and a ρ, so 11,906 values. This matches the roughly 50 KB checkpoint.

Training

sequenceDiagram
    participant L as DataLoader (batch 128, shuffle)
    participant M as BNN
    participant O as Adam (lr 0.001)
    loop 100 epochs
        loop each batch
            L->>M: X_batch
            M-->>L: ŷ (sampled weights)
            Note over M,O: loss = weighted BCE(ŷ, y) + λ · KL / N
            O->>M: backward and step
        end
        M->>M: validation pass (batch 256), record val loss
        M->>M: keep a copy of the weights if val loss is the best so far
    end
    M->>M: restore the best-validation weights
Loading

Per epoch the code records total loss, task (weighted BCE) loss, KL and validation loss, and writes them to outputs/bnn_training_curves.csv. At epoch 100 the total train loss is about 0.897, weighted task BCE about 0.879, KL about 369 and validation loss about 0.916. These losses are larger than in earlier versions because positives are up-weighted, so they are not comparable with unweighted BCE values.

The BNN trains for the full 100 epochs and then restores the weights from the epoch with the lowest validation loss (about 0.910 in the latest run).

Loss details. The KL term is divided by the number of training samples N, so λ = 1 is the standard ELBO. Positives are weighted by n_neg / n_pos (about 3.5, computed from the training labels). The weighting moves predicted probabilities upward, which is why the best decision threshold is no longer 0.5.

main() saves the final weights twice: bnn_model_v2.pth and a compatibility copy bnn_model.pth.


10. Stage 4: Evaluation

File: src/evaluate.py

flowchart TB
    A["Load test [H ‖ Z]<br/>+ bnn_model_v2.pth"] --> B["run_mc_inference<br/>T = 50"]
    B --> C["mean → risk score<br/>variance → uncertainty"]
    V["Validation MC scores<br/>→ threshold that maximizes F1"] --> D["Test metrics at the tuned threshold<br/>(and at 0.5 for comparison)"]
    C --> D
    D --> E[("evaluation_metrics.csv")]
    C --> F["plot_loss_curves"]
    C --> G["plot_tsne"]
    C --> H["plot_uncertainty_distribution"]
    F --> F1[("loss_vs_epoch.png")]
    G --> G1[("tsne_visualization.png")]
    H --> H1[("uncertainty_distribution.png")]
    D --> I["run_lambda_sweep"]
    I --> I1[("prior_regularization_sensitivity.png")]
    I --> J["run_ablation_studies"]
    J --> J1[("ablation_results.csv")]
Loading

Monte Carlo inference

sequenceDiagram
    participant E as evaluate.py
    participant M as BNN (eval mode)
    E->>M: batch X
    loop T = 50 passes
        M-->>E: ŷₜ (fresh weight sample each pass)
    end
    E->>E: risk = mean over passes
    E->>E: uncertainty = variance over passes
Loading

Note that model.eval() does not turn off weight sampling in BayesLinear, so every pass draws new weights, which is exactly what MC inference needs.

Experiments

Experiment Setup Output
Threshold tuning MC scores (T=50) on the validation split. The threshold in 0.05 to 0.95 (step 0.01) with the best F1 is chosen printed and stored in evaluation_metrics.csv
Final test evaluation bnn_model_v2.pth, T=50, tuned threshold; metrics at 0.5 are saved alongside evaluation_metrics.csv
Loss curves Reads bnn_training_curves.csv: train ELBO, validation loss, task BCE loss_vs_epoch.png
t-SNE 2,000 random test points (seed 42) in the 59-dim space, perplexity 30. Points above the 75th percentile of uncertainty are drawn as red X marks tsne_visualization.png
Uncertainty distribution KDE of variance for correct vs incorrect predictions (at the tuned threshold) uncertainty_distribution.png
λ sweep λ in {1e-2, 1e-1, 1, 10, 100, 1000, 10000}. Fresh seeded BNN per value, 25 epochs, lr 0.002, T=30, class-weighted loss, scored on validation AUROC prior_regularization_sensitivity.png (values are printed, not saved to CSV)
Ablation H only, Z only and [H ‖ Z]. Fresh seeded BNN each, 60 epochs, lr 0.001, λ=1, class-weighted loss. Threshold tuned on validation per configuration, T=50, scored on the test set ablation_results.csv

11. Pipeline orchestration (main.py)

src/main.py runs five scripts in order using the project's .venv interpreter if it exists (otherwise the current Python). It raises RuntimeError if any script exits non-zero.

stateDiagram-v2
    [*] --> DataPrep
    DataPrep --> GNN: exit 0
    GNN --> BayesNet: exit 0
    BayesNet --> BNN: exit 0
    BNN --> Evaluate: exit 0
    Evaluate --> [*]: pipeline complete
    DataPrep --> Failed: non-zero exit
    GNN --> Failed: non-zero exit
    BayesNet --> Failed: non-zero exit
    BNN --> Failed: non-zero exit
    Evaluate --> Failed: non-zero exit
    Failed --> [*]: RuntimeError
Loading

Scripts use relative paths such as data/processed, so run everything from the repository root.

Data flow between stages

flowchart LR
    DP["data_prep.py"] -->|"*_discrete.csv"| BN["bayes_network.py"]
    DP -->|"edge_index, node_features,<br/>node_labels, masks"| GN["gnn_model.py"]
    DP -->|"*_continuous.csv (Y column)"| BB["bnn_model.py"]
    GN -->|"graph_embeddings_*.csv"| BB
    BN -->|"latent_z_*.csv"| BB
    BB -->|"bnn_model_v2.pth,<br/>training curves"| EV["evaluate.py"]
    DP -->|"*_continuous.csv"| EV
    GN -->|"graph_embeddings_*.csv"| EV
    BN -->|"latent_z_*.csv"| EV
Loading

12. Every generated file

flowchart TB
    subgraph raw["data/raw"]
        R1["default_of_credit_card_clients.csv"]
    end
    subgraph proc["data/processed"]
        P1["train / val / test _discrete.csv"]
        P2["train / val / test _continuous.csv"]
        P3["edge_index.npy · node_features.npy · node_labels.npy"]
        P4["train_mask / val_mask / test_mask .npy"]
        P5["graph_embeddings_train / val / test .csv"]
        P6["latent_z_train / val / test .csv"]
    end
    subgraph mod["models/saved_models"]
        M1["bayesian_network.pkl"]
        M2["gat_risk_model.pt"]
    end
    subgraph out["outputs"]
        O1["evaluation_metrics.csv"]
        O2["ablation_results.csv"]
        O3["bnn_training_curves.csv"]
        O4["models/bnn_model.pth · bnn_model_v2.pth"]
        O5["figures/*.png (4 plots)"]
    end
    R1 --> P1 & P2 & P3 & P4
    P3 & P4 --> P5
    P1 --> P6
    P5 & P6 --> O4
    O4 --> O1 & O2 & O3 & O5
    P3 --> M2
    P1 --> M1
Loading

data/processed/

File Rows Columns Description
train_discrete.csv 20,999 24 Binned features plus Y. Used to learn the BN
val_discrete.csv 4,500 24 Same format
test_discrete.csv 4,500 24 Same format
train_continuous.csv 20,999 31 Log-transformed and standardized numerics, one-hot categoricals, Y
val_continuous.csv 4,500 31 Scaled with the training scaler
test_continuous.csv 4,500 31 Scaled with the training scaler
edge_index.npy n/a n/a int64, shape 2 × 223,740
node_features.npy 29,999 30 float32, scaled features in train, val, test order
node_labels.npy 29,999 n/a int64 labels
train_mask.npy, val_mask.npy, test_mask.npy 29,999 n/a Boolean node masks
graph_embeddings_*.csv split size 32 GNN_0 … GNN_31
latent_z_*.csv split size 27 z_{group}_{state_index}

(Row counts exclude the header line.)

Continuous columns (31): LIMIT_BAL, AGE, PAY_0, PAY_2–PAY_6, BILL_AMT1–6, PAY_AMT1–6, Y, SEX_2, EDUCATION_1–6, MARRIAGE_1–3. Node features are the same columns without Y (30).

Latent columns:

Columns Group Meaning
z_demo_0 … z_demo_3 Demographics P(AGE bin | Markov blanket)
z_credit_0 … z_credit_3 Credit P(LIMIT_BAL bin | Markov blanket)
z_pay_status_0 … z_pay_status_10 Repayment P(PAY_0 state | Markov blanket)
z_bill_0 … z_bill_3 Bills P(BILL_AMT1 bin | Markov blanket)
z_pay_amt_0 … z_pay_amt_3 Payments P(PAY_AMT1 bin | Markov blanket)

Each group's columns sum to about 1, because they are probability distributions.

Saved models

File Contents
models/saved_models/bayesian_network.pkl Pickled pgmpy DiscreteBayesianNetwork (structure and CPDs)
models/saved_models/gat_risk_model.pt GAT state_dict
outputs/models/bnn_model_v2.pth BNN state_dict (μ and ρ for 3 layers); loaded by evaluate.py first
outputs/models/bnn_model.pth Identical copy kept for compatibility

Output tables

File Columns
outputs/evaluation_metrics.csv Metric, Value, Value@0.5 for Threshold, Accuracy, Precision, Recall, F1-Score, AUROC, Mean Uncertainty. Value uses the validation-tuned threshold and Value@0.5 the default
outputs/ablation_results.csv Configuration, Input Dim, Threshold, Accuracy, Precision, Recall, F1-Score, AUROC, Recall@0.5, F1@0.5
outputs/bnn_training_curves.csv epoch, train_loss, task_loss, kl_loss, val_loss (100 rows, weighted-BCE scale)

.gitignore excludes data/processed/*.csv, outputs/*.csv, outputs/models/*.pth and outputs/figures/*.png, so a fresh clone regenerates them by running the pipeline.


13. Results

All values below come from the files in outputs/ after the latest full pipeline run (single run, seed 42).

Final model on the test set (4,500 customers)

The decision threshold (0.56) was chosen on the validation split to maximize F1. The default 0.5 threshold is shown for comparison.

Metric Tuned threshold (0.56) Default threshold (0.50)
Accuracy 0.7818 0.7538
Precision 0.5061 0.4603
Recall 0.5873 0.6526
F1-Score 0.5437 0.5399
AUROC 0.7733 0.7733
Mean MC uncertainty (variance) 0.00177 0.00177

AUROC does not depend on the threshold. Precision and accuracy fall because flagging more customers as defaults brings more false positives. If missing a defaulter costs more than reviewing an extra customer, a lower threshold (higher recall) is the right choice; the threshold is a single value that can be changed in find_best_threshold in evaluate.py.

Effect of the recall fixes

Metric Before (unweighted loss, threshold 0.5) After (weighted loss, tuned threshold)
Accuracy 0.814 0.782
Precision 0.639 0.506
Recall 0.37 0.59
F1-Score 0.465 0.544
AUROC 0.772 0.773
xychart-beta
    title "Test recall and F1, before vs after"
    x-axis ["Recall before", "Recall after", "F1 before", "F1 after"]
    y-axis "Score" 0 --> 0.7
    bar [0.37, 0.587, 0.465, 0.544]
Loading

Recall rose by about 22 points and F1 by about 8, with AUROC unchanged. The gain comes from the loss weighting and the threshold, not from a better ranking of customers.

Ablation: what does each branch add?

Each configuration uses its own validation-tuned threshold.

Configuration Input Threshold Accuracy Precision Recall F1 AUROC
Graph embeddings only (H) 32 0.54 0.7776 0.4979 0.5884 0.5393 0.7729
Bayesian latent only (Z) 27 0.48 0.6467 0.3346 0.6034 0.4305 0.6978
Full fusion [H ‖ Z] 59 0.52 0.7689 0.4827 0.6165 0.5414 0.7742
xychart-beta
    title "Test AUROC by input configuration"
    x-axis ["Z only", "H only", "H and Z"]
    y-axis "AUROC" 0.5 --> 0.8
    bar [0.698, 0.773, 0.774]
Loading

How to read this:

  • H carries nearly all of the signal. Fusing in Z adds about 0.001 AUROC and about 0.002 F1, which is within run-to-run noise.
  • The Z-only network used to predict no defaults at all (precision, recall and F1 of zero at the 0.5 threshold). With class weighting and a tuned threshold it now reaches recall 0.60 and F1 0.43, and its AUROC (0.70) is above chance.
  • Each ablation is a single run with one seed, so small differences are not reliable.

Comparison with the paper

Metric This repo (V2) Paper target
Accuracy 78.2% about 90%
Precision 50.6% about 89%
F1-Score 54.4% about 88%
AUROC 77.3% about 94%

The paper's numbers are not reproduced. The remaining gap is large and mostly in AUROC, which no threshold change can fix.

λ sweep (KL weight)

Validation AUROC from prior_regularization_sensitivity.png (25-epoch runs). The KL term is divided by the number of training samples, so λ = 1 is the standard ELBO.

λ Validation AUROC
0.01 0.7664
0.1 0.7655
1 0.7669
10 0.7642
100 0.5487
1000 0.5195
10000 0.5171
xychart-beta
    title "Validation AUROC vs KL weight (λ)"
    x-axis ["0.01", "0.1", "1", "10", "100", "1e3", "1e4"]
    y-axis "AUROC" 0.4 --> 0.8
    line [0.7664, 0.7655, 0.7669, 0.7642, 0.5487, 0.5195, 0.5171]
Loading

With the KL scaling fixed, AUROC is flat up to λ = 10 and falls to near chance from λ = 100, when the prior overwhelms the data term. Before the fix, AUROC collapsed already at λ = 0.01. The curve shows a plateau and then a drop, not the rise-then-fall shape described in the paper.

Uncertainty behaviour

uncertainty_distribution.png separates 3,518 correct predictions from 982 incorrect ones (at the tuned threshold). In the latest run the two variance distributions overlap almost completely, so the MC variance does not separate right from wrong predictions here. An earlier run with unweighted loss showed a heavier tail of high variance for wrong predictions, so this property is not stable across training setups. The mean variance is small (0.0018) because the posterior spread is small (ρ starts at -3.0). Treat the uncertainty output as unvalidated until it is checked with calibration measures (see Next steps).

Generated figures

Figure File Shows
Loss curves outputs/figures/loss_vs_epoch.png Total (ELBO) train loss, validation loss and weighted task BCE over 100 epochs
t-SNE outputs/figures/tsne_visualization.png 2D projection of the 59-dim fused space for 2,000 test points, with the highest-uncertainty quartile marked
λ sensitivity outputs/figures/prior_regularization_sensitivity.png Validation AUROC against λ on a log axis (λ from 0.01 to 10,000)
Uncertainty KDE outputs/figures/uncertainty_distribution.png Variance density for correct vs incorrect predictions at the tuned threshold

14. Hyperparameter reference

Component Setting Value
Data Split 70 / 15 / 15, stratified, seed 42
BN bins 4 quantile bins, edges from the training split
Money transform `sign(x) · log1p(
Graph k 5
Metric Euclidean, symmetrized, de-duplicated
GAT Layers 30 → 64 → 32, 1 head each
Dropout 0.5
Optimizer Adam, lr 0.005, weight decay 1e-4
Loss BCE with pos_weight = n_neg / n_pos (about 3.5)
Epochs and patience 100, patience 15 on validation loss; best epoch restored
Seed 42
Bayesian network Search Hill-climb, BIC, max_iter=100
Required edges PAY_6 → PAY_5 → … → PAY_0
Forbidden edges Y → any
Prior BDeu, equivalent sample size 10
Inference Variable elimination
BNN Architecture 59 → 64 → 32 → 1
Prior σ 0.1
ρ init -3.0
Loss weighted BCE (pos_weight about 3.5) + λ · KL / N, λ = 1
Checkpoint Best validation loss
Optimizer Adam, lr 0.001
Epochs and batch 100, batch 128 (train), 256 (val)
Seed 42
Inference MC passes T = 50 (30 in the λ sweep)
Threshold Tuned on validation to maximize F1 (0.05 to 0.95, step 0.01); 0.56 in the latest run
λ sweep Values 0.01, 0.1, 1, 10, 100, 1000, 10000
Per-run training 25 epochs, lr 0.002
Ablation Per-run training 60 epochs, lr 0.001, λ = 1, weighted loss, validation-tuned threshold
t-SNE Sample and perplexity 2,000 points, 30

15. Getting started

Prerequisites

  • Python 3.11 or newer
  • uv (recommended) or pip

Install

git clone https://github.com/Tetra4ge/GraphBayesRisk.git
cd GraphBayesRisk

Option A, pip (complete list):

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Option B, uv:

uv sync
uv pip install torch_geometric scikit-learn tqdm

pyproject.toml currently omits torch_geometric, scikit-learn and tqdm, so uv sync alone is not enough. See Known issues.

The dataset is already in data/raw/. To use a fresh copy, download default_of_credit_card_clients.csv from UCI into that folder.

Run everything

python src/main.py
gantt
    title Pipeline stages (relative cost, not measured times)
    dateFormat X
    axisFormat %s
    section Prepare
    data_prep.py (cleaning, splits, graph)     :a1, 0, 1
    section Embeddings
    gnn_model.py (GAT training)                :a2, 1, 2
    section Latent factors
    bayes_network.py (row-by-row inference)    :crit, a3, 3, 6
    section Model
    bnn_model.py (100 epochs)                  :a4, 9, 2
    section Evaluate
    evaluate.py (MC, t-SNE, sweep, ablation)   :a5, 11, 5
Loading

The Bayesian-network stage is the slowest. It runs variable-elimination queries for every row of all three splits, 5 groups per row, in a Python loop. The bars above are rough relative sizes, not timings; the code does not record timings.

Run one stage

python src/data_prep.py      # splits, two views, graph
python src/gnn_model.py      # GAT, writes graph_embeddings_*.csv
python src/bayes_network.py  # BN, writes latent_z_*.csv
python src/bnn_model.py      # BNN training
python src/evaluate.py       # needs a trained BNN checkpoint

Run them in this order from the repo root. Each stage reads files written by an earlier one, and bnn_model.py needs both the H and Z files.

Check the results

What Where
Test metrics outputs/evaluation_metrics.csv
Ablation outputs/ablation_results.csv
Training curves outputs/bnn_training_curves.csv
Figures outputs/figures/
Models models/saved_models/, outputs/models/

16. Dependencies

Package Version Used for
torch ≥ 2.13.0 BayesLinear, BNN and GAT training
torch_geometric ≥ 2.8.0 GATConv, GCNConv, Data
pgmpy ≥ 1.1.2 Structure learning, CPD estimation, inference
pandas ≥ 3.0.5 CSV handling
numpy ≥ 2.4.6 Arrays and NPY files
scikit-learn any StandardScaler, train_test_split, NearestNeighbors, metrics, t-SNE
matplotlib ≥ 3.11.1 Plots
seaborn ≥ 0.13.2 t-SNE scatter and KDE
tqdm any Progress bar in latent extraction

requirements.txt lists all nine. pyproject.toml lists only torch, pgmpy, pandas, numpy, matplotlib and seaborn.


17. Project history

timeline
    title Commit history (oldest to newest)
    V1 build : Data prep and Bayesian network pipeline
             : BNN model with training and evaluation
             : README and project docs
    V1 outputs : Latent features and BNN training outputs
               : Loss plots, λ sensitivity, t-SNE, saved weights
               : Methodology doc and flowchart images
    V2 planning : Dependencies and gitignore
                : New architecture with GNN integration
                : Plan v2 and BN refactoring plan
    V2 build : Graph construction pipeline
             : GAT model and embedding extraction
             : Feature fusion of H and Z
             : End-to-end test run and GAT + BNN integration
Loading
Commit Message
89781dd Generate and export processed datasets, implement BN pipeline
67b0277 Implement BNN with training and evaluation pipeline
9e5fcc7 Add project README
97bdb31, 2f87e10 Latent representations, loss plots, sensitivity analysis, t-SNE, weights
c2f9d49, 4102da2, 8200fcc Flowchart images and methodology doc
477ec83 Requirements and gitignore
e8e6769 New architecture with GNN integration strategy
a399c6a, 8e6ba35 Plan v2 and BN refactoring plan
ae37f7b Complete graph construction pipeline
66665bc GAT model and graph embedding extraction
ee2940e Feature fusion of GNN embeddings and BN latent features
cd4b9a4 End-to-end pipeline test and artifact refresh
b72ae48 GAT and BNN hybrid integration

18. Documentation map

File Content
docs/Abstract.md Project abstract, pipeline diagram and four core contributions
docs/METHODOLOGY.md How 23 raw features become latent factors and why they are fused (written for V1)
docs/ppt_slide_content.md Slide-by-slide presentation content
docs/Methodology.png Methodology flowchart image
replication_plan.md Original V1 replication plan
phase/plan_v2.md V2 architecture and four-phase plan
phase/Phase1_v2.md Graph construction
phase/Phase2_v2.md GNN model
phase/Phase3_v2.md Bayesian network refactor and fusion
phase/Phase4_v2.md BNN training and evaluation
papers/ The replicated paper and a related PDF
flowchart LR
    P1["Phase 1<br/>Graph construction"] --> P2["Phase 2<br/>GAT embeddings H"]
    P2 --> P3["Phase 3<br/>BN latent Z + fusion"]
    P3 --> P4["Phase 4<br/>BNN + evaluation"]
Loading

19. Known issues and limitations

Fixed in the fix branch

Issue Fix
GAT early stopping restored the last epoch, not the best (state_dict().copy() is shallow) The best state is now a deep clone of every tensor
pyproject.toml lacked torch_geometric, scikit-learn, tqdm, so uv sync could not run the pipeline Added with uv add; uv.lock regenerated
BN bin edges were computed on all rows before splitting Edges now come from the training split only
Duplicate removal happened after ID was dropped and removed 35 legitimate rows Duplicates are dropped with ID present, so only the one non-numeric row is removed (29,999 rows kept)
Misleading "undirected edges" log line Prints undirected pairs and directed entries separately
Low recall (about 0.37): unweighted loss and a fixed 0.5 threshold Class-weighted BCE (GAT and BNN) plus an F1-optimal threshold tuned on validation
λ sweep collapsed to chance for λ ≥ 0.01 (KL summed, not divided by N) KL is now divided by N; the sweep is flat up to λ = 10
BNN always kept the last epoch The best-validation weights are restored
.squeeze() broke on a batch of size 1 Replaced with .squeeze(-1)
Ablation and λ sweep were not seeded torch.manual_seed(42) before every model
main.py only worked from the repo root It changes into the repo root itself

Still open

# Issue Where Effect
1 Z adds little over H. Fusion improves AUROC by about 0.001 Ablation The BN branch is mainly an interpretability layer here
2 MC variance does not separate right from wrong predictions in the latest run uncertainty_distribution.png The uncertainty claim is unvalidated; calibration (ECE, reliability diagram) is needed
3 Precision and accuracy dropped when recall was raised Threshold choice Expected trade-off; pick the threshold by the real cost of missed defaults
4 Transductive graph. Validation and test nodes are in the graph, so new customers need the graph rebuilt or an inductive variant build_knn_graph Not deployable as-is for unseen customers
5 BN fitted and queried on the same training rows. Training-row Z values are in-sample extract_latent_features Train Z can look cleaner than val and test Z
6 BN extraction is slow. One inference call per row per group in a Python loop extract_latent_features Dominates pipeline runtime
7 MC inference cost. 50 forward passes per prediction run_mc_inference About 50 times a point-estimate model
8 Paper targets not reached (AUROC 0.77 against about 0.94) Results See Results
9 Duplicate checkpoint. bnn_model.pth and bnn_model_v2.pth are written identically bnn_model.main Redundant file
10 Empty notebooks. 01_eda.ipynb, 02_bn_experiments.ipynb, 03_bnn_experiments.ipynb are 0-byte files notebooks/ No interactive exploration yet
11 Single run, single seed for every reported number All experiments No confidence intervals
12 docs/METHODOLOGY.md and docs/ppt_slide_content.md describe V1 (57-dim fusion with raw features) docs/ Not updated for the GAT branch
13 __pycache__ files are tracked by git although .gitignore lists them src/__pycache__/ Noisy diffs

20. Suggested next steps

flowchart TB
    A["Measure calibration:<br/>ECE and reliability diagram (2)"] --> B["Multiple seeds,<br/>report mean ± std (11)"]
    B --> C["Add baselines:<br/>logistic regression, XGBoost, MLP (1)"]
    C --> D["Cross-fit Z to remove<br/>in-sample bias (5)"]
    D --> E["Speed up BN inference<br/>by caching queries (6)"]
    E --> F["Inductive GNN for<br/>unseen customers (4)"]
    F --> G["Fill notebooks,<br/>update METHODOLOGY (10, 12)"]
Loading

21. Dashboard

A read-only web dashboard over the pipeline outputs: React (Vite, TypeScript) in frontend/ and FastAPI in backend/. It does not train anything.

flowchart LR
    E["src/evaluate.py"] --> P[("outputs/predictions_test.csv")]
    E --> M[("outputs/evaluation_metrics.csv")]
    Z[("data/processed/latent_z_test.csv")]
    P --> B["backend (FastAPI)"]
    M --> B
    Z --> B
    B -- "JSON over /api" --> F["frontend (React)"]
Loading
Page Shows
/ Landing page
/results Cut-off slider that recomputes precision, recall, F1 and the confusion matrix, plus the before/after recall
/customers Ranked, filterable table of the 4,500 test customers; a row opens the 27 Bayesian-network factor values

Run it from the repo root:

uv sync
cd frontend && npm install && cp .env.example .env && cd ..
uv run uvicorn backend.main:app --reload --port 8000
cd frontend && npm run dev          # http://localhost:5173

Do not run uv sync inside backend/; it shares the root environment and would remove the pipeline packages. API reference and tests are in backend/README.md; the frontend is described in frontend/README.md; the plan is in plan/PLAN.md.

evaluate.py also writes outputs/predictions_test.csv (id, y_true, risk, uncertainty) for the dashboard. It comes from the same Monte Carlo pass as evaluation_metrics.csv, so the numbers match the tables above. The MC variance is shown as experimental, since it did not clearly separate right from wrong predictions in the latest run.


22. References

  1. Replicated paper: Uncertainty-Aware Robust Financial Risk Discrimination via Bayesian Network and Bayesian Neural Modeling (see papers/).
  2. Dataset paper: Yeh, I. C., & Lien, C. H. (2009). The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2), 2473-2480. Semantic Scholar
  3. Dataset: UCI ML Repository, Default of Credit Card Clients
  4. Libraries: PyTorch, PyTorch Geometric, pgmpy, scikit-learn

23. License

Apache License 2.0. See LICENSE.

About

Graph-Bayesian neural architecture for credit-card default prediction with uncertainty. Combines a GAT over a k-NN customer graph, a pgmpy Bayesian network for latent risk factors, and a Bayesian neural network with Monte Carlo sampling.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages