Paper replication, extended (V2). This repository replicates "Uncertainty-Aware Robust Financial Risk Discrimination via Bayesian Network and Bayesian Neural Modeling" on the UCI Default of Credit Card Clients dataset, then extends it with a Graph Attention Network (GAT) branch. The final model fuses graph embeddings H with Bayesian-network latent factors Z and classifies with a Bayesian Neural Network (BNN) that reports a risk score and an uncertainty estimate for every customer.
- What this project does
- Background and motivation
- Architecture at a glance
- Dataset
- Repository structure
- Stage 0: Data preparation
- Stage 1: Graph construction and GAT
- Stage 2: Bayesian network
- Stage 3: Fusion and Bayesian neural network
- Stage 4: Evaluation
- Pipeline orchestration (
main.py) - Every generated file
- Results
- Hyperparameter reference
- Getting started
- Dependencies
- Project history
- Documentation map
- Known issues and limitations
- Suggested next steps
- Dashboard
- References
- License
Given a credit-card client's demographics, credit limit, six months of repayment status, bill amounts and payment amounts, the pipeline:
- predicts whether the client will default next month, and
- says how uncertain it is about that prediction.
It gets there with three models that each contribute a different kind of information:
| Model | Library | Contribution | Output |
|---|---|---|---|
| Graph Attention Network | PyTorch Geometric | Learns from similar customers (neighbours in a k-NN graph) | H, 32 dims per customer |
| Bayesian Network | pgmpy | Models how features depend on each other and gives interpretable posteriors | Z, 27 dims per customer |
| Bayesian Neural Network | PyTorch (custom BayesLinear) |
Classifies the fused vector [H ‖ Z] with a distribution over every weight |
Risk score and variance |
flowchart LR
A["Raw customer record<br/>23 features"] --> B["Cleaning, splits,<br/>two data views"]
B --> C["k-NN graph<br/>+ GAT"]
B --> D["Binned data<br/>+ Bayesian Network"]
C -->|"H (32)"| E["Fusion<br/>H ‖ Z (59)"]
D -->|"Z (27)"| E
E --> F["Bayesian Neural Network"]
F --> G["50 Monte Carlo passes"]
G --> H["Risk score (mean)<br/>Uncertainty (variance)"]
A conventional credit model returns one number, such as "70% chance of default". It does not say whether the model has seen similar customers before or is extrapolating. Financial decisions need that distinction: ambiguous cases should go to a human reviewer.
The source paper combines a Bayesian Network (structured, interpretable) with a Bayesian Neural Network (flexible, uncertainty-aware). V1 of this repo implemented that paper directly, fusing 30 raw features with 27 latent factors. V2 replaces the raw features with graph embeddings, because the credit data has no natural relational structure and a k-NN similarity graph lets the model learn from each customer's neighbourhood.
flowchart TB
subgraph V1["V1: paper replication"]
direction LR
V1a["30 raw features"] --> V1c["Fusion 57-dim"]
V1b["Bayesian Network Z (27)"] --> V1c
V1c --> V1d["BNN"]
end
subgraph V2["V2: this repository"]
direction LR
V2a["k-NN graph + GAT, H (32)"] --> V2c["Fusion 59-dim"]
V2b["Bayesian Network Z (27)"] --> V2c
V2c --> V2d["BNN"]
end
V1 -->|"GNN branch replaces raw features"| V2
Bayesian network (paper Eq. 1 and 2)
p(z | G) = ∏ p(zᵢ | Pa(zᵢ))
p(y | x) = Σ_z p(y | z, x) · p(z | G)
G is the learned DAG, z the latent posteriors and Pa(zᵢ) the parents of a node.
BNN loss (paper Eq. 5 and 6)
L_task = -log p(y | x) (binary cross-entropy, positives up-weighted)
L = L_task + λ · KL( q(w) ‖ p(w) ) / N
q(w) is the learned Gaussian posterior over weights, p(w) = N(0, 0.1²) the prior and N the number of training samples. Dividing the KL term by N is the standard ELBO scaling, so λ = 1 is the true Bayesian objective. In this repo the positive class is also weighted by n_negatives / n_positives (about 3.5) to counter class imbalance.
Monte Carlo inference (paper Eq. 7)
ŷ = (1/T) Σₜ ŷₜ risk score
Var = (1/T) Σₜ (ŷₜ − ŷ)² uncertainty
with T = 50 stochastic forward passes.
flowchart TB
RAW[("data/raw<br/>UCI csv")]
subgraph S0["Stage 0 · data_prep.py"]
CLEAN["Clean<br/>(drop NaN, duplicates,<br/>clip negative PAY_AMT)"]
SPLIT["Stratified split<br/>70 / 15 / 15, seed 42"]
DISC["Discrete view<br/>4 quantile bins"]
CONT["Continuous view<br/>log, one-hot, scale"]
KNN["k-NN graph (k=5)<br/>edge_index + masks"]
CLEAN --> SPLIT --> DISC
SPLIT --> CONT --> KNN
end
subgraph S1["Stage 1 · gnn_model.py"]
GAT["GAT 30 → 64 → 32"]
H[/"H · graph_embeddings_*.csv"/]
GAT --> H
end
subgraph S2["Stage 2 · bayes_network.py"]
HC["Hill-climb + BIC"]
CPD["BDeu CPDs"]
VE["Variable elimination<br/>5 factor groups"]
Z[/"Z · latent_z_*.csv"/]
HC --> CPD --> VE --> Z
end
subgraph S3["Stage 3 · bnn_model.py"]
FUSE["Fuse H ‖ Z → 59"]
BNN["BayesLinear<br/>59 → 64 → 32 → 1"]
FUSE --> BNN
end
subgraph S4["Stage 4 · evaluate.py"]
MC["MC inference T=50"]
PLOTS["Loss, t-SNE,<br/>uncertainty plots"]
EXP["λ sweep +<br/>ablation"]
MC --> PLOTS
MC --> EXP
end
RAW --> CLEAN
KNN --> GAT
DISC --> HC
H --> FUSE
Z --> FUSE
BNN --> MC
| Property | Value |
|---|---|
| Name | Default of Credit Card Clients |
| Source | UCI Machine Learning Repository |
| Original paper | Yeh & Lien (2009), Expert Systems with Applications |
| Raw rows | 30,000 |
| Rows after cleaning | 29,999 (only the one row with a non-numeric value is dropped) |
| Features | 23 raw (25 columns in the file, including ID and Y) |
| Target | Y, default payment next month (1 = default) |
| Class balance | about 77.9% non-default and 22.1% default in the cleaned data |
| Region and period | Taiwan, April to September 2005 |
The raw file has two header rows. Row 1 holds generic labels (X1, X2, …) and row 2 holds the real names. The loader reads it with header=1.
| Feature | Description | Type |
|---|---|---|
LIMIT_BAL |
Credit limit in NT dollars (individual plus family credit) | Continuous |
SEX |
1 = male, 2 = female | Categorical |
EDUCATION |
1 = graduate school, 2 = university, 3 = high school, 4 = others (values 0, 5, 6 also occur) | Categorical |
MARRIAGE |
1 = married, 2 = single, 3 = others (value 0 also occurs) | Categorical |
AGE |
Age in years | Continuous |
PAY_0, PAY_2 … PAY_6 |
Repayment status, September (PAY_0) back to April (PAY_6). -1 = paid duly, 1 = one month late, … 9 = nine or more months late. The dataset has no PAY_1. |
Ordinal |
BILL_AMT1 … BILL_AMT6 |
Bill statement amounts, September back to April | Continuous |
PAY_AMT1 … PAY_AMT6 |
Previous payment amounts, September back to April | Continuous |
Y |
Default next month | Binary target |
flowchart LR
subgraph demo["demo (4)"]
d1["SEX"] --- d2["EDUCATION"] --- d3["MARRIAGE"] --- d4["AGE"]
end
subgraph credit["credit (1)"]
c1["LIMIT_BAL"]
end
subgraph pay["pay_status (6)"]
p1["PAY_0"] --- p2["PAY_2"] --- p3["PAY_3"] --- p4["PAY_4"] --- p5["PAY_5"] --- p6["PAY_6"]
end
subgraph bill["bill (6)"]
b1["BILL_AMT1 … 6"]
end
subgraph pa["pay_amt (6)"]
a1["PAY_AMT1 … 6"]
end
demo --> T(("Y"))
credit --> T
pay --> T
bill --> T
pa --> T
GraphBayesRisk/
├── README.md this file
├── LICENSE Apache 2.0
├── pyproject.toml uv project metadata
├── requirements.txt pip dependency list (includes torch_geometric)
├── uv.lock locked dependency versions
├── .python-version 3.11
├── .gitignore
├── replication_plan.md original V1 plan
│
├── src/
│ ├── main.py runs every stage in order
│ ├── data_prep.py cleaning, splits, two data views, k-NN graph
│ ├── gnn_model.py GAT and GCN models, training, embedding export
│ ├── bayes_network.py structure learning, CPDs, latent factor extraction
│ ├── bnn_model.py BayesLinear, BNN, FusedDataset, training loop
│ └── evaluate.py MC inference, metrics, plots, λ sweep, ablation
│
├── data/
│ ├── raw/ default_of_credit_card_clients.csv
│ └── processed/ splits, graph arrays, H and Z files
│
├── models/saved_models/ bayesian_network.pkl, gat_risk_model.pt
├── outputs/
│ ├── evaluation_metrics.csv final test metrics
│ ├── ablation_results.csv H only vs Z only vs fused
│ ├── bnn_training_curves.csv per-epoch losses
│ ├── models/ bnn_model.pth, bnn_model_v2.pth
│ └── figures/ four PNG plots
│
├── docs/ Abstract, METHODOLOGY, slide content, flowchart image
├── phase/ plan_v2.md and Phase1 to Phase4 notes
├── papers/ source paper PDFs
└── notebooks/ three empty placeholder notebooks
flowchart LR
main["main.py"] -. "subprocess" .-> dp["data_prep.py"]
main -. "subprocess" .-> gnn["gnn_model.py"]
main -. "subprocess" .-> bn["bayes_network.py"]
main -. "subprocess" .-> bnn["bnn_model.py"]
main -. "subprocess" .-> ev["evaluate.py"]
ev -->|"import BNN, FusedDataset, train_model"| bnn
main.py launches each script as a separate process. The only Python import between modules is evaluate.py importing from bnn_model.py; everything else communicates through files.
File: src/data_prep.py
flowchart TB
A["load_data<br/>header=1, keep ID,<br/>coerce to numeric"] --> B["clean_data<br/>dropna, drop_duplicates,<br/>drop ID, clip PAY_AMT ≥ 0,<br/>cast ints"]
B --> C["train_test_split<br/>70 / 30 stratified"]
C --> D["split the 30%<br/>50 / 50 stratified"]
D --> E{"same row indices<br/>for both views"}
E --> F["prepare_bn_data<br/>4 quantile bins<br/>(edges from train only)"]
E --> G["prepare_bnn_data<br/>signed log1p, one-hot,<br/>StandardScaler"]
F --> H[("train / val / test<br/>_discrete.csv")]
G --> I[("train / val / test<br/>_continuous.csv")]
I --> J["build_knn_graph<br/>k=5"]
J --> K[("edge_index.npy<br/>node_features.npy<br/>node_labels.npy<br/>3 masks")]
| Function | What it does |
|---|---|
load_data |
Reads the CSV with header=1, renames default payment next month to Y, coerces everything to numbers (bad cells become NaN). ID is kept until cleaning |
clean_data |
Drops NaN rows and exact duplicate records (with ID still present, so different customers with identical features are not merged), then drops ID, clips PAY_AMT1–PAY_AMT6 at zero, casts Y, SEX, EDUCATION, MARRIAGE, AGE and the PAY columns to integers |
prepare_bn_data |
Cuts AGE, LIMIT_BAL, BILL_AMT1–6 and PAY_AMT1–6 into 4 quantile bins (labels 0 to 3). Bin edges are computed on the training rows only and applied to all splits with pd.cut |
prepare_bnn_data |
Applies `sign(x) · log1p( |
build_knn_graph |
Concatenates train, val and test features and builds the graph described in Stage 1 |
main |
Runs the steps above and writes all files to data/processed/ |
- One split for both views. The stratified split (seed 42) is computed once on row indices, so row i of a discrete file and row i of the matching continuous file are the same customer.
- No scaler leakage.
StandardScaleris fit on the training split only, then applied to validation and test. One-hot columns andYare not scaled. - Train-only statistics. Both the scaler and the BN bin edges come from the training split, so no validation or test statistics reach the models.
- Split sizes: train 20,999, validation 4,500, test 4,500 (29,999 total).
Files: src/data_prep.py (graph) and src/gnn_model.py (model)
Credit data has no relationships between customers, so one is created from feature similarity.
flowchart LR
A["Scaled continuous features<br/>train + val + test<br/>29,999 × 30"] --> B["NearestNeighbors<br/>k+1 = 6, Euclidean"]
B --> C["Drop self match<br/>(column 0)"]
C --> D["Directed edges<br/>i → each of 5 neighbours"]
D --> E["Add reverse edges<br/>(undirected)"]
E --> F["np.unique<br/>remove duplicates"]
F --> G["edge_index<br/>shape 2 × 223,740"]
| Property | Value |
|---|---|
| Nodes | 29,999 (one per customer) |
| Node features | 30 scaled continuous features (no label) |
Edge entries in edge_index |
223,740 (both directions stored, so 111,870 undirected pairs) |
| Average degree | about 7.46 |
| Setting | Transductive: all nodes (train, val, test) are in one graph, and boolean masks pick which nodes count for loss and evaluation |
The graph uses only features, never labels, so the test labels cannot reach the GAT through the edges.
flowchart LR
X["x<br/>N × 30"] --> L1["GATConv<br/>30 → 64<br/>heads=1, dropout 0.5"]
L1 --> A1["ELU"]
A1 --> DR["Dropout 0.5"]
DR --> L2["GATConv<br/>64 → 32<br/>heads=1, concat=False"]
L2 --> A2["ELU"]
A2 --> HH(["H<br/>N × 32"])
A2 --> CL["Linear 32 → 1"]
CL --> SG["Sigmoid"]
SG --> O(["proxy risk"])
The proxy head exists only to give the GAT a supervised signal. The product of this stage is H, the 32-dim layer output.
gnn_model.py also defines GCNRiskModel (two GCNConv layers, ReLU) as a baseline. The pipeline does not use it.
sequenceDiagram
participant M as GATRiskModel
participant D as Graph data
participant O as Adam (lr 0.005, wd 1e-4)
loop up to 100 epochs
M->>D: forward on the whole graph (train mode)
M->>O: class-weighted BCE on train_mask nodes only, backward, step
M->>D: forward again (eval mode, no grad)
M-->>M: val loss and val AUC on val_mask nodes
alt val loss improved
M-->>M: save "best" state, reset patience
else no improvement
M-->>M: patience += 1 (stop at 15)
end
end
M->>M: load saved best state (cloned tensors)
| Function | Purpose |
|---|---|
load_graph_data |
Loads the NPY files and builds a PyG Data object with masks |
compute_metrics |
Accuracy, precision, recall, F1, ROC-AUC at a 0.5 threshold |
train_gnn |
Masked-loss training with early stopping on validation loss. The loss is BCE with positives weighted by n_neg / n_pos of the training nodes (about 3.5). The best state is deep-copied, so the restored model really is the best epoch |
extract_and_save_embeddings |
Runs the trained GAT once, slices H by mask, writes three CSVs and prints proxy-head test metrics |
main |
Seeds 42, picks CUDA if available, trains, saves models/saved_models/gat_risk_model.pt and the embeddings |
File: src/bayes_network.py
flowchart TB
A[("train_discrete.csv")] --> B["HillClimbSearch<br/>score: BIC, max_iter 100"]
K["ExpertKnowledge<br/>required: PAY chain<br/>forbidden: Y → anything"] --> B
B --> C["DAG (45 edges)"]
C --> D["DiscreteBayesianNetwork"]
A --> E["BayesianEstimator<br/>BDeu, ESS 10"]
D --> E
E --> F["CPDs + check_model"]
F --> G[("bayesian_network.pkl")]
F --> H["extract_latent_features<br/>per row, 5 groups"]
H --> I[("latent_z_train / val / test.csv")]
| Constraint | Edges | Reason |
|---|---|---|
| Required | PAY_6 → PAY_5 → PAY_4 → PAY_3 → PAY_2 → PAY_0 |
Repayment status follows time |
| Forbidden | Y → every other column |
The label is an outcome, so it can only be a child |
The search found 45 edges. Highlights from the saved model:
flowchart LR
PAY6["PAY_6"] --> PAY5["PAY_5"] --> PAY4["PAY_4"] --> PAY3["PAY_3"] --> PAY2["PAY_2"] --> PAY0["PAY_0"]
PAY0 --> Y(("Y"))
LIM["LIMIT_BAL"] --> Y
PAY2 --> B1["BILL_AMT1"]
PAY5 --> B4["BILL_AMT4"]
B5["BILL_AMT5"] --> PAY6
PAY2 --> A1["PAY_AMT1"]
PAY3 --> A2["PAY_AMT2"]
PAY4 --> A3["PAY_AMT3"]
PAY5 --> A4["PAY_AMT4"]
A5["PAY_AMT5"] --> PAY6
AGE["AGE"] --> EDU["EDUCATION"]
AGE --> MAR["MARRIAGE"]
AGE --> SEX["SEX"]
LIM --> AGE
LIM --> EDU
LIM --> SEX
Only PAY_0 and LIMIT_BAL are parents of Y. PAY_2 to PAY_5 each lead to the matching PAY_AMT, and the bill amounts and payment amounts form chains among themselves. The structure depends on the bin edges, so it differs slightly from earlier runs (the BN was learned on train-only bins).
For each customer and each of the five factor groups, the code asks the network for the posterior of the group's target variable, given the evidence in its Markov blanket.
sequenceDiagram
participant R as Customer row
participant G as Factor group
participant VE as VariableElimination
R->>G: pick target (e.g. PAY_0)
G->>G: evidence = Markov blanket of target, minus the group's own variables, minus Y
G->>VE: query(target, evidence)
alt query succeeds
VE-->>G: posterior over target states
else query fails
G->>VE: query(target) with no evidence (prior)
alt prior succeeds
VE-->>G: prior over target states
else prior fails
G-->>G: uniform distribution
end
end
G-->>R: columns z_group_0 … z_group_k
| Factor group | Target variable | Variables in the group | States | Z columns |
|---|---|---|---|---|
demo |
AGE |
SEX, EDUCATION, MARRIAGE, AGE | 4 | 4 |
credit |
LIMIT_BAL |
LIMIT_BAL | 4 | 4 |
pay_status |
PAY_0 |
PAY_0, PAY_2 … PAY_6 | 11 | 11 |
bill |
BILL_AMT1 |
BILL_AMT1–6 | 4 | 4 |
pay_amt |
PAY_AMT1 |
PAY_AMT1–6 | 4 | 4 |
| Total | 27 |
Excluding Y and the group's own variables from the evidence means a customer's own label and own target value never leak into Z.
File: src/bnn_model.py
flowchart LR
H["graph_embeddings_*.csv<br/>GNN_0 … GNN_31"] --> C["np.concatenate<br/>axis=1"]
Z["latent_z_*.csv<br/>z_* (27 columns)"] --> C
C --> X(["X · 59 dims"])
Y["*_continuous.csv<br/>column Y"] --> L(["y labels"])
X --> DS["FusedDataset"]
L --> DS
FusedDataset has three modes used by the ablation:
| Mode | Input | Dims |
|---|---|---|
both |
[H ‖ Z] |
59 |
graph_only |
H |
32 |
latent_only |
Z |
27 |
Only the Y column of the continuous CSV is read. The 30 raw continuous features are not part of the BNN input in V2.
flowchart TB
subgraph params["Learned parameters per weight and bias"]
mu["μ (mean)"]
rho["ρ (init -3.0)"]
end
rho --> sig["σ = log(1 + exp(ρ))"]
eps["ε ~ N(0, 1)"] --> w
mu --> w["w = μ + σ · ε"]
sig --> w
w --> lin["F.linear(x, w, b)"]
mu --> kl
sig --> kl["KL( N(μ, σ²) ‖ N(0, 0.1²) )<br/>closed form, summed"]
kl --> store["self.kl_divergence"]
BayesLinear keeps weight_mu, weight_rho, bias_mu and bias_rho. Means start uniform in ±1/√in_features. The KL term is recomputed on every forward pass and read back with BNN.get_kl() (the sum over the three layers).
flowchart LR
I["Input<br/>59"] --> F1["BayesLinear 59 → 64"] --> R1["ReLU"] --> F2["BayesLinear 64 → 32"] --> R2["ReLU"] --> F3["BayesLinear 32 → 1"] --> S["Sigmoid"] --> P(["ŷ ∈ (0, 1)"])
Trainable parameters: (59·64 + 64) + (64·32 + 32) + (32·1 + 1) = 5,953 weights and biases, each with a mean and a ρ, so 11,906 values. This matches the roughly 50 KB checkpoint.
sequenceDiagram
participant L as DataLoader (batch 128, shuffle)
participant M as BNN
participant O as Adam (lr 0.001)
loop 100 epochs
loop each batch
L->>M: X_batch
M-->>L: ŷ (sampled weights)
Note over M,O: loss = weighted BCE(ŷ, y) + λ · KL / N
O->>M: backward and step
end
M->>M: validation pass (batch 256), record val loss
M->>M: keep a copy of the weights if val loss is the best so far
end
M->>M: restore the best-validation weights
Per epoch the code records total loss, task (weighted BCE) loss, KL and validation loss, and writes them to outputs/bnn_training_curves.csv. At epoch 100 the total train loss is about 0.897, weighted task BCE about 0.879, KL about 369 and validation loss about 0.916. These losses are larger than in earlier versions because positives are up-weighted, so they are not comparable with unweighted BCE values.
The BNN trains for the full 100 epochs and then restores the weights from the epoch with the lowest validation loss (about 0.910 in the latest run).
Loss details. The KL term is divided by the number of training samples
N, soλ = 1is the standard ELBO. Positives are weighted byn_neg / n_pos(about 3.5, computed from the training labels). The weighting moves predicted probabilities upward, which is why the best decision threshold is no longer 0.5.
main() saves the final weights twice: bnn_model_v2.pth and a compatibility copy bnn_model.pth.
File: src/evaluate.py
flowchart TB
A["Load test [H ‖ Z]<br/>+ bnn_model_v2.pth"] --> B["run_mc_inference<br/>T = 50"]
B --> C["mean → risk score<br/>variance → uncertainty"]
V["Validation MC scores<br/>→ threshold that maximizes F1"] --> D["Test metrics at the tuned threshold<br/>(and at 0.5 for comparison)"]
C --> D
D --> E[("evaluation_metrics.csv")]
C --> F["plot_loss_curves"]
C --> G["plot_tsne"]
C --> H["plot_uncertainty_distribution"]
F --> F1[("loss_vs_epoch.png")]
G --> G1[("tsne_visualization.png")]
H --> H1[("uncertainty_distribution.png")]
D --> I["run_lambda_sweep"]
I --> I1[("prior_regularization_sensitivity.png")]
I --> J["run_ablation_studies"]
J --> J1[("ablation_results.csv")]
sequenceDiagram
participant E as evaluate.py
participant M as BNN (eval mode)
E->>M: batch X
loop T = 50 passes
M-->>E: ŷₜ (fresh weight sample each pass)
end
E->>E: risk = mean over passes
E->>E: uncertainty = variance over passes
Note that model.eval() does not turn off weight sampling in BayesLinear, so every pass draws new weights, which is exactly what MC inference needs.
| Experiment | Setup | Output |
|---|---|---|
| Threshold tuning | MC scores (T=50) on the validation split. The threshold in 0.05 to 0.95 (step 0.01) with the best F1 is chosen | printed and stored in evaluation_metrics.csv |
| Final test evaluation | bnn_model_v2.pth, T=50, tuned threshold; metrics at 0.5 are saved alongside |
evaluation_metrics.csv |
| Loss curves | Reads bnn_training_curves.csv: train ELBO, validation loss, task BCE |
loss_vs_epoch.png |
| t-SNE | 2,000 random test points (seed 42) in the 59-dim space, perplexity 30. Points above the 75th percentile of uncertainty are drawn as red X marks | tsne_visualization.png |
| Uncertainty distribution | KDE of variance for correct vs incorrect predictions (at the tuned threshold) | uncertainty_distribution.png |
| λ sweep | λ in {1e-2, 1e-1, 1, 10, 100, 1000, 10000}. Fresh seeded BNN per value, 25 epochs, lr 0.002, T=30, class-weighted loss, scored on validation AUROC | prior_regularization_sensitivity.png (values are printed, not saved to CSV) |
| Ablation | H only, Z only and [H ‖ Z]. Fresh seeded BNN each, 60 epochs, lr 0.001, λ=1, class-weighted loss. Threshold tuned on validation per configuration, T=50, scored on the test set |
ablation_results.csv |
src/main.py runs five scripts in order using the project's .venv interpreter if it exists (otherwise the current Python). It raises RuntimeError if any script exits non-zero.
stateDiagram-v2
[*] --> DataPrep
DataPrep --> GNN: exit 0
GNN --> BayesNet: exit 0
BayesNet --> BNN: exit 0
BNN --> Evaluate: exit 0
Evaluate --> [*]: pipeline complete
DataPrep --> Failed: non-zero exit
GNN --> Failed: non-zero exit
BayesNet --> Failed: non-zero exit
BNN --> Failed: non-zero exit
Evaluate --> Failed: non-zero exit
Failed --> [*]: RuntimeError
Scripts use relative paths such as data/processed, so run everything from the repository root.
flowchart LR
DP["data_prep.py"] -->|"*_discrete.csv"| BN["bayes_network.py"]
DP -->|"edge_index, node_features,<br/>node_labels, masks"| GN["gnn_model.py"]
DP -->|"*_continuous.csv (Y column)"| BB["bnn_model.py"]
GN -->|"graph_embeddings_*.csv"| BB
BN -->|"latent_z_*.csv"| BB
BB -->|"bnn_model_v2.pth,<br/>training curves"| EV["evaluate.py"]
DP -->|"*_continuous.csv"| EV
GN -->|"graph_embeddings_*.csv"| EV
BN -->|"latent_z_*.csv"| EV
flowchart TB
subgraph raw["data/raw"]
R1["default_of_credit_card_clients.csv"]
end
subgraph proc["data/processed"]
P1["train / val / test _discrete.csv"]
P2["train / val / test _continuous.csv"]
P3["edge_index.npy · node_features.npy · node_labels.npy"]
P4["train_mask / val_mask / test_mask .npy"]
P5["graph_embeddings_train / val / test .csv"]
P6["latent_z_train / val / test .csv"]
end
subgraph mod["models/saved_models"]
M1["bayesian_network.pkl"]
M2["gat_risk_model.pt"]
end
subgraph out["outputs"]
O1["evaluation_metrics.csv"]
O2["ablation_results.csv"]
O3["bnn_training_curves.csv"]
O4["models/bnn_model.pth · bnn_model_v2.pth"]
O5["figures/*.png (4 plots)"]
end
R1 --> P1 & P2 & P3 & P4
P3 & P4 --> P5
P1 --> P6
P5 & P6 --> O4
O4 --> O1 & O2 & O3 & O5
P3 --> M2
P1 --> M1
| File | Rows | Columns | Description |
|---|---|---|---|
train_discrete.csv |
20,999 | 24 | Binned features plus Y. Used to learn the BN |
val_discrete.csv |
4,500 | 24 | Same format |
test_discrete.csv |
4,500 | 24 | Same format |
train_continuous.csv |
20,999 | 31 | Log-transformed and standardized numerics, one-hot categoricals, Y |
val_continuous.csv |
4,500 | 31 | Scaled with the training scaler |
test_continuous.csv |
4,500 | 31 | Scaled with the training scaler |
edge_index.npy |
n/a | n/a | int64, shape 2 × 223,740 |
node_features.npy |
29,999 | 30 | float32, scaled features in train, val, test order |
node_labels.npy |
29,999 | n/a | int64 labels |
train_mask.npy, val_mask.npy, test_mask.npy |
29,999 | n/a | Boolean node masks |
graph_embeddings_*.csv |
split size | 32 | GNN_0 … GNN_31 |
latent_z_*.csv |
split size | 27 | z_{group}_{state_index} |
(Row counts exclude the header line.)
Continuous columns (31): LIMIT_BAL, AGE, PAY_0, PAY_2–PAY_6, BILL_AMT1–6, PAY_AMT1–6, Y, SEX_2, EDUCATION_1–6, MARRIAGE_1–3.
Node features are the same columns without Y (30).
Latent columns:
| Columns | Group | Meaning |
|---|---|---|
z_demo_0 … z_demo_3 |
Demographics | P(AGE bin | Markov blanket) |
z_credit_0 … z_credit_3 |
Credit | P(LIMIT_BAL bin | Markov blanket) |
z_pay_status_0 … z_pay_status_10 |
Repayment | P(PAY_0 state | Markov blanket) |
z_bill_0 … z_bill_3 |
Bills | P(BILL_AMT1 bin | Markov blanket) |
z_pay_amt_0 … z_pay_amt_3 |
Payments | P(PAY_AMT1 bin | Markov blanket) |
Each group's columns sum to about 1, because they are probability distributions.
| File | Contents |
|---|---|
models/saved_models/bayesian_network.pkl |
Pickled pgmpy DiscreteBayesianNetwork (structure and CPDs) |
models/saved_models/gat_risk_model.pt |
GAT state_dict |
outputs/models/bnn_model_v2.pth |
BNN state_dict (μ and ρ for 3 layers); loaded by evaluate.py first |
outputs/models/bnn_model.pth |
Identical copy kept for compatibility |
| File | Columns |
|---|---|
outputs/evaluation_metrics.csv |
Metric, Value, Value@0.5 for Threshold, Accuracy, Precision, Recall, F1-Score, AUROC, Mean Uncertainty. Value uses the validation-tuned threshold and Value@0.5 the default |
outputs/ablation_results.csv |
Configuration, Input Dim, Threshold, Accuracy, Precision, Recall, F1-Score, AUROC, Recall@0.5, F1@0.5 |
outputs/bnn_training_curves.csv |
epoch, train_loss, task_loss, kl_loss, val_loss (100 rows, weighted-BCE scale) |
.gitignoreexcludesdata/processed/*.csv,outputs/*.csv,outputs/models/*.pthandoutputs/figures/*.png, so a fresh clone regenerates them by running the pipeline.
All values below come from the files in outputs/ after the latest full pipeline run (single run, seed 42).
The decision threshold (0.56) was chosen on the validation split to maximize F1. The default 0.5 threshold is shown for comparison.
| Metric | Tuned threshold (0.56) | Default threshold (0.50) |
|---|---|---|
| Accuracy | 0.7818 | 0.7538 |
| Precision | 0.5061 | 0.4603 |
| Recall | 0.5873 | 0.6526 |
| F1-Score | 0.5437 | 0.5399 |
| AUROC | 0.7733 | 0.7733 |
| Mean MC uncertainty (variance) | 0.00177 | 0.00177 |
AUROC does not depend on the threshold. Precision and accuracy fall because flagging more customers as defaults brings more false positives. If missing a defaulter costs more than reviewing an extra customer, a lower threshold (higher recall) is the right choice; the threshold is a single value that can be changed in find_best_threshold in evaluate.py.
| Metric | Before (unweighted loss, threshold 0.5) | After (weighted loss, tuned threshold) |
|---|---|---|
| Accuracy | 0.814 | 0.782 |
| Precision | 0.639 | 0.506 |
| Recall | 0.37 | 0.59 |
| F1-Score | 0.465 | 0.544 |
| AUROC | 0.772 | 0.773 |
xychart-beta
title "Test recall and F1, before vs after"
x-axis ["Recall before", "Recall after", "F1 before", "F1 after"]
y-axis "Score" 0 --> 0.7
bar [0.37, 0.587, 0.465, 0.544]
Recall rose by about 22 points and F1 by about 8, with AUROC unchanged. The gain comes from the loss weighting and the threshold, not from a better ranking of customers.
Each configuration uses its own validation-tuned threshold.
| Configuration | Input | Threshold | Accuracy | Precision | Recall | F1 | AUROC |
|---|---|---|---|---|---|---|---|
| Graph embeddings only (H) | 32 | 0.54 | 0.7776 | 0.4979 | 0.5884 | 0.5393 | 0.7729 |
| Bayesian latent only (Z) | 27 | 0.48 | 0.6467 | 0.3346 | 0.6034 | 0.4305 | 0.6978 |
| Full fusion [H ‖ Z] | 59 | 0.52 | 0.7689 | 0.4827 | 0.6165 | 0.5414 | 0.7742 |
xychart-beta
title "Test AUROC by input configuration"
x-axis ["Z only", "H only", "H and Z"]
y-axis "AUROC" 0.5 --> 0.8
bar [0.698, 0.773, 0.774]
How to read this:
Hcarries nearly all of the signal. Fusing inZadds about 0.001 AUROC and about 0.002 F1, which is within run-to-run noise.- The
Z-only network used to predict no defaults at all (precision, recall and F1 of zero at the 0.5 threshold). With class weighting and a tuned threshold it now reaches recall 0.60 and F1 0.43, and its AUROC (0.70) is above chance. - Each ablation is a single run with one seed, so small differences are not reliable.
| Metric | This repo (V2) | Paper target |
|---|---|---|
| Accuracy | 78.2% | about 90% |
| Precision | 50.6% | about 89% |
| F1-Score | 54.4% | about 88% |
| AUROC | 77.3% | about 94% |
The paper's numbers are not reproduced. The remaining gap is large and mostly in AUROC, which no threshold change can fix.
Validation AUROC from prior_regularization_sensitivity.png (25-epoch runs). The KL term is divided by the number of training samples, so λ = 1 is the standard ELBO.
| λ | Validation AUROC |
|---|---|
| 0.01 | 0.7664 |
| 0.1 | 0.7655 |
| 1 | 0.7669 |
| 10 | 0.7642 |
| 100 | 0.5487 |
| 1000 | 0.5195 |
| 10000 | 0.5171 |
xychart-beta
title "Validation AUROC vs KL weight (λ)"
x-axis ["0.01", "0.1", "1", "10", "100", "1e3", "1e4"]
y-axis "AUROC" 0.4 --> 0.8
line [0.7664, 0.7655, 0.7669, 0.7642, 0.5487, 0.5195, 0.5171]
With the KL scaling fixed, AUROC is flat up to λ = 10 and falls to near chance from λ = 100, when the prior overwhelms the data term. Before the fix, AUROC collapsed already at λ = 0.01. The curve shows a plateau and then a drop, not the rise-then-fall shape described in the paper.
uncertainty_distribution.png separates 3,518 correct predictions from 982 incorrect ones (at the tuned threshold). In the latest run the two variance distributions overlap almost completely, so the MC variance does not separate right from wrong predictions here. An earlier run with unweighted loss showed a heavier tail of high variance for wrong predictions, so this property is not stable across training setups. The mean variance is small (0.0018) because the posterior spread is small (ρ starts at -3.0). Treat the uncertainty output as unvalidated until it is checked with calibration measures (see Next steps).
| Figure | File | Shows |
|---|---|---|
| Loss curves | outputs/figures/loss_vs_epoch.png |
Total (ELBO) train loss, validation loss and weighted task BCE over 100 epochs |
| t-SNE | outputs/figures/tsne_visualization.png |
2D projection of the 59-dim fused space for 2,000 test points, with the highest-uncertainty quartile marked |
| λ sensitivity | outputs/figures/prior_regularization_sensitivity.png |
Validation AUROC against λ on a log axis (λ from 0.01 to 10,000) |
| Uncertainty KDE | outputs/figures/uncertainty_distribution.png |
Variance density for correct vs incorrect predictions at the tuned threshold |
| Component | Setting | Value |
|---|---|---|
| Data | Split | 70 / 15 / 15, stratified, seed 42 |
| BN bins | 4 quantile bins, edges from the training split | |
| Money transform | `sign(x) · log1p( | |
| Graph | k | 5 |
| Metric | Euclidean, symmetrized, de-duplicated | |
| GAT | Layers | 30 → 64 → 32, 1 head each |
| Dropout | 0.5 | |
| Optimizer | Adam, lr 0.005, weight decay 1e-4 | |
| Loss | BCE with pos_weight = n_neg / n_pos (about 3.5) |
|
| Epochs and patience | 100, patience 15 on validation loss; best epoch restored | |
| Seed | 42 | |
| Bayesian network | Search | Hill-climb, BIC, max_iter=100 |
| Required edges | PAY_6 → PAY_5 → … → PAY_0 | |
| Forbidden edges | Y → any |
|
| Prior | BDeu, equivalent sample size 10 | |
| Inference | Variable elimination | |
| BNN | Architecture | 59 → 64 → 32 → 1 |
| Prior σ | 0.1 | |
| ρ init | -3.0 | |
| Loss | weighted BCE (pos_weight about 3.5) + λ · KL / N, λ = 1 |
|
| Checkpoint | Best validation loss | |
| Optimizer | Adam, lr 0.001 | |
| Epochs and batch | 100, batch 128 (train), 256 (val) | |
| Seed | 42 | |
| Inference | MC passes | T = 50 (30 in the λ sweep) |
| Threshold | Tuned on validation to maximize F1 (0.05 to 0.95, step 0.01); 0.56 in the latest run | |
| λ sweep | Values | 0.01, 0.1, 1, 10, 100, 1000, 10000 |
| Per-run training | 25 epochs, lr 0.002 | |
| Ablation | Per-run training | 60 epochs, lr 0.001, λ = 1, weighted loss, validation-tuned threshold |
| t-SNE | Sample and perplexity | 2,000 points, 30 |
- Python 3.11 or newer
uv(recommended) orpip
git clone https://github.com/Tetra4ge/GraphBayesRisk.git
cd GraphBayesRiskOption A, pip (complete list):
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtOption B, uv:
uv sync
uv pip install torch_geometric scikit-learn tqdmpyproject.toml currently omits torch_geometric, scikit-learn and tqdm, so uv sync alone is not enough. See Known issues.
The dataset is already in data/raw/. To use a fresh copy, download default_of_credit_card_clients.csv from UCI into that folder.
python src/main.pygantt
title Pipeline stages (relative cost, not measured times)
dateFormat X
axisFormat %s
section Prepare
data_prep.py (cleaning, splits, graph) :a1, 0, 1
section Embeddings
gnn_model.py (GAT training) :a2, 1, 2
section Latent factors
bayes_network.py (row-by-row inference) :crit, a3, 3, 6
section Model
bnn_model.py (100 epochs) :a4, 9, 2
section Evaluate
evaluate.py (MC, t-SNE, sweep, ablation) :a5, 11, 5
The Bayesian-network stage is the slowest. It runs variable-elimination queries for every row of all three splits, 5 groups per row, in a Python loop. The bars above are rough relative sizes, not timings; the code does not record timings.
python src/data_prep.py # splits, two views, graph
python src/gnn_model.py # GAT, writes graph_embeddings_*.csv
python src/bayes_network.py # BN, writes latent_z_*.csv
python src/bnn_model.py # BNN training
python src/evaluate.py # needs a trained BNN checkpointRun them in this order from the repo root. Each stage reads files written by an earlier one, and bnn_model.py needs both the H and Z files.
| What | Where |
|---|---|
| Test metrics | outputs/evaluation_metrics.csv |
| Ablation | outputs/ablation_results.csv |
| Training curves | outputs/bnn_training_curves.csv |
| Figures | outputs/figures/ |
| Models | models/saved_models/, outputs/models/ |
| Package | Version | Used for |
|---|---|---|
torch |
≥ 2.13.0 | BayesLinear, BNN and GAT training |
torch_geometric |
≥ 2.8.0 | GATConv, GCNConv, Data |
pgmpy |
≥ 1.1.2 | Structure learning, CPD estimation, inference |
pandas |
≥ 3.0.5 | CSV handling |
numpy |
≥ 2.4.6 | Arrays and NPY files |
scikit-learn |
any | StandardScaler, train_test_split, NearestNeighbors, metrics, t-SNE |
matplotlib |
≥ 3.11.1 | Plots |
seaborn |
≥ 0.13.2 | t-SNE scatter and KDE |
tqdm |
any | Progress bar in latent extraction |
requirements.txt lists all nine. pyproject.toml lists only torch, pgmpy, pandas, numpy, matplotlib and seaborn.
timeline
title Commit history (oldest to newest)
V1 build : Data prep and Bayesian network pipeline
: BNN model with training and evaluation
: README and project docs
V1 outputs : Latent features and BNN training outputs
: Loss plots, λ sensitivity, t-SNE, saved weights
: Methodology doc and flowchart images
V2 planning : Dependencies and gitignore
: New architecture with GNN integration
: Plan v2 and BN refactoring plan
V2 build : Graph construction pipeline
: GAT model and embedding extraction
: Feature fusion of H and Z
: End-to-end test run and GAT + BNN integration
| Commit | Message |
|---|---|
89781dd |
Generate and export processed datasets, implement BN pipeline |
67b0277 |
Implement BNN with training and evaluation pipeline |
9e5fcc7 |
Add project README |
97bdb31, 2f87e10 |
Latent representations, loss plots, sensitivity analysis, t-SNE, weights |
c2f9d49, 4102da2, 8200fcc |
Flowchart images and methodology doc |
477ec83 |
Requirements and gitignore |
e8e6769 |
New architecture with GNN integration strategy |
a399c6a, 8e6ba35 |
Plan v2 and BN refactoring plan |
ae37f7b |
Complete graph construction pipeline |
66665bc |
GAT model and graph embedding extraction |
ee2940e |
Feature fusion of GNN embeddings and BN latent features |
cd4b9a4 |
End-to-end pipeline test and artifact refresh |
b72ae48 |
GAT and BNN hybrid integration |
| File | Content |
|---|---|
| docs/Abstract.md | Project abstract, pipeline diagram and four core contributions |
| docs/METHODOLOGY.md | How 23 raw features become latent factors and why they are fused (written for V1) |
| docs/ppt_slide_content.md | Slide-by-slide presentation content |
| docs/Methodology.png | Methodology flowchart image |
| replication_plan.md | Original V1 replication plan |
| phase/plan_v2.md | V2 architecture and four-phase plan |
| phase/Phase1_v2.md | Graph construction |
| phase/Phase2_v2.md | GNN model |
| phase/Phase3_v2.md | Bayesian network refactor and fusion |
| phase/Phase4_v2.md | BNN training and evaluation |
| papers/ | The replicated paper and a related PDF |
flowchart LR
P1["Phase 1<br/>Graph construction"] --> P2["Phase 2<br/>GAT embeddings H"]
P2 --> P3["Phase 3<br/>BN latent Z + fusion"]
P3 --> P4["Phase 4<br/>BNN + evaluation"]
| Issue | Fix |
|---|---|
GAT early stopping restored the last epoch, not the best (state_dict().copy() is shallow) |
The best state is now a deep clone of every tensor |
pyproject.toml lacked torch_geometric, scikit-learn, tqdm, so uv sync could not run the pipeline |
Added with uv add; uv.lock regenerated |
| BN bin edges were computed on all rows before splitting | Edges now come from the training split only |
Duplicate removal happened after ID was dropped and removed 35 legitimate rows |
Duplicates are dropped with ID present, so only the one non-numeric row is removed (29,999 rows kept) |
| Misleading "undirected edges" log line | Prints undirected pairs and directed entries separately |
| Low recall (about 0.37): unweighted loss and a fixed 0.5 threshold | Class-weighted BCE (GAT and BNN) plus an F1-optimal threshold tuned on validation |
| λ sweep collapsed to chance for λ ≥ 0.01 (KL summed, not divided by N) | KL is now divided by N; the sweep is flat up to λ = 10 |
| BNN always kept the last epoch | The best-validation weights are restored |
.squeeze() broke on a batch of size 1 |
Replaced with .squeeze(-1) |
| Ablation and λ sweep were not seeded | torch.manual_seed(42) before every model |
main.py only worked from the repo root |
It changes into the repo root itself |
| # | Issue | Where | Effect |
|---|---|---|---|
| 1 | Z adds little over H. Fusion improves AUROC by about 0.001 | Ablation | The BN branch is mainly an interpretability layer here |
| 2 | MC variance does not separate right from wrong predictions in the latest run | uncertainty_distribution.png |
The uncertainty claim is unvalidated; calibration (ECE, reliability diagram) is needed |
| 3 | Precision and accuracy dropped when recall was raised | Threshold choice | Expected trade-off; pick the threshold by the real cost of missed defaults |
| 4 | Transductive graph. Validation and test nodes are in the graph, so new customers need the graph rebuilt or an inductive variant | build_knn_graph |
Not deployable as-is for unseen customers |
| 5 | BN fitted and queried on the same training rows. Training-row Z values are in-sample |
extract_latent_features |
Train Z can look cleaner than val and test Z |
| 6 | BN extraction is slow. One inference call per row per group in a Python loop | extract_latent_features |
Dominates pipeline runtime |
| 7 | MC inference cost. 50 forward passes per prediction | run_mc_inference |
About 50 times a point-estimate model |
| 8 | Paper targets not reached (AUROC 0.77 against about 0.94) | Results | See Results |
| 9 | Duplicate checkpoint. bnn_model.pth and bnn_model_v2.pth are written identically |
bnn_model.main |
Redundant file |
| 10 | Empty notebooks. 01_eda.ipynb, 02_bn_experiments.ipynb, 03_bnn_experiments.ipynb are 0-byte files |
notebooks/ |
No interactive exploration yet |
| 11 | Single run, single seed for every reported number | All experiments | No confidence intervals |
| 12 | docs/METHODOLOGY.md and docs/ppt_slide_content.md describe V1 (57-dim fusion with raw features) |
docs/ |
Not updated for the GAT branch |
| 13 | __pycache__ files are tracked by git although .gitignore lists them |
src/__pycache__/ |
Noisy diffs |
flowchart TB
A["Measure calibration:<br/>ECE and reliability diagram (2)"] --> B["Multiple seeds,<br/>report mean ± std (11)"]
B --> C["Add baselines:<br/>logistic regression, XGBoost, MLP (1)"]
C --> D["Cross-fit Z to remove<br/>in-sample bias (5)"]
D --> E["Speed up BN inference<br/>by caching queries (6)"]
E --> F["Inductive GNN for<br/>unseen customers (4)"]
F --> G["Fill notebooks,<br/>update METHODOLOGY (10, 12)"]
A read-only web dashboard over the pipeline outputs: React (Vite, TypeScript) in frontend/ and FastAPI in backend/. It does not train anything.
flowchart LR
E["src/evaluate.py"] --> P[("outputs/predictions_test.csv")]
E --> M[("outputs/evaluation_metrics.csv")]
Z[("data/processed/latent_z_test.csv")]
P --> B["backend (FastAPI)"]
M --> B
Z --> B
B -- "JSON over /api" --> F["frontend (React)"]
| Page | Shows |
|---|---|
/ |
Landing page |
/results |
Cut-off slider that recomputes precision, recall, F1 and the confusion matrix, plus the before/after recall |
/customers |
Ranked, filterable table of the 4,500 test customers; a row opens the 27 Bayesian-network factor values |
Run it from the repo root:
uv sync
cd frontend && npm install && cp .env.example .env && cd ..
uv run uvicorn backend.main:app --reload --port 8000
cd frontend && npm run dev # http://localhost:5173Do not run uv sync inside backend/; it shares the root environment and would remove the pipeline packages. API reference and tests are in backend/README.md; the frontend is described in frontend/README.md; the plan is in plan/PLAN.md.
evaluate.py also writes outputs/predictions_test.csv (id, y_true, risk, uncertainty) for the dashboard. It comes from the same Monte Carlo pass as evaluation_metrics.csv, so the numbers match the tables above. The MC variance is shown as experimental, since it did not clearly separate right from wrong predictions in the latest run.
- Replicated paper: Uncertainty-Aware Robust Financial Risk Discrimination via Bayesian Network and Bayesian Neural Modeling (see
papers/). - Dataset paper: Yeh, I. C., & Lien, C. H. (2009). The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2), 2473-2480. Semantic Scholar
- Dataset: UCI ML Repository, Default of Credit Card Clients
- Libraries: PyTorch, PyTorch Geometric, pgmpy, scikit-learn
Apache License 2.0. See LICENSE.