Uncertainty-Routed 3-Tier WAF

FastAPI web-application firewall with a cascading decision pipeline:

• Tier 1: deterministic regex rules.
• Tier 2: calibrated char-LSTM binary classifier.
• Tier 3: supervised specialist classifier on the tier-2 uncertain slice with Mondrian conformal abstention.

Training and evaluation are CSIC-only. The final test split stays untouched until report generation.

Requirements

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

CUDA is required for the artifact-backed tier-3 training and evaluation paths.

export WAF_INFERENCE_DEVICE=cuda

Data Preparation

python -m train.preprocess \
  --csic data/Web-Application-Attack-Datasets/OriginalDataSets/csic_2010 \
  --out data/unified.jsonl

python -m train.holdout \
  --input data/unified.jsonl \
  --output data/benign_holdout.jsonl

Training Pipeline

Run these steps in order.

1. Train Tier 2

python -m train.train_ml \
  --data data/unified.jsonl \
  --out waf/models \
  --sequence-handling masked_mean_max \
  --tier2-text-surface v2_payload_first

2. Calibrate Tier 2

python -m train.calibrate \
  --data data/unified.jsonl \
  --model waf/models

3. Build Tier-2 OOF Cache

python -m train.build_tier2_oof \
  --data data/unified.jsonl \
  --model waf/models \
  --out waf/models/tier2_oof_train.npz \
  --folds 5

4. Train Tier 3 Specialist

python -m train.train_specialist \
  --data data/unified.jsonl \
  --models waf/models \
  --folds 5 \
  --device cuda

5. Calibrate Tier 3 Conformal

python -m train.calibrate_specialist \
  --models waf/models \
  --alpha 0.10

6. Sync Runtime Profile

python -m train.sync_profile_thresholds \
  --models waf/models

Artifact Checklist

After a full run, these should exist:

waf/models/ml_waf.pt
waf/models/ml_config.json
waf/models/calibration_cache.npz
waf/models/tier2_oof_train.npz
waf/models/specialist.pt
waf/models/specialist_config.json
waf/models/specialist_oof_pred.npz
waf/models/specialist_conformal.json
data/benign_holdout.jsonl

Run The App

export WAF_INFERENCE_DEVICE=cuda
uvicorn waf.main:app --reload

Health:

curl http://127.0.0.1:8000/health

Inspect:

curl -X POST http://127.0.0.1:8000/inspect \
  -H 'content-type: application/json' \
  -d '{"method":"GET","url":"/?q=<script>alert(1)</script>","headers":{},"body":""}'

Evaluation

Preferred bundle:

python -m eval.reporting.report_bundle \
  --data data/unified.jsonl \
  --rules waf/rules/live.conf \
  --models waf/models \
  --include-http-latency

Key outputs:

eval/results/runs/<timestamp>/specialist_benchmark.json
eval/results/runs/<timestamp>/specialist_benchmark.csv
eval/results/runs/<timestamp>/specialist_audit.json
eval/results/runs/<timestamp>/adversarial.json
eval/results/runs/<timestamp>/latency_router.json
eval/results/runs/<timestamp>/latency_http.json
eval/results/report/specialist_audit.json

Focused commands:

python -m eval.experiments.benchmark --data data/unified.jsonl --rules waf/rules/live.conf --models waf/models
python -m eval.specialist.specialist_benchmark --data data/unified.jsonl --rules waf/rules/live.conf --models waf/models
python -m eval.specialist.specialist_audit --data data/unified.jsonl --rules waf/rules/live.conf --models waf/models
python -m eval.experiments.adversarial --data data/unified.jsonl --rules waf/rules/live.conf --models waf/models --n 100
python -m eval.experiments.latency --rules waf/rules/live.conf --models waf/models --mode both --rps 100 --duration 60

Dashboard

streamlit run dashboard/app.py

The dashboard reads:

data/runtime/decisions.jsonl
data/runtime/rule_installs.jsonl
eval/results/report/specialist_audit.json

Testing

Fast checks:

python -m compileall train waf eval tests dashboard
pytest -m "not integration"

Focused specialist/runtime checks:

pytest \
  tests/waf/e/test_conformal.py \
  tests/train/c/test_specialist_model.py \
  tests/waf/e/test_tier3_specialist.py \
  tests/waf/a/test_router_thresholds.py \
  tests/waf/a/test_router_tier3.py \
  tests/waf/b/test_runtime_state.py \
  tests/dashboard/test_dashboard_loaders.py \
  tests/eval/b/test_benchmark.py

Repo Layout

waf/              Runtime routing, policy, rule validation, specialist scoring
train/            Preprocessing, splits, tier training, calibration, profile sync
eval/             Benchmark, specialist audit, adversarial tests, latency, report bundle
dashboard/        Streamlit dashboard
tests/            Unit and integration tests