Calibrated severity risk for Israeli road accidents — a deliberately simple model behind a disproportionately rigorous production wrapper. The interesting artifact is not the classifier; it is the machinery that establishes what the classifier can and cannot honestly claim.
Status: v0.1.0. Ingestion, the data contract, conformal calibration, the fairness audit, the H3 hotspot surface, the serving container, and the deployment gate are in place and measured. The hosted demo, drift monitoring, and Terraform target are not yet deployed and are listed as such under Not built yet. Every number below is produced by a named command on 49,941 real accidents, and nothing is claimed that a command cannot reproduce.
Most portfolio ML repositories answer "can you fit a model?". That question is no longer interesting. This one answers a harder one: given a model that is genuinely weak, can you build a system that is still safe to deploy — and can you prove it?
The honesty is the deliverable. On this data, road and environment conditions predict accident severity badly: macro F1 0.458, and fatal-accident recall 0.024. A model that flags 2.4% of fatal accidents is not a model you ship behind a single-label API. So the system does not serve labels. It serves calibrated prediction sets with a distribution-free coverage guarantee, and a CI gate that refuses to ship when that guarantee, the fairness spread, or the latency budget breaks.
The single most useful result in this repository is a negative one, and it is in Results: the textbook conformal configuration delivers its promised 90% coverage while covering the fatal class 15.1% of the time.
flowchart TB
subgraph ingest["Ingestion — road_risk_il.data"]
CKAN["data.gov.il CKAN DataStore<br/>package 2023-puf · 5 years"]
CONTRACT{{"Data contract<br/>row drift · columns · target · codes"}}
CACHE[("Parquet cache<br/>49,941 accidents")]
CKAN -->|"paged datastore_search"| CONTRACT
CONTRACT -->|"pass"| CACHE
CONTRACT -.->|"fail"| STOP["DataContractError"]
end
subgraph features["Features — road_risk_il.features"]
TEMPORAL["SHAA bin → hour<br/>cyclical encoding"]
GEO["ITM EPSG:2039 → WGS84<br/>→ H3 hexagons"]
SPLIT{{"Administrative geography<br/>held out of the matrix"}}
end
subgraph model["Model — road_risk_il.models"]
GBM["XGBoost severity classifier<br/>no resampling"]
CONF["Conformal calibration<br/>LAC · class-conditional"]
GBM -->|"probabilities"| CONF
end
subgraph audit["Audit"]
FAIR["Fairness audit<br/>per-group recall + Wilson CIs"]
ABL["Conformal ablation<br/>6 configurations"]
end
subgraph serve["Serving"]
API["FastAPI /predict<br/>returns a SET, not a label"]
GATE{{"Deployment gate<br/>coverage · fairness · SLO"}}
end
CACHE --> TEMPORAL --> GBM
CACHE --> GEO --> HOT["H3 hotspot surface<br/>precise geocoding only"]
CACHE --> SPLIT -->|"audit only"| FAIR
CONF --> API
CONF --> FAIR
CONF --> ABL
FAIR --> GATE
CONF --> GATE
GATE -->|"pass"| SHIP["Container ships"]
GATE -->|"fail"| BLOCK["CI blocks the build"]
style STOP fill:#7f1d1d,color:#fff
style BLOCK fill:#7f1d1d,color:#fff
style SHIP fill:#14532d,color:#fff
style GATE fill:#78350f,color:#fff
style SPLIT fill:#1e3a5f,color:#fff
style CONTRACT fill:#78350f,color:#fff
All figures below come from just train and just ablate on the real PUF microdata: 49,941 injury
accidents, 2020–2024, split 29,964 train / 9,988 calibration / 9,989 test.
The model is weak, and that is reported rather than hidden
just train → macro F1 0.4583
| Severity | Recall | Test accidents |
|---|---|---|
| fatal (קטלנית) | 0.0237 | 337 |
| severe (קשה) | 0.3354 | 2,251 |
| slight (קלה) | 0.9637 | 7,401 |
The classifier finds 8 of 337 fatal accidents. Conditions recorded in the PUF — road type, lighting, weather, surface, speed limit, time — simply do not carry that signal. Impact speed, vehicle mass, restraint use, and time-to-care are what determine survival, and none of them are in this dataset. See ADR 0001.
just train → conformal coverage, nominal 0.90, LAC + class-conditional:
| Split | Marginal | Fatal | Severe | Slight | Mean set size |
|---|---|---|---|---|---|
| Random (guarantee applies) | 0.895 | 0.935 | 0.899 | 0.892 | 2.19 |
| Temporal shift (2024 held out) | 0.885 | 0.943 | 0.958 | 0.848 | 2.25 |
The random split is where conformal's exchangeability assumption holds, and coverage lands on nominal. The temporal split deliberately violates that assumption, because that is the situation a deployed model is actually in — and coverage on the largest class degrades to 0.848, a 5.2-point shortfall. Reporting only the first row would have been a nicer number and a less honest one.
just ablate → n_test 9,989, nominal coverage 0.90:
| Configuration | Marginal coverage | Worst-class coverage | Mean set size |
|---|---|---|---|
| LAC / marginal | 0.903 | 0.151 |
1.43 |
| LAC / class-conditional ← default | 0.895 | 0.892 | 2.19 |
| APS / marginal | 0.899 | 0.068 |
1.49 |
| APS / marginal +randomised | 0.904 | 0.181 |
1.48 |
| APS / class-conditional | 0.902 | 0.895 | 2.64 |
| APS / class-conditional +randomised | 0.958 | 0.908 | 2.44 |
Row 1 keeps its promise exactly — 90.3% marginal coverage — while covering fatal accidents 15.1% of the time. Row 3 manages 6.8%. Nothing is broken: a marginal guarantee is satisfied most cheaply by covering the abundant class and abandoning the rare one, and this data is 13.5:1 slight-to-fatal. A safety system built on the default configuration would pass its own tests and fail the people it exists to protect.
Class-conditional calibration fixes it, and the cost is visible in the last column: sets widen from 1.43 to 2.19 of 3 classes. That is the trade, stated rather than buried. See ADR 0003.
A night-time accident on a non-urban road, poor lighting, wet surface, high speed limit — from the running container:
{ "prediction_set": [1, 2], "most_likely": 3,
"options": [ {"code":1,"probability":0.240}, {"code":2,"probability":0.285}, {"code":3,"probability":0.474} ],
"coverage_level": 0.9, "calibration": "LAC / class-conditional" }The most likely single outcome is slight (p=0.474), and the calibrated set is {fatal, severe}
— which excludes it. That is not a bug. Under class-conditional calibration the fatal class carries a
lenient threshold and the common class a strict one, so the set is saying "I cannot certify the mild
outcome under these conditions." An argmax API would have answered "slight" and moved on.
just train reports this directly as the argmax-excluded rate: 0.094 — 9.4% of held-out sets
omit their most likely class. It is a reported metric rather than a footnote because a reader who
does not expect it will file it as a bug, and because a rate near zero would mean the per-class
thresholds had collapsed together and the calibration had quietly gone marginal. Pinned by
tests/test_conformal.py::test_class_conditional_sets_may_exclude_the_argmax.
just deploy-check → grouping by district (MAHOZ), 7 groups with ≥30 fatal accidents:
| Contract | Measured | Threshold |
|---|---|---|
| Fatal-recall spread | 0.078 | 0.15 |
| Fatal set-coverage spread | 0.079 | 0.15 |
Fatal recall ranges from 1.8% (Jerusalem district) to 9.6% (Southern district). The model is roughly five times better at flagging fatal accidents in some districts than others, on top of being poor at it everywhere. Per-group recalls carry Wilson confidence intervals, and groups with too few fatal accidents to estimate are excluded and counted rather than reported as noise.
An earlier version of this audit reported a spread of 0.170 and failed the gate on the strength of one group with 16 fatal accidents and 3 detections. The fix was to correct the metric, not to raise the threshold — recorded in ADR 0004 because the temptation to do the opposite was real.
just deploy-check on real data:
| Contract | Measured | Threshold | Verdict |
|---|---|---|---|
| Conformal coverage, worst class | 0.8920 | 0.8700 | PASS |
| Fatal-recall spread across MAHOZ | 0.0782 | 0.1500 | PASS |
| Fatal set-coverage spread across MAHOZ | 0.0785 | 0.1500 | PASS |
| Scoring latency p95 | 0.26 ms | 150 ms | PASS |
The gate's latency contract is measured on the in-process scoring path. The end-to-end HTTP
number is measured separately, against the running container — just loadtest, 400 requests:
| min | p50 | p95 | p99 | max | |
|---|---|---|---|---|---|
HTTP POST /predict |
1.17 ms | 1.36 ms | 1.79 ms | 1.99 ms | 2.33 ms |
Both are reported because they measure different things, and quoting the in-process figure as an end-to-end SLO would be the easy dishonesty here.
| Tests | 106 passing, 96.9% coverage · verified on Python 3.11 and 3.12 from a cold clone |
| Type checking | mypy --strict clean across 21 modules |
| Lint | ruff clean (E, F, I, N, UP, B, SIM, RUF, ANN, D) |
| Spatial correctness | reprojection tested against 5 control points to 0.03° |
| Offline/online parity | asserted by test, not assumed |
Works from a cold clone. Requires uv and Python 3.11 or 3.12.
git clone https://github.com/DataScientist13/road-risk-il && cd road-risk-il
just setup # deps, pre-commit, and the git-level commit-message guard
just test # 106 tests, fully offline
just ingest # ~50k accidents from data.gov.il, cached as Parquet
just train # fit + calibrate; writes artifacts/ and prints every number above
just ablate # reproduce the conformal ablation table
just deploy-check # the CI gate: coverage, fairness, latency
just hotspots # H3 hexagons ranked by harm, with the geocoding caveatNo API keys, no accounts, no manual downloads. just test never touches the network — CI runs
against a synthetic fixture that reproduces the real severity mix to within 2 percentage points.
Serve it:
just image && docker run -p 8000:8000 road-risk-il:local
curl -s localhost:8000/health
curl -s -X POST localhost:8000/predict -H 'content-type: application/json' \
-d '{"SHAA": 85, "TEURA": 4, "PNE_KVISH": 5, "MEHIRUT_MUTERET": 7}' | jqThe response carries the set, the per-class probabilities, the coverage level, the calibration mode, and a caveat stating that the set — not the top label — is the calibrated answer.
CBS injury-accident PUF microdata, 2020–2024, from data.gov.il package
2023-puf — open licence, no authentication.
| Year | Accidents |
|---|---|
| 2020 | 10,836 |
| 2021 | 11,554 |
| 2022 | 10,404 |
| 2023 | 8,832 |
| 2024 | 8,315 |
| Total | 49,941 |
Ingestion goes through the CKAN DataStore API rather than the file CDN, which sits behind an AWS WAF challenge; the DataStore SQL endpoint is not exposed, so aggregation is local. The client identifies itself, pauses between pages, and caches everything — this is public infrastructure. See ADR 0002.
Five coding quirks in this dataset will silently corrupt results if assumed away — coordinates that are metres not degrees, a time field that is a 15-minute bin not an hour, inverted severity codes, a geocoding-precision tier, and a dictionary resource that arrives malformed. All five are documented with their tests in ADR 0005.
Read this section before using anything here for a decision that matters.
- The model barely predicts fatality. Fatal recall is 0.024. Any use of the point prediction for triage would be indefensible. The prediction set is the usable output.
- Sets are wide — 2.19 of 3 classes on average. This is honest, not a bug: conditions are weakly informative, so a calibrated system declines to narrow. A tighter set would be a less truthful one.
- Coverage degrades under temporal shift, to 0.848 on the slight class when calibrating on 2020–2023 and testing on 2024. The 90% guarantee holds under exchangeability, and deployment forward in time is not exchangeable. Recalibrate on recent data before trusting it operationally.
- Injury accidents only. The PUF covers accidents with casualties reported to police. Property-damage-only and unreported accidents are absent, so this is not a model of "road risk" in general — it is a model of severity given a reported injury accident.
- This is not accident probability. It says nothing about how likely an accident is at a location. Conditioning on an accident having occurred is not the same as predicting one.
- Only 77.9% of accidents are precisely geocoded (38,916 of 49,941). Hotspot aggregation uses
tier-1
STATUS_IGUNonly, and every spatial output carries that caveat. Coarse geocoding is not randomly distributed, which plausibly biases the spatial view toward better-mapped areas — very likely against the peripheral localities that limitation 7 also fails to cover. - Fairness auditing is incomplete. District and locality audits ship; the socioeconomic-cluster audit needs a CBS locality-to-cluster join that is not yet wired in. Localities with fewer than 30 fatal accidents cannot be audited at all, which excludes exactly the small peripheral localities where the concern is sharpest.
MAHOZgroup 9 and Judea & Samaria carry small counts and are reported as the source codes them, without editorial adjustment in either direction.- Latency is measured on a laptop, single-client. HTTP p95 is 1.79 ms against a local container with no concurrency, no network hop, and a warm model. It says nothing about behaviour under load, and there is no autoscaling or concurrency test.
- No causal claim whatsoever. "Poor lighting associates with worse outcomes" is not "improving lighting will reduce severity". Nothing here supports an intervention decision.
Listed because a README that omits its gaps is a less useful README.
- Hosted demo (Hugging Face Space with an RTL Hebrew UI) — planned, not deployed.
- MLflow registry with staging→production promotion — the gate exists; the registry does not.
- Evidently drift monitoring and shadow scoring — designed, not wired.
- Docs site (mkdocs on GitHub Pages). The ADRs are markdown in
docs/adr/for now. - Terraform against a free-tier target, and kind-based k8s manifests validated in CI.
- SHAP explanations — the dependency is declared under the
explainextra but unused. - Socioeconomic-cluster fairness audit (needs the CBS locality join).
src/road_risk_il/
config.py settings and gate thresholds, env-overridable
data/ CKAN client · dictionary decoder · ingest + contract · fixture
features/ ITM→WGS84 + H3 · temporal decoding · matrix assembly
models/ XGBoost training · conformal · fairness · ablation
ops/gate.py the deployment gate
service/app.py FastAPI, returns prediction sets
docs/adr/ five decision records
tests/ 106 tests, fully offline
MIT. The underlying data is published by the Israeli Central Bureau of Statistics under an open licence; this repository redistributes none of it, only the code that fetches it.