feat(oracle): self-learning lead/finding prioritizer (/oracle) — the memory flywheel that decides what to hunt

[shivsin25]

The one-line case

The repo already stores every finding's outcome — and never learns from it. Oracle is the model that finally makes the hunt-memory flywheel decide what to hunt next.

Why this needs to exist (no hand-waving)

Every hunt writes outcomes to journal.jsonl — confirmed / rejected / false_positive, plus payout, severity, vuln class, endpoint, tech stack. That is a labelled dataset of what actually pays for this hunter on these surfaces. Yet the lead board still ranks purely by a hardcoded regex priority (high/med/low), identical for every program and every hunter. The maintainers already flagged this exact hole:

docs/TODOS.md TODO-6: "The memory → hunt feedback loop never spins up in practice."

Oracle closes that loop. It trains on the hunter's own journal and scores any lead by P(this becomes a real, valuable finding) — turning a dormant data asset into the prioritization brain.

Why it's provably better than what exists (the part that removes doubt)

The static regex scorer is context-blind: it thinks IDOR is always high-value and open-redirect is always low. Real programs disagree — one program dupes every IDOR; another pays big for an open-redirect→ATO chain. Oracle learns that program-specific reality.

oracle eval proves it on your own data with leave-one-out cross-validation against two baselines. On a history where IDOR is always duped and open-redirect always pays:

leave-one-out CV (16 confirmed / 18 rejected):
  model                acc   prec    rec     f1
  Oracle (learned)     1.0    1.0    1.0    1.0
  prior heuristic    0.471  0.471    1.0   0.64
  majority class     0.529      —      —      —
→ Oracle vs prior heuristic: +0.529 accuracy

Oracle correctly down-ranks the duped IDOR (30%) and up-ranks the paying open-redirect (81%) — the static scorer gets both backwards. The lift is measured, not asserted, and it's shown before you trust the model.

Why this model, specifically (engineering judgement)

Bug-bounty history is small (tens–hundreds of labelled findings). A gradient-boosted / neural stack would overfit that and drag in a dependency this repo deliberately avoids (requirements: just requests + mcp). So Oracle uses a smoothed Naive-Bayes log-odds model — the correct tool for the data regime:

• Calibrated & explainable — every score comes with the exact per-feature log-odds that produced it (class:idor -2.84 (seen 0✓/18✗)), so a hunter can see why. No black box.
• Cold-start prior — with little data it falls back to the same high-value intuition the regex scorer encodes, then blends toward the learned model as evidence accumulates (prior → blended → learned). Useful on day one, data-driven by day thirty.
• Pure-stdlib, deterministic — no new dependency, trivially testable, reproducible.

What it does

Command
/oracle train	learn from journal, save oracle_model.json, print CV report
/oracle eval	leave-one-out CV vs prior + majority baselines
/oracle rank <target>	score & rank the target's untouched leads (reuses lead_board.load_ledger)
/oracle score --class idor --url …	score one hypothetical lead, with explanation + expected value

Expected value = P(worth pursuing) × mean historical payout for that class, so ranking optimizes for money, not just probability. Read-only — Oracle never mutates the lead board.

It compounds with what's already here

/poc (merged, #144) and /replay (#152) produce the very outcome labels Oracle learns from: capture → confirm/replay → journal → Oracle gets smarter → better leads. This is the flywheel actually turning.

Safety & quality

• No network, no writes outside the memory dir, no lead-board mutation.
• Console reconfigures to UTF-8 so it can't crash a non-UTF-8 terminal.
• 22 tests (tests/test_oracle.py): feature extraction, that the model learns a real signal and stays a probability, explanations point at the right feature, serialization round-trips, cold-start falls back to prior, LOOCV beats the baselines, data loading, and all four CLI commands. Additive only: 4 files changed, 860 insertions(+).

Try it

tools/oracle.py train           # learn from your journal, see the CV lift
tools/oracle.py rank target.com # work the leads your history says pay

[shuvonsec]

Really like this - JSON persistence (no pickle), div-by-zero guarded throughout, cold-start handled and tested, and the tests actually exercise the learning signal + LOOCV-beats-baseline. Verified 22 tests pass. One accuracy ask (non-blocking):

• LOOCV in train/eval scores folds with _learned_score (pure learned, w=1), but production rank/score call model.score() which blends learned with the cold-start prior weighted w = n/(n+20). On the small histories Oracle targets (e.g. 34 samples -> w0.63) the deployed scorer is only ~63% learned, so the reported CV lift overstates what users actually get. Either run LOOCV through the deployed score(), or label the number as "learned-only upper bound."

[shivsin25]

Great catch, @shuvonsec — you're exactly right that the CV number was measuring _learned_score (w=1) while rank/score serve the blended score() (w = n/(n+20)), so on small histories the reported lift overstated production. Fixed in cf90867 — I did both things you suggested:

• LOOCV now runs through the deployed score() — each held-out fold is scored with the exact blended scorer users receive, so the headline Oracle (deployed) number reflects what they actually get (w included).
• Learned-only is kept as a labelled upper bound — reported separately as Oracle (learned-only, UB), the value the blend converges to as history grows.

The eval table and the headline lift now use the deployed number:

leave-one-out CV (10 confirmed / 10 rejected):
  model                        acc   prec    rec     f1
  Oracle (deployed)            ...            <- what users get (blended)
  Oracle (learned-only, UB)    ...            <- upper bound
  prior heuristic              ...
  majority class               ...
→ Oracle (deployed) vs prior heuristic: +X accuracy

Added 2 tests: one asserts both metrics are reported (and the UB ≥ deployed), and a regression guard that LOOCV actually calls score() (not _learned_score) so this can't silently regress. 24 tests pass.

Also merged the latest main and resolved the CLAUDE.md conflict (kept both the /dashboard and /oracle entries) — the PR is green/mergeable again. Thanks for the careful read