feat(verify): the report gate — prove-or-suppress (/verify) [Verification Spine 2/3]

[shivsin25]

The claim this unlocks

"Agentic Bug Hunter will not report a bug it cannot independently prove."

That single guarantee is the difference between a trusted security tool and AI slop. This PR is the gate that enforces it.

Where it sits — the Verification Spine (three linked PRs)

/poc   (#144, merged)   produces PROOF        — the reproducible evidence bundle
/verify (this PR)        the GATE              — re-derives a finding; PROVEN → reportable, else SUPPRESSED
/bench (PR #155)         MEASURES the gate     — proves its false-accept rate is ~0
report path (next PR)    ENFORCES it           — nothing unproven ever ships

/poc gives us proof, /verify decides on it, /bench measures how trustworthy that decision is, and the report wiring (next PR) makes it mandatory. Each is useful alone; together they're a verification system.

The policy (this is the whole product)

decide() is a pure, total function — every input path returns a verdict:

verdict	when	reportable
✅ PROVEN	bundle exists and re-derivation confirms the marker	yes
❌ REFUTED	re-derived, but the marker is gone / endpoint safe	no
🚫 UNPROVEN	no bundle, or no confirmable marker at all	no
⏸ INCONCLUSIVE	unreachable / missing secret / mutating method held	no

Only PROVEN ships. A test (test_only_proven_is_ever_reportable) enumerates every combination of inputs and asserts that no other verdict is ever reportable — so if someone adds a verdict later, they're forced to consciously decide its reportability. "Can't confirm" is deliberately not "probably fine": the burden of proof is on the finding. That stance is what drives false accepts toward zero — and it's exactly what /bench can measure.

Merge-safety: depends only on merged code

Re-derivation reuses only poc_bundler (already in main) — capture (SSRF-guarded) + parse_raw_request + sha256_hex. It deliberately does not import the still-open /replay branch, so this PR stands alone and passes CI on main today. It composes with /replay but does a different job: replay monitors a known bug over time; verify makes the one-shot "may this be reported?" decision and emits verification.json for the report step.

Safety

• SSRF-guarded re-send (tools/safe_http.py).
• PUT/DELETE/PATCH are never re-fired without --confirm-unsafe — a verification must not re-trigger a destructive action; it holds as INCONCLUSIVE.
• Secrets are read from env vars only and never written into verification.json (asserted by a test).

Quality

• 17 tests (tests/test_verifier.py): every policy branch, the "only PROVEN ships" guardrail, secret resolution, all four verdicts end-to-end offline (via monkeypatched capture), no-secret-leak, and --require-proof gate exit codes.
• Verified live against the repo's demo/app.py: PROVEN while up, UNPROVEN without a marker, INCONCLUSIVE against a dead port.
• Additive: 4 files changed, 599 insertions(+), nothing removed. Pure policy core, thin network layer.

Try it

tools/verifier.py check <poc-bundle-dir> --marker "the-string-that-proves-it" --require-proof
tools/verifier.py sweep findings/            # verify everything; suppress the unprovable

Next PR (PR-3) wires this gate into the report path so an unproven finding can never be written into a report — completing the Spine.

🤖 Generated with Claude Code

[shuvonsec]

Strong foundation - pure/total decide(), safe json-only deserialization, SSRF-guarded reuse of poc_bundler, and a clean stable interface for #157. Verified 17 tests pass. Two hardening asks on the false-verdict edges (the whole point of the gate), plus test gaps:

1. (false-PROVEN) A marker that merely appears in the fresh response mints PROVEN, with no cross-check against the original bundle response and status ignored - a server banner or a 500 page that reflects input would count as "proof." Please require the marker to also be present in the stored original response before treating fresh presence as proof, and record the fresh response's own sha (evidence_sha256 currently only stores the original's hash, so the record doesn't bind to what was re-fetched).
2. (false-SUPPRESS) The marker is matched in the response BODY only. Header-based classes (open-redirect Location, CORS Access-Control-Allow-Origin, Set-Cookie, cache poisoning) will be wrongly REFUTED and their real findings suppressed. poc already exposes response_headers - please also search headers, or explicitly document the body-only limitation so #157 doesn't assume every class is gate-able.
3. (tests) Add: the --confirm-unsafe positive path (an unsafe method actually re-fires when confirmed), load_marker precedence (override > manifest > sidecar), and find_bundles schema filtering.

Also: sweep --require-proof exits 1 if ANY bundle isn't reportable - confirm that blunt behavior is intended before it's wired into CI.

[shivsin25]

@shuvonsec thanks, these were the right things to push on — the whole point of the gate is the false edges. all fixed.

1. false-PROVEN: yeah, matching only the fresh response meant a banner or reflected-input-on-a-500 could pass. decide() now requires the marker to be in the ORIGINAL recorded response before its presence now counts as proof — so PROVEN needs it in both original and fresh. and you were right the record didn't bind to what was re-fetched, it only had the original hash; it now also stores rederived_sha256 + rederived_status.

2. false-SUPPRESS: good catch, body-only matching would've refuted every header-based class. now matching across headers + body on both the baseline and the fresh response, so open-redirect Location / CORS ACAO / Set-Cookie work. added a test with a Location-header marker that comes back PROVEN.

3. also added the initial-host SSRF guard while here — safe_urlopen only guards redirects and the bundle URL is untrusted, so a tampered bundle could've pointed the first hop at metadata/localhost. it's validated against the same blocklist before sending and holds INCONCLUSIVE.

tests: added the ones you asked for — --confirm-unsafe positive path (it actually re-fires now, asserted), load_marker precedence (override > manifest > sidecar), and find_bundles schema filtering — plus the false-PROVEN and header-marker cases. 26 pass.

on sweep --require-proof exiting 1 if any bundle isn't reportable: yeah that's intentional, it's meant as an all-or-nothing gate, but it's not wired into CI (unlike bench) — documented that in verify.md so it's a conscious choice if anyone adds it.

rebased on main + fixed the CLAUDE.md conflict too. and agreed on merge order — this (#156) should land before #157.