Verifier bake-off: no local verifier clears the second-gate bar (held-out NOT-RECOMMENDED)

[dshakes]

What

This is a pre-registered search for a verifier stronger than OpenJev noul to use as a second gate (specs/verifier-bakeoff.md, committed before any scoring). Every candidate runs locally and needs no API key.

• Data: the same 974 served MBPP answers and hidden-test oracle labels as Study B (111 are wrong). They are split by seeded sha256(id) into 488 dev and 486 held-out answers. The verifier and τ are chosen on dev; the verdict comes from held-out only.
• V0: OpenJev noul.
• V1: OpenJev noul with think: 1024 and samples: 4.
• V2: the proxy's JudgeGate prompt, mirrored verbatim and pinned by a parity test, sent to a local Qwen3-Coder-30B-A3B-Instruct (4-bit MLX, mlx-lm).
• V3: CodeT-style checking. The same coder writes 5 asserts from the task text only, never seeing the candidate, and the candidate runs against them in the fail-closed sandbox.

Result: NOT-RECOMMENDED (the bar is catch ≥ 0.30 and collateral ≤ 0.05)

held-out	catch	collateral	AUC	p50 latency
V0	0.0943	0.0600	0.6268	767 ms
V1	0.0755	0.0624	0.5976	6841 ms
V2	0.0000	0.0000	0.5917	670 ms
V3 (selected on dev)	0.0943	0.0670	0.6646	2870 ms

I checked that V2's zeros are genuine explicit score: 0 judgments; parse failures count as abstains, and there were none. It gives 0 to 17.1% of wrong answers and to 6.5% of right ones, so no τ clears the collateral cap. These candidates already passed Firstpass's gate, so the remaining wrong answers are the hard residual. A frontier-model judge hasn't been tried, because no key was available here.

Also

--fetch-priors now retries failed calls on resume. Previously a keyless first run left every task recorded as done, so a rerun with a valid key skipped them all. Covered by a regression test, which was mutation-tested.

Verification

• fmt 0, clippy -D warnings 0.
• cargo test --workspace --no-fail-fast: 898 passed, 0 failed.
• The two real-clock rate-limiter tests in proxy.rs failed intermittently under heavy concurrent CPU load from the bake-off. They passed 15 of 15 isolated runs, and this PR doesn't touch them.
• The bake-off rerun from caches is byte-identical.
• Mutation-tested: the verdict guard in verifier_bakeoff, and the resume fix.

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking Findings

1. V3 violates the pre-registered “task text only” protocol.
   The V3 prompt includes assert {signature_case} from task.visible_cases.first() (verifier_bakeoff.rs, verifier_bakeoff.rs). For MBPP, visible_cases are derived from reference test_list assertions, including expected outputs (dataset.rs). That conflicts with the spec’s V3 definition: tests are written “from the task text only” (specs/verifier-bakeoff.md). Since the checked-in report selects V3, this leakage can directly affect the study result.

2. The bakeoff can publish a formal verdict with missing or incomparable verifier coverage.
   V0 is only loaded from decision-scores.jsonl and is never required to exist or cover all candidates (verifier_bakeoff.rs). Then join silently drops candidates missing labels/scores (verifier_bakeoff.rs), and score_bakeoff selects across verifiers scored on whatever subset each happened to have (verifier_bakeoff.rs). A missing or partial cache can therefore make the “same 974 answers” comparison invalid without failing the run. The runner should require full/common coverage, or explicitly error before producing a verdict.

Not run: tests/lints, because this was an audit pass in a read-only workspace.

[github-actions]

🔒 Reviewer verdict: BLOCKING — Code and mirrors of judge.rs/decision_study.rs are consistent and well-tested, but the pre-registered spec's "memory use" metric is never collected or reported anywhere, an unmet Acceptance Criterion.

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Blocking Issues & Correctness Regressions

1. Silent cache write failure on non-existent scratch_dir (verifier_bakeoff.rs:600, decision_study.rs:121):
   verifier_bakeoff::run accepts scratch_dir but never calls std::fs::create_dir_all(scratch_dir). Because append_jsonl ignores errors on OpenOptions::open, running --verifier-bakeoff against a new directory silently fails to persist every intermediate cache (vrb-labels.jsonl, v1-scores.jsonl, v2-scores.jsonl, v3-tests.jsonl, v3-scores.jsonl). An interrupted run loses all progress without warning.

2. Unchecked missing decision-scores.jsonl silently corrupts V0 (verifier_bakeoff.rs:630, score_bakeoff:506):
   let v0_scores = decision_study::load_scores(&format!("{scratch}/decision-scores.jsonl")); does not verify the file exists or execute fallback calls. If scratch_dir lacks Study B's cached scores, v0_scores is empty; join drops all V0 tasks, yielding n=0, catch=0.0, collateral=0.0, and an erroneous UNDERPOWERED verdict for V0 instead of an explicit error directing the user to provide the cache or fetch V0.

3. Mangled doc comment and signature detachment (jev_replay.rs:687-705):
   fn resumable_ids was inserted between /// Load a priors JSONL file... and fn load_priors. resumable_ids inherits load_priors's error documentation (despite not returning a Result), while load_priors is left without its doc comment.

---

Non-Blocking Edge Cases & Deficiencies

4. Transient server errors permanently poison bake-off cache (verifier_bakeoff.rs:282, 324, 386):
   Unlike --fetch-priors (which now checks r.raw_ok and retries failures on resume), run_v2_scores, run_v3_tests, and run_openjev_scores only check contains_key(&c.id). If local endpoints (FIRSTPASS_OPENJEV_URL or FIRSTPASS_MLX_URL) drop requests or time out, score: None / empty tests are cached and permanently skipped on subsequent resumes.

5. n_dev and n_held_out in BakeoffReport overwrite per loop iteration (verifier_bakeoff.rs:521-522):
   In score_bakeoff, n_dev = dev_rows.len() and n_held_out = ho_rows.len() are updated inside for (name, scores) in &verifiers. If any verifier has missing records, the top-level report fields reflect only the row count of V3 rather than the dataset partition size.

6. Display precision rounds positive threshold to 0.0000 (verifier_bakeoff.rs:728, docs/benchmarks/verifier-bakeoff.md):
   render formats $\tau$ with {:.4}. For V1, the selected $\tau$ is a small positive float ($< 10^{-4}$), printed as 0.0000 while showing non-zero catch ($0.1207$) and collateral ($0.0488$). Because is_reject is score < tau, a literal $\tau = 0.0$ cannot reject scores in $[0, 1]$; scientific notation should be used for small $\tau$ to avoid the appearance of an arithmetic bug.

7. Untested orchestration and rendering logic (verifier_bakeoff.rs):
   score_bakeoff (cross-verifier tie-breaking, held-out metric routing) and render lack #[cfg(test)] coverage, contrary to repo standards (AGENTS.md).

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking

• verifier_bakeoff.rs: V3 score caching can permanently freeze a transient empty test-writer result. run_v3_tests retries empty raw_tests, but run_v3_scores skips any id already present in v3-scores.jsonl, even if the cached score is None. So: empty tests get scored as None, later tests are successfully regenerated, but scoring never reruns. Skip only completed scores, or key V3 scores by a hash/version of the generated tests.

• verifier_bakeoff.rs: V3 does not match the pre-registration. The spec says generated tests are written “from the task text only” (specs/verifier-bakeoff.md), but the implementation feeds the first visible/gate assertion into the prompt as a signature/example. That leaks more than task text and means the reported bake-off is not the experiment described by the preregistration.

High

• verifier_bakeoff.rs: V3 treats malformed/generated-bad tests as candidate failures. convert_assert only strips assert; then suite_score uses the normal trusted-test runner, whose eval catches any exception and counts it as a failed case (coding.rs). For model-generated verifier tests, syntax errors, wrong function names, or bad assertions are verifier failures and should be discarded/retried/abstained, not counted against the candidate. This can distort both catch and collateral.

Verification

I attempted a narrow test run, but the environment is read-only and rustup failed before compilation: could not create temp file ... Read-only file system.

[github-actions]

🔒 Reviewer verdict: BLOCKING — V3's sandbox-scoring resume check uses stale contains_key instead of the PR's own resumable_ids pattern, so a candidate that abstained (empty tests) on a prior run never gets rescored after test-writer retries succeed — undermining this PR's stated goal.

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Blocking

1. run_v3_scores breaks resumability by permanently poisoning the cache on failure (crates/firstpass-bench/src/verifier_bakeoff.rs:445-484)
   • Unlike run_openjev_scores, run_v2_scores, and run_v3_tests—which all use decision_study::resumable_ids to retry failed/empty records—run_v3_scores uses if scores.contains_key(&c.id) { continue; }.
   • If an initial run fails during run_v3_tests (e.g. server crash or timeout resulting in raw_tests: []), run_v3_scores records score: None, passed: 0, total: 0 into v3-scores.jsonl.
   • When the server is restarted and the command re-run, run_v3_tests successfully retries and generates tests, but run_v3_scores skips every retried candidate because scores.contains_key(&c.id) is true.
   • Impact: All retried candidates are permanently frozen as un-executed abstains (score: None, total: 0) in v3-scores.jsonl, corrupting the final evaluation unless the cache file is manually deleted.
   • Fix: Use resumable_ids in run_v3_scores (e.g. require r.total > 0 && r.score.is_some()), or avoid persisting records when cases.is_empty().

---

Correctness & Methodology

2. select_tau chooses positive collateral damage when catch rate is zero (crates/firstpass-bench/src/verifier_bakeoff.rs:348-382)

   • In select_tau:

     if collateral <= 0.05 && catch >= best.1 {
         best = (tau, catch, collateral);
     }

   • When all candidate thresholds or initial steps achieve catch = 0.0, any threshold with catch = 0.0 satisfies catch >= best.1. Because candidates are ordered ascending by τ, select_tau selects the highest τ that has collateral <= 0.05.
   • Impact: For any verifier where low scores fail to separate wrong answers, select_tau selects a τ that unnecessarily rejects up to 5% of valid answers (causing unnecessary escalations) instead of choosing τ = 0.0 (zero collateral).
   • Fix: Require catch > 0.0 to advance from τ = 0, or break equal-catch ties by minimizing collateral before breaking by higher τ.

3. Silent survivorship bias on missing candidates instead of recording abstains (crates/firstpass-bench/src/verifier_bakeoff.rs:328-343, 712-730)

   • In run(), V3 joins v3_scores and v3_tests via filter_map:

     v3_scores.iter().filter_map(|(k, s)| {
         let t = v3_tests.get(k)?;
         Some(...)
     })

   • In join(), candidates missing from scores are dropped (let s = scores.get(&c.id)?;).
   • If test generation was skipped for a candidate (e.g. missing signature or loader mismatch), that candidate is completely omitted from dev_rows and ho_rows rather than being evaluated as an abstain (score: None).
   • Impact: Shrinks n for that verifier and artificially alters catch/collateral rates via survivorship bias rather than penalizing the verifier's abstain_rate.

4. Offline/unreachable endpoints fail silently into a valid NOT-RECOMMENDED verdict (crates/firstpass-bench/src/verifier_bakeoff.rs:271-443)

   • chat_completion and fetch_one catch all transport errors, non-2xx responses, and timeouts, converting them to None.
   • If FIRSTPASS_OPENJEV_URL or FIRSTPASS_MLX_URL is offline, down, or misconfigured, all candidates fail immediately. No errors or warnings are raised; the run completes with code 0 and reports NOT-RECOMMENDED with abstain: 1.0000.
   • Recommendation: Add a pre-flight probe/ping to openjev_url and mlx_url before starting the batch loop, failing fast if the endpoint does not respond.

---

Edge Cases & Configuration

5. Hardcoded coder model name prevents use with common OpenAI-compatible servers (crates/firstpass-bench/src/verifier_bakeoff.rs:47)
   • CODER_MODEL is hardcoded to "mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit".
   • While FIRSTPASS_MLX_URL allows changing the endpoint, OpenAI-compatible backends like vLLM, Ollama, or llama.cpp reject requests whose model parameter does not match their loaded model name.
   • The spec specifies allowing alternative coder models of equal strength, but the CLI does not expose an env var (e.g. FIRSTPASS_CODER_MODEL) to override the string sent in chat_completion.

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking

• crates/firstpass-bench/src/verifier_bakeoff.rs:148: V3 violates the pre-registration’s “task text only” rule. run_v3_tests passes task.visible_cases.first() into build_test_writer_prompt, and MBPP visible_cases are actual converted assert expressions with expected outputs from test_list (dataset.rs:59). That means the test writer sees a real benchmark case, not just the task text/signature. Even though the reported result is negative, the committed bake-off is no longer measuring the registered V3 verifier. Derive only the function name/argument shape, or amend the spec/report honestly.

Other Findings

• crates/firstpass-bench/src/verifier_bakeoff.rs:738: load_v0_scores_or_err accepts any nonempty decision-scores.jsonl. A partial cache, wrong-candidate cache, or cache containing only raw_ok: false rows will silently become the V0 baseline. Since V0 is the baseline and is reused from Study B, this should validate coverage for the current candidate IDs and successful/parseable records, or fail loudly.

Verification

• git diff --check origin/main...HEAD reports one whitespace issue: docs/benchmarks/verifier-bakeoff.md:35: new blank line at EOF.
• I could not run Rust tests in this environment: cargo test -p firstpass-bench verifier_bakeoff -- --nocapture failed before build because rustup could not create temp files on the read-only filesystem.

[github-actions]

🔒 Reviewer verdict: BLOCKING — Unmet spec acceptance criterion (specs/verifier-bakeoff.md:38): report must record memory use alongside latency/model, but neither verifier_bakeoff.rs's BakeoffReport/render(), docs/benchmarks/verifier-bakeoff.md, nor the ADR 0013 addendum include a memory-use figure or a disclosed reason it's missing; otherwise the diff is correct, well-tested, and matches the rest of the pre-registered protocol. (Could not post the PR comment or run cargo test/clippy — gh/cargo commands required approval that was unavailable in this session.)

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Audit Assessment: PASS (Non-Blocking Findings Only)

Auditor: agent:audit-gemini (Independent 3rd Opinion)
Target: Unified diff in pr-diff.patch
Verdict: No Blocking Security or Correctness Regressions. The implementation adheres strictly to repository invariants (sandboxed execution, pure core separation, tamper-evident audit traces, forbid unsafe code, no fallible unwraps in library code).

Below are non-blocking correctness edge cases, operational inefficiencies, and documentation gaps identified for hardening.

---

1. Blocking Issues

None.

---

2. Correctness & Edge Cases (Non-Blocking)

2.1 Fail-Late Prerequisite Check in verifier_bakeoff::run

• Location: crates/firstpass-bench/src/verifier_bakeoff.rs:462-471
• Issue: run() invokes decision_study::run_oracle_labels(...) before validating the presence of the prerequisite V0 baseline cache via load_v0_scores_or_err(scratch)?.
• Impact: If --verifier-bakeoff is executed with a new scratch directory without first copying or creating decision-scores.jsonl, the harness will spend minutes executing the full MBPP candidate oracle suite in the Docker/Apple sandbox before failing fast with:

  ".../decision-scores.jsonl is missing or empty — run firstpass-bench --decision-study first..."
  

• Remediation: Move let v0_scores = load_v0_scores_or_err(scratch)?; to the top of run(), right after reading candidates and tasks_by_id.

2.2 parse_assert_lines Skips Compact Python Asserts

• Location: crates/firstpass-bench/src/verifier_bakeoff.rs:172-179
• Issue: Test line filtering relies on l.starts_with("assert "). If the local coder model emits compact call syntax like assert(f(1) == 1) without a space after assert, the line is discarded.
• Context: dataset::convert_assert already supports assert(...) (it strips "assert" then calls trim_start()).
• Remediation: Broaden the check to l.starts_with("assert ") || l.starts_with("assert(") or l.strip_prefix("assert").map(str::trim_start).

2.3 Spec Metric Omission: Memory Use

• Location: specs/verifier-bakeoff.md vs crates/firstpass-bench/src/verifier_bakeoff.rs:341-352
• Issue: The pre-registration spec states:

  "The pre-registration also records wall-clock latency per verification, the model used and memory use."

• Impact: BakeoffReport measures and reports latency percentiles (p50_ms, p95_ms) and coder_model, but omits memory consumption tracking (e.g., RSS of local servers). Neither verifier_bakeoff.rs nor docs/benchmarks/verifier-bakeoff.md includes this metric. If memory footprint is required by the evaluation protocol, this should be noted as omitted due to environment constraints or tracked via host metrics.

---

3. Documentation & Hygiene

3.1 Missing Artifact Index Entry in docs/benchmarks/README.md

• Location: docs/benchmarks/README.md
• Issue: docs/benchmarks/verifier-bakeoff.md was added as a benchmark artifact and referenced in CHANGELOG.md and docs/adr/0013-verified-predictive-routing.md, but the artifact table in docs/benchmarks/README.md was not updated to list verifier-bakeoff.md alongside decision-gate-study.md and prior-blend-replay.md.

---

4. Verified Invariants & Safety

• Execution Sandbox: Candidate answers and model-generated assertions are evaluated strictly via sb: &dyn Sandbox (suite_score). main.rs requires establish_sandbox(&sandbox_image()) and fails closed (exit(1)) if unavailable.
• Prompt Injection Defense: V2 judge prompt replicates proxy anti-injection markers (JUDGE_SYSTEM instruction fencing + <<<BEGIN_CANDIDATE). V3 test generation never passes candidate code to the test writer (only natural language task text + signature case).
• Split Determinism: is_dev derives split assignment from sha256_hex(id.as_bytes()), ensuring deterministic, leak-free 50/50 partition unaffected by row iteration order.
• Cache Resumability: resumable_ids properly discriminates between transient failures/abstains and successful calls, allowing restarts without poisoning the cache with failed attempts across all four verifiers.

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking

• crates/firstpass-bench/src/verifier_bakeoff.rs:741 only checks that decision-scores.jsonl is non-empty. A stale or partial V0 cache for different candidates passes, then join() silently drops non-matching IDs and V0 can be scored at n=0 or a biased subset despite the comment saying that must never happen. This can change verifier selection and invalidate the bake-off. Validate V0 coverage against the parsed candidate IDs, preferably requiring all candidates or failing with missing counts.

High

• crates/firstpass-bench/src/verifier_bakeoff.rs:157 says V3 writes exactly 5 tests, and the pre-registration says “writes 5 assert-based tests,” but parse_assert_lines accepts every parsed assert and line 400 scores all of them. If the model emits extra, repeated, or fewer asserts, V3’s denominator and threshold behavior no longer match the registered protocol. Enforce exactly 5, or make the deviation explicit in the spec/report before relying on the numbers.

I did not run tests; this was a diff audit only.

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Audit Verdict: Non-Blocking / Approved with Observations

No security vulnerabilities or blocking regressions were identified. The sandboxing of generated tests is fail-closed, the pre-registered protocol is strictly adhered to, and the resume bug in --fetch-priors / decision_study is correctly resolved.

Below are correctness edge cases, operational caveats, and non-blocking issues.

---

1. Correctness & Resumption Edge Cases

• load_v0_scores_or_err runs after expensive oracle labeling (verifier_bakeoff.rs:511-519):
  run_oracle_labels executes before load_v0_scores_or_err(scratch). If decision-scores.jsonl does not exist in scratch_dir, the runner executes up to 974 sandbox runs before immediately failing with "decision-scores.jsonl is missing or empty".

  • Fix: Check load_v0_scores_or_err at the start of verifier_bakeoff::run before invoking run_oracle_labels.

• select_tau tie-break selects nonzero collateral for zero catch (verifier_bakeoff.rs:388-410):
  select_tau initializes best = (0.0, 0.0, 0.0) and updates on collateral <= 0.05 && catch >= best.1. If no candidate $\tau$ achieves positive catch, any candidate $\tau > 0$ with catch == 0.0 and collateral <= 0.05 (e.g., 0.02) will overwrite tau = 0.0 due to 0.0 >= 0.0. This selects a threshold that rejects correct answers while catching no wrong ones.

  • Fix: Only allow $\tau$ to advance when catch > best.1, or when catch == best.1 && catch > 0.0.

• Silent complete failure on offline local servers (verifier_bakeoff.rs:200-220, 276-320):
  If openjev_url or mlx_url is unreachable (connection refused) or returns 5xx/404, fetch_one and chat_completion record failures as raw_ok: false / score: None for all 974 tasks without aborting. The bake-off proceeds to report all-abstain metrics and selects V0 by default.

  • Fix: Probe endpoints with a health check or abort if the first $N$ sequential calls fail.

• Missing V3 candidates dropped rather than recorded as abstains (verifier_bakeoff.rs:340-355):
  If a task is missing visible_cases.first() or mbpp_text, run_v3_tests and run_v3_scores continue without writing to the cache. In join, missing IDs are filtered out (scores.get(&c.id)?), reducing $n$ in the held-out table rather than counting as an abstain (score: None).

---

2. Operational & Configuration Rigidities

• Hardcoded CODER_MODEL constant (verifier_bakeoff.rs:46):
  CODER_MODEL is hardcoded to "mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit". While FIRSTPASS_MLX_URL allows changing the endpoint, the payload always requests this exact model string. Local OpenAI-compatible backends (e.g., vLLM, Ollama, LiteLLM) that validate model names will reject requests unless configured with an alias.

  • Fix: Allow overriding via FIRSTPASS_CODER_MODEL env var.

• V2 Judge rubric lacks request prompt parity (verifier_bakeoff.rs:90-100):
  As documented in code comments, build_judge_prompt only passes the rubric and candidate code (mirroring firstpass-proxy::judge::JudgeGate, which ignores _req). Because DEFAULT_RUBRIC states "The output should be ... directly responsive to the request", the judge model is asked to grade responsiveness to a prompt it was never provided.

---

3. Security Audit

• Sandbox isolation: V3 generated unit tests are executed using suite_score inside ContainerSandbox with strict memory (512MB), process (128), and wall-clock (10s) limits, isolated from the network and host filesystem (verify_isolating).
• Prompt Injection resistance: In V2, candidate outputs are enclosed in delimiter tags (<<<BEGIN_CANDIDATE ... >>>END_CANDIDATE) under a pinned system prompt instructing the model to treat candidate text strictly as data. In V3, candidate outputs are never included in the test generation prompt.
• Credentials: No secrets or credentials are leakable or committed; local endpoints require no authentication.