[napetrov]

Part of #687 (verifier no-reward, High).

Summary

 docker verifier _verify(task)
   proc = docker run verifier
   reward = read /logs/verifier/reward.txt
-  if reward is missing: return pass when exit 0
+  if reward is missing: _exit_code_verdict(returncode)
+     exit 0    -> RuntimeError("verifier exited 0 but wrote no reward.txt")
+     exit != 0 -> 0.0 (fail, as before)

_exit_code_verdict lives in harnesses/command.py, so docker_solver.py shrinks from 897 to 896 lines; scripts/growth_baseline.json is lowered to match (no baseline bump).

Evidence

• Before: test_docker_verify_exit_zero_without_a_reward_file_is_an_error_not_a_pass fails on intel/main: a verifier that exits 0 and writes no reward is scored as a pass.
• After: both new tests in tests/test_docker_verify_reward.py pass (exit 0 without reward -> error, non-zero exit without reward -> still a fail). tests/test_docker_task_md.py fixtures now answer the docker cp calls the verifier makes, since they relied on exit 0 = pass. make ci-local: growth ratchet and all other steps pass; the only failure is tests/test_process.py::test_sighup_kills_the_live_groups_too, a 30 s timeout under box load. It passes alone (1 passed).

Merge Danger

Door: two-way

Behaviour change: a task whose verifier exits 0 without writing a reward now errors the cell instead of scoring it as a pass. Tasks that relied on that silent pass will show as errors until their verifier writes reward.txt.

Blast Radius: medium

Affects every docker-solver task run. Expect a rise in error cells on any task whose verifier does not write a reward file; the fix is the verifier, not this check.

[Copilot]

🟢 Approval recommended

No active review issues remain for the change preventing Docker verifiers from passing without a reward file.

No open issues 🎉

What changed in this PR

0 new findings; nothing to fix.

This PR addresses the false-pass defect in #687: Docker verifiers that exit successfully without writing reward.txt now error the cell instead of receiving a passing score.

Changes:

• Add a missing-reward verdict helper that raises on exit code 0 and preserves a 0.0 score for non-zero exits.
• Add regression tests and update existing Docker fixtures to supply reward files.
• Document the behavior change and lower the Docker solver’s growth baseline.

File	Description
dokimi/​tests/​test_docker_verify_reward.py	Tests missing-reward behavior for zero and non-zero exits.
dokimi/​tests/​test_docker_task_md.py	Updates fixtures to provide the fake verifier’s reward file.
dokimi/​scripts/​growth_baseline.json	Lowers the Docker solver baseline to 896 lines.
dokimi/​dokimi/​harnesses/​docker_solver.py	Uses the new helper when no reward is available.
dokimi/​dokimi/​harnesses/​command.py	Adds explicit missing-reward exit-code handling.
dokimi/​changelog.d/​20261113_120500_napetrov_fsm687-verifier.md	Documents the corrected verifier behavior.

_{Review effort: Balanced}

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

CI at 97f70ef: 24 passing, 3 skipped.

Review details

• Commit reviewed: 97f70ef against merge base 74e61c6
• Coverage: 4 files in 1 groups
• Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
• Usage: 105k tokens (79k from cache), 24s
• Tool calls: 12 (7 file_read, 4 code_search, 1 pr_threads)
• Command harness verifier behavior and related documentation: 4 files

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.73%. Comparing base (f12057b) to head (97f70ef).
⚠️ Report is 2 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #695      +/-   ##
==========================================
+ Coverage   95.71%   95.73%   +0.01%     
==========================================
  Files         415      415              
  Lines       55346    55376      +30     
  Branches     9514     9519       +5     
==========================================
+ Hits        52977    53012      +35     
+ Misses       1466     1463       -3     
+ Partials      903      901       -2     

Files with missing lines	Coverage Δ
dokimi/dokimi/harnesses/command.py	97.27% <100.00%> (+0.16%)	⬆️
dokimi/dokimi/harnesses/docker_solver.py	97.96% <100.00%> (+0.22%)	⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vlad-perevezentsev]

Review at 1fb962d. The fix is right for the case it targets: a Docker verifier that exits 0 without reward.txt now errors the cell and leaves pass_rate, and a non-zero exit still scores 0. A reward file with no number in it is still scored as a fail, and two docs still describe the old fallback; details are in the line comments.

What I ran, at 1fb962d:

• make ci-local --offline: 0 failed, 21 passed, 19 skipped (network-only jobs); the test suite: 18,933 passed, 72 skipped.
• The CI smoke (docker-oracle and docker-probe on onetbb-histogram-local-aggregation, real Docker): 1/1 pass each, the reward read from reward.txt.
• Real Docker, docker-oracle on two copies of that task: test.sh that only exits 0 gives an errored cell (RuntimeError: verifier exited 0 but wrote no /logs/verifier/reward.txt, docker's "Could not find the file" in run.log); test.sh that writes PASS gives reward None with no error, counted as scored: 1, pass_rate: 0.0.
• Through run() with the fake docker: exit 0 with no file is errored in oracle mode (raised to the runner's _error_result); exits 1, 127 and 137 with no file score 0.0.

Checked, no issues

• Error path: in oracle mode without a sandbox run() re-raises and the runner records metrics.error; with a sandbox or an agent, run() records it itself. Both rows are errored and leave the pass_rate denominator.
• Non-zero exits: keeping 0.0 for 126/127/137 is right here, unlike the command harness: test.sh often ends with the solution's own binary, so a crash or a missing build is the solution's fail.
• Tests: the exit-0 test fails on the base (_verify returned 1.0); the _capture / _host_cli changes only let the fake write its reward.txt, so the existing tests still verify through a reward file.
• Growth gate: _exit_code_verdict lives in command.py and the tests in a new file because docker_solver.py and test_task_formats.py are at their baselines.

[vchamarthi]

Reviewed head 1fb962d. Nothing new to fix; 4 threads from vlad-perevezentsev are still open. 0 new findings. The exit-0 path errors the cell and non-zero exits still score 0.0, as the changed tests pin. Four open threads from vlad-perevezentsev still stand, so the PR is not ready to approve.

Verified

• Exit 0 without reward: A docker-* verifier that exits 0 with no /logs/verifier/reward.txt raises RuntimeError naming reward.txt from _verify, and the changed test asserts that raise.
• Exit 0 without reward on main: On main the same exit-0 verifier with no reward file does not raise, so the changed test fails there with DID NOT RAISE.
• Non-zero exit without reward: Exit codes 1, 126, 127 and 137 with no reward file score 0.0 on the head, and 0.0 on main.
• Run.log message: The missing-reward error writes /logs/verifier/reward.txt into the run log, as the changelog fragment says; main does not raise there.
• Exit-0 test pins the verdict: Reverting the verdict in docker_solver.py to 1.0 if exit 0 makes the exit-0 test in test_docker_verify_reward.py fail.
• Existing task verify tests: All 68 changed tests pass on the head, including the test_docker_task_md.py fixtures whose docker cp now falls through to a reward file.
• Full test run: With main merged in, the full test run has no failure that main lacks, and the file-length step passes with docker_solver.py at 896 lines.
• Merge: The PR merges onto the current main with no conflict.

Open threads from other reviewers:

• still holds on this head: #695 (comment): A reward.txt containing PASS makes _verify return None on the head, so run() records no reward and no error, and main does the same.
• still holds on this head: #695 (comment): docs/contributing-terminal-bench-task.md still says a missing reward.txt falls back to the exit code, and docs/coding-harnesses.md says a docker-* cell does too, while the head errors such a cell on exit 0.
• still holds on this head: #695 (comment): The fragment files the change under ### Fixed, but dokimi/changelog.d/README.md puts anything that moves scores for the same inputs under Changed.
• still holds on this head: #695 (comment): A docker cp that exits 1 after the verifier wrote reward.txt raises verifier exited 0 but wrote no /logs/verifier/reward.txt, and a docker cp timeout raises TimeoutExpired, which run() records as reward 0.0.

[xaleryb]

• Ran: head merged with current main, full suite plus ruff, mypy, bandit and the growth and comment-block gates. All green. The new exit-0 test fails on the pre-fix source, as it should.
• No new findings beyond the existing threads.
• The four open threads from vlad-perevezentsev still stand at 1fb962d, and none has a reply yet:
  • reward.txt with no number (PASS, empty) still scores as a fail rather than an error.
  • contributing-terminal-bench-task.md:156 and coding-harnesses.md:1101 still describe the exit-code fallback.
  • The fragment is filed under Fixed, but scores change for the same inputs, so it belongs under Changed.
  • A failed or timed-out docker cp is reported as a missing reward.txt or scored 0.0, and the comment at docker_solver.py:648 is stale.
• Left for the author: answer or fix those four threads.

[xaleryb]

Pushed 4 commits (1fb962d..aa35664):

• 81c4164: a docker reward.txt with no number in it (PASS, empty) errors the cell whatever the exit code, the same rule a command harness applies to its reward file.
• e5f4150: a docker cp of reward.txt that times out, or fails with anything other than docker's own no-such-file message, errors the cell with its own message and the cp exit code; a dead container still reports "not running at verify time". The stale comment is gone.
• bfdf48b: the fragment moves to Changed (pass rates are not comparable with earlier runs) and says what a task author has to do; docs/contributing-terminal-bench-task.md, docs/coding-harnesses.md and the self-test reference in dokimi-design-evals now describe the docker-* rule.
• aa35664: merges main.

On the merged tree: full suite (19942 passed), ruff, mypy, bandit, check_growth, docs check --strict and the CLI snapshot step pass; the mutants over each new branch are killed. A real docker-oracle run on five tiny tasks: PASS and an empty file error ("has no reward in it"), no file with exit 0 errors, no file with exit 1 scores 0.0, 1 passes.

[xaleryb]

Re-checked at aa35664 (our three fixes for the four review threads plus a merge of main): a reward.txt with no number and a failed or timed-out docker cp error the cell, a dead container still reports 'not running at verify time', the fragment is under Changed and the docs describe the docker-* rule; the full suite (19942 passed), ruff, mypy, bandit, check_growth and the docs gate pass, a real docker-oracle run matches, and CI is green.

[antonwolfy]

Review at aa35664. The exit-0, no-number and cp-failure rules hold, and the earlier threads are fixed. One point to fix before merge: the missing-file pattern for docker cp is too broad.

What I ran, on a Linux review host at aa35664:

• make ci-local --with-docker: 0 failed, 37 passed, 14 skipped; 19,961 tests passed, 19 skipped; changed-line coverage met
• A reproduction of the inline finding, with the Docker CLI faked: socket stderr with verifier exit 1 returns 0.0, with exit 0 raises the wrong message.
• Mutants of the new branches (cp timeout, copy failure, _check_reward_copy call, running inspect, each regex alternative): the reward-file tests kill them all except the no such file alternative. The cp timeout and the dead-container branch are pinned by test_cleanup_timeouts.py and test_container_agent.py.

Checked, no issues

• Commits: all four non-merge commits carry a Signed-off-by.
• Docs: coding-harnesses.md, the task contract, the self-test reference and the Changed fragment match _verify; no other doc still describes the exit-code fallback for docker-* cells (the remaining mentions are about the command harness).
• Verdicts: exit 0 without a file errors, a non-zero exit scores 0.0, PASS or an empty file errors for both exit codes, and a cp timeout errors.

[napetrov]

Re: review by vlad-perevezentsev

the four threads and the two docs from your review at 1fb962d
All four are fixed on head, and each thread is answered (87f8d38).

[napetrov]

Re: review by vchamarthi

4 threads from vlad-perevezentsev are still open
Answered in each thread, and fixed on head (87f8d38).

[napetrov]

Re: review by antonwolfy

the missing-file pattern for docker cp is too broad
Fixed in 62da132: a socket error no longer matches (87f8d38).

[napetrov]

Re: comment by xaleryb

Left for the author: answer or fix those four threads
Each thread is answered, and the fixes are on head (87f8d38).

[napetrov]

Re: comment by xaleryb

Pushed 4 commits (1fb962d..aa35664)
Noted; the head also has the intel/main merge and the socket fix (87f8d38).

[napetrov]

Re: vchamarthi's "not approving yet": the four vlad-perevezentsev threads are checked on head, and each is answered in its thread.
Open risks: none known. The docker cp socket pattern (exai-bot, antonwolfy) is fixed in 62da132.
Head is the merge of intel/main, and CI is pending on it.

[antonwolfy]

LGTM. The socket-error point from my earlier review is fixed: a missing Docker socket is now a failed docker cp, for either verifier exit code, and the two socket cases are pinned in the parametrization. The terminal-bench v1 carve-out in the docs now matches _verify. One nit inline.

What I ran, on a Linux review host at 74e61c6:

• make ci-local --with-docker: 0 failed, 38 passed, 14 skipped; 20,447 tests passed, 19 skipped; changed-line coverage 100% (19 lines).
• _reward_file_verdict on 75, 1.5, 100%, 1, 0.75, with no model or network involved.

Checked, no issues

• Commits: all nine non-merge commits carry a Signed-off-by.
• Socket pattern: _CP_NO_FILE_RE no longer has the bare no such file alternative. The socket, daemon and No such container: c1 stderr all raise docker cp ... failed (exit 1); Docker's and podman's missing-file wordings still reach the exit-code verdict.
• Docs: coding-harnesses.md, the task contract, the self-test reference and the Changed fragment agree with _verify: exit 0 without a file errors, a non-zero exit scores 0.0, and a terminal-bench v1 task keeps its pytest summary.
• Growth gate: docker_solver.py is under its baseline and tests/test_growth_ratchet.py passes.

[antonwolfy]

LGTM. I pushed a commit per nit, up to 97f70ef, which fix the nit from my review:

• 97f70ef: dokimi/dokimi/harnesses/command.py:464: the error now reads has no reward between 0 and 1 in it, and the changelog fragment and both docs say no number between 0 and 1 in it (PASS, empty, 75).

make ci-local at 97f70ef: 35 passed, 3 failed (host), 14 skipped.