fix(tbench2_env): bring Docker mode up to the canonical scoring contract

[nblintao]

Summary

#965 / #972 landed the canonical Terminal-Bench-2 contract in local mode (Tbench2Environment): scoring via the task's own tests/test.sh (pinned toolchain, /logs/verifier/reward.txt verdict), the task image's real WORKDIR, and verifier assets kept out of the agent's reach. Docker mode (Tbench2DockerEnvironment) only received the verifier-asset withholding half of #972 — its evaluate still ran bare pytest tests/ from /task (the task-source copy), skipping the canonical harness and the image's real WORKDIR. This PR closes that gap so both execution modes speak one scoring contract, and removes two silent-degradation paths found along the way.

Changes

• Canonical evaluate in Docker mode: stage tests/ at the official fixed path (/tests) for exactly the verify window (pre-wipe fail-closed, post-wipe of both /tests and /logs/verifier, as before), run tests/test.sh from the agent's workdir bounded by the task's own verifier budget (task.toml [verifier].timeout_sec, via coreutils timeout when present in the image), read the verdict from /logs/verifier/reward.txt. Task dirs without test.sh fall back to pytest, mirroring local mode's _evaluate_fallback. Command construction and marker parsing are factored into module helpers shared with local mode.
• Real task workdir: agent exec commands and scoring run in the task image's own WORKDIR, resolved from image metadata (Config.WorkingDir, which sees base-image WORKDIRs) with the task Dockerfile parse as fallback, then /task.
• exec quoting fix: commands now ride as a bash -c argv element instead of being spliced into a single-quoted shell string, which mangled any agent command containing a single quote.
• No host-execution fallback: a task dir that declares no [environment] docker_image is rejected at reset. The old silent fallback ran agent commands directly on the env-server host — a containment hole (arbitrary agent shell outside any container) and a scoring-fidelity hole at once. The containerless code paths (host subprocess exec, docker-class _evaluate_local) are removed with it.
• TB2_MODE=auto removed: it claimed to auto-detect Docker but unconditionally served local mode; unknown modes now fail at startup instead of silently serving local.

Who is affected?

Local mode — the default, and what HF Spaces run — is completely unchanged. The refactor generates byte-identical commands there, and the existing local-mode tests pass without modification. If you never set TB2_MODE=docker, you can stop reading.

For TB2_MODE=docker operators, the intended effect is simply: official TB2 tasks now score the official way (previously: bare pytest in the wrong directory, without the task's pinned toolchain — i.e. the old rewards were wrong). Scores will move; that's the fix working.

The only setups that need a change on their side:

• Custom task dirs whose tests hard-code the old layout (cwd /task, tests at /task/tests) and whose image declares a WORKDIR. Images without a WORKDIR keep /task as cwd, so most custom setups are untouched.
• Tasks with no docker_image in task.toml: these used to silently run agent commands directly on the server host — outside any container. That's a containment hole, so reset now rejects them with an error pointing at TB2_MODE=local. (If someone has a real use case for the old behavior, happy to gate it behind an explicit opt-in env var instead.)
• TB2_MODE=auto: never actually auto-detected anything (it silently ran local mode), so it now errors at startup; set local explicitly to keep the exact same behavior.

Nothing here can fail silently: every changed path either scores the official way or refuses loudly with an actionable message. One cosmetic note for downstream parsers: docker-mode evaluate now reports info["harness"] on the canonical path (same shape as local mode) instead of info["exit_code"]; the pytest-fallback path keeps exit_code.

Validation

• Unit tests (tests/envs/test_tbench2_env.py): 23 passed. Existing docker-mode tests updated to the new contract; new coverage for the pytest fallback without test.sh, workdir resolution precedence (image metadata → Dockerfile → /task), imageless reset rejection, and argv-form exec (single quotes survive byte-identical).
• Real-container golden run on an x86 docker host, driving a live TB2_MODE=docker server from this branch with a plain OpenEnv client (reset → stage the task's official solution/solve.sh → exec it → standard evaluate):
  • fix-git: reward 1.0, info.harness=tests/test.sh, and the in-container pwd probe returned /app/personal-site — the task's non-trivial image WORKDIR, resolved correctly (the old code would have run everything in /task).
  • chess-best-move: reward 1.0.
  • Negative control (no solution staged, straight to evaluate): reward 0.0 — the canonical harness genuinely ran and failed the tests.
  • Imageless task dir: reset raised RuntimeError: task imageless-task declares no [environment] docker_image … as intended.

🤖 Generated with Claude Code

---

Note

Medium Risk
Docker-only behavior changes (cwd, scoring, metrics) can break custom TB2 Docker deployments; local mode and shared eval helpers are intended unchanged. Removing host fallback and TB2_MODE=auto is a deliberate breaking change for misconfigured operators.

Overview
Aligns Docker mode with the canonical Terminal-Bench 2 scoring path already used in local mode, and removes silent misconfiguration and containment fallbacks.

Docker evaluate now stages tests/ at /tests, wipes /logs/verifier, runs the task’s tests/test.sh from the image WORKDIR (with task.toml verifier timeout via timeout when available), and reads /logs/verifier/reward.txt—replacing bare pytest from /task. Shared helpers (_canonical_eval_cmd, _parse_canonical_reward, fallback pytest path) are extracted for both modes.

Agent execution uses the resolved image WORKDIR (image Config.WorkingDir → task Dockerfile → /task). exec_run passes commands as ["bash", "-c", ...] so shell quoting in agent commands is preserved.

Stricter startup and reset: TB2_MODE=auto and unknown modes error at startup instead of defaulting to local. Tasks without docker_image fail at reset; host subprocess fallback and _evaluate_local are removed.

Docs/README updated for Docker mode behavior. Unit tests extended for canonical scoring, workdir, imageless reset, and argv exec.

^{Reviewed by Cursor Bugbot for commit 1a6f0c7. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 31d0fa6. Configure here.}

[cursor]

Docker fallback scoring lacks timeout

Medium Severity

Docker mode reads verifier_timeout_s for every evaluate, but only passes it into _canonical_eval_cmd. The pytest fallback via _fallback_eval_cmd has no in-shell timeout, and Docker exec has no server-side budget either, so a hung custom-task pytest can block the session indefinitely. Local mode still bounds that path through the terminal toolkit.

Additional Locations (1)

• envs/tbench2_env/server/tbench2_env_environment.py#L214-L228

 

^{Reviewed by Cursor Bugbot for commit 31d0fa6. Configure here.}