Isolate official SWE-bench agent commands with an optional Docker backend

.gitignore

@@ -9,3 +9,4 @@ dist/
 *.egg-info/
 .yada/
 eval-results/
+logs/

README-cn.md

@@ -20,6 +20,11 @@ Yada 目前处于 Alpha 阶段。仓库已经测试本地 Agent 闭环，但尚
 - [uv](https://docs.astral.sh/uv/)（推荐）
 - DeepSeek API Key
 
+直接运行 `yada` 和执行 `yada eval --case` 不需要 Docker；官方
+`yada eval --swebench` 评测要求安装并启动仍受维护的 Docker Desktop 或
+Docker Engine。安装、验证、版本策略与资源要求见
+[Docker requirements](docs/configuration.md#docker-requirements)。
+
 ## 快速开始
 
 ```bash
@@ -53,6 +58,8 @@ Trace 保存在目标仓库的 `.yada/runs/` 目录下。
 - [配置](docs/configuration.md)：其他安装方式、API Key、模型参数、命令策略和
   Trace Level。
 - [CLI 参考](docs/cli-reference.md)：`yada`、`yada eval` 和 `yada-trace`。
+- [评测生命周期](docs/evaluation.md)：`--case` 与 `--swebench` 会加载、修改、
+  缓存、评分和写入哪些内容。
 - [贡献指南](CONTRIBUTING.md)：开发环境、验证命令和基于 Rebase 的 PR 流程。
 - [架构](docs/dev/architecture.md)：Agent 循环、工具、Patch 事务、评测边界与
   安全不变量。

README.md

@@ -20,6 +20,11 @@ project does not claim a comparative benchmark result yet.
 - [uv](https://docs.astral.sh/uv/) (recommended)
 - A DeepSeek API key
 
+Docker is optional for direct runs and `yada eval --case`. Official
+`yada eval --swebench` runs require a running, maintained Docker Desktop or
+Docker Engine release; see [Docker requirements](docs/configuration.md#docker-requirements)
+for installation, verification, version policy, and resource guidance.
+
 ## Quick start
 
 ```bash
@@ -54,6 +59,8 @@ complete OS sandbox; use a disposable VM or container for unfamiliar projects.
 - [Configuration](docs/configuration.md): installation alternatives, API key,
   model settings, command policy, and trace levels.
 - [CLI reference](docs/cli-reference.md): `yada`, `yada eval`, and `yada-trace`.
+- [Evaluation lifecycle](docs/evaluation.md): what `--case` and `--swebench`
+  load, mutate, cache, grade, and write.
 - [Contributing](CONTRIBUTING.md): development setup, validation, and the rebase
   pull-request workflow.
 - [Architecture](docs/dev/architecture.md): agent loop, tools, patch transaction,

benchmarks/README.md

@@ -23,3 +23,8 @@ Cases under `swebench_verified/` use the canonical public problem statement and
 base commit. Their local grader is intended for fast development feedback. A
 published SWE-bench score must still come from the official Docker Harness.
 
+This local path is intentional rather than a duplicate of `--swebench`: it
+keeps prompt and tool regression tests fast, inspectable, and usable without
+Docker. Use `--swebench INSTANCE_ID` when environment parity and an
+official-compatible verdict matter; Yada then requires Docker before model
+inference and separates the Agent command container from the final grader.

docs/cli-reference.md

@@ -53,48 +53,85 @@ keyboard interruption. Argument errors also use the standard argparse status `2`
 ## `yada eval`
 
 `yada eval` runs an agent behind a benchmark-neutral adapter and persists a
-schema-versioned result even for ordinary adapter failures.
+schema-versioned result even for ordinary adapter failures. Unlike a direct
+`yada` run, an evaluation always has a task adapter and a grader.
+See [Evaluation lifecycle](evaluation.md) for the complete load, workspace,
+agent, grading, cache, and artifact sequence.
 
-### Checked-in development case
+### Choose a run mode
+
+```mermaid
+flowchart TD
+    start{"What do you want to run?"}
+    start -->|"Edit one repository"| direct["yada TASK --workspace REPO"]
+    start -->|"Run a reproducible evaluation"| eval["yada eval"]
+
+    eval --> source{"Which task source?"}
+    source -->|"Checked-in case directory"| case["--case PATH"]
+    source -->|"Official SWE-bench Verified"| swe["--swebench INSTANCE_ID"]
+
+    case --> local_adapter["Local adapter\ncase.json + host/uv environment"]
+    local_adapter --> local_result["Development/private result\nnot an official SWE-bench score"]
+
+    swe --> agent_container["Public instance image\nAgent commands in Docker"]
+    agent_container --> official["Separate official Harness\ngrading container"]
+    official --> official_result["Official-compatible SWE-bench result"]
+```
+
+`yada eval` deliberately exposes only two task selectors:
+
+| Invocation | Meaning | Native Yada command environment | Grader |
+| --- | --- | --- | --- |
+| `--case PATH` | Run a portable local case. A directory resolves to `PATH/case.json`; a manifest file may also be passed directly. | Host/uv environment declared by the case; Docker is not required by Yada | Command declared by the case manifest |
+| `--swebench INSTANCE_ID` | Load one SWE-bench Verified task and generate an official-compatible prediction. | Public SWE-bench instance image in Docker | Separate official SWE-bench Harness container |
+
+There is no `--local`, `--benchmark`, or separate `--manifest` selector. Pass a
+custom manifest directly to `--case`. A case is called local because its
+manifest controls preparation and grading; it may still fetch a Git repository
+and call the DeepSeek API. Each invocation runs exactly one task.
+
+#### Why `--case` is not redundant
+
+`--case` is the small, no-Docker evaluation path for prompt/tool regression,
+private repositories, and custom test subsets. It is fast, inspectable, and can
+run from fully local inputs. Its verdict belongs to the manifest and must not be
+published as an official SWE-bench score.
+
+`--swebench` is the comparability path. It fixes the dataset and Harness policy,
+requires a running Docker daemon before the first model request, runs native
+Yada commands in the public instance image, and grades the patch in an
+independent container that receives the evaluation tests. Collapsing these
+selectors would either force Docker onto ordinary Yada development or make
+local and official-looking results dangerously easy to confuse.
+
+### Checked-in development case (`--case`)
+
+Use a checked-in case for fast development and regression testing:
 
 ```bash
 uv run yada eval \
   --case benchmarks/swebench_verified/pytest-10051 \
   --agent yada \
-  --yes
+  --yes \
+  --trace-level debug
 ```
 
 The first run fetches the pinned pytest commit and builds the case's locked
 Python environment. Later runs reuse caches but create a fresh agent workspace.
-This local grader is a development feedback loop, not an official SWE-bench
-Docker score.
-
-### Common options
-
-| Option | Meaning | Default |
-| --- | --- | --- |
-| `--benchmark local\|swebench` | Benchmark adapter. | Inferred from `--case`, otherwise required |
-| `--case PATH` | Portable local case directory or `case.json`. | — |
-| `--instance ID` | Benchmark instance ID. | Manifest ID for local cases |
-| `--agent yada\|command` | Agent adapter. | `yada` |
-| `--output PATH` | Result JSON path. | `eval-results/<task>__<time>.json` |
-| `--artifact-dir PATH` | Workspace, logs, patch, trace, and grader artifacts. | Sibling `<result>.artifacts` |
-| `--run-id ID` | Stable correlation ID. | Generated |
-| `--max-steps N` | Model-turn budget. | `30` |
-| `--wall-time SECONDS` | Comparable wall-time budget. | `1800` |
-| `--max-output-tokens N` | Per-completion token limit. | `16384` |
+The case's grader is a development feedback loop, not the official SWE-bench
+Docker grader, even though the task originated in SWE-bench Verified.
 
-The native agent also accepts the model, thinking, timeout, command-policy, and
-trace-level options documented for `yada`. A deployment-level supervisor should
-enforce a hard wall-time limit for an in-process native agent.
-
-Evaluation exits with `0` for `resolved`, `1` for `unresolved`, and `2` for
-errors or non-verdict outcomes such as skipped grading.
+For a private or machine-local task, pass its manifest through the same entry
+point:
 
-### Local manifest
+```bash
+uv run yada eval \
+  --case /path/to/task.local.json \
+  --agent yada \
+  --yes
+```
 
-Use `--benchmark local --manifest FILE` for a private or machine-local task.
-A minimal version 1 manifest is:
+A minimal version 1 case manifest is:
 
 ```json
 {
@@ -151,14 +188,80 @@ Portable Git checkouts are cached under `--cache-dir` (default
 `.yada/cache/evals`) but never used directly as the mutable agent workspace.
 `install_workspace` accepts `editable` and `legacy-editable`.
 
-### External command agent
+### Official SWE-bench evaluation (`--swebench`)
+
+The official SWE-bench Harness is the single source for public instance
+metadata, the public instance image, and Docker grading. `--swebench` does not
+read the checked-in local case's `instance.json`, so there are no local and
+online copies to keep in sync. Gold patches, test patches, and hidden test IDs
+do not cross the public task boundary.
+
+Install and start a maintained Docker Desktop or Docker Engine release, then
+verify both `docker --version` and `docker info`. SWE-bench 4.1.0 does not state
+an exact minimum Docker version, so Yada checks working client/daemon behavior
+instead of enforcing an invented version number. Legacy Docker Toolbox and
+obsolete standalone clients are unsupported. See
+[Docker requirements](configuration.md#docker-requirements) for installation,
+platform checks, and resource guidance.
+
+The official `swebench` package is published on PyPI, so no SWE-bench Git clone
+or separately managed virtual environment is needed. Use uv's cached dependency
+overlay to run the pinned Harness version:
+
+```bash
+uv run --with 'swebench==4.1.0' yada eval \
+  --swebench pytest-dev__pytest-10051 \
+  --agent yada \
+  --yes \
+  --trace-level debug
+```
+
+The first run downloads the Harness and its dependencies into uv's cache;
+subsequent runs reuse them. This keeps Yada's core dependency-free and leaves
+the project environment unchanged. If `swebench==4.1.0` is already installed in
+Yada's active environment, the shorter `uv run yada eval ...` form also works.
+Yada checks the Docker CLI and daemon before dataset loading or model inference.
+
+The Harness first pulls or builds the public instance image. Yada exports its
+prepared `/testbed` as the mutable artifact workspace. For native
+`--agent yada`, file tools edit that workspace on the host while `run_command`
+executes against it in an ephemeral container made from the same image. Final
+grading uses another Harness-owned container; only that container receives the
+evaluation patch and script.
+
+During image preparation, Harness stdout and stderr are streamed to the
+terminal and flushed live to `swebench-agent-image.stdout.log` and
+`swebench-agent-image.stderr.log` in the artifact directory. A heartbeat is
+printed after each 30-second interval without output. This happens before the
+Agent phase, so `yada-trace.jsonl` is not the place to diagnose an image pull or
+build. Official grading is streamed the same way to `swebench.stdout.log` and
+`swebench.stderr.log`. See
+[Evaluation lifecycle](evaluation.md#official-swe-bench---swebench-instance_id)
+for the exact cache, workspace, container, and grading sequence.
+
+The adapter intentionally fixes the current public evaluation policy instead of
+exposing Harness internals as Yada CLI flags:
+
+| Setting | Built-in policy |
+| --- | --- |
+| Dataset | `princeton-nlp/SWE-bench_Verified` |
+| Split | `test` |
+| Grading | Official Docker Harness |
+| Native Agent commands | Public instance image in an ephemeral container |
+| Image cache | Keep environment images |
+| Grading timeout | 1800 seconds |
+| Docker namespace | `swebench`; automatically disabled on Apple Silicon so images build locally |
+
+Apple Silicon support in SWE-bench remains experimental. Building images locally
+takes substantially more time and disk space than running a checked-in case.
+
+### External command agent (`--agent command`)
 
 The command adapter runs a non-interactive argv without a shell:
 
 ```bash
 uv run yada eval \
-  --benchmark local \
-  --manifest /path/to/task.local.json \
+  --case /path/to/task.local.json \
   --agent command \
   --agent-name another-agent \
   --agent-command \
@@ -168,36 +271,32 @@ uv run yada eval \
 
 The template supports `{task}`, `{task_file}`, `{workspace}`, `{output_patch}`,
 and `{run_dir}`. If the command does not write `{output_patch}`, Yada collects
-the complete Git diff, including untracked files.
-
-### SWE-bench
+the complete Git diff, including untracked files. For `--swebench`, this
+external command is host-managed and is responsible for entering its own
+container; Yada's automatic Agent command container applies only to the native
+`--agent yada` adapter.
 
-The SWE-bench adapter loads public instance metadata from `--instance-file`, or
-from Hugging Face when the optional `datasets` package is installed. It discards
-gold patches, test patches, and hidden test IDs at the public task boundary.
+### Common evaluation options
 
-Install the official SWE-bench package and Docker separately, then run:
-
-```bash
-uv run yada eval \
-  --benchmark swebench \
-  --instance pytest-dev__pytest-10051 \
-  --instance-file benchmarks/swebench_verified/pytest-10051/instance.json \
-  --workspace /path/to/clean/pytest-base-repo \
-  --agent yada \
-  --yes \
-  --swebench-python /path/to/swebench-env/bin/python \
-  --output results/yada-pytest-10051.json
-```
+| Option | Meaning | Default |
+| --- | --- | --- |
+| `--case PATH` | Portable local case directory or `case.json`. | Mutually exclusive with `--swebench` |
+| `--swebench ID` | One official SWE-bench Verified instance. | Mutually exclusive with `--case` |
+| `--agent yada\|command` | Agent adapter. | `yada` |
+| `--output PATH` | Result JSON path. | `eval-results/<task>__<time>.json` |
+| `--artifact-dir PATH` | Workspace, logs, patch, trace, and grader artifacts. | Sibling `<result>.artifacts` |
+| `--run-id ID` | Stable correlation ID. | Generated |
+| `--max-steps N` | Model-turn budget. | `30` |
+| `--wall-time SECONDS` | Comparable wall-time budget. | `1800` |
+| `--max-output-tokens N` | Per-completion token limit. | `16384` |
 
-`--workspace` is optional; without it, Yada fetches the exact base commit. It
-prepares the candidate workspace but does not replace official Docker grading.
-Use `--grade-mode none` to test preparation and prediction generation without
-Docker; the result is `skipped`, never `resolved`.
+The native agent also accepts the model, thinking, timeout, command-policy, and
+trace-level options documented for `yada`. A deployment-level supervisor should
+enforce a hard wall-time limit for an in-process native agent. Use the same task,
+base commit, model budget, network policy, and grader when comparing agents.
 
-Other SWE-bench options are `--dataset-name`, `--split`, `--cache-level`,
-`--clean`, `--namespace`, and `--grade-timeout`. Use the same task, base commit,
-model budget, network policy, and official grader when comparing agents.
+Evaluation exits with `0` for `resolved`, `1` for `unresolved`, and `2` for
+errors or non-verdict outcomes such as skipped grading.
 
 ## `yada-trace`