diff --git a/.gitignore b/.gitignore index dbde422..c2d3459 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,4 @@ dist/ *.egg-info/ .yada/ eval-results/ +logs/ diff --git a/README-cn.md b/README-cn.md index 72f2c70..fc45408 100644 --- a/README-cn.md +++ b/README-cn.md @@ -20,6 +20,11 @@ Yada 目前处于 Alpha 阶段。仓库已经测试本地 Agent 闭环,但尚 - [uv](https://docs.astral.sh/uv/)(推荐) - DeepSeek API Key +直接运行 `yada` 和执行 `yada eval --case` 不需要 Docker;官方 +`yada eval --swebench` 评测要求安装并启动仍受维护的 Docker Desktop 或 +Docker Engine。安装、验证、版本策略与资源要求见 +[Docker requirements](docs/configuration.md#docker-requirements)。 + ## 快速开始 ```bash @@ -53,6 +58,8 @@ Trace 保存在目标仓库的 `.yada/runs/` 目录下。 - [配置](docs/configuration.md):其他安装方式、API Key、模型参数、命令策略和 Trace Level。 - [CLI 参考](docs/cli-reference.md):`yada`、`yada eval` 和 `yada-trace`。 +- [评测生命周期](docs/evaluation.md):`--case` 与 `--swebench` 会加载、修改、 + 缓存、评分和写入哪些内容。 - [贡献指南](CONTRIBUTING.md):开发环境、验证命令和基于 Rebase 的 PR 流程。 - [架构](docs/dev/architecture.md):Agent 循环、工具、Patch 事务、评测边界与 安全不变量。 diff --git a/README.md b/README.md index 8f5802e..a51e15f 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,11 @@ project does not claim a comparative benchmark result yet. - [uv](https://docs.astral.sh/uv/) (recommended) - A DeepSeek API key +Docker is optional for direct runs and `yada eval --case`. Official +`yada eval --swebench` runs require a running, maintained Docker Desktop or +Docker Engine release; see [Docker requirements](docs/configuration.md#docker-requirements) +for installation, verification, version policy, and resource guidance. + ## Quick start ```bash @@ -54,6 +59,8 @@ complete OS sandbox; use a disposable VM or container for unfamiliar projects. - [Configuration](docs/configuration.md): installation alternatives, API key, model settings, command policy, and trace levels. - [CLI reference](docs/cli-reference.md): `yada`, `yada eval`, and `yada-trace`. +- [Evaluation lifecycle](docs/evaluation.md): what `--case` and `--swebench` + load, mutate, cache, grade, and write. - [Contributing](CONTRIBUTING.md): development setup, validation, and the rebase pull-request workflow. - [Architecture](docs/dev/architecture.md): agent loop, tools, patch transaction, diff --git a/benchmarks/README.md b/benchmarks/README.md index abd9648..1bd8bfc 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -23,3 +23,8 @@ Cases under `swebench_verified/` use the canonical public problem statement and base commit. Their local grader is intended for fast development feedback. A published SWE-bench score must still come from the official Docker Harness. +This local path is intentional rather than a duplicate of `--swebench`: it +keeps prompt and tool regression tests fast, inspectable, and usable without +Docker. Use `--swebench INSTANCE_ID` when environment parity and an +official-compatible verdict matter; Yada then requires Docker before model +inference and separates the Agent command container from the final grader. diff --git a/docs/cli-reference.md b/docs/cli-reference.md index cd6f6c1..2e32401 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -53,48 +53,85 @@ keyboard interruption. Argument errors also use the standard argparse status `2` ## `yada eval` `yada eval` runs an agent behind a benchmark-neutral adapter and persists a -schema-versioned result even for ordinary adapter failures. +schema-versioned result even for ordinary adapter failures. Unlike a direct +`yada` run, an evaluation always has a task adapter and a grader. +See [Evaluation lifecycle](evaluation.md) for the complete load, workspace, +agent, grading, cache, and artifact sequence. -### Checked-in development case +### Choose a run mode + +```mermaid +flowchart TD + start{"What do you want to run?"} + start -->|"Edit one repository"| direct["yada TASK --workspace REPO"] + start -->|"Run a reproducible evaluation"| eval["yada eval"] + + eval --> source{"Which task source?"} + source -->|"Checked-in case directory"| case["--case PATH"] + source -->|"Official SWE-bench Verified"| swe["--swebench INSTANCE_ID"] + + case --> local_adapter["Local adapter\ncase.json + host/uv environment"] + local_adapter --> local_result["Development/private result\nnot an official SWE-bench score"] + + swe --> agent_container["Public instance image\nAgent commands in Docker"] + agent_container --> official["Separate official Harness\ngrading container"] + official --> official_result["Official-compatible SWE-bench result"] +``` + +`yada eval` deliberately exposes only two task selectors: + +| Invocation | Meaning | Native Yada command environment | Grader | +| --- | --- | --- | --- | +| `--case PATH` | Run a portable local case. A directory resolves to `PATH/case.json`; a manifest file may also be passed directly. | Host/uv environment declared by the case; Docker is not required by Yada | Command declared by the case manifest | +| `--swebench INSTANCE_ID` | Load one SWE-bench Verified task and generate an official-compatible prediction. | Public SWE-bench instance image in Docker | Separate official SWE-bench Harness container | + +There is no `--local`, `--benchmark`, or separate `--manifest` selector. Pass a +custom manifest directly to `--case`. A case is called local because its +manifest controls preparation and grading; it may still fetch a Git repository +and call the DeepSeek API. Each invocation runs exactly one task. + +#### Why `--case` is not redundant + +`--case` is the small, no-Docker evaluation path for prompt/tool regression, +private repositories, and custom test subsets. It is fast, inspectable, and can +run from fully local inputs. Its verdict belongs to the manifest and must not be +published as an official SWE-bench score. + +`--swebench` is the comparability path. It fixes the dataset and Harness policy, +requires a running Docker daemon before the first model request, runs native +Yada commands in the public instance image, and grades the patch in an +independent container that receives the evaluation tests. Collapsing these +selectors would either force Docker onto ordinary Yada development or make +local and official-looking results dangerously easy to confuse. + +### Checked-in development case (`--case`) + +Use a checked-in case for fast development and regression testing: ```bash uv run yada eval \ --case benchmarks/swebench_verified/pytest-10051 \ --agent yada \ - --yes + --yes \ + --trace-level debug ``` The first run fetches the pinned pytest commit and builds the case's locked Python environment. Later runs reuse caches but create a fresh agent workspace. -This local grader is a development feedback loop, not an official SWE-bench -Docker score. - -### Common options - -| Option | Meaning | Default | -| --- | --- | --- | -| `--benchmark local\|swebench` | Benchmark adapter. | Inferred from `--case`, otherwise required | -| `--case PATH` | Portable local case directory or `case.json`. | — | -| `--instance ID` | Benchmark instance ID. | Manifest ID for local cases | -| `--agent yada\|command` | Agent adapter. | `yada` | -| `--output PATH` | Result JSON path. | `eval-results/__