Add eval suite and upgrade api key load method

.gitignore

@@ -8,5 +8,9 @@ build/
 dist/
 *.egg-info/
 .yada/
+# Single-task and suite evaluation outputs, including workspaces and traces.
 eval-results/
+# Local background-run supervisor files from the documented suite command.
+eval_run.log
+eval_run.pid
 logs/

README-cn.md

@@ -32,12 +32,19 @@ git clone https://github.com/GenTang/Yada.git
 cd Yada
 uv sync --locked --dev
 
-export DEEPSEEK_API_KEY="sk-..."
+install -d -m 700 ~/.config/yada
+(umask 077; touch ~/.config/yada/deepseek_api_key)
+chmod 600 ~/.config/yada/deepseek_api_key
+${EDITOR:-vi} ~/.config/yada/deepseek_api_key
 
 uv run yada "修复 parser 的边界问题，并运行相关测试" \
   --workspace /path/to/repository
 ```
 
+文件中只写 API Key。对于 Docker、云主机或 Secret Manager 挂载，也可以使用
+`--api-key-file /run/secrets/deepseek_api_key`；详见
+[配置文档](docs/configuration.md#deepseek-credentials)。
+
 Yada 默认会在运行仓库命令前请求确认。只有在可信、一次性的隔离环境中才应使用
 `--yes`：
 

README.md

@@ -32,12 +32,19 @@ git clone https://github.com/GenTang/Yada.git
 cd Yada
 uv sync --locked --dev
 
-export DEEPSEEK_API_KEY="sk-..."
+install -d -m 700 ~/.config/yada
+(umask 077; touch ~/.config/yada/deepseek_api_key)
+chmod 600 ~/.config/yada/deepseek_api_key
+${EDITOR:-vi} ~/.config/yada/deepseek_api_key
 
 uv run yada "Fix the failing parser edge case and run the relevant tests" \
   --workspace /path/to/repository
 ```
 
+Enter only the API key in that file. Yada also accepts
+`--api-key-file /run/secrets/deepseek_api_key` for mounted secret stores; see
+[Configuration](docs/configuration.md#deepseek-credentials).
+
 Yada asks before running repository commands. Use `--yes` only inside a trusted,
 disposable environment:
 

benchmarks/suites/swebench-verified-canary-v1.json

@@ -0,0 +1,16 @@
+{
+  "schema_version": 1,
+  "suite_id": "swebench-verified-canary-v1",
+  "benchmark": "swebench-verified",
+  "description": "A small, versioned SWE-bench Verified canary for development signal before larger runs.",
+  "instances": [
+    "pytest-dev__pytest-10051",
+    "pytest-dev__pytest-10081",
+    "pytest-dev__pytest-10356",
+    "django__django-15987",
+    "sympy__sympy-19637",
+    "sphinx-doc__sphinx-9367",
+    "scikit-learn__scikit-learn-13439",
+    "pydata__xarray-6461"
+  ]
+}

docs/cli-reference.md

@@ -32,6 +32,7 @@ uv run yada --task-file issue.md --workspace /path/to/repository
 | `TASK` | Natural-language coding task. | — |
 | `--task-file PATH` | Read the task from a UTF-8 file. | — |
 | `--workspace PATH` | Target Git workspace. | Current directory |
+| `--api-key-file PATH` | Read the DeepSeek key from a private file. | Per-user config file, then compatibility environment fallback |
 | `--model NAME` | DeepSeek model name. | `DEEPSEEK_MODEL` or `deepseek-v4-pro` |
 | `--base-url URL` | DeepSeek-compatible API base URL. | `DEEPSEEK_BASE_URL` or `https://api.deepseek.com` |
 | `--reasoning-effort high\|max` | Thinking effort. | `max` |
@@ -290,6 +291,7 @@ container; Yada's automatic Agent command container applies only to the native
 | `--max-steps N` | Model-turn budget. | `30` |
 | `--wall-time SECONDS` | Comparable wall-time budget. | `1800` |
 | `--max-output-tokens N` | Per-completion token limit. | `16384` |
+| `--api-key-file PATH` | Private DeepSeek credential file for the native Yada agent. | Per-user config file, then compatibility environment fallback |
 | `--editing-strategy patch-only\|replace-first` | Native Yada editing policy. | `replace-first` |
 
 The native agent also accepts the model, thinking, timeout, command-policy, and

docs/configuration.md

@@ -28,21 +28,35 @@ After editable installation, use `.venv/bin/yada`; after `uv sync`, use
 
 ## DeepSeek credentials
 
-`DEEPSEEK_API_KEY` is required when the native Yada agent calls DeepSeek:
+The recommended setup is a private file at
+`~/.config/yada/deepseek_api_key`:
 
 ```bash
-export DEEPSEEK_API_KEY="sk-..."
+install -d -m 700 ~/.config/yada
+(umask 077; touch ~/.config/yada/deepseek_api_key)
+chmod 600 ~/.config/yada/deepseek_api_key
+${EDITOR:-vi} ~/.config/yada/deepseek_api_key
 ```
 
-PowerShell:
+Put only the API key in the file. On Linux and macOS, it must not grant any
+permissions to group or others (`0600` is recommended; `0400` is also valid).
+To use another location, pass it explicitly:
 
-```powershell
-$env:DEEPSEEK_API_KEY = "sk-..."
+```bash
+uv run yada "Fix the failing test" \
+  --api-key-file /run/secrets/deepseek_api_key \
+  --workspace /path/to/repository
 ```
 
-Do not put the key in a task file, trace, issue, or commit. Yada sends it only in
-the DeepSeek authorization header and removes secret-looking environment
-variables from repository subprocesses.
+Credential resolution order is:
+
+1. `--api-key-file PATH`;
+2. `DEEPSEEK_API_KEY_FILE`, containing a file path;
+3. `~/.config/yada/deepseek_api_key`; and
+4. `DEEPSEEK_API_KEY`, retained for compatibility.
+
+On Windows, the default file is `%APPDATA%\Yada\deepseek_api_key`. Never put
+the key in a command argument, task, trace, image, issue, or commit.
 
 ## Model and endpoint
 
@@ -240,9 +254,12 @@ network access:
 ```bash
 docker build -t yada .
 docker run --rm -it \
-  -e DEEPSEEK_API_KEY \
+  -v "$HOME/.config/yada/deepseek_api_key:/run/secrets/deepseek_api_key:ro" \
   -v "/path/to/repository:/workspace" \
-  yada "Fix the failing test" --workspace /workspace --yes
+  yada "Fix the failing test" \
+  --workspace /workspace \
+  --api-key-file /run/secrets/deepseek_api_key \
+  --yes
 ```
 
 Use a stronger sandbox when the repository or its dependencies are untrusted.

docs/dev/debugging.md

@@ -243,7 +243,9 @@ SWE-bench score. Use the official Docker grader for published results; see the
 
 ## Common failure signals
 
-- **`DEEPSEEK_API_KEY is not set`**: export the key in the shell launching Yada.
+- **`DeepSeek API key not found`**: create the private default credential file,
+  pass `--api-key-file`, or configure `DEEPSEEK_API_KEY_FILE`; see
+  [DeepSeek credentials](../configuration.md#deepseek-credentials).
 - **No request payload in a trace**: rerun with `--trace-level debug`.
 - **`finish_task` rejected**: run a successful `test` or `build` after the latest patch.
 - **Tool reports `ok` but tests failed**: inspect the command `exit_code`.

docs/evaluation.md

@@ -43,7 +43,8 @@ Before this pipeline starts, the CLI:
 2. resolves the result and artifact paths;
 3. creates a run ID containing UTC time and a random suffix;
 4. configures the native Yada or external command agent; and
-5. requires `DEEPSEEK_API_KEY` when the native Yada agent is selected.
+5. resolves a private API key file (or the compatibility environment fallback)
+   when the native Yada agent is selected.
 
 The default result name uses system-local time at minute precision. A collision
 adds `(1)`, `(2)`, and so on. The result and artifact directory always receive
@@ -84,6 +85,136 @@ Without Docker, use direct `yada ...` for normal repository work or
 may choose to invoke Docker itself, but Docker is not a Yada requirement for
 this path.
 
+## Choose the smallest useful evaluation scale
+
+Evaluation should expand in explicit stages. A larger stage costs more time,
+model tokens, network traffic, and Docker storage; it does not replace the
+faster signal from the stages before it.
+
+| Scale | Use it for | Typical task set | Meaning |
+| --- | --- | --- | --- |
+| Local case | Developing or debugging a prompt, tool, adapter, or grader change | One checked-in `--case` recipe | Fast project-specific regression signal; not an official SWE-bench result |
+| Canary suite | Checking that an evaluation-affecting change behaves plausibly across projects before spending on a pilot | The versioned Canary-8 manifest | Repeatable development signal with aggregate metrics and resumable execution |
+| Pilot sample | Estimating variance, failures, and operational cost before a full run | A fixed, reviewed sample larger than the canary | Preflight evidence; record and version the sample rather than selecting tasks during the run |
+| Full benchmark | Producing the final result after the implementation and run configuration are frozen | All 500 SWE-bench Verified tasks | The only full-dataset result, still requiring complete methodology and provenance for any comparison |
+
+Run the checked-in Canary-8 suite with the pinned Harness environment:
+
+```bash
+uv run --with 'swebench==4.1.0' \
+  python scripts/eval_suite.py run \
+  benchmarks/suites/swebench-verified-canary-v1.json \
+  --api-key-file ~/.config/yada/deepseek_api_key
+```
+
+The development-only runner calls the existing single-task command once per
+instance, sequentially:
+
+```text
+python -m yada eval --swebench INSTANCE_ID
+```
+
+It does not add a public `yada eval-suite` command or code under `src/yada`.
+Every instance runs once unless the whole suite receives `--repeat N`; there are
+no per-instance repeat overrides. Model, budget, and secret-file options also
+apply uniformly to every attempt. Only the secret path may enter suite run
+metadata; the key value is never copied.
+
+The runner prints the new suite directory under `eval-results/suites/`. That
+directory contains a manifest snapshot, run metadata, isolated attempt
+directories, and deterministic `summary.json` and `summary.md` files. It is
+already covered by the repository's `eval-results/` ignore rule, so raw
+workspaces, traces, Harness logs, and Docker data are not committed.
+
+After Ctrl-C or another interruption, resume the printed directory:
+
+```bash
+uv run --with 'swebench==4.1.0' \
+  python scripts/eval_suite.py run \
+  benchmarks/suites/swebench-verified-canary-v1.json \
+  --resume eval-results/suites/SUITE_DIRECTORY
+```
+
+Resume requires the same manifest hash, Yada commit, model, budgets, and global
+repeat count. Completed attempts are skipped. An attempt whose atomic Yada
+result was written immediately before interruption is recovered; an incomplete
+workspace is retained for diagnosis and the retry uses a new execution
+directory. Ordinary resolved, unresolved, and error outcomes never stop later
+tasks.
+
+The summaries report outcome counts and resolution rate; per-attempt and
+per-instance steps, token usage, and Agent duration; repeated-run min, max, and
+median; semantic patch IDs and patch convergence; and result, artifact, and
+trace paths. They record only safe run configuration and never copy API keys
+from any credential source. Canary and pilot summaries are development
+evidence, not official leaderboard scores.
+
+## Long-running suites on a remote host
+
+Yada calls a remote model API and runs SWE-bench in Docker, so this workload
+does not need a GPU. Prefer an `x86_64` Linux host: upstream SWE-bench recommends
+at least 8 CPU cores, 16 GB RAM, and 120 GB of free storage, and describes ARM
+support as experimental. The suite runner is intentionally sequential, so a
+larger machine improves an individual build or test but does not make multiple
+instances run concurrently.
+
+For AWS, a current-generation general-purpose Intel instance such as
+`m8i.2xlarge` (8 vCPU, 32 GiB), with `m7i.2xlarge` as a broadly available
+fallback, is a practical starting point. Use an encrypted persistent `gp3` EBS
+volume rather than instance storage. The upstream 120 GB figure covers the
+Harness baseline; Yada also retains result workspaces and traces. Start around
+300 GB for canary and pilot work, and size a full 500-task volume from measured
+pilot growth—500 GB to 1 TB is a safer initial range when retaining every raw
+workspace. Monitor both `df -h` and `docker system df`.
+
+Use an EC2 instance role, not long-lived AWS access keys. Give it only
+`secretsmanager:GetSecretValue` for the DeepSeek secret, the permissions needed
+for Systems Manager, and optional access to a dedicated result bucket. Systems
+Manager Session Manager allows administration without an inbound SSH rule.
+Materialize the DeepSeek value into a private memory-backed file before the run:
+
+```bash
+install -d -m 700 /dev/shm/yada
+umask 077
+aws secretsmanager get-secret-value \
+  --secret-id yada/deepseek-api-key \
+  --query SecretString \
+  --output text \
+  --no-cli-pager \
+  > /dev/shm/yada/deepseek_api_key
+chmod 600 /dev/shm/yada/deepseek_api_key
+```
+
+Then run with unbuffered logs and place both the checkout and output directory
+on EBS:
+
+```bash
+nohup env PYTHONUNBUFFERED=1 \
+  uv run --with 'swebench==4.1.0' \
+  python -u scripts/eval_suite.py run \
+  benchmarks/suites/swebench-verified-canary-v1.json \
+  --api-key-file /dev/shm/yada/deepseek_api_key \
+  --repeat 3 \
+  --max-steps 60 \
+  --output-dir /data/yada-results/canary-v1-r3 \
+  > /data/yada-results/canary-v1-r3.log 2>&1 < /dev/null &
+```
+
+Use On-Demand for the first long run. Spot can reduce compute cost after resume
+has been tested, but interruption notice is short and a terminated instance can
+lose its root volume by default. Keep the suite directory on persistent EBS,
+preserve that volume on termination, and expect the current attempt to resume
+in a new execution directory. Copy `summary.json`, `summary.md`, and any needed
+diagnostics to encrypted object storage; raw traces and workspaces may contain
+sensitive source or model reasoning and should not be uploaded indiscriminately.
+
+References: [SWE-bench Docker setup](https://www.swebench.com/SWE-bench/guides/docker_setup/),
+[EC2 general-purpose instances](https://aws.amazon.com/ec2/instance-types/general-purpose/),
+[EBS gp3](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html),
+[EC2 IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html),
+[Secrets Manager GetSecretValue](https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html),
+and [Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html).
+
 ## Local case: `--case PATH`
 
 Example: