|
| 1 | +# Pythinker Benchmark |
| 2 | + |
| 3 | +Pythinker Benchmark is the native local-fixture harness for comparing how a configured Pythinker model handles small, deterministic coding tasks. It runs inside the current Pythinker session, uses the active toolset and approval runtime, materializes each task into an isolated workspace, executes the agent turn, then runs the task's verification command and writes replayable artifacts. |
| 4 | + |
| 5 | +It is designed for repeatable local checks, not hosted leaderboard scoring. SWE-style input is supported as trusted local fixtures; it is not a full SWE-bench Docker runner. |
| 6 | + |
| 7 | +## Commands |
| 8 | + |
| 9 | +Run the default core suite: |
| 10 | + |
| 11 | +```sh |
| 12 | +/benchmark start |
| 13 | +``` |
| 14 | + |
| 15 | +Run a single task or named suite: |
| 16 | + |
| 17 | +```sh |
| 18 | +/benchmark start --task core-safe-path-join |
| 19 | +/benchmark start --suite pythinker-smoke |
| 20 | +``` |
| 21 | + |
| 22 | +Estimate a run without making model calls: |
| 23 | + |
| 24 | +```sh |
| 25 | +/benchmark estimate --suite pythinker-core |
| 26 | +``` |
| 27 | + |
| 28 | +Inspect available tasks and saved runs: |
| 29 | + |
| 30 | +```sh |
| 31 | +/benchmark list |
| 32 | +/benchmark show <run-id> |
| 33 | +/benchmark report --suite pythinker-core |
| 34 | +``` |
| 35 | + |
| 36 | +Compare configured models and export report data: |
| 37 | + |
| 38 | +```sh |
| 39 | +/benchmark compare --models model-a,model-b --suite pythinker-core --repeat 3 |
| 40 | +/benchmark export --suite pythinker-core --format csv |
| 41 | +``` |
| 42 | + |
| 43 | +Discover candidate tasks from an allowlisted online source: |
| 44 | + |
| 45 | +```sh |
| 46 | +/benchmark discover --source terminal-bench --difficulty hard --limit 5 --output ./candidate-tasks.jsonl |
| 47 | +``` |
| 48 | + |
| 49 | +Namespaced aliases are available for interactive completion: `/benchmark:start`, `/benchmark:all`, `/benchmark:estimate`, `/benchmark:list`, `/benchmark:show`, `/benchmark:report`, `/benchmark:compare`, and `/benchmark:swe`. There is no `/benchmark:export` or `/benchmark:discover` alias. |
| 50 | + |
| 51 | +The supported flags are: |
| 52 | + |
| 53 | +| Flag | Applies to | Behavior | |
| 54 | +| --- | --- | --- | |
| 55 | +| `--model <model-key>` | `start`, `estimate`, `swe` | Uses a configured model instead of the active/default model. | |
| 56 | +| `--models <model-a,model-b>` | `compare` | Runs each selected task for at least two distinct configured models. | |
| 57 | +| `--task <task-id>` | `start`, `estimate`, `compare` | Runs, estimates, or compares one bundled task. Mutually exclusive with `--suite`. | |
| 58 | +| `--suite <suite-name>` | `start`, `estimate`, `report`, `export`, `compare` | Selects a bundled suite or filters report/export output. | |
| 59 | +| `--repeat <n>` | `start`, `estimate`, `compare`, `swe` | Runs or estimates each selected task multiple times. | |
| 60 | +| `--timeout-seconds <n>` | `start`, `compare`, `swe` | Overrides the task timeout for the agent turn. | |
| 61 | +| `--format json\|csv` | `export` | Selects the export format. | |
| 62 | +| `--output <path>` | `start`, `compare`, `show`, `report`, `export`, `swe`; `.jsonl` only for `discover` | Uses a custom benchmark artifact root for run/report/export commands. For `discover`, it only writes a provisional manifest when the path suffix is `.jsonl`. | |
| 63 | +| `--dataset <path.jsonl>` | `swe` | Loads SWE-style local fixture records from a JSONL file. | |
| 64 | +| `--instance <instance-id>` | `swe` | Runs only one instance from the dataset. | |
| 65 | +| `--trusted-dataset true` | `swe` | Required acknowledgement before dataset verification commands can run. | |
| 66 | +| `--source <allowlisted>` | `discover` | Selects an allowlisted benchmark source such as `terminal-bench`. | |
| 67 | +| `--difficulty <difficulty>` | `discover` | Filters discovered candidates by difficulty. Defaults to `hard`. | |
| 68 | +| `--limit <n>` | `discover` | Limits discovered candidates. Defaults to `5`. | |
| 69 | + |
| 70 | +`--max-concurrency` is parsed but must remain `1` in the current implementation. `--judges` is parsed but only `off` is supported. |
| 71 | + |
| 72 | +## Architecture |
| 73 | + |
| 74 | +The benchmark integration has four layers: |
| 75 | + |
| 76 | +1. Slash commands in `src/pythinker_code/soul/slash.py` register `/benchmark` and the namespaced aliases. |
| 77 | +2. `src/pythinker_code/benchmark/commands.py` parses arguments, resolves the active model, expands tasks or suites, and dispatches each run. |
| 78 | +3. `src/pythinker_code/benchmark/runner.py` prepares the workspace, temporarily applies the task's `max_steps` to the soul loop, overrides the runtime work directory, calls `PythinkerSoul.turn`, runs verification, and restores the previous runtime state in a `finally` block. |
| 79 | +4. `src/pythinker_code/benchmark/records.py` and `src/pythinker_code/benchmark/report.py` persist run metadata, traces, summaries, context and Wire slices, and Markdown reports. |
| 80 | + |
| 81 | +Task and suite definitions are regular bundled JSON files under `src/pythinker_code/benchmark/bundled/`. `src/pythinker_code/benchmark/tasks.py` validates bundled task shape and rejects unsafe workspace paths before files are materialized. |
| 82 | + |
| 83 | +## Bundled suites |
| 84 | + |
| 85 | +`pythinker-core` is the default suite. It covers deterministic local coding tasks that expose common agent failure modes: |
| 86 | + |
| 87 | +- `core-atomic-transfer`: transactional rollback, missing accounts, insufficient funds, and non-positive transfer amounts. |
| 88 | +- `core-dedup-order`: ordered de-duplication for lists and generators, including falsey values. |
| 89 | +- `core-explicit-none-metadata`: explicit `None` handling without losing valid falsey metadata values. |
| 90 | +- `core-safe-path-join`: path traversal defense, absolute-path rejection, sibling-prefix escapes, and symlink escapes. |
| 91 | + |
| 92 | +`pythinker-smoke` contains smaller tasks for validating the runner itself: |
| 93 | + |
| 94 | +- `smoke-edit-readme` |
| 95 | +- `smoke-fix-python-test` |
| 96 | +- `smoke-add-small-function` |
| 97 | + |
| 98 | +## Task schema |
| 99 | + |
| 100 | +A bundled task JSON object contains: |
| 101 | + |
| 102 | +```json |
| 103 | +{ |
| 104 | + "id": "core-safe-path-join", |
| 105 | + "title": "Safe Path Join", |
| 106 | + "description": "Reject path traversal while allowing paths inside the root.", |
| 107 | + "prompt": "Fix paths.safe_join ...", |
| 108 | + "workspace": { |
| 109 | + "files": { |
| 110 | + "paths.py": "...", |
| 111 | + "test_paths.py": "..." |
| 112 | + } |
| 113 | + }, |
| 114 | + "verification": { |
| 115 | + "type": "command", |
| 116 | + "command": "python -m pytest test_paths.py -q" |
| 117 | + }, |
| 118 | + "limits": { |
| 119 | + "timeout_seconds": 120, |
| 120 | + "max_steps": 60 |
| 121 | + }, |
| 122 | + "tags": ["core", "security"] |
| 123 | +} |
| 124 | +``` |
| 125 | + |
| 126 | +Workspace paths must be relative, non-empty, and must not contain `..` path segments. Verification type is currently `command`. |
| 127 | + |
| 128 | +## Publishable comparisons |
| 129 | + |
| 130 | +Use `/benchmark compare` when comparing configured models: |
| 131 | + |
| 132 | +```sh |
| 133 | +/benchmark compare --models model-a,model-b --suite pythinker-core --repeat 3 |
| 134 | +``` |
| 135 | + |
| 136 | +Export the saved report rows when you need machine-readable results: |
| 137 | + |
| 138 | +```sh |
| 139 | +/benchmark export --suite pythinker-core --format csv |
| 140 | +``` |
| 141 | + |
| 142 | +Reports include publishability warnings. Treat warnings as blockers for public claims, not as lint. Local fixture runs are useful for regression and internal comparison, but they are not SWE-bench Docker evaluations. |
| 143 | + |
| 144 | +## Online discovery and quiz fixtures |
| 145 | + |
| 146 | +`/benchmark discover` fetches metadata from allowlisted benchmark sources and writes provisional manifests. It does not execute source-provided commands and does not make discovered tasks trusted: |
| 147 | + |
| 148 | +```sh |
| 149 | +/benchmark discover --source terminal-bench --difficulty hard --limit 5 --output ./candidate-tasks.jsonl |
| 150 | +``` |
| 151 | + |
| 152 | +The `--output` flag writes a manifest only when the path suffix is `.jsonl`. These JSONL records preserve the source URL and are marked `trusted: false`. Online quiz fixture records use deterministic review metadata: |
| 153 | + |
| 154 | +```json |
| 155 | +{ |
| 156 | + "verification": { |
| 157 | + "type": "answer_contains", |
| 158 | + "expected_substrings": ["terminal-bench", "hard"] |
| 159 | + }, |
| 160 | + "trusted": false, |
| 161 | + "workspace": { |
| 162 | + "files": {} |
| 163 | + } |
| 164 | +} |
| 165 | +``` |
| 166 | + |
| 167 | +These manifests are for dataset review and offline conversion first. They are not runnable through `/benchmark:swe`, and `answer_contains` is not executed by the benchmark runner. Convert reviewed tasks into trusted local fixtures with workspace files and a local verification command before running them. |
| 168 | + |
| 169 | +## SWE-style local fixtures |
| 170 | + |
| 171 | +`/benchmark:swe` loads newline-delimited JSON records with local workspace files and a verification command. The command is executed on the local machine after the agent turn, so the slash command refuses to run unless `--trusted-dataset true` is present: |
| 172 | + |
| 173 | +```sh |
| 174 | +/benchmark:swe --dataset ./cases.jsonl --trusted-dataset true |
| 175 | +``` |
| 176 | + |
| 177 | +Each record must include `instance_id`, `repo`, `base_commit`, `problem_statement`, `workspace.files`, and `verification.command`. Optional `FAIL_TO_PASS`, `PASS_TO_PASS`, and `limits` fields are folded into the generated local fixture task. |
| 178 | + |
| 179 | +## Artifacts |
| 180 | + |
| 181 | +By default, benchmark runs are written under the Pythinker share directory in `benchmarks/<run-id>/`. A custom root can be supplied with `--output <path>`. |
| 182 | + |
| 183 | +Each run directory contains: |
| 184 | + |
| 185 | +| File or directory | Contents | |
| 186 | +| --- | --- | |
| 187 | +| `run.json` | Run metadata: command, model key, provider key, task id, suite name, repeat index, timestamps, and final status. | |
| 188 | +| `summary.json` | Runtime summary: duration, step count, tool calls, changed files, token counts, verification status, and exit reason. | |
| 189 | +| `report.md` | Human-readable report for the run. | |
| 190 | +| `trace.jsonl` | Benchmark event trace, including workspace preparation, model message, and verification result. | |
| 191 | +| `workspace/` | The materialized local task workspace after the run. | |
| 192 | +| `context.jsonl` and `wire.jsonl` | Slices copied from the active session for replay and debugging. | |
| 193 | + |
| 194 | +Generated verification caches such as `__pycache__`, `.pytest_cache`, `.ruff_cache`, and `.mypy_cache` are excluded from changed-file summaries. |
0 commit comments