diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 4c0dba2b4b..be39b91d53 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -2,6 +2,9 @@ Classified by what is being measured. +- [deepswe/](deepswe/): coding task quality. Ten harnesses on the same 113 + tasks and model, with results, tested versions, public integration references, + release dates, and comparison limits. - [performance/](performance/): the footprint of the CLI itself. Disk, startup, memory at rest, memory as parallel sessions multiply, memory over a long session, and behaviour through interrupts and crashes. Seven agent diff --git a/docs/benchmarks/deepswe/README.md b/docs/benchmarks/deepswe/README.md index 9e5fed668c..0d1d1afb18 100644 --- a/docs/benchmarks/deepswe/README.md +++ b/docs/benchmarks/deepswe/README.md @@ -32,12 +32,77 @@ are senior-dev alone, not a comparison. | Verifiers | official DeepSWE at `0b9fabb` | | Budget | 3 h per task | | Isolation | four shards per harness, a dedicated OpenRouter key per harness | -| Ran | nine harnesses on 2026-09-11; senior-dev on 2026-09-12 | +| Ran | nine harnesses on 2026-09-10–11 UTC; senior-dev on 2026-09-12 UTC | [`arms.csv`](arms.csv) has one row per harness: solved, reward rate, valid grades, invalid outcomes, mean F2P and P2P, OpenRouter spend, cost per task, mean agent seconds and model. +## Tested versions and public references + +The nine competing harness versions below were checked against all 113 saved +task records per harness. senior-dev uses its public integration and first +stable release references, explained below. Release dates come from the linked +upstream releases; they are not installation dates. These references describe +the historical comparison and its public lineage. + +| harness | tested version / public reference | source revision / integration commit | release date (UTC) | +| --- | --- | --- | --- | +| senior-dev | First stable: codeaf [v0.5.0](https://github.com/Agent-Field/CodeAF/releases/tag/v0.5.0) | [`dd0fcc654`](https://github.com/Agent-Field/CodeAF/commit/dd0fcc654a4a77b4cbf6beae27e6af1cfe4e4c28) — [#1488](https://github.com/Agent-Field/CodeAF/pull/1488) | 2026-09-30 | +| mini-swe-agent | 2.4.6 | `a83fcae` | [2026-07-23](https://github.com/SWE-agent/mini-swe-agent/releases/tag/v2.4.6) | +| codex (Codex CLI) | 0.154.0 | `6b9826e` | [2026-09-09](https://github.com/openai/codex/releases/tag/rust-v0.154.0) | +| pi | 0.85.1 | `d981de1` | [2026-09-05](https://github.com/earendil-works/pi/releases/tag/v0.85.1) | +| omp (Oh My Pi) | 18.1.17 | `3b3a6dc` | [2026-09-10](https://github.com/can1357/oh-my-pi/releases/tag/v18.1.17) | +| opencode | 1.18.30 | `3104c14` | [2026-09-09](https://github.com/anomalyco/opencode/releases/tag/v1.18.30) | +| kilo | 7.6.2 | `3d04228` | [2026-09-10](https://github.com/Kilo-Org/kilocode/releases/tag/v7.6.2) | +| claude-code | 2.1.267 | — | [2026-09-09](https://github.com/anthropics/claude-code/releases/tag/v2.1.267) | +| deepseek-harness (`@deepseek-ai/dsh`) | 0.1.5-rc.1 | — | [2026-09-10](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.1) | +| muse-code | 1.1.1-R2514.1 | — | Not independently established | + +[`versions.csv`](versions.csv) records the same versions alongside source +revisions where recorded, public release timestamps and source URLs. Its +`harness` values join to `arms.csv`; revisions are stored in full where available. +Empty revision, release timestamp, or URL fields mean that information was not +established for that row. + +For senior-dev, `version` names the first stable codeaf release containing the +agent, and `source_revision` names the public integration commit from #1488. +That commit is contained in v0.5.0; it is distinct from the release tag's commit, +`4949397498f775a80de16d3e6429b0f6f933f54c`. The benchmark was run before the +integration. This row traces where the measured agent became publicly available +in codeaf; it does not describe a new benchmark run on v0.5.0. + +Muse Code's version comes from the frozen benchmark records. Its historical +publication date and whether a newer build was available at launch were not +independently established. + +## Release timing and scope + +Most competing harness versions were released within a week of the campaign. +mini-swe-agent 2.4.6 was about seven weeks old, but was still the latest +[published release](https://github.com/SWE-agent/mini-swe-agent/releases) +at launch. The comparison does not establish how newer versions perform. + +- **OpenCode was 1.x.** Version 1.18.30 was the latest production 1.x + release at launch; [1.18.31](https://github.com/anomalyco/opencode/releases/tag/v1.18.31) + followed on September 14. V2 development already existed, as a + [contemporaneous upstream issue](https://github.com/anomalyco/opencode/issues/42878) + shows. This row does not measure OpenCode 2. +- **Claude Code was pinned before a patch release.** Version + [2.1.268](https://github.com/anthropics/claude-code/releases/tag/v2.1.268) + was published on September 10 at 20:30:54 UTC, before 112 of this arm's 113 + tasks started. Its release notes include a fix for HTTP 400 responses on + third-party Anthropic-compatible endpoints. The campaign used such an + endpoint; whether that fix would change the result has not been established. + Version [2.1.269](https://github.com/anthropics/claude-code/releases/tag/v2.1.269) + arrived after all tasks had started, near the end of the arm. +- **DeepSeek Harness had a newer release candidate.** Version + [0.1.5-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.2) + was published on September 10 at 15:09:34 UTC, before this arm ran. The + frozen package lock selected npm's `latest` tag at 0.1.5-rc.1. The rc.2 + release notes describe feedback dialogs and interface layout changes; no + run on rc.2 is included here. + ## How the columns in the README are derived - **solved**: tasks the verifier passed, out of 113. diff --git a/docs/benchmarks/deepswe/versions.csv b/docs/benchmarks/deepswe/versions.csv new file mode 100644 index 0000000000..cc8086dcea --- /dev/null +++ b/docs/benchmarks/deepswe/versions.csv @@ -0,0 +1,11 @@ +harness,version,source_revision,release_published_at_utc,source_url +senior-dev,v0.5.0,dd0fcc654a4a77b4cbf6beae27e6af1cfe4e4c28,2026-09-30T02:08:08Z,https://github.com/Agent-Field/CodeAF/releases/tag/v0.5.0 +mini-swe-agent,2.4.6,a83fcae82d2a08f0ee0c688f9d137b3566c097f8,2026-07-23T03:12:33Z,https://github.com/SWE-agent/mini-swe-agent/releases/tag/v2.4.6 +codex,0.154.0,6b9826e3aa83b1a5947db50f4332cb9c65f1b340,2026-09-09T22:35:38Z,https://github.com/openai/codex/releases/tag/rust-v0.154.0 +pi,0.85.1,d981de1229ef899957bbe968bc8dcda02a21f477,2026-09-05T12:29:01Z,https://github.com/earendil-works/pi/releases/tag/v0.85.1 +omp,18.1.17,3b3a6dc9bbd85102ce19d0b1c11bf6870915f6ec,2026-09-10T19:33:46Z,https://github.com/can1357/oh-my-pi/releases/tag/v18.1.17 +opencode,1.18.30,3104c1428ec91f809e5ab86631300de41eb6952e,2026-09-09T03:34:27Z,https://github.com/anomalyco/opencode/releases/tag/v1.18.30 +kilo,7.6.2,3d04228b6a642acb3daf68a649269618b6018250,2026-09-10T17:39:12Z,https://github.com/Kilo-Org/kilocode/releases/tag/v7.6.2 +deepseek-harness,0.1.5-rc.1,,2026-09-10T03:09:00Z,https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.1 +claude-code,2.1.267,,2026-09-09T19:58:14Z,https://github.com/anthropics/claude-code/releases/tag/v2.1.267 +muse-code,1.1.1-R2514.1,,, diff --git a/docs/changes/unreleased/1877-deepswe-versions.md b/docs/changes/unreleased/1877-deepswe-versions.md new file mode 100644 index 0000000000..cd96c4e35f --- /dev/null +++ b/docs/changes/unreleased/1877-deepswe-versions.md @@ -0,0 +1,14 @@ +--- +kind: changed +title: DeepSWE results identify the tested harness versions and release timing +pr: 1877 +surface: [docs] +invalidates: + - "The DeepSWE comparison listed scores without the tested harness versions, leaving its OpenCode generation unclear. The results page now identifies the nine competing versions, including OpenCode 1.18.30, and senior-dev's public integration in #1488 and first stable release v0.5.0; versions.csv records versions, revisions and public release references." + - "The setup dated all competing runs to September 11. Saved task metadata spans September 10–11 UTC; senior-dev ran on September 12 UTC." + - "The benchmark index listed only CLI footprint measurements. It now also links the DeepSWE task-quality comparison and its version evidence." +--- + +The comparison names the measured versions and the patch releases available at +launch without extending the results to newer versions. Published scores are +unchanged.