From 4ffc7414a2930496096534d86a9e8f1d2d4cf1de Mon Sep 17 00:00:00 2001 From: ZeroPoint95 <329227198+ZeroPoint95@users.noreply.github.com> Date: Mon, 19 Oct 2026 11:03:36 -0400 Subject: [PATCH 1/3] docs: record DeepSWE harness versions and release context --- docs/benchmarks/README.md | 3 ++ docs/benchmarks/deepswe/README.md | 71 +++++++++++++++++++++++++++- docs/benchmarks/deepswe/versions.csv | 11 +++++ 3 files changed, 84 insertions(+), 1 deletion(-) create mode 100644 docs/benchmarks/deepswe/versions.csv diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 4c0dba2b4b..6caf20b1ed 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -2,6 +2,9 @@ Classified by what is being measured. +- [deepswe/](deepswe/): coding task quality. Ten harnesses on the same 113 + tasks and model, with results, tested versions, release dates, artifact + checksums, and comparison limits. - [performance/](performance/): the footprint of the CLI itself. Disk, startup, memory at rest, memory as parallel sessions multiply, memory over a long session, and behaviour through interrupts and crashes. Seven agent diff --git a/docs/benchmarks/deepswe/README.md b/docs/benchmarks/deepswe/README.md index 9e5fed668c..aa0da356dd 100644 --- a/docs/benchmarks/deepswe/README.md +++ b/docs/benchmarks/deepswe/README.md @@ -32,12 +32,81 @@ are senior-dev alone, not a comparison. | Verifiers | official DeepSWE at `0b9fabb` | | Budget | 3 h per task | | Isolation | four shards per harness, a dedicated OpenRouter key per harness | -| Ran | nine harnesses on 2026-09-11; senior-dev on 2026-09-12 | +| Ran | nine harnesses on 2026-09-10–11 UTC; senior-dev on 2026-09-12 UTC | [`arms.csv`](arms.csv) has one row per harness: solved, reward rate, valid grades, invalid outcomes, mean F2P and P2P, OpenRouter spend, cost per task, mean agent seconds and model. +## Versions tested + +These are the versions used for the comparison above, checked against all 113 +saved task records per harness. Release dates come from the linked upstream +releases; they are not installation dates. The dates and versions describe this +campaign, not the versions available today. + +| harness | version tested | release date (UTC) | +| --- | --- | --- | +| senior-dev | Benchmark build `8ae964c` | Not a release tag | +| mini-swe-agent | 2.4.6 | [2026-07-23](https://github.com/SWE-agent/mini-swe-agent/releases/tag/v2.4.6) | +| codex (Codex CLI) | 0.154.0 | [2026-09-09](https://github.com/openai/codex/releases/tag/rust-v0.154.0) | +| pi | 0.85.1 | [2026-09-05](https://github.com/earendil-works/pi/releases/tag/v0.85.1) | +| omp (Oh My Pi) | 18.1.17 | [2026-09-10](https://github.com/can1357/oh-my-pi/releases/tag/v18.1.17) | +| opencode | 1.18.30 | [2026-09-09](https://github.com/anomalyco/opencode/releases/tag/v1.18.30) | +| kilo | 7.6.2 | [2026-09-10](https://github.com/Kilo-Org/kilocode/releases/tag/v7.6.2) | +| claude-code | 2.1.267 | [2026-09-09](https://github.com/anthropics/claude-code/releases/tag/v2.1.267) | +| deepseek-harness (`@deepseek-ai/dsh`) | 0.1.5-rc.1 | [2026-09-10](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.1) | +| muse-code | 1.1.1-R2514.1 | Not independently established | + +[`versions.csv`](versions.csv) records the same versions alongside source +revisions where recorded, release timestamps and source URLs, the artifact +checksums from frozen provenance records, and the first start and last finish +in each arm's saved task metadata. Its `harness` values join to `arms.csv`. +Those time bounds include setup and grading as recorded; they are not the mean +agent times in `arms.csv`. + +The artifact kind states which bytes the checksum identifies: an executable, +release archive, prepared Python payload, or npm package tarball. An archive +checksum is not the checksum of its unpacked executable. The npm checksum is +SHA-1 as recorded in its package lock; the others are SHA-256. Empty source +revision, release timestamp, or URL fields mean that information was not +established for that artifact. A source URL identifies its upstream release or +package, not necessarily a download of the prepared benchmark payload. + +senior-dev's build identifier names the measured agent before its integration +into codeaf; it is not a codeaf release tag. Muse Code's frozen lock records the +build selected from [Meta's stable channel](https://api.meta.ai/muse-code/channels/muse-stable) +and its [versioned artifact manifest](https://lookaside.facebook.com/lookaside/muse/download/?channel=muse&version=1.1.1-R2514.1&file=manifest.json). +Its historical publication date and whether a newer build was available at +launch were not independently established. + +## Release timing and scope + +Most tested versions were released within a week of the campaign. +mini-swe-agent 2.4.6 was about seven weeks old, but was still the latest +[published release](https://github.com/SWE-agent/mini-swe-agent/releases) +at launch. The comparison does not establish how newer versions perform. + +- **OpenCode was 1.x.** Version 1.18.30 was the latest production 1.x + release at launch; [1.18.31](https://github.com/anomalyco/opencode/releases/tag/v1.18.31) + followed on September 14. V2 development already existed, as a + [contemporaneous upstream issue](https://github.com/anomalyco/opencode/issues/42878) + shows. This row does not measure OpenCode 2. +- **Claude Code was pinned before a patch release.** Version + [2.1.268](https://github.com/anthropics/claude-code/releases/tag/v2.1.268) + was published on September 10 at 20:30:54 UTC, before 112 of this arm's 113 + tasks started. Its release notes include a fix for HTTP 400 responses on + third-party Anthropic-compatible endpoints. The campaign used such an + endpoint; whether that fix would change the result has not been established. + Version [2.1.269](https://github.com/anthropics/claude-code/releases/tag/v2.1.269) + arrived after all tasks had started, near the end of the arm. +- **DeepSeek Harness had a newer release candidate.** Version + [0.1.5-rc.2](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.2) + was published on September 10 at 15:09:34 UTC, before this arm ran. The + frozen package lock selected npm's `latest` tag at 0.1.5-rc.1. The rc.2 + release notes describe feedback dialogs and interface layout changes; no + run on rc.2 is included here. + ## How the columns in the README are derived - **solved**: tasks the verifier passed, out of 113. diff --git a/docs/benchmarks/deepswe/versions.csv b/docs/benchmarks/deepswe/versions.csv new file mode 100644 index 0000000000..e710052c80 --- /dev/null +++ b/docs/benchmarks/deepswe/versions.csv @@ -0,0 +1,11 @@ +harness,version,source_revision,release_published_at_utc,source_url,artifact,artifact_kind,checksum_algorithm,checksum,first_recorded_start_utc,last_recorded_finish_utc +senior-dev,8ae964c,8ae964c,,,codeaf-8ae964c,linux-amd64-executable,sha256,316f928e0654489a4760174046b9baa2d67b04add9b3f6a623b2a4f6b155f0d6,2026-09-12T03:03:35Z,2026-09-12T08:15:20Z +mini-swe-agent,2.4.6,a83fcae82d2a08f0ee0c688f9d137b3566c097f8,2026-07-23T03:12:33Z,https://github.com/SWE-agent/mini-swe-agent/releases/tag/v2.4.6,mini-swe-agent-v2.4.6-linux-amd64-py312.tar.gz,prepared-linux-amd64-python-payload,sha256,7e84fe6b6812ce2440e8a945d3909183301551eb9322b0e6393d3e5797633398,2026-09-10T18:50:38Z,2026-09-10T23:36:40Z +codex,0.154.0,6b9826e3aa83b1a5947db50f4332cb9c65f1b340,2026-09-09T22:35:38Z,https://github.com/openai/codex/releases/tag/rust-v0.154.0,codex-x86_64-unknown-linux-musl.tar.gz,linux-amd64-release-archive,sha256,d7e18b2597ae8f242f5f31ee9e90deef48dbc9edd634d9868fb6435d08c07f02,2026-09-11T02:16:30Z,2026-09-11T06:59:39Z +pi,0.85.1,d981de1229ef899957bbe968bc8dcda02a21f477,2026-09-05T12:29:01Z,https://github.com/earendil-works/pi/releases/tag/v0.85.1,pi-linux-x64.tar.gz,linux-amd64-release-archive,sha256,494e498f47d74d21f40b3386f6a5e921a3d49531a169cab55bbdaca0ea1fe25a,2026-09-10T18:54:11Z,2026-09-11T01:55:09Z +omp,18.1.17,3b3a6dc9bbd85102ce19d0b1c11bf6870915f6ec,2026-09-10T19:33:46Z,https://github.com/can1357/oh-my-pi/releases/tag/v18.1.17,omp-linux-x64,linux-amd64-executable,sha256,040c254ddeb30f6d592be67d1469dc67fcac14e933fa04dbf690f42820ffd820,2026-09-11T00:25:52Z,2026-09-11T07:02:28Z +opencode,1.18.30,3104c1428ec91f809e5ab86631300de41eb6952e,2026-09-09T03:34:27Z,https://github.com/anomalyco/opencode/releases/tag/v1.18.30,opencode-linux-x64.tar.gz,linux-amd64-release-archive,sha256,55007246858165496ff85ba1c2b648f7421e8e2013bf4189a680c9ff8e699d17,2026-09-11T07:26:49Z,2026-09-11T12:28:57Z +kilo,7.6.2,3d04228b6a642acb3daf68a649269618b6018250,2026-09-10T17:39:12Z,https://github.com/Kilo-Org/kilocode/releases/tag/v7.6.2,kilo-linux-x64.tar.gz,linux-amd64-release-archive,sha256,74cba87e7b4b0ae53cd40c76e464432e6ae75f810a753ff88436f666a08d3858,2026-09-11T12:49:43Z,2026-09-11T20:07:06Z +deepseek-harness,0.1.5-rc.1,,2026-09-10T03:09:00Z,https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.1,dsh-0.1.5-rc.1.tgz,npm-package-tarball,sha1,6bcdb554bf2eef837666e37f5bd5fa494eb053e4,2026-09-11T08:02:37Z,2026-09-11T13:24:33Z +claude-code,2.1.267,,2026-09-09T19:58:14Z,https://github.com/anthropics/claude-code/releases/tag/v2.1.267,claude-linux-x64.tar.gz,linux-amd64-release-archive,sha256,449bc65dc82d68694103a801c8ac54b6549f9e12f469217c9e0d5560c1085023,2026-09-10T18:45:33Z,2026-09-11T19:23:13Z +muse-code,1.1.1-R2514.1,,,https://lookaside.facebook.com/lookaside/muse/download/?channel=muse&version=1.1.1-R2514.1&file=manifest.json,muse-x86-linux,linux-amd64-executable,sha256,47908f2e5b0ac5b0c0ba147a48b30a9e48c0aebea3c5cc891ad34d95425a4993,2026-09-10T18:45:08Z,2026-09-11T22:28:15Z From 93e68bafbadb78c3e9d558978cb3e481ba7cb039 Mon Sep 17 00:00:00 2001 From: ZeroPoint95 <329227198+ZeroPoint95@users.noreply.github.com> Date: Mon, 19 Oct 2026 11:06:28 -0400 Subject: [PATCH 2/3] docs: add DeepSWE version evidence change entry (#1877) --- docs/changes/unreleased/1877-deepswe-versions.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 docs/changes/unreleased/1877-deepswe-versions.md diff --git a/docs/changes/unreleased/1877-deepswe-versions.md b/docs/changes/unreleased/1877-deepswe-versions.md new file mode 100644 index 0000000000..5c78d0ff94 --- /dev/null +++ b/docs/changes/unreleased/1877-deepswe-versions.md @@ -0,0 +1,14 @@ +--- +kind: changed +title: DeepSWE results identify the tested harness versions and release timing +pr: 1877 +surface: [docs] +invalidates: + - "The DeepSWE comparison listed scores without the tested harness versions, leaving its OpenCode generation unclear. The results page now identifies all ten versions, including OpenCode 1.18.30, with public release dates and links; versions.csv records source revisions, artifact checksums and saved attempt time bounds." + - "The setup dated all competing runs to September 11. Saved task metadata spans September 10–11 UTC; senior-dev ran on September 12 UTC." + - "The benchmark index listed only CLI footprint measurements. It now also links the DeepSWE task-quality comparison and its version evidence." +--- + +The comparison names the measured versions and the patch releases available at +launch without extending the results to newer versions. Published scores are +unchanged. From 9a8b41cdf8dcfc729870709581ca20161a6acb16 Mon Sep 17 00:00:00 2001 From: ZeroPoint95 <329227198+ZeroPoint95@users.noreply.github.com> Date: Mon, 19 Oct 2026 12:23:06 -0400 Subject: [PATCH 3/3] docs: use public senior-dev lineage and simplify version records --- docs/benchmarks/README.md | 4 +- docs/benchmarks/deepswe/README.md | 76 +++++++++---------- docs/benchmarks/deepswe/versions.csv | 22 +++--- .../unreleased/1877-deepswe-versions.md | 2 +- 4 files changed, 50 insertions(+), 54 deletions(-) diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 6caf20b1ed..be39b91d53 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -3,8 +3,8 @@ Classified by what is being measured. - [deepswe/](deepswe/): coding task quality. Ten harnesses on the same 113 - tasks and model, with results, tested versions, release dates, artifact - checksums, and comparison limits. + tasks and model, with results, tested versions, public integration references, + release dates, and comparison limits. - [performance/](performance/): the footprint of the CLI itself. Disk, startup, memory at rest, memory as parallel sessions multiply, memory over a long session, and behaviour through interrupts and crashes. Seven agent diff --git a/docs/benchmarks/deepswe/README.md b/docs/benchmarks/deepswe/README.md index aa0da356dd..0d1d1afb18 100644 --- a/docs/benchmarks/deepswe/README.md +++ b/docs/benchmarks/deepswe/README.md @@ -38,51 +38,47 @@ are senior-dev alone, not a comparison. grades, invalid outcomes, mean F2P and P2P, OpenRouter spend, cost per task, mean agent seconds and model. -## Versions tested - -These are the versions used for the comparison above, checked against all 113 -saved task records per harness. Release dates come from the linked upstream -releases; they are not installation dates. The dates and versions describe this -campaign, not the versions available today. - -| harness | version tested | release date (UTC) | -| --- | --- | --- | -| senior-dev | Benchmark build `8ae964c` | Not a release tag | -| mini-swe-agent | 2.4.6 | [2026-07-23](https://github.com/SWE-agent/mini-swe-agent/releases/tag/v2.4.6) | -| codex (Codex CLI) | 0.154.0 | [2026-09-09](https://github.com/openai/codex/releases/tag/rust-v0.154.0) | -| pi | 0.85.1 | [2026-09-05](https://github.com/earendil-works/pi/releases/tag/v0.85.1) | -| omp (Oh My Pi) | 18.1.17 | [2026-09-10](https://github.com/can1357/oh-my-pi/releases/tag/v18.1.17) | -| opencode | 1.18.30 | [2026-09-09](https://github.com/anomalyco/opencode/releases/tag/v1.18.30) | -| kilo | 7.6.2 | [2026-09-10](https://github.com/Kilo-Org/kilocode/releases/tag/v7.6.2) | -| claude-code | 2.1.267 | [2026-09-09](https://github.com/anthropics/claude-code/releases/tag/v2.1.267) | -| deepseek-harness (`@deepseek-ai/dsh`) | 0.1.5-rc.1 | [2026-09-10](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.1) | -| muse-code | 1.1.1-R2514.1 | Not independently established | +## Tested versions and public references + +The nine competing harness versions below were checked against all 113 saved +task records per harness. senior-dev uses its public integration and first +stable release references, explained below. Release dates come from the linked +upstream releases; they are not installation dates. These references describe +the historical comparison and its public lineage. + +| harness | tested version / public reference | source revision / integration commit | release date (UTC) | +| --- | --- | --- | --- | +| senior-dev | First stable: codeaf [v0.5.0](https://github.com/Agent-Field/CodeAF/releases/tag/v0.5.0) | [`dd0fcc654`](https://github.com/Agent-Field/CodeAF/commit/dd0fcc654a4a77b4cbf6beae27e6af1cfe4e4c28) — [#1488](https://github.com/Agent-Field/CodeAF/pull/1488) | 2026-09-30 | +| mini-swe-agent | 2.4.6 | `a83fcae` | [2026-07-23](https://github.com/SWE-agent/mini-swe-agent/releases/tag/v2.4.6) | +| codex (Codex CLI) | 0.154.0 | `6b9826e` | [2026-09-09](https://github.com/openai/codex/releases/tag/rust-v0.154.0) | +| pi | 0.85.1 | `d981de1` | [2026-09-05](https://github.com/earendil-works/pi/releases/tag/v0.85.1) | +| omp (Oh My Pi) | 18.1.17 | `3b3a6dc` | [2026-09-10](https://github.com/can1357/oh-my-pi/releases/tag/v18.1.17) | +| opencode | 1.18.30 | `3104c14` | [2026-09-09](https://github.com/anomalyco/opencode/releases/tag/v1.18.30) | +| kilo | 7.6.2 | `3d04228` | [2026-09-10](https://github.com/Kilo-Org/kilocode/releases/tag/v7.6.2) | +| claude-code | 2.1.267 | — | [2026-09-09](https://github.com/anthropics/claude-code/releases/tag/v2.1.267) | +| deepseek-harness (`@deepseek-ai/dsh`) | 0.1.5-rc.1 | — | [2026-09-10](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.1) | +| muse-code | 1.1.1-R2514.1 | — | Not independently established | [`versions.csv`](versions.csv) records the same versions alongside source -revisions where recorded, release timestamps and source URLs, the artifact -checksums from frozen provenance records, and the first start and last finish -in each arm's saved task metadata. Its `harness` values join to `arms.csv`. -Those time bounds include setup and grading as recorded; they are not the mean -agent times in `arms.csv`. - -The artifact kind states which bytes the checksum identifies: an executable, -release archive, prepared Python payload, or npm package tarball. An archive -checksum is not the checksum of its unpacked executable. The npm checksum is -SHA-1 as recorded in its package lock; the others are SHA-256. Empty source -revision, release timestamp, or URL fields mean that information was not -established for that artifact. A source URL identifies its upstream release or -package, not necessarily a download of the prepared benchmark payload. - -senior-dev's build identifier names the measured agent before its integration -into codeaf; it is not a codeaf release tag. Muse Code's frozen lock records the -build selected from [Meta's stable channel](https://api.meta.ai/muse-code/channels/muse-stable) -and its [versioned artifact manifest](https://lookaside.facebook.com/lookaside/muse/download/?channel=muse&version=1.1.1-R2514.1&file=manifest.json). -Its historical publication date and whether a newer build was available at -launch were not independently established. +revisions where recorded, public release timestamps and source URLs. Its +`harness` values join to `arms.csv`; revisions are stored in full where available. +Empty revision, release timestamp, or URL fields mean that information was not +established for that row. + +For senior-dev, `version` names the first stable codeaf release containing the +agent, and `source_revision` names the public integration commit from #1488. +That commit is contained in v0.5.0; it is distinct from the release tag's commit, +`4949397498f775a80de16d3e6429b0f6f933f54c`. The benchmark was run before the +integration. This row traces where the measured agent became publicly available +in codeaf; it does not describe a new benchmark run on v0.5.0. + +Muse Code's version comes from the frozen benchmark records. Its historical +publication date and whether a newer build was available at launch were not +independently established. ## Release timing and scope -Most tested versions were released within a week of the campaign. +Most competing harness versions were released within a week of the campaign. mini-swe-agent 2.4.6 was about seven weeks old, but was still the latest [published release](https://github.com/SWE-agent/mini-swe-agent/releases) at launch. The comparison does not establish how newer versions perform. diff --git a/docs/benchmarks/deepswe/versions.csv b/docs/benchmarks/deepswe/versions.csv index e710052c80..cc8086dcea 100644 --- a/docs/benchmarks/deepswe/versions.csv +++ b/docs/benchmarks/deepswe/versions.csv @@ -1,11 +1,11 @@ -harness,version,source_revision,release_published_at_utc,source_url,artifact,artifact_kind,checksum_algorithm,checksum,first_recorded_start_utc,last_recorded_finish_utc -senior-dev,8ae964c,8ae964c,,,codeaf-8ae964c,linux-amd64-executable,sha256,316f928e0654489a4760174046b9baa2d67b04add9b3f6a623b2a4f6b155f0d6,2026-09-12T03:03:35Z,2026-09-12T08:15:20Z -mini-swe-agent,2.4.6,a83fcae82d2a08f0ee0c688f9d137b3566c097f8,2026-07-23T03:12:33Z,https://github.com/SWE-agent/mini-swe-agent/releases/tag/v2.4.6,mini-swe-agent-v2.4.6-linux-amd64-py312.tar.gz,prepared-linux-amd64-python-payload,sha256,7e84fe6b6812ce2440e8a945d3909183301551eb9322b0e6393d3e5797633398,2026-09-10T18:50:38Z,2026-09-10T23:36:40Z -codex,0.154.0,6b9826e3aa83b1a5947db50f4332cb9c65f1b340,2026-09-09T22:35:38Z,https://github.com/openai/codex/releases/tag/rust-v0.154.0,codex-x86_64-unknown-linux-musl.tar.gz,linux-amd64-release-archive,sha256,d7e18b2597ae8f242f5f31ee9e90deef48dbc9edd634d9868fb6435d08c07f02,2026-09-11T02:16:30Z,2026-09-11T06:59:39Z -pi,0.85.1,d981de1229ef899957bbe968bc8dcda02a21f477,2026-09-05T12:29:01Z,https://github.com/earendil-works/pi/releases/tag/v0.85.1,pi-linux-x64.tar.gz,linux-amd64-release-archive,sha256,494e498f47d74d21f40b3386f6a5e921a3d49531a169cab55bbdaca0ea1fe25a,2026-09-10T18:54:11Z,2026-09-11T01:55:09Z -omp,18.1.17,3b3a6dc9bbd85102ce19d0b1c11bf6870915f6ec,2026-09-10T19:33:46Z,https://github.com/can1357/oh-my-pi/releases/tag/v18.1.17,omp-linux-x64,linux-amd64-executable,sha256,040c254ddeb30f6d592be67d1469dc67fcac14e933fa04dbf690f42820ffd820,2026-09-11T00:25:52Z,2026-09-11T07:02:28Z -opencode,1.18.30,3104c1428ec91f809e5ab86631300de41eb6952e,2026-09-09T03:34:27Z,https://github.com/anomalyco/opencode/releases/tag/v1.18.30,opencode-linux-x64.tar.gz,linux-amd64-release-archive,sha256,55007246858165496ff85ba1c2b648f7421e8e2013bf4189a680c9ff8e699d17,2026-09-11T07:26:49Z,2026-09-11T12:28:57Z -kilo,7.6.2,3d04228b6a642acb3daf68a649269618b6018250,2026-09-10T17:39:12Z,https://github.com/Kilo-Org/kilocode/releases/tag/v7.6.2,kilo-linux-x64.tar.gz,linux-amd64-release-archive,sha256,74cba87e7b4b0ae53cd40c76e464432e6ae75f810a753ff88436f666a08d3858,2026-09-11T12:49:43Z,2026-09-11T20:07:06Z -deepseek-harness,0.1.5-rc.1,,2026-09-10T03:09:00Z,https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.1,dsh-0.1.5-rc.1.tgz,npm-package-tarball,sha1,6bcdb554bf2eef837666e37f5bd5fa494eb053e4,2026-09-11T08:02:37Z,2026-09-11T13:24:33Z -claude-code,2.1.267,,2026-09-09T19:58:14Z,https://github.com/anthropics/claude-code/releases/tag/v2.1.267,claude-linux-x64.tar.gz,linux-amd64-release-archive,sha256,449bc65dc82d68694103a801c8ac54b6549f9e12f469217c9e0d5560c1085023,2026-09-10T18:45:33Z,2026-09-11T19:23:13Z -muse-code,1.1.1-R2514.1,,,https://lookaside.facebook.com/lookaside/muse/download/?channel=muse&version=1.1.1-R2514.1&file=manifest.json,muse-x86-linux,linux-amd64-executable,sha256,47908f2e5b0ac5b0c0ba147a48b30a9e48c0aebea3c5cc891ad34d95425a4993,2026-09-10T18:45:08Z,2026-09-11T22:28:15Z +harness,version,source_revision,release_published_at_utc,source_url +senior-dev,v0.5.0,dd0fcc654a4a77b4cbf6beae27e6af1cfe4e4c28,2026-09-30T02:08:08Z,https://github.com/Agent-Field/CodeAF/releases/tag/v0.5.0 +mini-swe-agent,2.4.6,a83fcae82d2a08f0ee0c688f9d137b3566c097f8,2026-07-23T03:12:33Z,https://github.com/SWE-agent/mini-swe-agent/releases/tag/v2.4.6 +codex,0.154.0,6b9826e3aa83b1a5947db50f4332cb9c65f1b340,2026-09-09T22:35:38Z,https://github.com/openai/codex/releases/tag/rust-v0.154.0 +pi,0.85.1,d981de1229ef899957bbe968bc8dcda02a21f477,2026-09-05T12:29:01Z,https://github.com/earendil-works/pi/releases/tag/v0.85.1 +omp,18.1.17,3b3a6dc9bbd85102ce19d0b1c11bf6870915f6ec,2026-09-10T19:33:46Z,https://github.com/can1357/oh-my-pi/releases/tag/v18.1.17 +opencode,1.18.30,3104c1428ec91f809e5ab86631300de41eb6952e,2026-09-09T03:34:27Z,https://github.com/anomalyco/opencode/releases/tag/v1.18.30 +kilo,7.6.2,3d04228b6a642acb3daf68a649269618b6018250,2026-09-10T17:39:12Z,https://github.com/Kilo-Org/kilocode/releases/tag/v7.6.2 +deepseek-harness,0.1.5-rc.1,,2026-09-10T03:09:00Z,https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.1 +claude-code,2.1.267,,2026-09-09T19:58:14Z,https://github.com/anthropics/claude-code/releases/tag/v2.1.267 +muse-code,1.1.1-R2514.1,,, diff --git a/docs/changes/unreleased/1877-deepswe-versions.md b/docs/changes/unreleased/1877-deepswe-versions.md index 5c78d0ff94..cd96c4e35f 100644 --- a/docs/changes/unreleased/1877-deepswe-versions.md +++ b/docs/changes/unreleased/1877-deepswe-versions.md @@ -4,7 +4,7 @@ title: DeepSWE results identify the tested harness versions and release timing pr: 1877 surface: [docs] invalidates: - - "The DeepSWE comparison listed scores without the tested harness versions, leaving its OpenCode generation unclear. The results page now identifies all ten versions, including OpenCode 1.18.30, with public release dates and links; versions.csv records source revisions, artifact checksums and saved attempt time bounds." + - "The DeepSWE comparison listed scores without the tested harness versions, leaving its OpenCode generation unclear. The results page now identifies the nine competing versions, including OpenCode 1.18.30, and senior-dev's public integration in #1488 and first stable release v0.5.0; versions.csv records versions, revisions and public release references." - "The setup dated all competing runs to September 11. Saved task metadata spans September 10–11 UTC; senior-dev ran on September 12 UTC." - "The benchmark index listed only CLI footprint measurements. It now also links the DeepSWE task-quality comparison and its version evidence." ---