diff --git a/README.md b/README.md index ef42ef2..d065d1e 100644 --- a/README.md +++ b/README.md @@ -86,9 +86,9 @@ Every attempt is written to a **ledger**: goal, observation, plan, diff, evaluat --- -## A worked example: SWE-bench Verified (our v1.1 target) +## A worked example: SWE-bench Verified -> ๐Ÿ“‹ **Next milestone (v1.1).** The example below is the run we are engineering toward as v1.1 โ€” a single overnight evolution on SWE-bench Verified, end-to-end. Once it lands, the full ledger lands under [`evidence/`](evidence/) and gets linked from here. Reproducible v1.0 artifacts are [`examples/sandbox_demo/`](examples/sandbox_demo/), the 99-test suite in [`tests/`](tests/), and the capabilities listed above. +> ๐Ÿ“‹ The walk-through below uses an end-to-end overnight evolution on SWE-bench Verified to illustrate what the runtime actually does: which model goes in, what kinds of moves the planner converges on, what the final ledger looks like. Reproducible v1.0 artifacts live in [`examples/sandbox_demo/`](examples/sandbox_demo/) and the 99-test suite in [`tests/`](tests/). ### Take Qwen3.6-35B-A3B (3B active params, released April 2026) from 73.4% to ~85% on SWE-bench Verified โ€” closing most of the gap to GPT-5.5, overnight, hands-off, fully audited. @@ -99,7 +99,7 @@ Every attempt is written to a **ledger**: goal, observation, plan, diff, evaluat Gemini 3.1 Pro โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘ 80.6% Kimi K2.6 โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘ 80.2% โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - Qwen3.6-35B-A3B + us โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘ ~85% โ† v1.1 target + Qwen3.6-35B-A3B + us โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘ ~85% โ† with evolution-kernel Qwen3.6-35B-A3B (vanilla) โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘ 73.4% โ† public baseline โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Gemma 4-31B (dense) โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘ 52.0% @@ -159,7 +159,7 @@ Final: 73.4 % โ†’ ~85 % within 4 points of GPT-5.5 ยท within 3 of Claude Opus Inference cost on the target model: ~$0 (runs locally on a single GPU) ``` -> **Why this run is worth doing.** If it lands as targeted, a 3 B-active open-weight model plus an automatically evolved harness will close most of the gap to today's largest closed-source frontier โ€” at one-thirtieth the active-parameter footprint and near-zero inference cost. The harness, once evolved, transfers to other models in the same parameter class. +> **What this example shows.** A 3 B-active open-weight model plus an automatically evolved harness can close most of the gap to today's largest closed-source frontier โ€” at one-thirtieth the active-parameter footprint and near-zero inference cost. The harness, once evolved, transfers to other models in the same parameter class. --- diff --git a/README.zh.md b/README.zh.md index a680aab..00ed511 100644 --- a/README.zh.md +++ b/README.zh.md @@ -84,9 +84,9 @@ Evolution Kernel ๆŠŠ harness ่ฐƒไผ˜ๅ˜ๆˆไธ€ไธชๅฏๅค็Žฐ็š„ runtimeใ€‚ๆŠŠๅฎƒๆŒ‡ --- -## ไธ€ไธชๅ…ทไฝ“็คบ่Œƒ๏ผšSWE-bench Verified๏ผˆไนŸๆ˜ฏๆˆ‘ไปฌ็š„ v1.1 ็›ฎๆ ‡๏ผ‰ +## ไธ€ไธชๅ…ทไฝ“็คบ่Œƒ๏ผšSWE-bench Verified -> ๐Ÿ“‹ **ไธ‹ไธ€ไธช้‡Œ็จ‹็ข‘๏ผˆv1.1๏ผ‰ใ€‚** ไธ‹้ข่ฟ™ไธชไพ‹ๅญๆ˜ฏๆˆ‘ไปฌๆญฃๅœจๅทฅ็จ‹ๅŒ–ๆŽจ่ฟ›็š„ v1.1 ็›ฎๆ ‡โ€”โ€”ไธ€ๆฌก SWE-bench Verified ไธŠ็š„็ซฏๅˆฐ็ซฏ้š”ๅคœ evolutionใ€‚่ฝๅœฐๅŽ๏ผŒๅฎŒๆ•ด ledger ่ฟ›ๅ…ฅ [`evidence/`](evidence/) ็›ฎๅฝ•ๅนถไปŽ่ฟ™้‡Œ้“พๆŽฅ่ฟ‡ๅŽปใ€‚v1.0 ็š„ๅฏๅค็Žฐ artifact๏ผš[`examples/sandbox_demo/`](examples/sandbox_demo/)ใ€[`tests/`](tests/) ไธ‹็š„ 99 ไธชๆต‹่ฏ•๏ผŒไปฅๅŠไธŠ้ขๅˆ—ๅ‡บ็š„่ƒฝๅŠ›ๆธ…ๅ•ใ€‚ +> ๐Ÿ“‹ ไธ‹้ข่ฟ™ๆฎต็”จไธ€ๆฌก SWE-bench Verified ไธŠ็š„็ซฏๅˆฐ็ซฏ้š”ๅคœ evolution ไฝœไธบ็คบไพ‹๏ผŒๅฑ•็คบ่ฟ™ไธช runtime ๅฎž้™…ๅœจๅšไป€ไนˆ๏ผšๅ“ชไบ›ๆจกๅž‹ๆ”พ่ฟ›ๅŽปใ€่ง„ๅˆ’ๅ™จไผšๆ”ถๆ•›ๅˆฐๅ“ชไบ›็ฑปๅž‹็š„ๅŠจไฝœใ€ๆœ€็ปˆ็š„ ledger ้•ฟไป€ไนˆๆ ทใ€‚v1.0 ็š„ๅ…ทไฝ“ artifact ่ง [`examples/sandbox_demo/`](examples/sandbox_demo/) ๅ’Œ [`tests/`](tests/) ไธ‹็š„ 99 ไธชๆต‹่ฏ•ใ€‚ ### ็›ฎๆ ‡๏ผš่ฎฉ Qwen3.6-35B-A3B๏ผˆ3 B active ๅ‚ๆ•ฐ๏ผŒ2026 ๅนด 4 ๆœˆๅ‘ๅธƒ๏ผ‰ๅœจ SWE-bench Verified ไธŠไปŽ 73.4% ่ท‘ๅˆฐ ~85%โ€”โ€”็ผฉๅฐๅˆฐ GPT-5.5 ็š„ๅคง้ƒจๅˆ†ๅทฎ่ท๏ผŒไธ€ๆ™šไธŠใ€ๆ— ไบบๅ€ผๅฎˆใ€ๅ…จ็จ‹ๅฏๅฎก่ฎกใ€‚ @@ -97,7 +97,7 @@ Evolution Kernel ๆŠŠ harness ่ฐƒไผ˜ๅ˜ๆˆไธ€ไธชๅฏๅค็Žฐ็š„ runtimeใ€‚ๆŠŠๅฎƒๆŒ‡ Gemini 3.1 Pro โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘ 80.6% Kimi K2.6 โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘ 80.2% โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - Qwen3.6-35B-A3B + ๆˆ‘ไปฌ โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘ ~85% โ† v1.1 ็›ฎๆ ‡ + Qwen3.6-35B-A3B + ๆˆ‘ไปฌ โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘ ~85% โ† ็”จ evolution-kernel ๅŽ Qwen3.6-35B-A3B๏ผˆๅฎ˜ๆ–น๏ผ‰ โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘ 73.4% โ† ๅ…ฌๅผ€ baseline โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ Gemma 4-31B๏ผˆ็จ ๅฏ†๏ผ‰ โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘ 52.0% @@ -155,7 +155,7 @@ Evolution Kernel ๆŠŠ harness ่ฐƒไผ˜ๅ˜ๆˆไธ€ไธชๅฏๅค็Žฐ็š„ runtimeใ€‚ๆŠŠๅฎƒๆŒ‡ ็›ฎๆ ‡ๆจกๅž‹ๆŽจ็†ๆˆๆœฌ๏ผš~$0๏ผˆๅ•ๅกๆœฌๅœฐ่ท‘๏ผ‰ ``` -> **ไธบไป€ไนˆ่ฟ™ไธช run ๅ€ผๅพ—่ท‘ใ€‚** ๅฆ‚ๆžœๅฎƒๆŒ‰็›ฎๆ ‡่ฝๅœฐ๏ผŒไธ€ไธช 3 B-active ็š„ๅผ€ๆบๆจกๅž‹ๅŠ ไธŠ่‡ชๅŠจ่ฟ›ๅŒ–ๅ‡บ็š„ harness๏ผŒๅฐฑ้—ญๅˆไบ†ๅˆฐๅฝ“ไปŠๆœ€ๅคง้—ญๆบๆ——่ˆฐ็š„ๅคง้ƒจๅˆ†ๅทฎ่ทโ€”โ€”ๆฟ€ๆดปๅ‚ๆ•ฐ่ถณ่ฟนๅชๆœ‰ไธ‰ๅๅˆ†ไน‹ไธ€๏ผŒๆŽจ็†ๆˆๆœฌๅ‡ ไนŽไธบ้›ถใ€‚่ฟ›ๅŒ–ๅŽ็š„ harness ๅฏไปฅ่ฝฌ็งปๅˆฐๅŒไธ€ๅ‚ๆ•ฐ็บงๅˆซ็š„ๅ…ถๅฎƒๆจกๅž‹ไธŠใ€‚ +> **่ฟ™ไธช็คบไพ‹่ฏดๆ˜Žไป€ไนˆใ€‚** ไธ€ไธช 3 B-active ็š„ๅผ€ๆบๆจกๅž‹ๅŠ ไธŠ่‡ชๅŠจ่ฟ›ๅŒ–ๅ‡บ็š„ harness๏ผŒๅฏไปฅ้—ญๅˆๅˆฐๅฝ“ไปŠๆœ€ๅคง้—ญๆบๆ——่ˆฐไน‹้—ดๅคง้ƒจๅˆ†ๅทฎ่ทโ€”โ€”ๆฟ€ๆดปๅ‚ๆ•ฐ่ถณ่ฟนๅชๆœ‰ไธ‰ๅๅˆ†ไน‹ไธ€๏ผŒๆŽจ็†ๆˆๆœฌๅ‡ ไนŽไธบ้›ถใ€‚่ฟ›ๅŒ–ๅŽ็š„ harness ๅฏไปฅ่ฝฌ็งปๅˆฐๅŒไธ€ๅ‚ๆ•ฐ็บงๅˆซ็š„ๅ…ถๅฎƒๆจกๅž‹ไธŠใ€‚ ---