docs: replace SWE-Bench example with GSM8K math tutoring (en + zh) - #12
Merged
Conversation
Swap "See it in action" section in both READMEs: SWE-Bench Qwen3-Coder-7B → GSM8K Qwen3-7B-Instruct (51.8%→96.2%, $34.10, 25 gens). Math tutoring is a more universally legible value story — every parent understands the market, the baseline, and the outcome. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Updates the project’s “See it in action” documentation examples to use a GSM8K math-tutoring benchmark narrative (EN + ZH) instead of the prior SWE-Bench example, and aligns the referenced config fields (mission, command, allowed_paths) across both READMEs.
Changes:
- Replaces the README demo story/metrics from SWE-Bench (Qwen3-Coder-7B) to GSM8K (Qwen3-7B-Instruct), including updated generation-by-generation walkthroughs.
- Updates the sample config snippets in both
README.mdandREADME.zh.mdto reference the new mission/benchmark and mutation scope paths.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| README.md | Replaces SWE-Bench example narrative + updates config snippet fields to GSM8K math-solver framing. |
| README.zh.md | Mirrors the English README updates with a Chinese version of the GSM8K narrative and matching config snippet fields. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+66
to
+69
| command: "python3 scripts/run_gsm8k.py --model qwen3-7b-instruct --sample 100 --json" | ||
|
|
||
| mutation_scope: | ||
| allowed_paths: ["src/agent_harness/"] | ||
| allowed_paths: ["src/math_solver_harness/"] |
Comment on lines
257
to
260
| evidence_sources: | ||
| - type: shell # stdout goes into observation.json | ||
| command: "python3 scripts/run_benchmark.py --sample 50 --json" | ||
| command: "python3 scripts/run_gsm8k.py --model qwen3-7b-instruct --sample 100 --json" | ||
| - type: file # file contents go into observation.json |
| mutation_scope: | ||
| allowed_paths: | ||
| - "src/agent_harness/" # changes outside this list are auto-rejected | ||
| - "src/math_solver_harness/" # changes outside this list are auto-rejected |
Comment on lines
+66
to
+69
| command: "python3 scripts/run_gsm8k.py --model qwen3-7b-instruct --sample 100 --json" | ||
|
|
||
| mutation_scope: | ||
| allowed_paths: ["src/agent_harness/"] | ||
| allowed_paths: ["src/math_solver_harness/"] |
Comment on lines
257
to
260
| evidence_sources: | ||
| - type: shell # stdout 写入 observation.json | ||
| command: "python3 scripts/run_benchmark.py --sample 50 --json" | ||
| command: "python3 scripts/run_gsm8k.py --model qwen3-7b-instruct --sample 100 --json" | ||
| - type: file # 文件内容写入 observation.json |
| mutation_scope: | ||
| allowed_paths: | ||
| - "src/agent_harness/" # 不在列表里的改动自动拒绝 | ||
| - "src/math_solver_harness/" # 不在列表里的改动自动拒绝 |
Add a note above each config snippet explaining that scripts/run_gsm8k.py and src/math_solver_harness/ are paths in the user's target project, not this repo. Point to examples/evolution.yml for a runnable demo. Addresses Copilot review comments on PR #12. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
README.mdandREADME.zh.mdmission,command, andallowed_pathsupdated to match in both READMEsWhy GSM8K
GSM8K is a better anchor example for this README:
Test plan
🤖 Generated with Claude Code