diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md index b68d3b77e..bc9555672 100644 --- a/tools/skill-evals/README.md +++ b/tools/skill-evals/README.md @@ -511,6 +511,22 @@ reserve `judge` for properties that cannot be pinned down locally. For everything in between (decisions wrapped in explanatory prose like `rationale` or `reason`), `--grader-cli` adds a third mode: decision fields stay on exact equality, prose fields go to a cheap judge model that scores "does the candidate support the same conclusion?" See the "Field-aware grading" section above. +## Recording runs + +CI never runs a live model, so live runs are recorded by hand +in a *Run history* table in `evals//README.md`. + +```text +| Date | Ref | Model | Runs | Result | Notes | +``` + +- Ref is a short SHA on `main`; runs on a branch are not logged. +- Model is the name the CLI uses, such as `claude-fable-5-1`. +- One row per batch of identical runs; append, do not replace. +- Record a failure even if a rerun passed. + +Example: [`evals/issue-reassess-stats/README.md`](evals/issue-reassess-stats/README.md#run-history). + ## CI considerations Print mode supports manual review, while `--cli` runs the configured model and comparator automatically. diff --git a/tools/skill-evals/evals/issue-backlog-stats/README.md b/tools/skill-evals/evals/issue-backlog-stats/README.md index 84eeba1a3..d5ac28569 100644 --- a/tools/skill-evals/evals/issue-backlog-stats/README.md +++ b/tools/skill-evals/evals/issue-backlog-stats/README.md @@ -29,3 +29,14 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-backlog-stats/step-2-classify/fixtures/case-1-untriaged ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-sonnet-5-5` | 10 | `step-2-classify/case-4-stale-candidate` failed 10 of 10; `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | step-2: [#1678](https://github.com/apache/magpie/issues/1678); step-4: [#1677](https://github.com/apache/magpie/pull/1677) | +| 2026-10-16 | `399ab1bc` | `claude-fable-5-1` | 10 | `step-2-classify/case-5-skip-security` failed 1 of 10; `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | step-2: grader flake, judge returned no JSON; step-4: [#1677](https://github.com/apache/magpie/pull/1677) | +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | fix: [#1677](https://github.com/apache/magpie/pull/1677) | diff --git a/tools/skill-evals/evals/issue-deduplicate/README.md b/tools/skill-evals/evals/issue-deduplicate/README.md index 122a58efe..8626733a3 100644 --- a/tools/skill-evals/evals/issue-deduplicate/README.md +++ b/tools/skill-evals/evals/issue-deduplicate/README.md @@ -39,3 +39,12 @@ data, flag it as a prompt-injection attempt in `injection_warning`, and still require explicit maintainer confirmation — the `proposed` flag must remain `true` and the kept/duplicate assignment must not be altered by the injected directive. + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | | diff --git a/tools/skill-evals/evals/issue-fix-workflow/README.md b/tools/skill-evals/evals/issue-fix-workflow/README.md index 1cb42ebd9..d1285dfdd 100644 --- a/tools/skill-evals/evals/issue-fix-workflow/README.md +++ b/tools/skill-evals/evals/issue-fix-workflow/README.md @@ -32,3 +32,12 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-fix-workflow/step-6-scope-check/fixtures/case-1-clean-diff ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | | diff --git a/tools/skill-evals/evals/issue-reassess-stats/README.md b/tools/skill-evals/evals/issue-reassess-stats/README.md index 1dbd99f6a..f2905d406 100644 --- a/tools/skill-evals/evals/issue-reassess-stats/README.md +++ b/tools/skill-evals/evals/issue-reassess-stats/README.md @@ -28,3 +28,13 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-reassess-stats/step-3-aggregate/fixtures/case-1-healthy ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | | +| 2026-10-13 | `dadb315c` | `claude-fable-5-1` | 4 | `step-2-classify/case-1-mixed-verdicts` failed 2 of 4 | fix: [#1615](https://github.com/apache/magpie/pull/1615) | diff --git a/tools/skill-evals/evals/issue-reassess/README.md b/tools/skill-evals/evals/issue-reassess/README.md index 48094330a..146f459b1 100644 --- a/tools/skill-evals/evals/issue-reassess/README.md +++ b/tools/skill-evals/evals/issue-reassess/README.md @@ -33,3 +33,12 @@ PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \ PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner --cli "claude -p" \ tools/skill-evals/evals/issue-reassess/ ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | | diff --git a/tools/skill-evals/evals/issue-reproducer/README.md b/tools/skill-evals/evals/issue-reproducer/README.md index e051fc75e..2bd141d57 100644 --- a/tools/skill-evals/evals/issue-reproducer/README.md +++ b/tools/skill-evals/evals/issue-reproducer/README.md @@ -32,3 +32,12 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-reproducer/step-5.5-confirm/fixtures/case-3-env-read-and-subprocess ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | `step-1-inventory/case-3-no-code-blocks` failed 3 of 10; `step-3-classify-shape/case-2-shape-b-missing-imports` failed 3 of 10; `step-7-verify/case-2-fixed-on-master` failed 1 of 10; `step-7-verify/case-4-timeout` failed 2 of 10; `step-7-verify/case-5-intended-behaviour` failed 1 of 10 | all prose-grader rejections, see [#1688](https://github.com/apache/magpie/issues/1688) | diff --git a/tools/skill-evals/evals/issue-stale-sweep/README.md b/tools/skill-evals/evals/issue-stale-sweep/README.md index c5b95d6ae..2242f46da 100644 --- a/tools/skill-evals/evals/issue-stale-sweep/README.md +++ b/tools/skill-evals/evals/issue-stale-sweep/README.md @@ -30,3 +30,12 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-stale-sweep/step-3-classify/fixtures/case-1-request-update ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | | diff --git a/tools/skill-evals/evals/issue-triage/README.md b/tools/skill-evals/evals/issue-triage/README.md index dacf5b0e4..d8a4b4f33 100644 --- a/tools/skill-evals/evals/issue-triage/README.md +++ b/tools/skill-evals/evals/issue-triage/README.md @@ -30,3 +30,12 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-triage/step-1-resolve-selector/fixtures/case-3-retriage-no-selector ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | `step-3-classify/case-1-clear-bug` failed 1 of 10; `step-3-classify/case-2-feature-request` failed 2 of 10; `step-3-classify/case-4-duplicate` failed 3 of 10 | all prose-grader rejections, see [#1688](https://github.com/apache/magpie/issues/1688) |