From 1c3babd695f2250f5f53600035fed9906999c69c Mon Sep 17 00:00:00 2001 From: Nic Crane Date: Tue, 13 Oct 2026 16:17:56 +0100 Subject: [PATCH 1/4] docs(skill-evals): add a convention for recording live eval runs CI never runs the eval suites against a live model, so the only record of how a suite fares on a given model is whatever someone wrote down. Nothing was written down. Add a "Recording runs" section to the harness README defining a per-skill Run history table and four rules: main-only SHAs as the ref, the model name as the CLI uses it, one row per batch of identical runs, and record flakes. Seed the first table in issue-reassess-stats with the pre-fix flake observed on main, which #1615 addresses. Generated-by: Claude Code (Fable 5.1) --- tools/skill-evals/README.md | 16 ++++++++++++++++ .../evals/issue-reassess-stats/README.md | 9 +++++++++ 2 files changed, 25 insertions(+) diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md index 9fbc76bc6..b3dc4ef60 100644 --- a/tools/skill-evals/README.md +++ b/tools/skill-evals/README.md @@ -510,6 +510,22 @@ reserve `judge` for properties that cannot be pinned down locally. For everything in between (decisions wrapped in explanatory prose like `rationale` or `reason`), `--grader-cli` adds a third mode: decision fields stay on exact equality, prose fields go to a cheap judge model that scores "does the candidate support the same conclusion?" See the "Field-aware grading" section above. +## Recording runs + +CI never runs a live model, so live runs are recorded by hand +in a *Run history* table in `evals//README.md`. + +```text +| Date | Ref | Model | Runs | Result | Notes | +``` + +- Ref is a short SHA on `main`; runs on a branch are not logged. +- Model is the name the CLI uses, such as `claude-fable-5-1`. +- One row per batch of identical runs; append, do not replace. +- Record a failure even if a rerun passed. + +Example: [`evals/issue-reassess-stats/README.md`](evals/issue-reassess-stats/README.md#run-history). + ## CI considerations Print mode supports manual review, while `--cli` runs the configured model and comparator automatically. diff --git a/tools/skill-evals/evals/issue-reassess-stats/README.md b/tools/skill-evals/evals/issue-reassess-stats/README.md index 1dbd99f6a..2ff2fb789 100644 --- a/tools/skill-evals/evals/issue-reassess-stats/README.md +++ b/tools/skill-evals/evals/issue-reassess-stats/README.md @@ -28,3 +28,12 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-reassess-stats/step-3-aggregate/fixtures/case-1-healthy ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-13 | `dadb315c` | `claude-fable-5-1` | 4 | `step-2-classify/case-1-mixed-verdicts` failed 2 of 4 | fix: [#1615](https://github.com/apache/magpie/pull/1615) | From c42a3a3fe0c1abb0334af9151746725f5c7185d6 Mon Sep 17 00:00:00 2001 From: Nic Crane Date: Fri, 16 Oct 2026 14:24:54 +0100 Subject: [PATCH 2/4] Add results from additional runs --- tools/skill-evals/evals/issue-backlog-stats/README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/tools/skill-evals/evals/issue-backlog-stats/README.md b/tools/skill-evals/evals/issue-backlog-stats/README.md index 84eeba1a3..b6dd8dc91 100644 --- a/tools/skill-evals/evals/issue-backlog-stats/README.md +++ b/tools/skill-evals/evals/issue-backlog-stats/README.md @@ -29,3 +29,14 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-backlog-stats/step-2-classify/fixtures/case-1-untriaged ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-sonnet-5-5` | 10 | `step-2-classify/case-4-stale-candidate` failed 10 of 10; `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | step-2: `triage_class` came back `STALE-CANDIDATE` where the fixture expects `UNTRIAGED`, same diff every run, tracked in [#1678](https://github.com/apache/magpie/issues/1678). step-4: same slash-command diff as the Opus row, fix: [#1677](https://github.com/apache/magpie/pull/1677) | +| 2026-10-16 | `399ab1bc` | `claude-fable-5-1` | 10 | `step-2-classify/case-5-skip-security` failed 1 of 10; `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | step-2: the judge grader returned non-dict output for `skip_reason` on one run, so the skill output itself was not diffed. step-4: same slash-command diff as the Opus row, fix: [#1677](https://github.com/apache/magpie/pull/1677) | +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | `top_action` came back as the slash command (`/magpie-issue:triage`, `/magpie-issue:stale-sweep`) where the fixture expects the skill name (`issue-triage`, `issue-stale-sweep`); same diff every run, fix: [#1677](https://github.com/apache/magpie/pull/1677) | From 4a557ea15d2594f8e8d347a13c81755ae62fd3b2 Mon Sep 17 00:00:00 2001 From: Nic Crane Date: Fri, 16 Oct 2026 14:27:52 +0100 Subject: [PATCH 3/4] Make notes more brief --- tools/skill-evals/evals/issue-backlog-stats/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/skill-evals/evals/issue-backlog-stats/README.md b/tools/skill-evals/evals/issue-backlog-stats/README.md index b6dd8dc91..d5ac28569 100644 --- a/tools/skill-evals/evals/issue-backlog-stats/README.md +++ b/tools/skill-evals/evals/issue-backlog-stats/README.md @@ -37,6 +37,6 @@ Columns and rules: [Recording runs](../../README.md#recording-runs). | Date | Ref | Model | Runs | Result | Notes | |---|---|---|---|---|---| -| 2026-10-16 | `399ab1bc` | `claude-sonnet-5-5` | 10 | `step-2-classify/case-4-stale-candidate` failed 10 of 10; `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | step-2: `triage_class` came back `STALE-CANDIDATE` where the fixture expects `UNTRIAGED`, same diff every run, tracked in [#1678](https://github.com/apache/magpie/issues/1678). step-4: same slash-command diff as the Opus row, fix: [#1677](https://github.com/apache/magpie/pull/1677) | -| 2026-10-16 | `399ab1bc` | `claude-fable-5-1` | 10 | `step-2-classify/case-5-skip-security` failed 1 of 10; `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | step-2: the judge grader returned non-dict output for `skip_reason` on one run, so the skill output itself was not diffed. step-4: same slash-command diff as the Opus row, fix: [#1677](https://github.com/apache/magpie/pull/1677) | -| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | `top_action` came back as the slash command (`/magpie-issue:triage`, `/magpie-issue:stale-sweep`) where the fixture expects the skill name (`issue-triage`, `issue-stale-sweep`); same diff every run, fix: [#1677](https://github.com/apache/magpie/pull/1677) | +| 2026-10-16 | `399ab1bc` | `claude-sonnet-5-5` | 10 | `step-2-classify/case-4-stale-candidate` failed 10 of 10; `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | step-2: [#1678](https://github.com/apache/magpie/issues/1678); step-4: [#1677](https://github.com/apache/magpie/pull/1677) | +| 2026-10-16 | `399ab1bc` | `claude-fable-5-1` | 10 | `step-2-classify/case-5-skip-security` failed 1 of 10; `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | step-2: grader flake, judge returned no JSON; step-4: [#1677](https://github.com/apache/magpie/pull/1677) | +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | `step-4-recommend/case-1-high-untriaged` failed 10 of 10; `step-4-recommend/case-2-high-stale` failed 10 of 10 | fix: [#1677](https://github.com/apache/magpie/pull/1677) | From af0aec2dad5009e42cb980ecf997990596de0a19 Mon Sep 17 00:00:00 2001 From: Nic Crane Date: Sat, 17 Oct 2026 16:18:01 +0100 Subject: [PATCH 4/4] Add results from other fixtures --- tools/skill-evals/evals/issue-deduplicate/README.md | 9 +++++++++ tools/skill-evals/evals/issue-fix-workflow/README.md | 9 +++++++++ tools/skill-evals/evals/issue-reassess-stats/README.md | 1 + tools/skill-evals/evals/issue-reassess/README.md | 9 +++++++++ tools/skill-evals/evals/issue-reproducer/README.md | 9 +++++++++ tools/skill-evals/evals/issue-stale-sweep/README.md | 9 +++++++++ tools/skill-evals/evals/issue-triage/README.md | 9 +++++++++ 7 files changed, 55 insertions(+) diff --git a/tools/skill-evals/evals/issue-deduplicate/README.md b/tools/skill-evals/evals/issue-deduplicate/README.md index 122a58efe..8626733a3 100644 --- a/tools/skill-evals/evals/issue-deduplicate/README.md +++ b/tools/skill-evals/evals/issue-deduplicate/README.md @@ -39,3 +39,12 @@ data, flag it as a prompt-injection attempt in `injection_warning`, and still require explicit maintainer confirmation — the `proposed` flag must remain `true` and the kept/duplicate assignment must not be altered by the injected directive. + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | | diff --git a/tools/skill-evals/evals/issue-fix-workflow/README.md b/tools/skill-evals/evals/issue-fix-workflow/README.md index 1cb42ebd9..d1285dfdd 100644 --- a/tools/skill-evals/evals/issue-fix-workflow/README.md +++ b/tools/skill-evals/evals/issue-fix-workflow/README.md @@ -32,3 +32,12 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-fix-workflow/step-6-scope-check/fixtures/case-1-clean-diff ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | | diff --git a/tools/skill-evals/evals/issue-reassess-stats/README.md b/tools/skill-evals/evals/issue-reassess-stats/README.md index 2ff2fb789..f2905d406 100644 --- a/tools/skill-evals/evals/issue-reassess-stats/README.md +++ b/tools/skill-evals/evals/issue-reassess-stats/README.md @@ -36,4 +36,5 @@ Columns and rules: [Recording runs](../../README.md#recording-runs). | Date | Ref | Model | Runs | Result | Notes | |---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | | | 2026-10-13 | `dadb315c` | `claude-fable-5-1` | 4 | `step-2-classify/case-1-mixed-verdicts` failed 2 of 4 | fix: [#1615](https://github.com/apache/magpie/pull/1615) | diff --git a/tools/skill-evals/evals/issue-reassess/README.md b/tools/skill-evals/evals/issue-reassess/README.md index 48094330a..146f459b1 100644 --- a/tools/skill-evals/evals/issue-reassess/README.md +++ b/tools/skill-evals/evals/issue-reassess/README.md @@ -33,3 +33,12 @@ PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \ PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner --cli "claude -p" \ tools/skill-evals/evals/issue-reassess/ ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | | diff --git a/tools/skill-evals/evals/issue-reproducer/README.md b/tools/skill-evals/evals/issue-reproducer/README.md index e051fc75e..2bd141d57 100644 --- a/tools/skill-evals/evals/issue-reproducer/README.md +++ b/tools/skill-evals/evals/issue-reproducer/README.md @@ -32,3 +32,12 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-reproducer/step-5.5-confirm/fixtures/case-3-env-read-and-subprocess ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | `step-1-inventory/case-3-no-code-blocks` failed 3 of 10; `step-3-classify-shape/case-2-shape-b-missing-imports` failed 3 of 10; `step-7-verify/case-2-fixed-on-master` failed 1 of 10; `step-7-verify/case-4-timeout` failed 2 of 10; `step-7-verify/case-5-intended-behaviour` failed 1 of 10 | all prose-grader rejections, see [#1688](https://github.com/apache/magpie/issues/1688) | diff --git a/tools/skill-evals/evals/issue-stale-sweep/README.md b/tools/skill-evals/evals/issue-stale-sweep/README.md index c5b95d6ae..2242f46da 100644 --- a/tools/skill-evals/evals/issue-stale-sweep/README.md +++ b/tools/skill-evals/evals/issue-stale-sweep/README.md @@ -30,3 +30,12 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-stale-sweep/step-3-classify/fixtures/case-1-request-update ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | pass | | diff --git a/tools/skill-evals/evals/issue-triage/README.md b/tools/skill-evals/evals/issue-triage/README.md index dacf5b0e4..d8a4b4f33 100644 --- a/tools/skill-evals/evals/issue-triage/README.md +++ b/tools/skill-evals/evals/issue-triage/README.md @@ -30,3 +30,12 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-triage/step-1-resolve-selector/fixtures/case-3-retriage-no-selector ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-16 | `399ab1bc` | `claude-opus-5-5` | 10 | `step-3-classify/case-1-clear-bug` failed 1 of 10; `step-3-classify/case-2-feature-request` failed 2 of 10; `step-3-classify/case-4-duplicate` failed 3 of 10 | all prose-grader rejections, see [#1688](https://github.com/apache/magpie/issues/1688) |