From ab1edd859250324cfbae6bfa8668c28ed8cf48d9 Mon Sep 17 00:00:00 2001 From: Jon Gallant <2163001+jongio@users.noreply.github.com> Date: Mon, 16 Nov 2026 09:45:46 -0800 Subject: [PATCH 01/11] docs(evals): add Ship / No-ship PR comment verdict mockups for review Add three interactive PR comment mockups combining @rwoll's Ship / No-ship verdict table and collapsible
structure with Vally 0.21.0 score, turn, wall-time, and output-token comparison telemetry. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/pr-comment-verdict-mockups.md | 226 +++++++++++++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 docs/pr-comment-verdict-mockups.md diff --git a/docs/pr-comment-verdict-mockups.md b/docs/pr-comment-verdict-mockups.md new file mode 100644 index 0000000..a9434da --- /dev/null +++ b/docs/pr-comment-verdict-mockups.md @@ -0,0 +1,226 @@ +# Skill Eval PR Comment Mock-ups: Ship / No-Ship Verdicts & Collapsible Telemetry + +> **Status:** Draft mock-ups for PM & engineering review +> **Reference:** [`@rwoll` Gist โ€” Mock-up: skill eval PR comment with a ship / no ship verdict per skill](https://gist.github.com/rwoll/5dafeff236e21fca14a71a7301ebc385) +> **Live comments on this PR:** See the 3 rendered PR comments on this pull request timeline to experience the exact collapsed vs. expanded GitHub UI behavior. + +## Goals + +1. **Clear Ship / No-Ship decision at the top:** Reviewers immediately see `โœ… Ship`, `๐Ÿ›‘ No ship`, or `โš ๏ธ Re-run` per skill without scrolling through large telemetry tables. +2. **Progressive disclosure via `
`:** Keep the default comment view compact while preserving all real Vally `0.21.0` metrics (`Score`, `Turns`, `Agent Wall Time`, `Total Duration`, `Input / Output Tokens`, `%` deltas, `Skills Invoked`, and artifact `[Evidence]` links) one click away. +3. **Distinguish modified skills from unrelated flakes:** Surface `โš ๏ธ Re-run` when a skill not touched by the PR fails a single-trial scenario. + +--- + +## Mock-up 1: Single-Skill PR (`โœ… Ship` โ€” Real `pr-description` Vally `0.21.0` Run) + +This shows how a single-skill PR (such as [`#21`](https://github.com/github/plugin-marketplace-starter/pull/21)) renders with real Vally `0.21.0` (`claude-haiku-5.5`) baseline (`63a64377`) and candidate (`ab6a6e4d`) runs. + +--- + +## Evals: โœ… Ship + +| Skill | Verdict | Why | +| --- | --- | --- | +| **pr-description** | โœ… **Ship** | 1 scenario fixed (`1/2` โ†’ `2/2` passing) ยท score `93.8%` โ†’ `100.0%` ยท `-31.5%` output tokens ยท `-6.6%` agent wall time | + +
+โœ… pr-description: 1 fixed (1/2 โ†’ 2/2 on claude-haiku-5.5) + +| | Scenario | Model | Score | Turns | Wall Time | Output Tokens | What happened | Evidence | +| --- | --- | --- | --- | ---: | ---: | ---: | --- | --- | +| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 87.5% โ†’ 100.0% | 2 โ†’ 2 | 11.1s โ†’ 7.1s (-36.2%) | 918 โ†’ 402 (-56.2%) | 3/4 โ†’ 4/4 checks passed (fixed `program` check: includes `## Risks / Rollback`, excludes `#42` from `## Changes`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | +| โœ… Passed | `ignores-bug-and-review-question` | `claude-haiku-5.5` | 100.0% โ†’ 100.0% | 1 โ†’ 1 | 6.5s โ†’ 9.3s (+44.2%) | 529 โ†’ 589 (+11.3%) | Unchanged and passing (4/4 checks passed) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | + +Scenarios passing, before โ†’ after: `claude-haiku-5.5` 1/2 โ†’ 2/2. + +
+ +
+๐Ÿ“Š Full comparison metrics, token usage & run evidence + +### Baseline vs. Candidate comparison + +| Metric | Baseline | Candidate | Delta | +| --- | --- | --- | --- | +| Outcome | failed | passed | fixed | +| Passed | 1 | 2 | +1 | +| Failed | 1 | 0 | -1 | +| Avg Score | 93.8% | 100.0% | +6.3% | +| Turns | 3 | 3 | 0 | +| Agent Wall Time | 17.5s | 16.4s | -1.1s (-6.6%) | +| Total Duration | 38.5s | 34.1s | -4.4s (-11.4%) | +| Total Tokens | 73,199 (71,752 in / 1,447 out) | 73,008 (72,017 in / 991 out) | -191 (-0.3%) | +| Output Tokens | 1,447 | 991 | -456 (-31.5%) | + +### Quality & token summary (Candidate) + +| Eval | Model | Verdict | Score | Skills Invoked | Turns | Wall Time | Tokens | +| --- | --- | --- | --- | --- | ---: | ---: | --- | +| `pr-description` | `claude-haiku-5.5` | passed | 1.00 (threshold 1.00) | `pr-description` | 3 | 16.4s | 73,008 (72,017 in / 991 out) | + +**Selection:** Native eval paths or tags +**Candidate:** passed ยท 2 trials ยท [Evidence](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) +**Baseline:** failed ยท 2 trials ยท [Evidence](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208576029) +**Revisions:** Before [`0b6bf97`](https://github.com/github/plugin-marketplace-starter/commit/0b6bf97) โ†’ After [`bd250bf`](https://github.com/github/plugin-marketplace-starter/commit/bd250bf6a15f1478dbf0520852088d3cfa8fe31b) ยท [Workflow run](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109) + +
+ +--- + +## Mock-up 2: Multi-Skill Suite PR (`๐Ÿ›‘ No ship` โ€” Mixed Verdicts + `Everything else & full telemetry`) + +This combines `@rwoll`'s multi-skill gist scenario (`โœ… Ship`, `๐Ÿ›‘ No ship`, `โš ๏ธ Re-run`, and `Everything else`) with per-scenario and aggregate telemetry inside `
`. + +--- + +## Evals: ๐Ÿ›‘ No ship + +| Skill | Verdict | Why | +| --- | --- | --- | +| **pr-description** | โœ… **Ship** | 1 scenario fixed (`1/2` โ†’ `2/2` passing) ยท `-31.5%` output tokens | +| **release-plugins** | ๐Ÿ›‘ **No ship** | 1 scenario broke, 1 fixed (`4/5` โ†’ `4/5` passing) | +| try-plugin-locally *(not changed)* | โš ๏ธ **Re-run** | 1 scenario broke, but this PR does not change this skill | + +
+โœ… pr-description: 1 fixed + +| | Scenario | Model | Score | Output Tokens | What happened | Evidence | +| --- | --- | --- | --- | ---: | --- | --- | +| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 87.5% โ†’ 100.0% | 918 โ†’ 402 (-56.2%) | 3/4 โ†’ 4/4 checks passed (`program` check fixed: includes `## Risks / Rollback`, excludes `#42` from `## Changes`) | [artifact](#) | + +Scenarios passing, before โ†’ after: `claude-haiku-5.5` 1/2 โ†’ 2/2. + +
+ +
+๐Ÿ›‘ release-plugins: 1 broke, 1 fixed + +| | Scenario | Model | Score | Output Tokens | What happened | Evidence | +| --- | --- | --- | --- | ---: | --- | --- | +| ๐Ÿ“‰ Broke | `preserves-unreleased-initial-version` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 410 โ†’ 390 (-4.9%) | Check failed: `skill-invocation` (skill was not invoked) | [artifact](#) | +| ๐Ÿ“ˆ Fixed | `asks-for-unknown-release-base` | `claude-haiku-5.5` | 50.0% โ†’ 100.0% | 620 โ†’ 310 (-50.0%) | 2/4 โ†’ 4/4 checks passed (`output-matches` fixed) | [artifact](#) | + +Scenarios passing, before โ†’ after: `claude-haiku-5.5` 4/5 โ†’ 4/5. + +
+ +
+โš ๏ธ try-plugin-locally (not changed): 1 broke + +| | Scenario | Model | Score | Output Tokens | What happened | Evidence | +| --- | --- | --- | --- | ---: | --- | --- | +| ๐Ÿ“‰ Broke | `prepares-current-checkout-session` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 512 โ†’ 488 (-4.7%) | Check failed: `program` (command omitted local plugin path) | [artifact](#) | + +This PR does not change this skill, so re-run to rule out single-trial noise. + +
+ +
+Everything else & full telemetry + +Unchanged and passing: `update-remote-pins` (3/3), `add-plugin` (2/2), `submit-plugin` (2/2). + +Not checked: 6 skills have no evals (`eval/compare`, `eval/create`, `eval/improve`, `eval/review`, `eval/run`, `eval/setup`). + +16 scenarios on 1 model (`claude-haiku-5.5`), 1 run each, no retries. A single flip can be noise. + +### Baseline vs. Candidate comparison + +| Eval | Baseline | Candidate | Score Delta | Turns Delta | Duration Delta | Token Delta | Transition | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `pr-description` | failed (0.94) | passed (1.00) | +0.06 | 0 | -4.4s (-11.4%) | -191 (-0.3%; out: -456 / -31.5%) | fixed | +| `release-plugins` | failed (0.85) | failed (0.85) | 0.00 | 0 | -1.2s (-3.1%) | -310 (-0.4%; out: -330 / -18.2%) | regressed | +| `try-plugin-locally` | passed (1.00) | failed (0.88) | -0.12 | 0 | +0.8s (+4.2%) | -24 (-0.1%) | regressed | +| `update-remote-pins` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.3s (-1.2%) | -12 (0.0%) | unchanged | +| `add-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | +0.2s (+1.0%) | +8 (0.0%) | unchanged | +| `submit-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.1s (-0.5%) | 0 (0.0%) | unchanged | + +Before is `0b6bf97`, After is `bd250bf`. [Candidate evidence](#) ยท [Baseline evidence](#) ยท [Workflow run](#). + +
+ +--- + +## Mock-up 3: Ultra-Minimal Variant (Exact `@rwoll` Gist Tables + Separate Telemetry Collapsible) + +This variant keeps the top table and per-skill `
` tables **identical** to `@rwoll`'s 5-column gist (zero token/duration columns in the scenario tables), and places all numeric score, token, turn, and duration tables in a single bottom `
` block. + +--- + +| Skill | Verdict | Why | +| --- | --- | --- | +| **pr-description** | โœ… Ship | 1 scenario fixed | +| **release-plugins** | ๐Ÿ›‘ No ship | 1 scenario broke, 1 fixed | +| try-plugin-locally (not changed) | โš ๏ธ Re-run | 1 scenario broke, but this PR does not change this skill | + +
+โœ… pr-description: 1 fixed + +| | Scenario | Model | What happened | Evidence | +| --- | --- | --- | --- | --- | +| ๐Ÿ“ˆ Fixed | drafts-structured-pr-body | Claude Haiku 5.5 | Now includes the `## Risks / Rollback` section and excludes `#42` from `## Changes`. | [log](#) | + +Scenarios passing, before โ†’ after: Claude Haiku 5.5 1/2 โ†’ 2/2. + +
+ +
+๐Ÿ›‘ release-plugins: 1 broke, 1 fixed + +| | Scenario | Model | What happened | Evidence | +| --- | --- | --- | --- | --- | +| ๐Ÿ“‰ Broke | preserves-unreleased-initial-version | Claude Haiku 5.5 | The skill was not used when it should have been (`skill-invocation`). | [log](#) | +| ๐Ÿ“ˆ Fixed | asks-for-unknown-release-base | Claude Haiku 5.5 | Now asks for the missing release base. | [log](#) | + +Scenarios passing, before โ†’ after: Claude Haiku 5.5 4/5 โ†’ 4/5. + +
+ +
+โš ๏ธ try-plugin-locally (not changed): 1 broke + +| | Scenario | Model | What happened | Evidence | +| --- | --- | --- | --- | --- | +| ๐Ÿ“‰ Broke | prepares-current-checkout-session | Claude Haiku 5.5 | The command left out the plugin's local path. | [log](#) | + +This PR does not change this skill, so re-run to rule out noise. + +
+ +
+Everything else + +Unchanged and passing: `update-remote-pins` (3/3), `add-plugin` (2/2), `submit-plugin` (2/2). + +Not checked: 6 skills have no evals (`eval/compare`, `eval/create`, `eval/improve`, `eval/review`, `eval/run`, `eval/setup`). + +16 scenarios on 1 model, 1 run each, no retries. A single flip can be noise. + +Before is `0b6bf97`, After is `bd250bf`. [Workflow run](#). + +
+ +
+๐Ÿ“Š Quality, token & duration telemetry (Baseline vs. Candidate) + +| Metric | Baseline | Candidate | Delta | +| --- | --- | --- | --- | +| Outcome | failed | passed | fixed | +| Passed | 1 | 2 | +1 | +| Failed | 1 | 0 | -1 | +| Avg Score | 93.8% | 100.0% | +6.3% | +| Turns | 3 | 3 | 0 | +| Agent Wall Time | 17.5s | 16.4s | -1.1s (-6.6%) | +| Total Duration | 38.5s | 34.1s | -4.4s (-11.4%) | +| Total Tokens | 73,199 (71,752 in / 1,447 out) | 73,008 (72,017 in / 991 out) | -191 (-0.3%) | +| Output Tokens | 1,447 | 991 | -456 (-31.5%) | + +| Case | Baseline | Candidate | Score Delta | Turns | Output Tokens | Wall Time | Transition | +| --- | --- | --- | --- | ---: | ---: | ---: | --- | +| `pr-description / drafts-structured-pr-body` | FAIL (87.5%) | PASS (100.0%) | +12.5% | 2 โ†’ 2 | 918 โ†’ 402 (-56.2%) | 11.1s โ†’ 7.1s (-36.2%) | Fixed | +| `pr-description / ignores-bug-and-review-question` | PASS (100.0%) | PASS (100.0%) | 0.0% | 1 โ†’ 1 | 529 โ†’ 589 (+11.3%) | 6.5s โ†’ 9.3s (+44.2%) | Passed | + +**Candidate artifact:** [Evidence](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) ยท **Baseline artifact:** [Evidence](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208576029) + +
From 7bab71590543102e6c95ba1dba07882394c9b2f8 Mon Sep 17 00:00:00 2001 From: Jon Gallant <2163001+jongio@users.noreply.github.com> Date: Mon, 16 Nov 2026 10:02:07 -0800 Subject: [PATCH 02/11] docs(evals): replace 'Everything else' prose with structured coverage table and consistent aside labels Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/pr-comment-verdict-mockups.md | 133 ++++++++++++++--------------- 1 file changed, 64 insertions(+), 69 deletions(-) diff --git a/docs/pr-comment-verdict-mockups.md b/docs/pr-comment-verdict-mockups.md index a9434da..90f51c8 100644 --- a/docs/pr-comment-verdict-mockups.md +++ b/docs/pr-comment-verdict-mockups.md @@ -8,16 +8,12 @@ 1. **Clear Ship / No-Ship decision at the top:** Reviewers immediately see `โœ… Ship`, `๐Ÿ›‘ No ship`, or `โš ๏ธ Re-run` per skill without scrolling through large telemetry tables. 2. **Progressive disclosure via `
`:** Keep the default comment view compact while preserving all real Vally `0.21.0` metrics (`Score`, `Turns`, `Agent Wall Time`, `Total Duration`, `Input / Output Tokens`, `%` deltas, `Skills Invoked`, and artifact `[Evidence]` links) one click away. -3. **Distinguish modified skills from unrelated flakes:** Surface `โš ๏ธ Re-run` when a skill not touched by the PR fails a single-trial scenario. +3. **Structured, scannable asides:** Replace loose prose in `Everything else` with a structured `๐Ÿ“‹ Unchanged skills & suite coverage` table and a consistent `๐Ÿ“Š Telemetry, token deltas & run evidence` collapsible across all variants. --- ## Mock-up 1: Single-Skill PR (`โœ… Ship` โ€” Real `pr-description` Vally `0.21.0` Run) -This shows how a single-skill PR (such as [`#21`](https://github.com/github/plugin-marketplace-starter/pull/21)) renders with real Vally `0.21.0` (`claude-haiku-5.5`) baseline (`63a64377`) and candidate (`ab6a6e4d`) runs. - ---- - ## Evals: โœ… Ship | Skill | Verdict | Why | @@ -25,21 +21,19 @@ This shows how a single-skill PR (such as [`#21`](https://github.com/github/plug | **pr-description** | โœ… **Ship** | 1 scenario fixed (`1/2` โ†’ `2/2` passing) ยท score `93.8%` โ†’ `100.0%` ยท `-31.5%` output tokens ยท `-6.6%` agent wall time |
-โœ… pr-description: 1 fixed (1/2 โ†’ 2/2 on claude-haiku-5.5) +โœ… pr-description โ€” 1 fixed (1/2 โ†’ 2/2 passing) | | Scenario | Model | Score | Turns | Wall Time | Output Tokens | What happened | Evidence | | --- | --- | --- | --- | ---: | ---: | ---: | --- | --- | | ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 87.5% โ†’ 100.0% | 2 โ†’ 2 | 11.1s โ†’ 7.1s (-36.2%) | 918 โ†’ 402 (-56.2%) | 3/4 โ†’ 4/4 checks passed (fixed `program` check: includes `## Risks / Rollback`, excludes `#42` from `## Changes`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | | โœ… Passed | `ignores-bug-and-review-question` | `claude-haiku-5.5` | 100.0% โ†’ 100.0% | 1 โ†’ 1 | 6.5s โ†’ 9.3s (+44.2%) | 529 โ†’ 589 (+11.3%) | Unchanged and passing (4/4 checks passed) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | -Scenarios passing, before โ†’ after: `claude-haiku-5.5` 1/2 โ†’ 2/2. +Pass rate (before โ†’ after): claude-haiku-5.5 1/2 โ†’ 2/2 (1 trial/scenario, 0 retries)
-๐Ÿ“Š Full comparison metrics, token usage & run evidence - -### Baseline vs. Candidate comparison +๐Ÿ“Š Telemetry, token deltas & run evidence โ€” score 93.8% โ†’ 100.0%, -31.5% output tokens | Metric | Baseline | Candidate | Delta | | --- | --- | --- | --- | @@ -53,26 +47,18 @@ Scenarios passing, before โ†’ after: `claude-haiku-5.5` 1/2 โ†’ 2/2. | Total Tokens | 73,199 (71,752 in / 1,447 out) | 73,008 (72,017 in / 991 out) | -191 (-0.3%) | | Output Tokens | 1,447 | 991 | -456 (-31.5%) | -### Quality & token summary (Candidate) - -| Eval | Model | Verdict | Score | Skills Invoked | Turns | Wall Time | Tokens | -| --- | --- | --- | --- | --- | ---: | ---: | --- | -| `pr-description` | `claude-haiku-5.5` | passed | 1.00 (threshold 1.00) | `pr-description` | 3 | 16.4s | 73,008 (72,017 in / 991 out) | +| Scenario | Baseline | Candidate | Score Delta | Turns | Output Tokens | Wall Time | Transition | +| --- | --- | --- | --- | ---: | ---: | ---: | --- | +| `pr-description / drafts-structured-pr-body` | FAIL (87.5%) | PASS (100.0%) | +12.5% | 2 โ†’ 2 | 918 โ†’ 402 (-56.2%) | 11.1s โ†’ 7.1s (-36.2%) | Fixed | +| `pr-description / ignores-bug-and-review-question` | PASS (100.0%) | PASS (100.0%) | 0.0% | 1 โ†’ 1 | 529 โ†’ 589 (+11.3%) | 6.5s โ†’ 9.3s (+44.2%) | Passed | -**Selection:** Native eval paths or tags -**Candidate:** passed ยท 2 trials ยท [Evidence](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) -**Baseline:** failed ยท 2 trials ยท [Evidence](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208576029) -**Revisions:** Before [`0b6bf97`](https://github.com/github/plugin-marketplace-starter/commit/0b6bf97) โ†’ After [`bd250bf`](https://github.com/github/plugin-marketplace-starter/commit/bd250bf6a15f1478dbf0520852088d3cfa8fe31b) ยท [Workflow run](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109) +Evidence: Candidate artifact ยท Baseline artifact ยท Before 0b6bf97 โ†’ After bd250bf ยท Workflow run
--- -## Mock-up 2: Multi-Skill Suite PR (`๐Ÿ›‘ No ship` โ€” Mixed Verdicts + `Everything else & full telemetry`) - -This combines `@rwoll`'s multi-skill gist scenario (`โœ… Ship`, `๐Ÿ›‘ No ship`, `โš ๏ธ Re-run`, and `Everything else`) with per-scenario and aggregate telemetry inside `
`. - ---- +## Mock-up 2: Multi-Skill Suite PR (`๐Ÿ›‘ No ship` โ€” Inline Score/Tokens in Scenario Tables) ## Evals: ๐Ÿ›‘ No ship @@ -83,52 +69,58 @@ This combines `@rwoll`'s multi-skill gist scenario (`โœ… Ship`, `๐Ÿ›‘ No ship`, | try-plugin-locally *(not changed)* | โš ๏ธ **Re-run** | 1 scenario broke, but this PR does not change this skill |
-โœ… pr-description: 1 fixed +โœ… pr-description โ€” 1 fixed (1/2 โ†’ 2/2 passing) | | Scenario | Model | Score | Output Tokens | What happened | Evidence | | --- | --- | --- | --- | ---: | --- | --- | -| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 87.5% โ†’ 100.0% | 918 โ†’ 402 (-56.2%) | 3/4 โ†’ 4/4 checks passed (`program` check fixed: includes `## Risks / Rollback`, excludes `#42` from `## Changes`) | [artifact](#) | +| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 87.5% โ†’ 100.0% | 918 โ†’ 402 (-56.2%) | 3/4 โ†’ 4/4 checks passed (fixed `program` check: includes `## Risks / Rollback`, excludes `#42` from `## Changes`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | -Scenarios passing, before โ†’ after: `claude-haiku-5.5` 1/2 โ†’ 2/2. +Pass rate (before โ†’ after): claude-haiku-5.5 1/2 โ†’ 2/2
-๐Ÿ›‘ release-plugins: 1 broke, 1 fixed +๐Ÿ›‘ release-plugins โ€” 1 broke, 1 fixed (4/5 โ†’ 4/5 passing) | | Scenario | Model | Score | Output Tokens | What happened | Evidence | | --- | --- | --- | --- | ---: | --- | --- | | ๐Ÿ“‰ Broke | `preserves-unreleased-initial-version` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 410 โ†’ 390 (-4.9%) | Check failed: `skill-invocation` (skill was not invoked) | [artifact](#) | | ๐Ÿ“ˆ Fixed | `asks-for-unknown-release-base` | `claude-haiku-5.5` | 50.0% โ†’ 100.0% | 620 โ†’ 310 (-50.0%) | 2/4 โ†’ 4/4 checks passed (`output-matches` fixed) | [artifact](#) | -Scenarios passing, before โ†’ after: `claude-haiku-5.5` 4/5 โ†’ 4/5. +Pass rate (before โ†’ after): claude-haiku-5.5 4/5 โ†’ 4/5
-โš ๏ธ try-plugin-locally (not changed): 1 broke +โš ๏ธ try-plugin-locally (not changed in PR) โ€” 1 broke (2/2 โ†’ 1/2 passing) | | Scenario | Model | Score | Output Tokens | What happened | Evidence | | --- | --- | --- | --- | ---: | --- | --- | | ๐Ÿ“‰ Broke | `prepares-current-checkout-session` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 512 โ†’ 488 (-4.7%) | Check failed: `program` (command omitted local plugin path) | [artifact](#) | -This PR does not change this skill, so re-run to rule out single-trial noise. +Pass rate (before โ†’ after): claude-haiku-5.5 2/2 โ†’ 1/2 ยท This PR does not modify try-plugin-locally; re-run to rule out single-trial variance.
-Everything else & full telemetry +๐Ÿ“‹ Unchanged skills & suite coverage โ€” 3 passing (7/7 scenarios), 6 without evals -Unchanged and passing: `update-remote-pins` (3/3), `add-plugin` (2/2), `submit-plugin` (2/2). +| Skill | Status | Scenarios (Before โ†’ After) | Details | +| --- | --- | --- | --- | +| `update-remote-pins` | โœ… Unchanged | 3/3 โ†’ 3/3 | All scenarios passing on `claude-haiku-5.5` | +| `add-plugin` | โœ… Unchanged | 2/2 โ†’ 2/2 | All scenarios passing on `claude-haiku-5.5` | +| `submit-plugin` | โœ… Unchanged | 2/2 โ†’ 2/2 | All scenarios passing on `claude-haiku-5.5` | +| `eval/*` *(6 skills)* | โšช Not evaluated | โ€” | No evals configured (`compare`, `create`, `improve`, `review`, `run`, `setup`) | -Not checked: 6 skills have no evals (`eval/compare`, `eval/create`, `eval/improve`, `eval/review`, `eval/run`, `eval/setup`). +Run config: 16 scenarios ยท claude-haiku-5.5 ยท 1 trial/scenario ยท 0 retries (single-trial flips on unmodified skills can reflect model variance) -16 scenarios on 1 model (`claude-haiku-5.5`), 1 run each, no retries. A single flip can be noise. +
-### Baseline vs. Candidate comparison +
+๐Ÿ“Š Telemetry, token deltas & run evidence โ€” 6 skills evaluated (16 scenarios) -| Eval | Baseline | Candidate | Score Delta | Turns Delta | Duration Delta | Token Delta | Transition | -| --- | --- | --- | --- | --- | --- | --- | --- | +| Skill | Baseline | Candidate | Score Delta | Turns Delta | Duration Delta | Token Delta | Transition | +| --- | --- | --- | --- | ---: | ---: | --- | --- | | `pr-description` | failed (0.94) | passed (1.00) | +0.06 | 0 | -4.4s (-11.4%) | -191 (-0.3%; out: -456 / -31.5%) | fixed | | `release-plugins` | failed (0.85) | failed (0.85) | 0.00 | 0 | -1.2s (-3.1%) | -310 (-0.4%; out: -330 / -18.2%) | regressed | | `try-plugin-locally` | passed (1.00) | failed (0.88) | -0.12 | 0 | +0.8s (+4.2%) | -24 (-0.1%) | regressed | @@ -136,73 +128,72 @@ Not checked: 6 skills have no evals (`eval/compare`, `eval/create`, `eval/improv | `add-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | +0.2s (+1.0%) | +8 (0.0%) | unchanged | | `submit-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.1s (-0.5%) | 0 (0.0%) | unchanged | -Before is `0b6bf97`, After is `bd250bf`. [Candidate evidence](#) ยท [Baseline evidence](#) ยท [Workflow run](#). +Evidence: Candidate artifact ยท Baseline artifact ยท Before 0b6bf97 โ†’ After bd250bf ยท Workflow run
--- -## Mock-up 3: Ultra-Minimal Variant (Exact `@rwoll` Gist Tables + Separate Telemetry Collapsible) - -This variant keeps the top table and per-skill `
` tables **identical** to `@rwoll`'s 5-column gist (zero token/duration columns in the scenario tables), and places all numeric score, token, turn, and duration tables in a single bottom `
` block. +## Mock-up 3: Multi-Skill Suite PR (`๐Ÿ›‘ No ship` โ€” Minimal 5-Column Scenario Tables + Structured Coverage & Telemetry Asides) ---- +## Evals: ๐Ÿ›‘ No ship | Skill | Verdict | Why | | --- | --- | --- | -| **pr-description** | โœ… Ship | 1 scenario fixed | -| **release-plugins** | ๐Ÿ›‘ No ship | 1 scenario broke, 1 fixed | -| try-plugin-locally (not changed) | โš ๏ธ Re-run | 1 scenario broke, but this PR does not change this skill | +| **pr-description** | โœ… **Ship** | 1 scenario fixed | +| **release-plugins** | ๐Ÿ›‘ **No ship** | 1 scenario broke, 1 fixed | +| try-plugin-locally *(not changed)* | โš ๏ธ **Re-run** | 1 scenario broke, but this PR does not change this skill |
-โœ… pr-description: 1 fixed +โœ… pr-description โ€” 1 fixed (1/2 โ†’ 2/2 passing) | | Scenario | Model | What happened | Evidence | | --- | --- | --- | --- | --- | -| ๐Ÿ“ˆ Fixed | drafts-structured-pr-body | Claude Haiku 5.5 | Now includes the `## Risks / Rollback` section and excludes `#42` from `## Changes`. | [log](#) | +| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | Now includes the `## Risks / Rollback` section and excludes `#42` from `## Changes`. | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | -Scenarios passing, before โ†’ after: Claude Haiku 5.5 1/2 โ†’ 2/2. +Pass rate (before โ†’ after): claude-haiku-5.5 1/2 โ†’ 2/2
-๐Ÿ›‘ release-plugins: 1 broke, 1 fixed +๐Ÿ›‘ release-plugins โ€” 1 broke, 1 fixed (4/5 โ†’ 4/5 passing) | | Scenario | Model | What happened | Evidence | | --- | --- | --- | --- | --- | -| ๐Ÿ“‰ Broke | preserves-unreleased-initial-version | Claude Haiku 5.5 | The skill was not used when it should have been (`skill-invocation`). | [log](#) | -| ๐Ÿ“ˆ Fixed | asks-for-unknown-release-base | Claude Haiku 5.5 | Now asks for the missing release base. | [log](#) | +| ๐Ÿ“‰ Broke | `preserves-unreleased-initial-version` | `claude-haiku-5.5` | The skill was not used when it should have been (`skill-invocation`). | [artifact](#) | +| ๐Ÿ“ˆ Fixed | `asks-for-unknown-release-base` | `claude-haiku-5.5` | Now asks for the missing release base (`output-matches`). | [artifact](#) | -Scenarios passing, before โ†’ after: Claude Haiku 5.5 4/5 โ†’ 4/5. +Pass rate (before โ†’ after): claude-haiku-5.5 4/5 โ†’ 4/5
-โš ๏ธ try-plugin-locally (not changed): 1 broke +โš ๏ธ try-plugin-locally (not changed in PR) โ€” 1 broke (2/2 โ†’ 1/2 passing) | | Scenario | Model | What happened | Evidence | | --- | --- | --- | --- | --- | -| ๐Ÿ“‰ Broke | prepares-current-checkout-session | Claude Haiku 5.5 | The command left out the plugin's local path. | [log](#) | +| ๐Ÿ“‰ Broke | `prepares-current-checkout-session` | `claude-haiku-5.5` | The command left out the plugin's local path (`program`). | [artifact](#) | -This PR does not change this skill, so re-run to rule out noise. +Pass rate (before โ†’ after): claude-haiku-5.5 2/2 โ†’ 1/2 ยท This PR does not modify try-plugin-locally; re-run to rule out single-trial variance.
-Everything else - -Unchanged and passing: `update-remote-pins` (3/3), `add-plugin` (2/2), `submit-plugin` (2/2). +๐Ÿ“‹ Unchanged skills & suite coverage โ€” 3 passing (7/7 scenarios), 6 without evals -Not checked: 6 skills have no evals (`eval/compare`, `eval/create`, `eval/improve`, `eval/review`, `eval/run`, `eval/setup`). - -16 scenarios on 1 model, 1 run each, no retries. A single flip can be noise. +| Skill | Status | Scenarios (Before โ†’ After) | Details | +| --- | --- | --- | --- | +| `update-remote-pins` | โœ… Unchanged | 3/3 โ†’ 3/3 | All scenarios passing on `claude-haiku-5.5` | +| `add-plugin` | โœ… Unchanged | 2/2 โ†’ 2/2 | All scenarios passing on `claude-haiku-5.5` | +| `submit-plugin` | โœ… Unchanged | 2/2 โ†’ 2/2 | All scenarios passing on `claude-haiku-5.5` | +| `eval/*` *(6 skills)* | โšช Not evaluated | โ€” | No evals configured (`compare`, `create`, `improve`, `review`, `run`, `setup`) | -Before is `0b6bf97`, After is `bd250bf`. [Workflow run](#). +Run config: 16 scenarios ยท claude-haiku-5.5 ยท 1 trial/scenario ยท 0 retries (single-trial flips on unmodified skills can reflect model variance)
-๐Ÿ“Š Quality, token & duration telemetry (Baseline vs. Candidate) +๐Ÿ“Š Telemetry, token deltas & run evidence โ€” 6 skills evaluated (16 scenarios) | Metric | Baseline | Candidate | Delta | | --- | --- | --- | --- | @@ -216,11 +207,15 @@ Before is `0b6bf97`, After is `bd250bf`. [Workflow run](#). | Total Tokens | 73,199 (71,752 in / 1,447 out) | 73,008 (72,017 in / 991 out) | -191 (-0.3%) | | Output Tokens | 1,447 | 991 | -456 (-31.5%) | -| Case | Baseline | Candidate | Score Delta | Turns | Output Tokens | Wall Time | Transition | -| --- | --- | --- | --- | ---: | ---: | ---: | --- | -| `pr-description / drafts-structured-pr-body` | FAIL (87.5%) | PASS (100.0%) | +12.5% | 2 โ†’ 2 | 918 โ†’ 402 (-56.2%) | 11.1s โ†’ 7.1s (-36.2%) | Fixed | -| `pr-description / ignores-bug-and-review-question` | PASS (100.0%) | PASS (100.0%) | 0.0% | 1 โ†’ 1 | 529 โ†’ 589 (+11.3%) | 6.5s โ†’ 9.3s (+44.2%) | Passed | +| Skill | Baseline | Candidate | Score Delta | Turns Delta | Duration Delta | Token Delta | Transition | +| --- | --- | --- | --- | ---: | ---: | --- | --- | +| `pr-description` | failed (0.94) | passed (1.00) | +0.06 | 0 | -4.4s (-11.4%) | -191 (-0.3%; out: -456 / -31.5%) | fixed | +| `release-plugins` | failed (0.85) | failed (0.85) | 0.00 | 0 | -1.2s (-3.1%) | -310 (-0.4%; out: -330 / -18.2%) | regressed | +| `try-plugin-locally` | passed (1.00) | failed (0.88) | -0.12 | 0 | +0.8s (+4.2%) | -24 (-0.1%) | regressed | +| `update-remote-pins` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.3s (-1.2%) | -12 (0.0%) | unchanged | +| `add-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | +0.2s (+1.0%) | +8 (0.0%) | unchanged | +| `submit-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.1s (-0.5%) | 0 (0.0%) | unchanged | -**Candidate artifact:** [Evidence](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) ยท **Baseline artifact:** [Evidence](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208576029) +Evidence: Candidate artifact ยท Baseline artifact ยท Before 0b6bf97 โ†’ After bd250bf ยท Workflow run
From 92bf10b7236be05d7445ba07f99ef8afb05a43f6 Mon Sep 17 00:00:00 2001 From: Jon Gallant <2163001+jongio@users.noreply.github.com> Date: Mon, 16 Nov 2026 10:05:28 -0800 Subject: [PATCH 03/11] docs(evals): consolidate to single unified PR comment mockup Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/pr-comment-verdict-mockups.md | 142 +---------------------------- 1 file changed, 2 insertions(+), 140 deletions(-) diff --git a/docs/pr-comment-verdict-mockups.md b/docs/pr-comment-verdict-mockups.md index 90f51c8..1f1c909 100644 --- a/docs/pr-comment-verdict-mockups.md +++ b/docs/pr-comment-verdict-mockups.md @@ -1,155 +1,17 @@ -# Skill Eval PR Comment Mock-ups: Ship / No-Ship Verdicts & Collapsible Telemetry - -> **Status:** Draft mock-ups for PM & engineering review -> **Reference:** [`@rwoll` Gist โ€” Mock-up: skill eval PR comment with a ship / no ship verdict per skill](https://gist.github.com/rwoll/5dafeff236e21fca14a71a7301ebc385) -> **Live comments on this PR:** See the 3 rendered PR comments on this pull request timeline to experience the exact collapsed vs. expanded GitHub UI behavior. - -## Goals - -1. **Clear Ship / No-Ship decision at the top:** Reviewers immediately see `โœ… Ship`, `๐Ÿ›‘ No ship`, or `โš ๏ธ Re-run` per skill without scrolling through large telemetry tables. -2. **Progressive disclosure via `
`:** Keep the default comment view compact while preserving all real Vally `0.21.0` metrics (`Score`, `Turns`, `Agent Wall Time`, `Total Duration`, `Input / Output Tokens`, `%` deltas, `Skills Invoked`, and artifact `[Evidence]` links) one click away. -3. **Structured, scannable asides:** Replace loose prose in `Everything else` with a structured `๐Ÿ“‹ Unchanged skills & suite coverage` table and a consistent `๐Ÿ“Š Telemetry, token deltas & run evidence` collapsible across all variants. - ---- - -## Mock-up 1: Single-Skill PR (`โœ… Ship` โ€” Real `pr-description` Vally `0.21.0` Run) - -## Evals: โœ… Ship - -| Skill | Verdict | Why | -| --- | --- | --- | -| **pr-description** | โœ… **Ship** | 1 scenario fixed (`1/2` โ†’ `2/2` passing) ยท score `93.8%` โ†’ `100.0%` ยท `-31.5%` output tokens ยท `-6.6%` agent wall time | - -
-โœ… pr-description โ€” 1 fixed (1/2 โ†’ 2/2 passing) - -| | Scenario | Model | Score | Turns | Wall Time | Output Tokens | What happened | Evidence | -| --- | --- | --- | --- | ---: | ---: | ---: | --- | --- | -| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 87.5% โ†’ 100.0% | 2 โ†’ 2 | 11.1s โ†’ 7.1s (-36.2%) | 918 โ†’ 402 (-56.2%) | 3/4 โ†’ 4/4 checks passed (fixed `program` check: includes `## Risks / Rollback`, excludes `#42` from `## Changes`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | -| โœ… Passed | `ignores-bug-and-review-question` | `claude-haiku-5.5` | 100.0% โ†’ 100.0% | 1 โ†’ 1 | 6.5s โ†’ 9.3s (+44.2%) | 529 โ†’ 589 (+11.3%) | Unchanged and passing (4/4 checks passed) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | - -Pass rate (before โ†’ after): claude-haiku-5.5 1/2 โ†’ 2/2 (1 trial/scenario, 0 retries) - -
- -
-๐Ÿ“Š Telemetry, token deltas & run evidence โ€” score 93.8% โ†’ 100.0%, -31.5% output tokens - -| Metric | Baseline | Candidate | Delta | -| --- | --- | --- | --- | -| Outcome | failed | passed | fixed | -| Passed | 1 | 2 | +1 | -| Failed | 1 | 0 | -1 | -| Avg Score | 93.8% | 100.0% | +6.3% | -| Turns | 3 | 3 | 0 | -| Agent Wall Time | 17.5s | 16.4s | -1.1s (-6.6%) | -| Total Duration | 38.5s | 34.1s | -4.4s (-11.4%) | -| Total Tokens | 73,199 (71,752 in / 1,447 out) | 73,008 (72,017 in / 991 out) | -191 (-0.3%) | -| Output Tokens | 1,447 | 991 | -456 (-31.5%) | - -| Scenario | Baseline | Candidate | Score Delta | Turns | Output Tokens | Wall Time | Transition | -| --- | --- | --- | --- | ---: | ---: | ---: | --- | -| `pr-description / drafts-structured-pr-body` | FAIL (87.5%) | PASS (100.0%) | +12.5% | 2 โ†’ 2 | 918 โ†’ 402 (-56.2%) | 11.1s โ†’ 7.1s (-36.2%) | Fixed | -| `pr-description / ignores-bug-and-review-question` | PASS (100.0%) | PASS (100.0%) | 0.0% | 1 โ†’ 1 | 529 โ†’ 589 (+11.3%) | 6.5s โ†’ 9.3s (+44.2%) | Passed | - -Evidence: Candidate artifact ยท Baseline artifact ยท Before 0b6bf97 โ†’ After bd250bf ยท Workflow run - -
- ---- - -## Mock-up 2: Multi-Skill Suite PR (`๐Ÿ›‘ No ship` โ€” Inline Score/Tokens in Scenario Tables) - ## Evals: ๐Ÿ›‘ No ship | Skill | Verdict | Why | | --- | --- | --- | -| **pr-description** | โœ… **Ship** | 1 scenario fixed (`1/2` โ†’ `2/2` passing) ยท `-31.5%` output tokens | +| **pr-description** | โœ… **Ship** | 1 scenario fixed (`1/2` โ†’ `2/2` passing ยท `-31.5%` output tokens) | | **release-plugins** | ๐Ÿ›‘ **No ship** | 1 scenario broke, 1 fixed (`4/5` โ†’ `4/5` passing) | | try-plugin-locally *(not changed)* | โš ๏ธ **Re-run** | 1 scenario broke, but this PR does not change this skill |
โœ… pr-description โ€” 1 fixed (1/2 โ†’ 2/2 passing) -| | Scenario | Model | Score | Output Tokens | What happened | Evidence | -| --- | --- | --- | --- | ---: | --- | --- | -| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 87.5% โ†’ 100.0% | 918 โ†’ 402 (-56.2%) | 3/4 โ†’ 4/4 checks passed (fixed `program` check: includes `## Risks / Rollback`, excludes `#42` from `## Changes`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | - -Pass rate (before โ†’ after): claude-haiku-5.5 1/2 โ†’ 2/2 - -
- -
-๐Ÿ›‘ release-plugins โ€” 1 broke, 1 fixed (4/5 โ†’ 4/5 passing) - -| | Scenario | Model | Score | Output Tokens | What happened | Evidence | -| --- | --- | --- | --- | ---: | --- | --- | -| ๐Ÿ“‰ Broke | `preserves-unreleased-initial-version` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 410 โ†’ 390 (-4.9%) | Check failed: `skill-invocation` (skill was not invoked) | [artifact](#) | -| ๐Ÿ“ˆ Fixed | `asks-for-unknown-release-base` | `claude-haiku-5.5` | 50.0% โ†’ 100.0% | 620 โ†’ 310 (-50.0%) | 2/4 โ†’ 4/4 checks passed (`output-matches` fixed) | [artifact](#) | - -Pass rate (before โ†’ after): claude-haiku-5.5 4/5 โ†’ 4/5 - -
- -
-โš ๏ธ try-plugin-locally (not changed in PR) โ€” 1 broke (2/2 โ†’ 1/2 passing) - -| | Scenario | Model | Score | Output Tokens | What happened | Evidence | -| --- | --- | --- | --- | ---: | --- | --- | -| ๐Ÿ“‰ Broke | `prepares-current-checkout-session` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 512 โ†’ 488 (-4.7%) | Check failed: `program` (command omitted local plugin path) | [artifact](#) | - -Pass rate (before โ†’ after): claude-haiku-5.5 2/2 โ†’ 1/2 ยท This PR does not modify try-plugin-locally; re-run to rule out single-trial variance. - -
- -
-๐Ÿ“‹ Unchanged skills & suite coverage โ€” 3 passing (7/7 scenarios), 6 without evals - -| Skill | Status | Scenarios (Before โ†’ After) | Details | -| --- | --- | --- | --- | -| `update-remote-pins` | โœ… Unchanged | 3/3 โ†’ 3/3 | All scenarios passing on `claude-haiku-5.5` | -| `add-plugin` | โœ… Unchanged | 2/2 โ†’ 2/2 | All scenarios passing on `claude-haiku-5.5` | -| `submit-plugin` | โœ… Unchanged | 2/2 โ†’ 2/2 | All scenarios passing on `claude-haiku-5.5` | -| `eval/*` *(6 skills)* | โšช Not evaluated | โ€” | No evals configured (`compare`, `create`, `improve`, `review`, `run`, `setup`) | - -Run config: 16 scenarios ยท claude-haiku-5.5 ยท 1 trial/scenario ยท 0 retries (single-trial flips on unmodified skills can reflect model variance) - -
- -
-๐Ÿ“Š Telemetry, token deltas & run evidence โ€” 6 skills evaluated (16 scenarios) - -| Skill | Baseline | Candidate | Score Delta | Turns Delta | Duration Delta | Token Delta | Transition | -| --- | --- | --- | --- | ---: | ---: | --- | --- | -| `pr-description` | failed (0.94) | passed (1.00) | +0.06 | 0 | -4.4s (-11.4%) | -191 (-0.3%; out: -456 / -31.5%) | fixed | -| `release-plugins` | failed (0.85) | failed (0.85) | 0.00 | 0 | -1.2s (-3.1%) | -310 (-0.4%; out: -330 / -18.2%) | regressed | -| `try-plugin-locally` | passed (1.00) | failed (0.88) | -0.12 | 0 | +0.8s (+4.2%) | -24 (-0.1%) | regressed | -| `update-remote-pins` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.3s (-1.2%) | -12 (0.0%) | unchanged | -| `add-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | +0.2s (+1.0%) | +8 (0.0%) | unchanged | -| `submit-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.1s (-0.5%) | 0 (0.0%) | unchanged | - -Evidence: Candidate artifact ยท Baseline artifact ยท Before 0b6bf97 โ†’ After bd250bf ยท Workflow run - -
- ---- - -## Mock-up 3: Multi-Skill Suite PR (`๐Ÿ›‘ No ship` โ€” Minimal 5-Column Scenario Tables + Structured Coverage & Telemetry Asides) - -## Evals: ๐Ÿ›‘ No ship - -| Skill | Verdict | Why | -| --- | --- | --- | -| **pr-description** | โœ… **Ship** | 1 scenario fixed | -| **release-plugins** | ๐Ÿ›‘ **No ship** | 1 scenario broke, 1 fixed | -| try-plugin-locally *(not changed)* | โš ๏ธ **Re-run** | 1 scenario broke, but this PR does not change this skill | - -
-โœ… pr-description โ€” 1 fixed (1/2 โ†’ 2/2 passing) - | | Scenario | Model | What happened | Evidence | | --- | --- | --- | --- | --- | -| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | Now includes the `## Risks / Rollback` section and excludes `#42` from `## Changes`. | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | +| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 3/4 โ†’ 4/4 checks passed: now includes `## Risks / Rollback` and excludes `#42` from `## Changes` (`-56.2%` output tokens). | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | Pass rate (before โ†’ after): claude-haiku-5.5 1/2 โ†’ 2/2 From cffee502e55c0fbc27ed4f94147ff9629b05cef6 Mon Sep 17 00:00:00 2001 From: Jon Gallant <2163001+jongio@users.noreply.github.com> Date: Mon, 16 Nov 2026 10:06:05 -0800 Subject: [PATCH 04/11] docs(evals): set single PR comment mockup to Mock-up 2 layout Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/pr-comment-verdict-mockups.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/pr-comment-verdict-mockups.md b/docs/pr-comment-verdict-mockups.md index 1f1c909..8cb6bdd 100644 --- a/docs/pr-comment-verdict-mockups.md +++ b/docs/pr-comment-verdict-mockups.md @@ -2,16 +2,16 @@ | Skill | Verdict | Why | | --- | --- | --- | -| **pr-description** | โœ… **Ship** | 1 scenario fixed (`1/2` โ†’ `2/2` passing ยท `-31.5%` output tokens) | +| **pr-description** | โœ… **Ship** | 1 scenario fixed (`1/2` โ†’ `2/2` passing) ยท `-31.5%` output tokens | | **release-plugins** | ๐Ÿ›‘ **No ship** | 1 scenario broke, 1 fixed (`4/5` โ†’ `4/5` passing) | | try-plugin-locally *(not changed)* | โš ๏ธ **Re-run** | 1 scenario broke, but this PR does not change this skill |
โœ… pr-description โ€” 1 fixed (1/2 โ†’ 2/2 passing) -| | Scenario | Model | What happened | Evidence | -| --- | --- | --- | --- | --- | -| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 3/4 โ†’ 4/4 checks passed: now includes `## Risks / Rollback` and excludes `#42` from `## Changes` (`-56.2%` output tokens). | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | +| | Scenario | Model | Score | Output Tokens | What happened | Evidence | +| --- | --- | --- | --- | ---: | --- | --- | +| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 87.5% โ†’ 100.0% | 918 โ†’ 402 (-56.2%) | 3/4 โ†’ 4/4 checks passed (fixed `program` check: includes `## Risks / Rollback`, excludes `#42` from `## Changes`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | Pass rate (before โ†’ after): claude-haiku-5.5 1/2 โ†’ 2/2 @@ -20,10 +20,10 @@
๐Ÿ›‘ release-plugins โ€” 1 broke, 1 fixed (4/5 โ†’ 4/5 passing) -| | Scenario | Model | What happened | Evidence | -| --- | --- | --- | --- | --- | -| ๐Ÿ“‰ Broke | `preserves-unreleased-initial-version` | `claude-haiku-5.5` | The skill was not used when it should have been (`skill-invocation`). | [artifact](#) | -| ๐Ÿ“ˆ Fixed | `asks-for-unknown-release-base` | `claude-haiku-5.5` | Now asks for the missing release base (`output-matches`). | [artifact](#) | +| | Scenario | Model | Score | Output Tokens | What happened | Evidence | +| --- | --- | --- | --- | ---: | --- | --- | +| ๐Ÿ“‰ Broke | `preserves-unreleased-initial-version` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 410 โ†’ 390 (-4.9%) | Check failed: `skill-invocation` (skill was not invoked) | [artifact](#) | +| ๐Ÿ“ˆ Fixed | `asks-for-unknown-release-base` | `claude-haiku-5.5` | 50.0% โ†’ 100.0% | 620 โ†’ 310 (-50.0%) | 2/4 โ†’ 4/4 checks passed (`output-matches` fixed) | [artifact](#) | Pass rate (before โ†’ after): claude-haiku-5.5 4/5 โ†’ 4/5 @@ -32,9 +32,9 @@
โš ๏ธ try-plugin-locally (not changed in PR) โ€” 1 broke (2/2 โ†’ 1/2 passing) -| | Scenario | Model | What happened | Evidence | -| --- | --- | --- | --- | --- | -| ๐Ÿ“‰ Broke | `prepares-current-checkout-session` | `claude-haiku-5.5` | The command left out the plugin's local path (`program`). | [artifact](#) | +| | Scenario | Model | Score | Output Tokens | What happened | Evidence | +| --- | --- | --- | --- | ---: | --- | --- | +| ๐Ÿ“‰ Broke | `prepares-current-checkout-session` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 512 โ†’ 488 (-4.7%) | Check failed: `program` (command omitted local plugin path) | [artifact](#) | Pass rate (before โ†’ after): claude-haiku-5.5 2/2 โ†’ 1/2 ยท This PR does not modify try-plugin-locally; re-run to rule out single-trial variance. From b3f752cb68419475e98dca2813cb24645cd1e899 Mon Sep 17 00:00:00 2001 From: Jon Gallant <2163001+jongio@users.noreply.github.com> Date: Mon, 16 Nov 2026 11:12:49 -0800 Subject: [PATCH 05/11] docs(evals): address Ross and Ronnie feedback with pure Vally-driven mockup - Order actionable No-ship / Re-run sections above --- and passing/telemetry sections below - Keep all fields strictly derived from Vally results.jsonl (no git-diff or repo-scan heuristics) - Reconcile 6-skill / 16-scenario aggregate totals and Outcome with the No-ship headline - Replace # placeholder links with real workflow artifact URLs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/pr-comment-verdict-mockups.md | 57 +++++++++++++++--------------- 1 file changed, 29 insertions(+), 28 deletions(-) diff --git a/docs/pr-comment-verdict-mockups.md b/docs/pr-comment-verdict-mockups.md index 8cb6bdd..e49e5cb 100644 --- a/docs/pr-comment-verdict-mockups.md +++ b/docs/pr-comment-verdict-mockups.md @@ -2,55 +2,56 @@ | Skill | Verdict | Why | | --- | --- | --- | -| **pr-description** | โœ… **Ship** | 1 scenario fixed (`1/2` โ†’ `2/2` passing) ยท `-31.5%` output tokens | | **release-plugins** | ๐Ÿ›‘ **No ship** | 1 scenario broke, 1 fixed (`4/5` โ†’ `4/5` passing) | -| try-plugin-locally *(not changed)* | โš ๏ธ **Re-run** | 1 scenario broke, but this PR does not change this skill | +| **try-plugin-locally** | โš ๏ธ **Re-run** | 1 scenario errored (`execution-error` ยท `2/2` โ†’ `1/2` passing) | +| **pr-description** | โœ… **Ship** | 1 scenario fixed (`1/2` โ†’ `2/2` passing) ยท `-31.5%` output tokens |
-โœ… pr-description โ€” 1 fixed (1/2 โ†’ 2/2 passing) +๐Ÿ›‘ release-plugins โ€” 1 broke, 1 fixed (4/5 โ†’ 4/5 passing) | | Scenario | Model | Score | Output Tokens | What happened | Evidence | | --- | --- | --- | --- | ---: | --- | --- | -| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 87.5% โ†’ 100.0% | 918 โ†’ 402 (-56.2%) | 3/4 โ†’ 4/4 checks passed (fixed `program` check: includes `## Risks / Rollback`, excludes `#42` from `## Changes`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | +| ๐Ÿ“‰ Broke | `preserves-unreleased-initial-version` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 410 โ†’ 390 (-4.9%) | 4/4 โ†’ 3/4 checks passed (failed: `skill-invocation`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | +| ๐Ÿ“ˆ Fixed | `asks-for-unknown-release-base` | `claude-haiku-5.5` | 50.0% โ†’ 100.0% | 620 โ†’ 310 (-50.0%) | 2/4 โ†’ 4/4 checks passed (fixed: `output-matches`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | -Pass rate (before โ†’ after): claude-haiku-5.5 1/2 โ†’ 2/2 +Pass rate (before โ†’ after): claude-haiku-5.5 4/5 โ†’ 4/5
-๐Ÿ›‘ release-plugins โ€” 1 broke, 1 fixed (4/5 โ†’ 4/5 passing) +โš ๏ธ try-plugin-locally โ€” 1 errored (2/2 โ†’ 1/2 passing) | | Scenario | Model | Score | Output Tokens | What happened | Evidence | | --- | --- | --- | --- | ---: | --- | --- | -| ๐Ÿ“‰ Broke | `preserves-unreleased-initial-version` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 410 โ†’ 390 (-4.9%) | Check failed: `skill-invocation` (skill was not invoked) | [artifact](#) | -| ๐Ÿ“ˆ Fixed | `asks-for-unknown-release-base` | `claude-haiku-5.5` | 50.0% โ†’ 100.0% | 620 โ†’ 310 (-50.0%) | 2/4 โ†’ 4/4 checks passed (`output-matches` fixed) | [artifact](#) | +| โš ๏ธ Error | `prepares-current-checkout-session` | `claude-haiku-5.5` | 100.0% โ†’ โ€” | 512 โ†’ โ€” | Trial execution error (`execution-error`); re-run workflow to verify | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | -Pass rate (before โ†’ after): claude-haiku-5.5 4/5 โ†’ 4/5 +Pass rate (before โ†’ after): claude-haiku-5.5 2/2 โ†’ 1/2 (1 execution error)
+--- +
-โš ๏ธ try-plugin-locally (not changed in PR) โ€” 1 broke (2/2 โ†’ 1/2 passing) +โœ… pr-description โ€” 1 fixed (1/2 โ†’ 2/2 passing) | | Scenario | Model | Score | Output Tokens | What happened | Evidence | | --- | --- | --- | --- | ---: | --- | --- | -| ๐Ÿ“‰ Broke | `prepares-current-checkout-session` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 512 โ†’ 488 (-4.7%) | Check failed: `program` (command omitted local plugin path) | [artifact](#) | +| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 87.5% โ†’ 100.0% | 918 โ†’ 402 (-56.2%) | 3/4 โ†’ 4/4 checks passed (fixed: `program`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | -Pass rate (before โ†’ after): claude-haiku-5.5 2/2 โ†’ 1/2 ยท This PR does not modify try-plugin-locally; re-run to rule out single-trial variance. +Pass rate (before โ†’ after): claude-haiku-5.5 1/2 โ†’ 2/2
-๐Ÿ“‹ Unchanged skills & suite coverage โ€” 3 passing (7/7 scenarios), 6 without evals +๐Ÿ“‹ Unchanged evals & run configuration โ€” 3 passing (7/7 scenarios) | Skill | Status | Scenarios (Before โ†’ After) | Details | | --- | --- | --- | --- | | `update-remote-pins` | โœ… Unchanged | 3/3 โ†’ 3/3 | All scenarios passing on `claude-haiku-5.5` | | `add-plugin` | โœ… Unchanged | 2/2 โ†’ 2/2 | All scenarios passing on `claude-haiku-5.5` | | `submit-plugin` | โœ… Unchanged | 2/2 โ†’ 2/2 | All scenarios passing on `claude-haiku-5.5` | -| `eval/*` *(6 skills)* | โšช Not evaluated | โ€” | No evals configured (`compare`, `create`, `improve`, `review`, `run`, `setup`) | -Run config: 16 scenarios ยท claude-haiku-5.5 ยท 1 trial/scenario ยท 0 retries (single-trial flips on unmodified skills can reflect model variance) +Run config: 16 scenarios across 6 evals ยท claude-haiku-5.5 ยท 1 trial/scenario ยท 0 skipped ยท 0 retries
@@ -59,23 +60,23 @@ | Metric | Baseline | Candidate | Delta | | --- | --- | --- | --- | -| Outcome | failed | passed | fixed | -| Passed | 1 | 2 | +1 | -| Failed | 1 | 0 | -1 | -| Avg Score | 93.8% | 100.0% | +6.3% | -| Turns | 3 | 3 | 0 | -| Agent Wall Time | 17.5s | 16.4s | -1.1s (-6.6%) | -| Total Duration | 38.5s | 34.1s | -4.4s (-11.4%) | -| Total Tokens | 73,199 (71,752 in / 1,447 out) | 73,008 (72,017 in / 991 out) | -191 (-0.3%) | -| Output Tokens | 1,447 | 991 | -456 (-31.5%) | +| Outcome | failed | failed (๐Ÿ›‘ No ship) | 2 fixed, 1 broke, 1 error | +| Scenarios Passed | 14 / 16 | 14 / 16 | 0 (+2 fixed, -2 broke/error) | +| Scenarios Failed / Errored | 2 / 16 | 2 / 16 | 0 | +| Skills Passing | 4 / 6 | 4 / 6 | 0 | +| Avg Score | 96.5% | 95.5% | -1.0% | +| Turns | 25 | 24 | -1 | +| Total Duration | 159.2s | 154.2s | -5.0s (-3.1%) | +| Total Tokens | 319,399 (312,539 in / 6,860 out) | 318,870 (312,840 in / 6,030 out) | -529 (-0.2%) | +| Output Tokens | 6,860 | 6,030 | -830 (-12.1%) | | Skill | Baseline | Candidate | Score Delta | Turns Delta | Duration Delta | Token Delta | Transition | | --- | --- | --- | --- | ---: | ---: | --- | --- | -| `pr-description` | failed (0.94) | passed (1.00) | +0.06 | 0 | -4.4s (-11.4%) | -191 (-0.3%; out: -456 / -31.5%) | fixed | | `release-plugins` | failed (0.85) | failed (0.85) | 0.00 | 0 | -1.2s (-3.1%) | -310 (-0.4%; out: -330 / -18.2%) | regressed | -| `try-plugin-locally` | passed (1.00) | failed (0.88) | -0.12 | 0 | +0.8s (+4.2%) | -24 (-0.1%) | regressed | -| `update-remote-pins` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.3s (-1.2%) | -12 (0.0%) | unchanged | -| `add-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | +0.2s (+1.0%) | +8 (0.0%) | unchanged | +| `try-plugin-locally` | passed (1.00) | error (โ€”) | โ€” | -1 | +0.8s (+4.2%) | -24 (-0.1%; out: -44 / -5.2%) | error | +| `pr-description` | failed (0.94) | passed (1.00) | +0.06 | 0 | -4.4s (-11.4%) | -191 (-0.3%; out: -456 / -31.5%) | fixed | +| `update-remote-pins` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.3s (-1.2%) | -12 (0.0%; out: -4 / -0.3%) | unchanged | +| `add-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | +0.2s (+1.0%) | +8 (0.0%; out: +4 / +0.5%) | unchanged | | `submit-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.1s (-0.5%) | 0 (0.0%) | unchanged | Evidence: Candidate artifact ยท Baseline artifact ยท Before 0b6bf97 โ†’ After bd250bf ยท Workflow run From df256c245cb6a2842f1deb5b57234dcf2053bae8 Mon Sep 17 00:00:00 2001 From: Jon Gallant <2163001+jongio@users.noreply.github.com> Date: Mon, 16 Nov 2026 12:14:18 -0800 Subject: [PATCH 06/11] feat(evals): implement Ship / No-ship verdict summary and collapsible report sections MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add `summarizeSkillVerdicts` and collapsible `
` sections to `renderReport` in `.github/scripts/eval-report.mjs` when comparing a baseline and candidate run. - Order `๐Ÿ›‘ No ship` and `โš ๏ธ Re-run` skill sections above `---`, and place `โœ… Ship`, `๐Ÿ“‹ Unchanged evals & run configuration`, and `๐Ÿ“Š Telemetry, token deltas & run evidence` below `---`. - Align aggregate `Outcome` with the headline verdict, remove temporary RFC mockup doc, and add unit tests in `eval-report.test.mjs`. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .github/scripts/eval-report.mjs | 294 ++++++++++++++++++++++++--- .github/scripts/eval-report.test.mjs | 42 ++++ docs/checks-and-evals.md | 9 +- docs/pr-comment-verdict-mockups.md | 84 -------- 4 files changed, 317 insertions(+), 112 deletions(-) delete mode 100644 docs/pr-comment-verdict-mockups.md diff --git a/.github/scripts/eval-report.mjs b/.github/scripts/eval-report.mjs index 4c935f6..05a8218 100644 --- a/.github/scripts/eval-report.mjs +++ b/.github/scripts/eval-report.mjs @@ -278,6 +278,8 @@ export function summarizeRun(records) { group.failures.add(sanitizeGraderId(detail)); } } + group.gradersTotal = (group.gradersTotal ?? 0) + (grade?.details?.length ?? 0); + group.gradersPassed = (group.gradersPassed ?? 0) + (grade?.details ?? []).filter((d) => d.passed).length; } if (!trialCount) throw new Error("No cases were evaluated."); @@ -306,6 +308,8 @@ export function summarizeRun(records) { : {}), } : undefined, + gradersPassed: group.gradersPassed ?? 0, + gradersTotal: group.gradersTotal ?? 0, failures: [...group.failures], }; }); @@ -505,11 +509,182 @@ function formatDurationDelta(deltaMs, pct) { return `${sign}${formatDuration(Math.abs(deltaMs))}${pctText}`; } +export function summarizeSkillVerdicts(comparison) { + const bySkill = new Map(); + for (const item of comparison.cases) { + const skillName = item.candidate?.evalName ?? item.baseline?.evalName ?? "eval"; + let group = bySkill.get(skillName); + if (!group) { + group = { + skillName, + cases: [], + fixed: 0, + broke: 0, + errored: 0, + changed: false, + basePassed: 0, + baseTotal: 0, + candPassed: 0, + candTotal: 0, + baseOutputTokens: 0, + candOutputTokens: 0, + hasOutputTokens: true, + }; + bySkill.set(skillName, group); + } + group.cases.push(item); + if (item.baseline) { + group.baseTotal++; + if (item.baseline.passed) group.basePassed++; + } + if (item.candidate) { + group.candTotal++; + if (item.candidate.passed) group.candPassed++; + } + const baseOut = item.baseline?.tokenUsage?.outputTokens; + const candOut = item.candidate?.tokenUsage?.outputTokens; + if (typeof baseOut === "number" && typeof candOut === "number") { + group.baseOutputTokens += baseOut; + group.candOutputTokens += candOut; + } else { + group.hasOutputTokens = false; + } + if (item.candidate?.status === "ERROR / NOT EVALUATED" || !item.candidate) { + group.errored++; + group.changed = true; + } else if (item.transition === "Fixed") { + group.fixed++; + group.changed = true; + } else if (item.transition === "Regressed") { + group.broke++; + group.changed = true; + } else if (item.transition !== "Unchanged") { + group.changed = true; + } + } + + const severityOrder = { "๐Ÿ›‘ No ship": 0, "โš ๏ธ Re-run": 1, "โœ… Ship": 2 }; + return [...bySkill.values()] + .map((group) => { + const hasBehavioralFailure = group.cases.some((c) => c.candidate?.status === "BEHAVIORAL FAILURE"); + const verdict = group.candTotal === 0 || (group.errored > 0 && !hasBehavioralFailure && group.broke === 0) + ? "โš ๏ธ Re-run" + : group.candPassed === group.candTotal && group.broke === 0 && group.errored === 0 + ? "โœ… Ship" + : "๐Ÿ›‘ No ship"; + const reasons = []; + if (group.fixed > 0 && group.broke > 0) { + reasons.push(`${group.broke} ${group.broke === 1 ? "scenario" : "scenarios"} broke, ${group.fixed} fixed (${group.basePassed}/${group.baseTotal} โ†’ ${group.candPassed}/${group.candTotal} passing)`); + } else if (group.broke > 0) { + reasons.push(`${group.broke} ${group.broke === 1 ? "scenario" : "scenarios"} broke (${group.basePassed}/${group.baseTotal} โ†’ ${group.candPassed}/${group.candTotal} passing)`); + } else if (group.errored > 0) { + reasons.push(`${group.errored} ${group.errored === 1 ? "scenario" : "scenarios"} errored (${group.basePassed}/${group.baseTotal} โ†’ ${group.candPassed}/${group.candTotal} passing)`); + } else if (group.fixed > 0) { + reasons.push(`${group.fixed} ${group.fixed === 1 ? "scenario" : "scenarios"} fixed (${group.basePassed}/${group.baseTotal} โ†’ ${group.candPassed}/${group.candTotal} passing)`); + } else { + reasons.push(`${group.candPassed}/${group.candTotal} scenarios passing`); + } + if (group.hasOutputTokens && group.baseOutputTokens > 0) { + const outDelta = group.candOutputTokens - group.baseOutputTokens; + const outPct = (outDelta / group.baseOutputTokens) * 100; + if (Math.abs(outPct) >= 0.1) { + const sign = outPct > 0 ? "+" : ""; + reasons.push(`${sign}${outPct.toFixed(1)}% output tokens`); + } + } + const summaryCounts = []; + if (group.broke > 0) summaryCounts.push(`${group.broke} broke`); + if (group.errored > 0) summaryCounts.push(`${group.errored} errored`); + if (group.fixed > 0) summaryCounts.push(`${group.fixed} fixed`); + if (summaryCounts.length === 0) summaryCounts.push(`${group.candPassed}/${group.candTotal} passing`); + return { + ...group, + verdict, + summaryLabel: `${summaryCounts.join(", ")} (${group.basePassed}/${group.baseTotal} โ†’ ${group.candPassed}/${group.candTotal} passing)`, + why: reasons.join(" ยท "), + }; + }) + .sort((a, b) => (severityOrder[a.verdict] ?? 3) - (severityOrder[b.verdict] ?? 3)); +} + +function describeCaseChange(item) { + const baseFailures = item.baseline?.failures ?? []; + const candFailures = item.candidate?.failures ?? []; + if (item.candidate?.status === "ERROR / NOT EVALUATED") { + return "Trial execution error; re-run workflow to verify"; + } + const hasCounts = Boolean(item.baseline?.gradersTotal && item.candidate?.gradersTotal); + const countPrefix = hasCounts + ? `${item.baseline.gradersPassed}/${item.baseline.gradersTotal} โ†’ ${item.candidate.gradersPassed}/${item.candidate.gradersTotal} checks passed` + : undefined; + if (item.transition === "Fixed") { + const resolved = baseFailures.filter((f) => !candFailures.includes(f)); + const prefix = countPrefix ?? "Now passes all checks"; + return resolved.length + ? `${prefix} (fixed: ${resolved.map((f) => `\`${escapeCell(f)}\``).join(", ")})` + : prefix; + } + if (item.transition === "Regressed") { + const newlyBroken = candFailures.filter((f) => !baseFailures.includes(f)); + const prefix = countPrefix ?? "Check failed"; + return newlyBroken.length + ? `${prefix} (failed: ${newlyBroken.map((f) => `\`${escapeCell(f)}\``).join(", ")})` + : prefix; + } + if (item.candidate?.passed) { + return item.candidate.gradersTotal + ? `Unchanged (${item.candidate.gradersPassed}/${item.candidate.gradersTotal} checks passed)` + : "Unchanged and passing"; + } + return candFailures.length + ? `Still failing (${candFailures.map((f) => `\`${escapeCell(f)}\``).join(", ")})` + : "Unchanged"; +} + +function renderSkillDetailsBlock(skill, defaultModel) { + const verdictEmoji = skill.verdict.split(" ")[0]; + const modelLabel = defaultModel || "default"; + return [ + "
", + `${verdictEmoji} ${escapeCell(skill.skillName)} โ€” ${skill.summaryLabel}`, + "", + "| | Scenario | Model | Score | Output Tokens | What happened |", + "| --- | --- | --- | --- | ---: | --- |", + ...skill.cases.map((item) => { + const badge = item.candidate?.status === "ERROR / NOT EVALUATED" + ? "โš ๏ธ Error" + : item.transition === "Fixed" + ? "๐Ÿ“ˆ Fixed" + : item.transition === "Regressed" + ? "๐Ÿ“‰ Broke" + : item.candidate?.passed + ? "โœ… Passed" + : "โŒ Failed"; + const rawCaseName = item.candidate?.name ?? item.baseline?.name ?? item.key; + const model = item.candidate?.models?.[0] ?? item.baseline?.models?.[0] ?? modelLabel; + const scoreText = `${formatScore(item.baseline?.score)} โ†’ ${item.candidate?.status === "ERROR / NOT EVALUATED" ? "โ€”" : formatScore(item.candidate?.score)}`; + const baseOut = item.baseline?.tokenUsage?.outputTokens; + const candOut = item.candidate?.tokenUsage?.outputTokens; + const outText = typeof baseOut === "number" && typeof candOut === "number" && item.candidate?.status !== "ERROR / NOT EVALUATED" + ? `${baseOut.toLocaleString("en-US")} โ†’ ${candOut.toLocaleString("en-US")} (${formatTokenDelta(item.outputTokenDelta, item.outputTokenDeltaPct)})` + : typeof baseOut === "number" + ? `${baseOut.toLocaleString("en-US")} โ†’ โ€”` + : "โ€”"; + return `| ${badge} | \`${escapeCell(rawCaseName)}\` | \`${escapeCell(model)}\` | ${scoreText} | ${outText} | ${describeCaseChange(item)} |`; + }), + "", + `Pass rate (before โ†’ after): ${escapeCell(modelLabel)} ${skill.basePassed}/${skill.baseTotal} โ†’ ${skill.candPassed}/${skill.candTotal}`, + "", + "
", + "", + ]; +} + export function renderReport(summary, path, { baselineSummary, baselinePath } = {}) { const modelHeader = summary.models?.length ? ` | Model: \`${escapeCell(summary.models.join(", "))}\`` : ""; - const lines = [ + const singleRunLines = [ "## Behavioral eval report", "", `Saved run: \`${escapeCell(path)}\`${modelHeader}`, @@ -533,41 +708,110 @@ export function renderReport(summary, path, { baselineSummary, baselinePath } = row.failures.map((grader) => `- ${escapeCell(row.name)} / ${escapeCell(sanitizeGraderId(grader))}: check failed`)); const errorLines = summary.errors.map((error) => `- ERROR: ${escapeCell(error)}`); if (failureLines.length || errorLines.length) { - lines.push(...failureLines, ...errorLines, ""); + singleRunLines.push(...failureLines, ...errorLines, ""); + } + + if (!baselineSummary) { + return singleRunLines.join("\n"); } - if (baselineSummary) { - const comparison = compareSummaries(baselineSummary, summary); + const comparison = compareSummaries(baselineSummary, summary); + const skillVerdicts = summarizeSkillVerdicts(comparison); + const hasNoShip = skillVerdicts.some((s) => s.verdict === "๐Ÿ›‘ No ship") || summary.failedCount > 0; + const hasRerun = skillVerdicts.some((s) => s.verdict === "โš ๏ธ Re-run") || summary.errorCount > 0; + const overallVerdict = hasNoShip ? "๐Ÿ›‘ No ship" : hasRerun ? "โš ๏ธ Re-run" : "โœ… Ship"; + const baseOutcome = (baselineSummary.failedCount + baselineSummary.errorCount) === 0 ? "passed" : "failed"; + const candOutcome = (summary.failedCount + summary.errorCount) === 0 ? `passed (${overallVerdict})` : `failed (${overallVerdict})`; + const outcomeDelta = baseOutcome === "failed" && (summary.failedCount + summary.errorCount) === 0 + ? "fixed" + : baseOutcome === "passed" && (summary.failedCount + summary.errorCount) > 0 + ? "regressed" + : `${overallVerdict}`; + const defaultModel = summary.models?.[0] ?? baselineSummary.models?.[0] ?? "default"; + + const actionableSkills = skillVerdicts.filter((s) => s.verdict !== "โœ… Ship"); + const passingChangedSkills = skillVerdicts.filter((s) => s.verdict === "โœ… Ship" && (s.changed || skillVerdicts.length === 1)); + const unchangedPassingSkills = skillVerdicts.filter((s) => s.verdict === "โœ… Ship" && !s.changed && skillVerdicts.length > 1); + + const lines = [ + `## Evals: ${overallVerdict}`, + "", + "| Skill | Verdict | Why |", + "| --- | --- | --- |", + ...skillVerdicts.map((s) => { + const boldVerdict = s.verdict.replace(/^(โœ…|๐Ÿ›‘|โš ๏ธ)\s+(.*)$/u, "$1 **$2**"); + return `| **${escapeCell(s.skillName)}** | ${boldVerdict} | ${s.why} |`; + }), + "", + ]; + + for (const skill of actionableSkills) { + lines.push(...renderSkillDetailsBlock(skill, defaultModel)); + } + + if (actionableSkills.length > 0) { + lines.push("---", ""); + } + + for (const skill of passingChangedSkills) { + lines.push(...renderSkillDetailsBlock(skill, defaultModel)); + } + + if (unchangedPassingSkills.length > 0) { + const unchangedScenarios = unchangedPassingSkills.reduce((sum, s) => sum + s.candTotal, 0); lines.push( - "## Baseline vs. Candidate Comparison", + "
", + `๐Ÿ“‹ Unchanged evals & run configuration โ€” ${unchangedPassingSkills.length} passing (${unchangedScenarios}/${unchangedScenarios} scenarios)`, "", - ...(baselinePath ? [`Baseline: \`${escapeCell(baselinePath)}\` | Candidate: \`${escapeCell(path)}\``, ""] : []), - "| Metric | Baseline | Candidate | Delta |", + "| Skill | Status | Scenarios (Before โ†’ After) | Details |", "| --- | --- | --- | --- |", - `| Passed | ${baselineSummary.passedCount} | ${summary.passedCount} | ${formatSignedInt(comparison.deltas.passed)} |`, - `| Failed | ${baselineSummary.failedCount + baselineSummary.errorCount} | ${summary.failedCount + summary.errorCount} | ${formatSignedInt(comparison.deltas.failed)} |`, - `| Avg Score | ${formatScore(baselineSummary.averageScore)} | ${formatScore(summary.averageScore)} | ${formatScoreDelta(comparison.deltas.averageScore)} |`, - `| Total Turns | ${baselineSummary.totalTurns ?? "โ€”"} | ${summary.totalTurns ?? "โ€”"} | ${formatSignedInt(comparison.deltas.totalTurns)} |`, - `| Total Tokens | ${formatTokens(baselineSummary.tokenUsage)} | ${formatTokens(summary.tokenUsage)} | ${formatTokenDelta(comparison.deltas.totalTokens, comparison.deltas.totalTokensPct)} |`, - ...(typeof baselineSummary.tokenUsage?.outputTokens === "number" && typeof summary.tokenUsage?.outputTokens === "number" - ? [`| Output Tokens | ${baselineSummary.tokenUsage.outputTokens.toLocaleString("en-US")} | ${summary.tokenUsage.outputTokens.toLocaleString("en-US")} | ${formatTokenDelta(comparison.deltas.outputTokens, comparison.deltas.outputTokensPct)} |`] - : []), - `| Agent Wall Time | ${formatDuration(baselineSummary.totalWallTimeMs)} | ${formatDuration(summary.totalWallTimeMs)} | ${formatDurationDelta(comparison.deltas.totalWallTimeMs)} |`, - `| Total Duration | ${formatDuration(baselineSummary.totalDurationMs)} | ${formatDuration(summary.totalDurationMs)} | ${formatDurationDelta(comparison.deltas.totalDurationMs)} |`, + ...unchangedPassingSkills.map((s) => + `| \`${escapeCell(s.skillName)}\` | โœ… Unchanged | ${s.basePassed}/${s.baseTotal} โ†’ ${s.candPassed}/${s.candTotal} | All scenarios passing on \`${escapeCell(defaultModel)}\` |`, + ), "", - "| Case | Baseline | Candidate | Score Delta | Turn Delta | Token Delta | Wall Time Delta | Transition |", - "| --- | --- | --- | --- | --- | --- | --- | --- |", - ...comparison.cases.map((item) => { - const baseCell = item.baseline ? `${item.baseline.status} (${formatScore(item.baseline.score)})` : "โ€”"; - const candCell = item.candidate ? `${item.candidate.status} (${formatScore(item.candidate.score)})` : "โ€”"; - return `| ${escapeCell(item.key)} | ${baseCell} | ${candCell} | ${formatScoreDelta(item.scoreDelta)} | ${formatSignedInt(item.turnsDelta)} | ${formatTokenDelta(item.tokenDelta, item.tokenDeltaPct)} | ${formatDurationDelta(item.wallTimeDeltaMs, item.wallTimeDeltaPct)} | ${item.transition} |`; - }), + `Run config: ${comparison.cases.length} scenarios across ${skillVerdicts.length} ${skillVerdicts.length === 1 ? "eval" : "evals"} ยท ${escapeCell(defaultModel)}`, "", - "> **Note:** Single-run score, token, and duration deltas are observed signals for review rather than multi-trial statistical proof.", + "
", "", ); } + lines.push( + "
", + `๐Ÿ“Š Telemetry, token deltas & run evidence โ€” ${skillVerdicts.length} ${skillVerdicts.length === 1 ? "skill" : "skills"} evaluated (${comparison.cases.length} ${comparison.cases.length === 1 ? "scenario" : "scenarios"})`, + "", + "## Baseline vs. Candidate Comparison", + "", + ...(baselinePath ? [`Baseline: \`${escapeCell(baselinePath)}\` | Candidate: \`${escapeCell(path)}\``, ""] : []), + "| Metric | Baseline | Candidate | Delta |", + "| --- | --- | --- | --- |", + `| Outcome | ${baseOutcome} | ${candOutcome} | ${outcomeDelta} |`, + `| Passed | ${baselineSummary.passedCount} | ${summary.passedCount} | ${formatSignedInt(comparison.deltas.passed)} |`, + `| Failed | ${baselineSummary.failedCount + baselineSummary.errorCount} | ${summary.failedCount + summary.errorCount} | ${formatSignedInt(comparison.deltas.failed)} |`, + `| Avg Score | ${formatScore(baselineSummary.averageScore)} | ${formatScore(summary.averageScore)} | ${formatScoreDelta(comparison.deltas.averageScore)} |`, + `| Total Turns | ${baselineSummary.totalTurns ?? "โ€”"} | ${summary.totalTurns ?? "โ€”"} | ${formatSignedInt(comparison.deltas.totalTurns)} |`, + `| Total Tokens | ${formatTokens(baselineSummary.tokenUsage)} | ${formatTokens(summary.tokenUsage)} | ${formatTokenDelta(comparison.deltas.totalTokens, comparison.deltas.totalTokensPct)} |`, + ...(typeof baselineSummary.tokenUsage?.outputTokens === "number" && typeof summary.tokenUsage?.outputTokens === "number" + ? [`| Output Tokens | ${baselineSummary.tokenUsage.outputTokens.toLocaleString("en-US")} | ${summary.tokenUsage.outputTokens.toLocaleString("en-US")} | ${formatTokenDelta(comparison.deltas.outputTokens, comparison.deltas.outputTokensPct)} |`] + : []), + `| Agent Wall Time | ${formatDuration(baselineSummary.totalWallTimeMs)} | ${formatDuration(summary.totalWallTimeMs)} | ${formatDurationDelta(comparison.deltas.totalWallTimeMs)} |`, + `| Total Duration | ${formatDuration(baselineSummary.totalDurationMs)} | ${formatDuration(summary.totalDurationMs)} | ${formatDurationDelta(comparison.deltas.totalDurationMs)} |`, + "", + "| Case | Baseline | Candidate | Score Delta | Turn Delta | Token Delta | Wall Time Delta | Transition |", + "| --- | --- | --- | --- | --- | --- | --- | --- |", + ...comparison.cases.map((item) => { + const baseCell = item.baseline ? `${item.baseline.status} (${formatScore(item.baseline.score)})` : "โ€”"; + const candCell = item.candidate ? `${item.candidate.status} (${formatScore(item.candidate.score)})` : "โ€”"; + return `| ${escapeCell(item.key)} | ${baseCell} | ${candCell} | ${formatScoreDelta(item.scoreDelta)} | ${formatSignedInt(item.turnsDelta)} | ${formatTokenDelta(item.tokenDelta, item.tokenDeltaPct)} | ${formatDurationDelta(item.wallTimeDeltaMs, item.wallTimeDeltaPct)} | ${item.transition} |`; + }), + "", + "> **Note:** Single-run score, token, and duration deltas are observed signals for review rather than multi-trial statistical proof.", + "", + ...singleRunLines, + "
", + "", + ); + return lines.join("\n"); } diff --git a/.github/scripts/eval-report.test.mjs b/.github/scripts/eval-report.test.mjs index 125ae00..6b293f7 100644 --- a/.github/scripts/eval-report.test.mjs +++ b/.github/scripts/eval-report.test.mjs @@ -16,6 +16,7 @@ import { loadResults, renderReport, summarizeRun, + summarizeSkillVerdicts, } from "./eval-report.mjs"; const root = fileURLToPath(new URL("../../", import.meta.url)); @@ -255,6 +256,10 @@ test("compareSummaries and renderReport surface fixed failures, regressions, and baselineSummary: baseline, baselinePath: "baseline/results.jsonl", }); + assert.match(markdown, /## Evals: โœ… Ship/); + assert.match(markdown, /\| \*\*pr-description\*\* \| โœ… \*\*Ship\*\* \| 1 scenario fixed \(1\/2 โ†’ 2\/2 passing\) ยท -64\.0% output tokens \|/); + assert.match(markdown, /โœ… pr-description<\/strong> โ€” 1 fixed \(1\/2 โ†’ 2\/2 passing\)<\/summary>/); + assert.match(markdown, /๐Ÿ“Š Telemetry, token deltas & run evidence<\/strong>/); assert.match(markdown, /## Baseline vs\. Candidate Comparison/); assert.match(markdown, /\| Total Turns \| 3 \| 3 \| 0 \|/); assert.match(markdown, /\| Total Tokens \| 6,960 \(in: 5,570, out: 1,390\) \| 3,300 \(in: 2,800, out: 500\) \| -3,660 \(-52\.6%\) \|/); @@ -263,6 +268,43 @@ test("compareSummaries and renderReport surface fixed failures, regressions, and assert.match(markdown, /\| pr-description \/ pr-description-drafts-structured-pr-body \| BEHAVIORAL FAILURE \(50\.0%\) \| PASS \(100\.0%\) \| \+50\.0% \| 0 \| -2,410 \(-54\.8%\) \| -2\.4s \(-46\.2%\) \| Fixed \|/); assert.match(markdown, /\| pr-description \/ pr-description-ignores-bug-and-review-question \| PASS \(100\.0%\) \| PASS \(100\.0%\) \| 0\.0% \| 0 \| -1,250 \(-48\.8%\) \| -1\.3s \(-41\.9%\) \| Reduced tokens at parity \|/); assert.match(markdown, /Single-run score, token, and duration deltas are observed signals for review/); + + // Multi-skill mixed verdict ordering: No ship and Re-run above ---, Ship and Unchanged/Telemetry below --- + const multiSkillBase = summarizeRun([ + makeTrial({ runId: "ms-base", evalName: "pr-description", stimulus: "drafts-body", passed: false, gradeScore: 0.5, outputTokens: 900, inputTokens: 2000, totalTokens: 2900, details: [{ name: "program", passed: false, score: 0 }, { name: "completed", passed: true, score: 1 }] }), + makeTrial({ runId: "ms-base", evalName: "release-plugins", stimulus: "preserves-version", passed: true, gradeScore: 1, outputTokens: 400, inputTokens: 2000, totalTokens: 2400, details: [{ name: "skill-invocation", passed: true, score: 1 }] }), + makeTrial({ runId: "ms-base", evalName: "try-plugin-locally", stimulus: "prepares-session", passed: true, gradeScore: 1, outputTokens: 500, inputTokens: 2000, totalTokens: 2500 }), + makeTrial({ runId: "ms-base", evalName: "update-remote-pins", stimulus: "updates-pins", passed: true, gradeScore: 1, outputTokens: 300, inputTokens: 1500, totalTokens: 1800 }), + makeFooter("ms-base"), + ]); + const multiSkillCand = summarizeRun([ + makeTrial({ runId: "ms-cand", evalName: "pr-description", stimulus: "drafts-body", passed: true, gradeScore: 1, outputTokens: 400, inputTokens: 2000, totalTokens: 2400, details: [{ name: "program", passed: true, score: 1 }, { name: "completed", passed: true, score: 1 }] }), + makeTrial({ runId: "ms-cand", evalName: "release-plugins", stimulus: "preserves-version", passed: false, gradeScore: 0, outputTokens: 390, inputTokens: 2000, totalTokens: 2390, details: [{ name: "skill-invocation", passed: false, score: 0 }] }), + makeTrial({ runId: "ms-cand", evalName: "try-plugin-locally", stimulus: "prepares-session", status: "error", passed: false, gradeScore: 0, outputTokens: 0, inputTokens: 0, totalTokens: 0 }), + makeTrial({ runId: "ms-cand", evalName: "update-remote-pins", stimulus: "updates-pins", passed: true, gradeScore: 1, outputTokens: 300, inputTokens: 1500, totalTokens: 1800 }), + makeFooter("ms-cand", true), + ]); + const multiSkillComparison = compareSummaries(multiSkillBase, multiSkillCand); + const verdicts = summarizeSkillVerdicts(multiSkillComparison); + assert.deepEqual(verdicts.map((v) => [v.skillName, v.verdict]), [ + ["release-plugins", "๐Ÿ›‘ No ship"], + ["try-plugin-locally", "โš ๏ธ Re-run"], + ["pr-description", "โœ… Ship"], + ["update-remote-pins", "โœ… Ship"], + ]); + const multiSkillMarkdown = renderReport(multiSkillCand, "cand/results.jsonl", { + baselineSummary: multiSkillBase, + baselinePath: "base/results.jsonl", + }); + assert.match(multiSkillMarkdown, /## Evals: ๐Ÿ›‘ No ship/); + const hrIndex = multiSkillMarkdown.indexOf("\n---\n"); + assert.ok(hrIndex > 0, "expected --- separator between actionable and passing sections"); + assert.ok(multiSkillMarkdown.indexOf("๐Ÿ›‘ release-plugins") < hrIndex); + assert.ok(multiSkillMarkdown.indexOf("โš ๏ธ try-plugin-locally") < hrIndex); + assert.ok(multiSkillMarkdown.indexOf("โœ… pr-description") > hrIndex); + assert.ok(multiSkillMarkdown.indexOf("๐Ÿ“‹ Unchanged evals & run configuration") > hrIndex); + assert.ok(multiSkillMarkdown.indexOf("๐Ÿ“Š Telemetry, token deltas & run evidence") > hrIndex); + assert.match(multiSkillMarkdown, /\| Outcome \| failed \| failed \(๐Ÿ›‘ No ship\) \|/); }); prDescriptionTest("eval-report CLI compares the shared pr-description baseline and candidate fixtures without writing GITHUB_STEP_SUMMARY", async (t) => { diff --git a/docs/checks-and-evals.md b/docs/checks-and-evals.md index 0a54480..bf89005 100644 --- a/docs/checks-and-evals.md +++ b/docs/checks-and-evals.md @@ -308,9 +308,12 @@ matching `evals//skills//eval.yaml` changes: 3. **Post sticky PR comment (`report`):** run `github/evals-action` with `command: report`, `report-id: evals`, and `baseline-report-id: evals-baseline` in a separate `pull-requests: write` - job to publish one sticky PR comment with the **Baseline vs. Candidate - comparison** table (Outcome, Passed/Failed, Score delta, Duration delta, and - Token `in/out` + `%` delta) alongside per-eval results. + job to publish one sticky PR comment with a top-level **Ship / No-ship / + Re-run** verdict summary table, collapsible per-skill `
` sections + (ordering actionable `๐Ÿ›‘ No ship` and `โš ๏ธ Re-run` sections above `---`), and + collapsible **Baseline vs. Candidate** telemetry (`Outcome`, `Passed/Failed`, + `Score delta`, `Turns delta`, `Duration delta`, and `Token in/out + %` + delta). > **Action version note:** `baseline-report-id` (`evals-baseline`) is provided > by [github/evals-action#34](https://github.com/github/evals-action/pull/34) diff --git a/docs/pr-comment-verdict-mockups.md b/docs/pr-comment-verdict-mockups.md deleted file mode 100644 index e49e5cb..0000000 --- a/docs/pr-comment-verdict-mockups.md +++ /dev/null @@ -1,84 +0,0 @@ -## Evals: ๐Ÿ›‘ No ship - -| Skill | Verdict | Why | -| --- | --- | --- | -| **release-plugins** | ๐Ÿ›‘ **No ship** | 1 scenario broke, 1 fixed (`4/5` โ†’ `4/5` passing) | -| **try-plugin-locally** | โš ๏ธ **Re-run** | 1 scenario errored (`execution-error` ยท `2/2` โ†’ `1/2` passing) | -| **pr-description** | โœ… **Ship** | 1 scenario fixed (`1/2` โ†’ `2/2` passing) ยท `-31.5%` output tokens | - -
-๐Ÿ›‘ release-plugins โ€” 1 broke, 1 fixed (4/5 โ†’ 4/5 passing) - -| | Scenario | Model | Score | Output Tokens | What happened | Evidence | -| --- | --- | --- | --- | ---: | --- | --- | -| ๐Ÿ“‰ Broke | `preserves-unreleased-initial-version` | `claude-haiku-5.5` | 100.0% โ†’ 75.0% | 410 โ†’ 390 (-4.9%) | 4/4 โ†’ 3/4 checks passed (failed: `skill-invocation`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | -| ๐Ÿ“ˆ Fixed | `asks-for-unknown-release-base` | `claude-haiku-5.5` | 50.0% โ†’ 100.0% | 620 โ†’ 310 (-50.0%) | 2/4 โ†’ 4/4 checks passed (fixed: `output-matches`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | - -Pass rate (before โ†’ after): claude-haiku-5.5 4/5 โ†’ 4/5 - -
- -
-โš ๏ธ try-plugin-locally โ€” 1 errored (2/2 โ†’ 1/2 passing) - -| | Scenario | Model | Score | Output Tokens | What happened | Evidence | -| --- | --- | --- | --- | ---: | --- | --- | -| โš ๏ธ Error | `prepares-current-checkout-session` | `claude-haiku-5.5` | 100.0% โ†’ โ€” | 512 โ†’ โ€” | Trial execution error (`execution-error`); re-run workflow to verify | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | - -Pass rate (before โ†’ after): claude-haiku-5.5 2/2 โ†’ 1/2 (1 execution error) - -
- ---- - -
-โœ… pr-description โ€” 1 fixed (1/2 โ†’ 2/2 passing) - -| | Scenario | Model | Score | Output Tokens | What happened | Evidence | -| --- | --- | --- | --- | ---: | --- | --- | -| ๐Ÿ“ˆ Fixed | `drafts-structured-pr-body` | `claude-haiku-5.5` | 87.5% โ†’ 100.0% | 918 โ†’ 402 (-56.2%) | 3/4 โ†’ 4/4 checks passed (fixed: `program`) | [artifact](https://github.com/github/plugin-marketplace-starter/actions/runs/43829816109/artifacts/14208216970) | - -Pass rate (before โ†’ after): claude-haiku-5.5 1/2 โ†’ 2/2 - -
- -
-๐Ÿ“‹ Unchanged evals & run configuration โ€” 3 passing (7/7 scenarios) - -| Skill | Status | Scenarios (Before โ†’ After) | Details | -| --- | --- | --- | --- | -| `update-remote-pins` | โœ… Unchanged | 3/3 โ†’ 3/3 | All scenarios passing on `claude-haiku-5.5` | -| `add-plugin` | โœ… Unchanged | 2/2 โ†’ 2/2 | All scenarios passing on `claude-haiku-5.5` | -| `submit-plugin` | โœ… Unchanged | 2/2 โ†’ 2/2 | All scenarios passing on `claude-haiku-5.5` | - -Run config: 16 scenarios across 6 evals ยท claude-haiku-5.5 ยท 1 trial/scenario ยท 0 skipped ยท 0 retries - -
- -
-๐Ÿ“Š Telemetry, token deltas & run evidence โ€” 6 skills evaluated (16 scenarios) - -| Metric | Baseline | Candidate | Delta | -| --- | --- | --- | --- | -| Outcome | failed | failed (๐Ÿ›‘ No ship) | 2 fixed, 1 broke, 1 error | -| Scenarios Passed | 14 / 16 | 14 / 16 | 0 (+2 fixed, -2 broke/error) | -| Scenarios Failed / Errored | 2 / 16 | 2 / 16 | 0 | -| Skills Passing | 4 / 6 | 4 / 6 | 0 | -| Avg Score | 96.5% | 95.5% | -1.0% | -| Turns | 25 | 24 | -1 | -| Total Duration | 159.2s | 154.2s | -5.0s (-3.1%) | -| Total Tokens | 319,399 (312,539 in / 6,860 out) | 318,870 (312,840 in / 6,030 out) | -529 (-0.2%) | -| Output Tokens | 6,860 | 6,030 | -830 (-12.1%) | - -| Skill | Baseline | Candidate | Score Delta | Turns Delta | Duration Delta | Token Delta | Transition | -| --- | --- | --- | --- | ---: | ---: | --- | --- | -| `release-plugins` | failed (0.85) | failed (0.85) | 0.00 | 0 | -1.2s (-3.1%) | -310 (-0.4%; out: -330 / -18.2%) | regressed | -| `try-plugin-locally` | passed (1.00) | error (โ€”) | โ€” | -1 | +0.8s (+4.2%) | -24 (-0.1%; out: -44 / -5.2%) | error | -| `pr-description` | failed (0.94) | passed (1.00) | +0.06 | 0 | -4.4s (-11.4%) | -191 (-0.3%; out: -456 / -31.5%) | fixed | -| `update-remote-pins` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.3s (-1.2%) | -12 (0.0%; out: -4 / -0.3%) | unchanged | -| `add-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | +0.2s (+1.0%) | +8 (0.0%; out: +4 / +0.5%) | unchanged | -| `submit-plugin` | passed (1.00) | passed (1.00) | 0.00 | 0 | -0.1s (-0.5%) | 0 (0.0%) | unchanged | - -Evidence: Candidate artifact ยท Baseline artifact ยท Before 0b6bf97 โ†’ After bd250bf ยท Workflow run - -
From 3835217fb1e48cc6e75142aab0aadae944e2994a Mon Sep 17 00:00:00 2001 From: Jon Gallant <2163001+jongio@users.noreply.github.com> Date: Mon, 16 Nov 2026 12:35:52 -0800 Subject: [PATCH 07/11] fix(evals): guard trial-count mismatches, footer execution errors, and Outcome labels - Require matching per-case trial counts before reporting output-token percentage deltas in summarizeSkillVerdicts. - Include run-level summary.errors in the Re-run verdict check. - Derive candidate Outcome and transition labels from overallVerdict. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .github/scripts/eval-report.mjs | 30 +++++++++++++++++++--------- .github/scripts/eval-report.test.mjs | 13 ++++++++++++ 2 files changed, 34 insertions(+), 9 deletions(-) diff --git a/.github/scripts/eval-report.mjs b/.github/scripts/eval-report.mjs index 05a8218..0e3ba98 100644 --- a/.github/scripts/eval-report.mjs +++ b/.github/scripts/eval-report.mjs @@ -543,7 +543,10 @@ export function summarizeSkillVerdicts(comparison) { } const baseOut = item.baseline?.tokenUsage?.outputTokens; const candOut = item.candidate?.tokenUsage?.outputTokens; - if (typeof baseOut === "number" && typeof candOut === "number") { + const sameCaseTrials = Boolean( + item.baseline && item.candidate && (item.baseline.trials ?? 1) === (item.candidate.trials ?? 1), + ); + if (sameCaseTrials && typeof baseOut === "number" && typeof candOut === "number") { group.baseOutputTokens += baseOut; group.candOutputTokens += candOut; } else { @@ -718,15 +721,24 @@ export function renderReport(summary, path, { baselineSummary, baselinePath } = const comparison = compareSummaries(baselineSummary, summary); const skillVerdicts = summarizeSkillVerdicts(comparison); const hasNoShip = skillVerdicts.some((s) => s.verdict === "๐Ÿ›‘ No ship") || summary.failedCount > 0; - const hasRerun = skillVerdicts.some((s) => s.verdict === "โš ๏ธ Re-run") || summary.errorCount > 0; + const hasRerun = skillVerdicts.some((s) => s.verdict === "โš ๏ธ Re-run") + || summary.errorCount > 0 + || summary.errors.length > 0 + || baselineSummary.errors.length > 0; const overallVerdict = hasNoShip ? "๐Ÿ›‘ No ship" : hasRerun ? "โš ๏ธ Re-run" : "โœ… Ship"; - const baseOutcome = (baselineSummary.failedCount + baselineSummary.errorCount) === 0 ? "passed" : "failed"; - const candOutcome = (summary.failedCount + summary.errorCount) === 0 ? `passed (${overallVerdict})` : `failed (${overallVerdict})`; - const outcomeDelta = baseOutcome === "failed" && (summary.failedCount + summary.errorCount) === 0 - ? "fixed" - : baseOutcome === "passed" && (summary.failedCount + summary.errorCount) > 0 - ? "regressed" - : `${overallVerdict}`; + const baseClean = (baselineSummary.failedCount + baselineSummary.errorCount) === 0 && baselineSummary.errors.length === 0; + const candClean = (summary.failedCount + summary.errorCount) === 0 && summary.errors.length === 0; + const baseOutcome = baseClean ? "passed" : "failed"; + const candOutcome = overallVerdict === "โœ… Ship" + ? "passed (โœ… Ship)" + : overallVerdict === "๐Ÿ›‘ No ship" + ? "failed (๐Ÿ›‘ No ship)" + : `${candClean ? "passed" : "failed"} (โš ๏ธ Re-run)`; + const outcomeDelta = overallVerdict === "โœ… Ship" + ? (baseOutcome === "failed" ? "fixed" : "unchanged") + : overallVerdict === "๐Ÿ›‘ No ship" + ? (baseOutcome === "passed" || skillVerdicts.some((s) => s.broke > 0) ? "regressed" : "๐Ÿ›‘ No ship") + : "โš ๏ธ Re-run"; const defaultModel = summary.models?.[0] ?? baselineSummary.models?.[0] ?? "default"; const actionableSkills = skillVerdicts.filter((s) => s.verdict !== "โœ… Ship"); diff --git a/.github/scripts/eval-report.test.mjs b/.github/scripts/eval-report.test.mjs index 6b293f7..3204138 100644 --- a/.github/scripts/eval-report.test.mjs +++ b/.github/scripts/eval-report.test.mjs @@ -484,6 +484,19 @@ test("eval-report redacts raw grader evidence and error strings, requires comple assert.match(mismatchedTrialsReport, /\| Total Tokens \| 1,100 \(in: 900, out: 200\) \| 400 \(in: 300, out: 100\) \| โ€” \|/); assert.match(mismatchedTrialsReport, /\| Agent Wall Time \| 2\.2s \| 900ms \| โ€” \|/); assert.match(mismatchedTrialsReport, /\| Total Duration \| 2\.2s \| 900ms \| โ€” \|/); + assert.doesNotMatch(mismatchedTrialsReport, /output tokens/); + + // Footer execution error without trial failure still triggers Re-run and Outcome alignment + const footerErrCand = summarizeRun([ + makeTrial({ runId: "f-cand", stimulus: "repeat-case", passed: true, gradeScore: 1, durationMs: 900, inputTokens: 300, outputTokens: 100, totalTokens: 400 }), + makeFooter("f-cand", true), + ]); + const footerErrReport = renderReport(footerErrCand, "cand/results.jsonl", { + baselineSummary: singleTrialCand, + baselinePath: "base/results.jsonl", + }); + assert.match(footerErrReport, /## Evals: โš ๏ธ Re-run/); + assert.match(footerErrReport, /\| Outcome \| passed \| failed \(โš ๏ธ Re-run\) \| โš ๏ธ Re-run \|/); const extraCaseBase = summarizeRun([ makeTrial({ runId: "e-base", stimulus: "repeat-case", passed: true, gradeScore: 1, durationMs: 900, inputTokens: 300, outputTokens: 100, totalTokens: 400 }), From 6766cef4c541d8e8f332a57cd3c22fff12529966 Mon Sep 17 00:00:00 2001 From: Jon Gallant <2163001+jongio@users.noreply.github.com> Date: Mon, 16 Nov 2026 12:54:49 -0800 Subject: [PATCH 08/11] fix(evals): escape ampersands/backticks, preserve failedCount, and label missing/added cases Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .github/scripts/eval-report.mjs | 54 ++++++++++++++++++++++++--------- 1 file changed, 40 insertions(+), 14 deletions(-) diff --git a/.github/scripts/eval-report.mjs b/.github/scripts/eval-report.mjs index 0e3ba98..27671fd 100644 --- a/.github/scripts/eval-report.mjs +++ b/.github/scripts/eval-report.mjs @@ -9,7 +9,12 @@ const stripTerminalControls = (text) => stripVTControlCharacters(String(text ?? // eslint-disable-next-line no-control-regex .replace(/[\u0000-\u001f\u007f-\u009f\u202a-\u202e\u2066-\u2069]+/gu, " ") .trim(); -const escapeCell = (text) => stripTerminalControls(text).replaceAll("|", "\\|").replaceAll("<", "<").replaceAll(">", ">"); +const escapeCell = (text) => stripTerminalControls(text) + .replaceAll("&", "&") + .replaceAll("|", "\\|") + .replaceAll("`", "\\`") + .replaceAll("<", "<") + .replaceAll(">", ">"); const finiteNonNegative = (value) => typeof value === "number" && Number.isFinite(value) && value >= 0 ? value : undefined; const sanitizeIdentifier = (value, fallback = "check") => { const cleaned = String(value ?? fallback).replace(/[^A-Za-z0-9._:-]+/g, "-").replace(/^-+|-+$/g, ""); @@ -237,6 +242,7 @@ export function summarizeRun(records) { if (grade.passed) { group.passedCount++; } else { + group.failedCount = (group.failedCount ?? 0) + 1; group.allPassed = false; } group.scoreSum += grade.score; @@ -292,6 +298,7 @@ export function summarizeRun(records) { name: group.name, trials: group.trials, passedCount: group.passedCount, + failedCount: group.failedCount ?? 0, status, passed, score: !group.hasFatal && group.allScored ? group.scoreSum / group.trials : undefined, @@ -569,10 +576,12 @@ export function summarizeSkillVerdicts(comparison) { const severityOrder = { "๐Ÿ›‘ No ship": 0, "โš ๏ธ Re-run": 1, "โœ… Ship": 2 }; return [...bySkill.values()] .map((group) => { - const hasBehavioralFailure = group.cases.some((c) => c.candidate?.status === "BEHAVIORAL FAILURE"); + const hasBehavioralFailure = group.cases.some( + (c) => c.candidate?.status === "BEHAVIORAL FAILURE" || (c.candidate?.failedCount ?? 0) > 0, + ); const verdict = group.candTotal === 0 || (group.errored > 0 && !hasBehavioralFailure && group.broke === 0) ? "โš ๏ธ Re-run" - : group.candPassed === group.candTotal && group.broke === 0 && group.errored === 0 + : group.candPassed === group.candTotal && group.broke === 0 && group.errored === 0 && !hasBehavioralFailure ? "โœ… Ship" : "๐Ÿ›‘ No ship"; const reasons = []; @@ -613,9 +622,22 @@ export function summarizeSkillVerdicts(comparison) { function describeCaseChange(item) { const baseFailures = item.baseline?.failures ?? []; const candFailures = item.candidate?.failures ?? []; - if (item.candidate?.status === "ERROR / NOT EVALUATED") { + if (!item.candidate) { + return "Scenario not evaluated in candidate run"; + } + if (item.candidate.status === "ERROR / NOT EVALUATED") { return "Trial execution error; re-run workflow to verify"; } + if (item.transition === "Added") { + if (item.candidate.passed) { + return item.candidate.gradersTotal + ? `New scenario (${item.candidate.gradersPassed}/${item.candidate.gradersTotal} checks passed)` + : "New scenario passing"; + } + return candFailures.length + ? `New scenario failed (${candFailures.map((f) => `\`${escapeCell(f)}\``).join(", ")})` + : "New scenario failed"; + } const hasCounts = Boolean(item.baseline?.gradersTotal && item.candidate?.gradersTotal); const countPrefix = hasCounts ? `${item.baseline.gradersPassed}/${item.baseline.gradersTotal} โ†’ ${item.candidate.gradersPassed}/${item.candidate.gradersTotal} checks passed` @@ -654,18 +676,22 @@ function renderSkillDetailsBlock(skill, defaultModel) { "| | Scenario | Model | Score | Output Tokens | What happened |", "| --- | --- | --- | --- | ---: | --- |", ...skill.cases.map((item) => { - const badge = item.candidate?.status === "ERROR / NOT EVALUATED" - ? "โš ๏ธ Error" - : item.transition === "Fixed" - ? "๐Ÿ“ˆ Fixed" - : item.transition === "Regressed" - ? "๐Ÿ“‰ Broke" - : item.candidate?.passed - ? "โœ… Passed" - : "โŒ Failed"; + const badge = !item.candidate + ? "โš ๏ธ Missing" + : item.candidate.status === "ERROR / NOT EVALUATED" + ? "โš ๏ธ Error" + : item.transition === "Added" + ? (item.candidate.passed ? "๐Ÿ†• Added" : "โŒ Failed") + : item.transition === "Fixed" + ? "๐Ÿ“ˆ Fixed" + : item.transition === "Regressed" + ? "๐Ÿ“‰ Broke" + : item.candidate.passed + ? "โœ… Passed" + : "โŒ Failed"; const rawCaseName = item.candidate?.name ?? item.baseline?.name ?? item.key; const model = item.candidate?.models?.[0] ?? item.baseline?.models?.[0] ?? modelLabel; - const scoreText = `${formatScore(item.baseline?.score)} โ†’ ${item.candidate?.status === "ERROR / NOT EVALUATED" ? "โ€”" : formatScore(item.candidate?.score)}`; + const scoreText = `${formatScore(item.baseline?.score)} โ†’ ${!item.candidate || item.candidate.status === "ERROR / NOT EVALUATED" ? "โ€”" : formatScore(item.candidate.score)}`; const baseOut = item.baseline?.tokenUsage?.outputTokens; const candOut = item.candidate?.tokenUsage?.outputTokens; const outText = typeof baseOut === "number" && typeof candOut === "number" && item.candidate?.status !== "ERROR / NOT EVALUATED" From ab402b681193e774b836603c0fe93d9cbd314b5d Mon Sep 17 00:00:00 2001 From: Jon Gallant <2163001+jongio@users.noreply.github.com> Date: Mon, 16 Nov 2026 13:15:45 -0800 Subject: [PATCH 09/11] fix(evals): preserve candidate output tokens when baseline is missing and use per-skill models Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .github/scripts/eval-report.mjs | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/.github/scripts/eval-report.mjs b/.github/scripts/eval-report.mjs index 27671fd..48d1ae9 100644 --- a/.github/scripts/eval-report.mjs +++ b/.github/scripts/eval-report.mjs @@ -525,6 +525,7 @@ export function summarizeSkillVerdicts(comparison) { group = { skillName, cases: [], + models: new Set(), fixed: 0, broke: 0, errored: 0, @@ -540,6 +541,9 @@ export function summarizeSkillVerdicts(comparison) { bySkill.set(skillName, group); } group.cases.push(item); + for (const model of [...(item.candidate?.models ?? []), ...(item.baseline?.models ?? [])]) { + if (model) group.models.add(model); + } if (item.baseline) { group.baseTotal++; if (item.baseline.passed) group.basePassed++; @@ -611,6 +615,7 @@ export function summarizeSkillVerdicts(comparison) { if (summaryCounts.length === 0) summaryCounts.push(`${group.candPassed}/${group.candTotal} passing`); return { ...group, + models: [...group.models], verdict, summaryLabel: `${summaryCounts.join(", ")} (${group.basePassed}/${group.baseTotal} โ†’ ${group.candPassed}/${group.candTotal} passing)`, why: reasons.join(" ยท "), @@ -668,7 +673,7 @@ function describeCaseChange(item) { function renderSkillDetailsBlock(skill, defaultModel) { const verdictEmoji = skill.verdict.split(" ")[0]; - const modelLabel = defaultModel || "default"; + const skillModelLabel = skill.models?.length ? skill.models.join(", ") : (defaultModel || "default"); return [ "
", `${verdictEmoji} ${escapeCell(skill.skillName)} โ€” ${skill.summaryLabel}`, @@ -690,19 +695,23 @@ function renderSkillDetailsBlock(skill, defaultModel) { ? "โœ… Passed" : "โŒ Failed"; const rawCaseName = item.candidate?.name ?? item.baseline?.name ?? item.key; - const model = item.candidate?.models?.[0] ?? item.baseline?.models?.[0] ?? modelLabel; + const caseModels = item.candidate?.models?.length ? item.candidate.models : item.baseline?.models; + const model = caseModels?.length ? caseModels.join(", ") : skillModelLabel; const scoreText = `${formatScore(item.baseline?.score)} โ†’ ${!item.candidate || item.candidate.status === "ERROR / NOT EVALUATED" ? "โ€”" : formatScore(item.candidate.score)}`; const baseOut = item.baseline?.tokenUsage?.outputTokens; const candOut = item.candidate?.tokenUsage?.outputTokens; - const outText = typeof baseOut === "number" && typeof candOut === "number" && item.candidate?.status !== "ERROR / NOT EVALUATED" - ? `${baseOut.toLocaleString("en-US")} โ†’ ${candOut.toLocaleString("en-US")} (${formatTokenDelta(item.outputTokenDelta, item.outputTokenDeltaPct)})` - : typeof baseOut === "number" - ? `${baseOut.toLocaleString("en-US")} โ†’ โ€”` - : "โ€”"; + const candHasOut = typeof candOut === "number" && item.candidate && item.candidate.status !== "ERROR / NOT EVALUATED"; + const outText = typeof baseOut === "number" && candHasOut + ? `${baseOut.toLocaleString("en-US")} โ†’ ${candOut.toLocaleString("en-US")}${typeof item.outputTokenDelta === "number" ? ` (${formatTokenDelta(item.outputTokenDelta, item.outputTokenDeltaPct)})` : ""}` + : candHasOut + ? `โ€” โ†’ ${candOut.toLocaleString("en-US")}` + : typeof baseOut === "number" + ? `${baseOut.toLocaleString("en-US")} โ†’ โ€”` + : "โ€”"; return `| ${badge} | \`${escapeCell(rawCaseName)}\` | \`${escapeCell(model)}\` | ${scoreText} | ${outText} | ${describeCaseChange(item)} |`; }), "", - `Pass rate (before โ†’ after): ${escapeCell(modelLabel)} ${skill.basePassed}/${skill.baseTotal} โ†’ ${skill.candPassed}/${skill.candTotal}`, + `Pass rate (before โ†’ after): ${escapeCell(skillModelLabel)} ${skill.basePassed}/${skill.baseTotal} โ†’ ${skill.candPassed}/${skill.candTotal}`, "", "
", "", @@ -803,9 +812,10 @@ export function renderReport(summary, path, { baselineSummary, baselinePath } = "", "| Skill | Status | Scenarios (Before โ†’ After) | Details |", "| --- | --- | --- | --- |", - ...unchangedPassingSkills.map((s) => - `| \`${escapeCell(s.skillName)}\` | โœ… Unchanged | ${s.basePassed}/${s.baseTotal} โ†’ ${s.candPassed}/${s.candTotal} | All scenarios passing on \`${escapeCell(defaultModel)}\` |`, - ), + ...unchangedPassingSkills.map((s) => { + const skillModel = s.models?.length ? s.models.join(", ") : defaultModel; + return `| \`${escapeCell(s.skillName)}\` | โœ… Unchanged | ${s.basePassed}/${s.baseTotal} โ†’ ${s.candPassed}/${s.candTotal} | All scenarios passing on \`${escapeCell(skillModel)}\` |`; + }), "", `Run config: ${comparison.cases.length} scenarios across ${skillVerdicts.length} ${skillVerdicts.length === 1 ? "eval" : "evals"} ยท ${escapeCell(defaultModel)}`, "", From e6130c1c9161a14903907996fc5b00b30e883906 Mon Sep 17 00:00:00 2001 From: Jon Gallant <2163001+jongio@users.noreply.github.com> Date: Mon, 16 Nov 2026 13:37:39 -0800 Subject: [PATCH 10/11] refactor(evals): trim unused case-transition and multi-model branches Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .github/scripts/eval-report.mjs | 79 ++++++++++----------------------- 1 file changed, 24 insertions(+), 55 deletions(-) diff --git a/.github/scripts/eval-report.mjs b/.github/scripts/eval-report.mjs index 48d1ae9..98d4342 100644 --- a/.github/scripts/eval-report.mjs +++ b/.github/scripts/eval-report.mjs @@ -242,7 +242,6 @@ export function summarizeRun(records) { if (grade.passed) { group.passedCount++; } else { - group.failedCount = (group.failedCount ?? 0) + 1; group.allPassed = false; } group.scoreSum += grade.score; @@ -298,7 +297,6 @@ export function summarizeRun(records) { name: group.name, trials: group.trials, passedCount: group.passedCount, - failedCount: group.failedCount ?? 0, status, passed, score: !group.hasFatal && group.allScored ? group.scoreSum / group.trials : undefined, @@ -525,7 +523,6 @@ export function summarizeSkillVerdicts(comparison) { group = { skillName, cases: [], - models: new Set(), fixed: 0, broke: 0, errored: 0, @@ -541,9 +538,6 @@ export function summarizeSkillVerdicts(comparison) { bySkill.set(skillName, group); } group.cases.push(item); - for (const model of [...(item.candidate?.models ?? []), ...(item.baseline?.models ?? [])]) { - if (model) group.models.add(model); - } if (item.baseline) { group.baseTotal++; if (item.baseline.passed) group.basePassed++; @@ -580,12 +574,10 @@ export function summarizeSkillVerdicts(comparison) { const severityOrder = { "๐Ÿ›‘ No ship": 0, "โš ๏ธ Re-run": 1, "โœ… Ship": 2 }; return [...bySkill.values()] .map((group) => { - const hasBehavioralFailure = group.cases.some( - (c) => c.candidate?.status === "BEHAVIORAL FAILURE" || (c.candidate?.failedCount ?? 0) > 0, - ); + const hasBehavioralFailure = group.cases.some((c) => c.candidate?.status === "BEHAVIORAL FAILURE"); const verdict = group.candTotal === 0 || (group.errored > 0 && !hasBehavioralFailure && group.broke === 0) ? "โš ๏ธ Re-run" - : group.candPassed === group.candTotal && group.broke === 0 && group.errored === 0 && !hasBehavioralFailure + : group.candPassed === group.candTotal && group.broke === 0 && group.errored === 0 ? "โœ… Ship" : "๐Ÿ›‘ No ship"; const reasons = []; @@ -615,7 +607,6 @@ export function summarizeSkillVerdicts(comparison) { if (summaryCounts.length === 0) summaryCounts.push(`${group.candPassed}/${group.candTotal} passing`); return { ...group, - models: [...group.models], verdict, summaryLabel: `${summaryCounts.join(", ")} (${group.basePassed}/${group.baseTotal} โ†’ ${group.candPassed}/${group.candTotal} passing)`, why: reasons.join(" ยท "), @@ -627,22 +618,9 @@ export function summarizeSkillVerdicts(comparison) { function describeCaseChange(item) { const baseFailures = item.baseline?.failures ?? []; const candFailures = item.candidate?.failures ?? []; - if (!item.candidate) { - return "Scenario not evaluated in candidate run"; - } - if (item.candidate.status === "ERROR / NOT EVALUATED") { + if (item.candidate?.status === "ERROR / NOT EVALUATED") { return "Trial execution error; re-run workflow to verify"; } - if (item.transition === "Added") { - if (item.candidate.passed) { - return item.candidate.gradersTotal - ? `New scenario (${item.candidate.gradersPassed}/${item.candidate.gradersTotal} checks passed)` - : "New scenario passing"; - } - return candFailures.length - ? `New scenario failed (${candFailures.map((f) => `\`${escapeCell(f)}\``).join(", ")})` - : "New scenario failed"; - } const hasCounts = Boolean(item.baseline?.gradersTotal && item.candidate?.gradersTotal); const countPrefix = hasCounts ? `${item.baseline.gradersPassed}/${item.baseline.gradersTotal} โ†’ ${item.candidate.gradersPassed}/${item.candidate.gradersTotal} checks passed` @@ -673,7 +651,7 @@ function describeCaseChange(item) { function renderSkillDetailsBlock(skill, defaultModel) { const verdictEmoji = skill.verdict.split(" ")[0]; - const skillModelLabel = skill.models?.length ? skill.models.join(", ") : (defaultModel || "default"); + const modelLabel = defaultModel || "default"; return [ "
", `${verdictEmoji} ${escapeCell(skill.skillName)} โ€” ${skill.summaryLabel}`, @@ -681,37 +659,29 @@ function renderSkillDetailsBlock(skill, defaultModel) { "| | Scenario | Model | Score | Output Tokens | What happened |", "| --- | --- | --- | --- | ---: | --- |", ...skill.cases.map((item) => { - const badge = !item.candidate - ? "โš ๏ธ Missing" - : item.candidate.status === "ERROR / NOT EVALUATED" - ? "โš ๏ธ Error" - : item.transition === "Added" - ? (item.candidate.passed ? "๐Ÿ†• Added" : "โŒ Failed") - : item.transition === "Fixed" - ? "๐Ÿ“ˆ Fixed" - : item.transition === "Regressed" - ? "๐Ÿ“‰ Broke" - : item.candidate.passed - ? "โœ… Passed" - : "โŒ Failed"; + const badge = item.candidate?.status === "ERROR / NOT EVALUATED" + ? "โš ๏ธ Error" + : item.transition === "Fixed" + ? "๐Ÿ“ˆ Fixed" + : item.transition === "Regressed" + ? "๐Ÿ“‰ Broke" + : item.candidate?.passed + ? "โœ… Passed" + : "โŒ Failed"; const rawCaseName = item.candidate?.name ?? item.baseline?.name ?? item.key; - const caseModels = item.candidate?.models?.length ? item.candidate.models : item.baseline?.models; - const model = caseModels?.length ? caseModels.join(", ") : skillModelLabel; - const scoreText = `${formatScore(item.baseline?.score)} โ†’ ${!item.candidate || item.candidate.status === "ERROR / NOT EVALUATED" ? "โ€”" : formatScore(item.candidate.score)}`; + const model = item.candidate?.models?.[0] ?? item.baseline?.models?.[0] ?? modelLabel; + const scoreText = `${formatScore(item.baseline?.score)} โ†’ ${item.candidate?.status === "ERROR / NOT EVALUATED" ? "โ€”" : formatScore(item.candidate?.score)}`; const baseOut = item.baseline?.tokenUsage?.outputTokens; const candOut = item.candidate?.tokenUsage?.outputTokens; - const candHasOut = typeof candOut === "number" && item.candidate && item.candidate.status !== "ERROR / NOT EVALUATED"; - const outText = typeof baseOut === "number" && candHasOut - ? `${baseOut.toLocaleString("en-US")} โ†’ ${candOut.toLocaleString("en-US")}${typeof item.outputTokenDelta === "number" ? ` (${formatTokenDelta(item.outputTokenDelta, item.outputTokenDeltaPct)})` : ""}` - : candHasOut - ? `โ€” โ†’ ${candOut.toLocaleString("en-US")}` - : typeof baseOut === "number" - ? `${baseOut.toLocaleString("en-US")} โ†’ โ€”` - : "โ€”"; + const outText = typeof baseOut === "number" && typeof candOut === "number" && item.candidate?.status !== "ERROR / NOT EVALUATED" + ? `${baseOut.toLocaleString("en-US")} โ†’ ${candOut.toLocaleString("en-US")} (${formatTokenDelta(item.outputTokenDelta, item.outputTokenDeltaPct)})` + : typeof baseOut === "number" + ? `${baseOut.toLocaleString("en-US")} โ†’ โ€”` + : "โ€”"; return `| ${badge} | \`${escapeCell(rawCaseName)}\` | \`${escapeCell(model)}\` | ${scoreText} | ${outText} | ${describeCaseChange(item)} |`; }), "", - `Pass rate (before โ†’ after): ${escapeCell(skillModelLabel)} ${skill.basePassed}/${skill.baseTotal} โ†’ ${skill.candPassed}/${skill.candTotal}`, + `Pass rate (before โ†’ after): ${escapeCell(modelLabel)} ${skill.basePassed}/${skill.baseTotal} โ†’ ${skill.candPassed}/${skill.candTotal}`, "", "
", "", @@ -812,10 +782,9 @@ export function renderReport(summary, path, { baselineSummary, baselinePath } = "", "| Skill | Status | Scenarios (Before โ†’ After) | Details |", "| --- | --- | --- | --- |", - ...unchangedPassingSkills.map((s) => { - const skillModel = s.models?.length ? s.models.join(", ") : defaultModel; - return `| \`${escapeCell(s.skillName)}\` | โœ… Unchanged | ${s.basePassed}/${s.baseTotal} โ†’ ${s.candPassed}/${s.candTotal} | All scenarios passing on \`${escapeCell(skillModel)}\` |`; - }), + ...unchangedPassingSkills.map((s) => + `| \`${escapeCell(s.skillName)}\` | โœ… Unchanged | ${s.basePassed}/${s.baseTotal} โ†’ ${s.candPassed}/${s.candTotal} | All scenarios passing on \`${escapeCell(defaultModel)}\` |`, + ), "", `Run config: ${comparison.cases.length} scenarios across ${skillVerdicts.length} ${skillVerdicts.length === 1 ? "eval" : "evals"} ยท ${escapeCell(defaultModel)}`, "", From 4295c26e5fd5ccfe7ced109db4c8ed030186c2e5 Mon Sep 17 00:00:00 2001 From: Jon Gallant <2163001+jongio@users.noreply.github.com> Date: Mon, 16 Nov 2026 14:18:24 -0800 Subject: [PATCH 11/11] fix(evals): encode backticks as `, preserve unchanged skill grouping on token noise, and omit undefined delta parens Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .github/scripts/eval-report.mjs | 13 +++++++++---- docs/checks-and-evals.md | 16 ++++++++-------- 2 files changed, 17 insertions(+), 12 deletions(-) diff --git a/.github/scripts/eval-report.mjs b/.github/scripts/eval-report.mjs index 98d4342..1933759 100644 --- a/.github/scripts/eval-report.mjs +++ b/.github/scripts/eval-report.mjs @@ -12,7 +12,7 @@ const stripTerminalControls = (text) => stripVTControlCharacters(String(text ?? const escapeCell = (text) => stripTerminalControls(text) .replaceAll("&", "&") .replaceAll("|", "\\|") - .replaceAll("`", "\\`") + .replaceAll("`", "`") .replaceAll("<", "<") .replaceAll(">", ">"); const finiteNonNegative = (value) => typeof value === "number" && Number.isFinite(value) && value >= 0 ? value : undefined; @@ -566,7 +566,12 @@ export function summarizeSkillVerdicts(comparison) { } else if (item.transition === "Regressed") { group.broke++; group.changed = true; - } else if (item.transition !== "Unchanged") { + } else if ( + item.transition === "Improved score" + || item.transition === "Added" + || item.transition === "Removed" + || Math.abs(item.outputTokenDeltaPct ?? 0) >= 10 + ) { group.changed = true; } } @@ -674,11 +679,11 @@ function renderSkillDetailsBlock(skill, defaultModel) { const baseOut = item.baseline?.tokenUsage?.outputTokens; const candOut = item.candidate?.tokenUsage?.outputTokens; const outText = typeof baseOut === "number" && typeof candOut === "number" && item.candidate?.status !== "ERROR / NOT EVALUATED" - ? `${baseOut.toLocaleString("en-US")} โ†’ ${candOut.toLocaleString("en-US")} (${formatTokenDelta(item.outputTokenDelta, item.outputTokenDeltaPct)})` + ? `${baseOut.toLocaleString("en-US")} โ†’ ${candOut.toLocaleString("en-US")}${typeof item.outputTokenDelta === "number" ? ` (${formatTokenDelta(item.outputTokenDelta, item.outputTokenDeltaPct)})` : ""}` : typeof baseOut === "number" ? `${baseOut.toLocaleString("en-US")} โ†’ โ€”` : "โ€”"; - return `| ${badge} | \`${escapeCell(rawCaseName)}\` | \`${escapeCell(model)}\` | ${scoreText} | ${outText} | ${describeCaseChange(item)} |`; + return `| ${badge} | ${escapeCell(rawCaseName)} | ${escapeCell(model)} | ${scoreText} | ${outText} | ${describeCaseChange(item)} |`; }), "", `Pass rate (before โ†’ after): ${escapeCell(modelLabel)} ${skill.basePassed}/${skill.baseTotal} โ†’ ${skill.candPassed}/${skill.candTotal}`, diff --git a/docs/checks-and-evals.md b/docs/checks-and-evals.md index bf89005..a512d63 100644 --- a/docs/checks-and-evals.md +++ b/docs/checks-and-evals.md @@ -309,17 +309,17 @@ matching `evals//skills//eval.yaml` changes: `command: report`, `report-id: evals`, and `baseline-report-id: evals-baseline` in a separate `pull-requests: write` job to publish one sticky PR comment with a top-level **Ship / No-ship / - Re-run** verdict summary table, collapsible per-skill `
` sections - (ordering actionable `๐Ÿ›‘ No ship` and `โš ๏ธ Re-run` sections above `---`), and - collapsible **Baseline vs. Candidate** telemetry (`Outcome`, `Passed/Failed`, - `Score delta`, `Turns delta`, `Duration delta`, and `Token in/out + %` - delta). + Re-run** verdict summary table and a collapsible **Telemetry, token deltas & + run evidence** `
` section (`Outcome`, `Passed/Failed`, `Score + delta`, `Turns delta`, `Duration delta`, and `Token in/out + %` delta). > **Action version note:** `baseline-report-id` (`evals-baseline`) is provided > by [github/evals-action#34](https://github.com/github/evals-action/pull/34) -> (verified commit `3c551330898c1467b8ddf9ef7c7f6e6c6af489d6`). Pin -> `github/evals-action@3c551330898c1467b8ddf9ef7c7f6e6c6af489d6` until `#34` -> is merged and published to `github/evals-action@v0`. +> (`3c551330898c1467b8ddf9ef7c7f6e6c6af489d6`), and the top-level **Ship / +> No-ship** verdict table layout is added in +> [github/evals-action#35](https://github.com/github/evals-action/pull/35). Pin +> `github/evals-action` to that commit until published to +> `github/evals-action@v0`. ```yaml name: Skill eval comparison