Skip to content

feat(evals): add Ship / No-ship verdict summary and collapsible comparison sections - #23

Merged
jongio merged 11 commits into
mainfrom
feat/ship-verdict-summary
Nov 16, 2026
Merged

jongio merged 11 commits into
mainfrom
feat/ship-verdict-summary

Conversation

@jongio

@jongio jongio commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Implements the Ship / No-ship / Re-run verdict summary table and collapsible <details> layout in .github/scripts/eval-report.mjs (and updates docs/checks-and-evals.md to match the companion github/evals-action PR):

  • Top-level Verdict Table: Leads comparison reports with ## Evals: ✅ Ship / 🛑 No ship / ⚠️ Re-run and a severity-ordered | Skill | Verdict | Why | table derived strictly from Vally results.jsonl.
  • Actionable vs. Informational <details> layout: Places 🛑 No ship and ⚠️ Re-run per-skill scenario <details> above ---, and places ✅ Ship, 📋 Unchanged evals & run configuration, and 📊 Telemetry, token deltas & run evidence below ---.
  • Consistent Aggregate Outcome & Totals: Aligns the Outcome row in Baseline vs. Candidate Comparison with the headline verdict and sums all evaluated scenarios/skills.

Changes

  • .github/scripts/eval-report.mjs: Added summarizeSkillVerdicts, describeCaseChange, renderSkillDetailsBlock, and verdict-first collapsible layout in renderReport.
  • .github/scripts/eval-report.test.mjs: Added unit tests verifying single-skill ✅ Ship and multi-skill mixed-verdict ordering around ---.
  • docs/checks-and-evals.md: Updated PR comparison documentation.
  • Removed temporary docs/pr-comment-verdict-mockups.md RFC file (rendered preview remains in #issuecomment-6924338386).

Verification

  • node --test --experimental-test-coverage .github/scripts/eval-report.test.mjs .github/scripts/pr-description.test.mjs: passed (7/7 tests, 96.23% line / 84.16% branch coverage)

Add three interactive PR comment mockups combining @rwoll's Ship / No-ship
verdict table and collapsible <details> structure with Vally 0.21.0
score, turn, wall-time, and output-token comparison telemetry.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio

jongio commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Evals: 🛑 No ship

Skill Verdict Why
release-plugins 🛑 No ship 1 scenario broke, 1 fixed (4/5 → 4/5 passing)
try-plugin-locally ⚠️ Re-run 1 scenario errored (execution-error · 2/2 → 1/2 passing)
pr-description ✅ Ship 1 scenario fixed (1/2 → 2/2 passing) · -31.5% output tokens
🛑 release-plugins — 1 broke, 1 fixed (4/5 → 4/5 passing)
Scenario Model Score Output Tokens What happened Evidence
📉 Broke preserves-unreleased-initial-version claude-haiku-5.5 100.0% → 75.0% 410 → 390 (-4.9%) 4/4 → 3/4 checks passed (failed: skill-invocation) artifact
📈 Fixed asks-for-unknown-release-base claude-haiku-5.5 50.0% → 100.0% 620 → 310 (-50.0%) 2/4 → 4/4 checks passed (fixed: output-matches) artifact

Pass rate (before → after): claude-haiku-5.5 4/5 → 4/5

⚠️ try-plugin-locally — 1 errored (2/2 → 1/2 passing)
Scenario Model Score Output Tokens What happened Evidence
⚠️ Error prepares-current-checkout-session claude-haiku-5.5 100.0% → — 512 → — Trial execution error (execution-error); re-run workflow to verify artifact

Pass rate (before → after): claude-haiku-5.5 2/2 → 1/2 (1 execution error)


✅ pr-description — 1 fixed (1/2 → 2/2 passing)
Scenario Model Score Output Tokens What happened Evidence
📈 Fixed drafts-structured-pr-body claude-haiku-5.5 87.5% → 100.0% 918 → 402 (-56.2%) 3/4 → 4/4 checks passed (fixed: program) artifact

Pass rate (before → after): claude-haiku-5.5 1/2 → 2/2

📋 Unchanged evals & run configuration — 3 passing (7/7 scenarios)
Skill Status Scenarios (Before → After) Details
update-remote-pins ✅ Unchanged 3/3 → 3/3 All scenarios passing on claude-haiku-5.5
add-plugin ✅ Unchanged 2/2 → 2/2 All scenarios passing on claude-haiku-5.5
submit-plugin ✅ Unchanged 2/2 → 2/2 All scenarios passing on claude-haiku-5.5

Run config: 16 scenarios across 6 evals · claude-haiku-5.5 · 1 trial/scenario · 0 skipped · 0 retries

📊 Telemetry, token deltas & run evidence — 6 skills evaluated (16 scenarios)
Metric Baseline Candidate Delta
Outcome failed failed (🛑 No ship) 2 fixed, 1 broke, 1 error
Scenarios Passed 14 / 16 14 / 16 0 (+2 fixed, -2 broke/error)
Scenarios Failed / Errored 2 / 16 2 / 16 0
Skills Passing 4 / 6 4 / 6 0
Avg Score 96.5% 95.5% -1.0%
Turns 25 24 -1
Total Duration 159.2s 154.2s -5.0s (-3.1%)
Total Tokens 319,399 (312,539 in / 6,860 out) 318,870 (312,840 in / 6,030 out) -529 (-0.2%)
Output Tokens 6,860 6,030 -830 (-12.1%)
Skill Baseline Candidate Score Delta Turns Delta Duration Delta Token Delta Transition
release-plugins failed (0.85) failed (0.85) 0.00 0 -1.2s (-3.1%) -310 (-0.4%; out: -330 / -18.2%) regressed
try-plugin-locally passed (1.00) error (—) — -1 +0.8s (+4.2%) -24 (-0.1%; out: -44 / -5.2%) error
pr-description failed (0.94) passed (1.00) +0.06 0 -4.4s (-11.4%) -191 (-0.3%; out: -456 / -31.5%) fixed
update-remote-pins passed (1.00) passed (1.00) 0.00 0 -0.3s (-1.2%) -12 (0.0%; out: -4 / -0.3%) unchanged
add-plugin passed (1.00) passed (1.00) 0.00 0 +0.2s (+1.0%) +8 (0.0%; out: +4 / +0.5%) unchanged
submit-plugin passed (1.00) passed (1.00) 0.00 0 -0.1s (-0.5%) 0 (0.0%) unchanged

Evidence: Candidate artifact · Baseline artifact · Before 0b6bf97 → After bd250bf · Workflow run

jongio and others added 3 commits November 16, 2026 10:02
… table and consistent aside labels

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread docs/pr-comment-verdict-mockups.md Resolved
Comment thread docs/pr-comment-verdict-mockups.md Resolved
Comment thread docs/pr-comment-verdict-mockups.md Resolved
@rwoll

rwoll commented Nov 16, 2026

Copy link
Copy Markdown

Evals: 🛑 No ship

Skill Verdict Why
pr-description ✅ Ship 1 scenario fixed (1/2 → 2/2 passing) · -31.5% output tokens
release-plugins 🛑 No ship 1 scenario broke, 1 fixed (4/5 → 4/5 passing)
try-plugin-locally (not changed) ⚠️ Re-run 1 scenario broke, but this PR does not change this skill
🛑 release-plugins — 1 broke, 1 fixed (4/5 → 4/5 passing)
Scenario Model Score Output Tokens What happened Evidence
📉 Broke preserves-unreleased-initial-version claude-haiku-5.5 100.0% → 75.0% 410 → 390 (-4.9%) Check failed: skill-invocation (skill was not invoked) artifact
📈 Fixed asks-for-unknown-release-base claude-haiku-5.5 50.0% → 100.0% 620 → 310 (-50.0%) 2/4 → 4/4 checks passed (output-matches fixed) artifact

Pass rate (before → after): claude-haiku-5.5 4/5 → 4/5

⚠️ try-plugin-locally (not changed in PR) — 1 broke (2/2 → 1/2 passing)
Scenario Model Score Output Tokens What happened Evidence
📉 Broke prepares-current-checkout-session claude-haiku-5.5 100.0% → 75.0% 512 → 488 (-4.7%) Check failed: program (command omitted local plugin path) artifact

Pass rate (before → after): claude-haiku-5.5 2/2 → 1/2 · This PR does not modify try-plugin-locally; re-run to rule out single-trial variance.


✅ pr-description — 1 fixed (1/2 → 2/2 passing)
Scenario Model Score Output Tokens What happened Evidence
📈 Fixed drafts-structured-pr-body claude-haiku-5.5 87.5% → 100.0% 918 → 402 (-56.2%) 3/4 → 4/4 checks passed (fixed program check: includes ## Risks / Rollback, excludes #42 from ## Changes) artifact

Pass rate (before → after): claude-haiku-5.5 1/2 → 2/2

📋 Unchanged skills & suite coverage — 3 passing (7/7 scenarios), 6 without evals
Skill Status Scenarios (Before → After) Details
update-remote-pins ✅ Unchanged 3/3 → 3/3 All scenarios passing on claude-haiku-5.5
add-plugin ✅ Unchanged 2/2 → 2/2 All scenarios passing on claude-haiku-5.5
submit-plugin ✅ Unchanged 2/2 → 2/2 All scenarios passing on claude-haiku-5.5
eval/* (6 skills) ⚪ Not evaluated — No evals configured (compare, create, improve, review, run, setup)

Run config: 16 scenarios · claude-haiku-5.5 · 1 trial/scenario · 0 retries (single-trial flips on unmodified skills can reflect model variance)

📊 Telemetry, token deltas & run evidence — 6 skills evaluated (16 scenarios)
Metric Baseline Candidate Delta
Outcome failed passed fixed
Passed 1 2 +1
Failed 1 0 -1
Avg Score 93.8% 100.0% +6.3%
Turns 3 3 0
Agent Wall Time 17.5s 16.4s -1.1s (-6.6%)
Total Duration 38.5s 34.1s -4.4s (-11.4%)
Total Tokens 73,199 (71,752 in / 1,447 out) 73,008 (72,017 in / 991 out) -191 (-0.3%)
Output Tokens 1,447 991 -456 (-31.5%)
Skill Baseline Candidate Score Delta Turns Delta Duration Delta Token Delta Transition
pr-description failed (0.94) passed (1.00) +0.06 0 -4.4s (-11.4%) -191 (-0.3%; out: -456 / -31.5%) fixed
release-plugins failed (0.85) failed (0.85) 0.00 0 -1.2s (-3.1%) -310 (-0.4%; out: -330 / -18.2%) regressed
try-plugin-locally passed (1.00) failed (0.88) -0.12 0 +0.8s (+4.2%) -24 (-0.1%) regressed
update-remote-pins passed (1.00) passed (1.00) 0.00 0 -0.3s (-1.2%) -12 (0.0%) unchanged
add-plugin passed (1.00) passed (1.00) 0.00 0 +0.2s (+1.0%) +8 (0.0%) unchanged
submit-plugin passed (1.00) passed (1.00) 0.00 0 -0.1s (-0.5%) 0 (0.0%) unchanged

Evidence: Candidate artifact · Baseline artifact · Before 0b6bf97 → After bd250bf · Workflow run

jongio and others added 2 commits November 16, 2026 11:12
…mockup

- Order actionable No-ship / Re-run sections above --- and passing/telemetry sections below
- Keep all fields strictly derived from Vally results.jsonl (no git-diff or repo-scan heuristics)
- Reconcile 6-skill / 16-scenario aggregate totals and Outcome with the No-ship headline
- Replace # placeholder links with real workflow artifact URLs

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… report sections

- Add `summarizeSkillVerdicts` and collapsible `<details>` sections to
  `renderReport` in `.github/scripts/eval-report.mjs` when comparing a
  baseline and candidate run.
- Order `🛑 No ship` and `⚠️ Re-run` skill sections above `---`, and
  place `✅ Ship`, `📋 Unchanged evals & run configuration`, and
  `📊 Telemetry, token deltas & run evidence` below `---`.
- Align aggregate `Outcome` with the headline verdict, remove temporary
  RFC mockup doc, and add unit tests in `eval-report.test.mjs`.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio jongio changed the title docs(evals): RFC mockups for Ship / No-ship skill eval PR comments feat(evals): add Ship / No-ship verdict summary and collapsible comparison sections Nov 16, 2026
@jongio
jongio marked this pull request as ready for review November 16, 2026 20:14
@jongio
jongio requested a review from a team as a code owner November 16, 2026 20:14
Copilot AI adaptive review requested due to automatic review settings November 16, 2026 20:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The report can misstate token savings when trial counts differ and present successful outcomes despite execution errors, incomplete comparisons, or regressions.

3 open issues
What changed in this PR

Updates the local eval comparison reporter with a verdict-first layout, making actionable results easier to find while keeping detailed evidence available.

Changes:

  • Adds severity-ordered Ship / No ship / Re-run verdicts and per-skill explanations.
  • Groups scenario results and telemetry into collapsible sections, with actionable skills first.
  • Adds verdict and layout tests and updates comparison-report documentation.
File Description
docs/​checks-and-evals.md Describes the verdict summary and collapsible report layout.
.github/​scripts/​eval-report.test.mjs Adds single-skill and mixed-verdict rendering assertions.
.github/​scripts/​eval-report.mjs Adds skill verdicts, grader counts, collapsible sections, and an aggregate Outcome row.

Review effort: Balanced (adaptive)

Comment thread .github/scripts/eval-report.mjs Resolved
Comment thread .github/scripts/eval-report.mjs Resolved
Comment thread .github/scripts/eval-report.mjs Resolved
…d Outcome labels

- Require matching per-case trial counts before reporting output-token
  percentage deltas in summarizeSkillVerdicts.
- Include run-level summary.errors in the Re-run verdict check.
- Derive candidate Outcome and transition labels from overallVerdict.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio

jongio commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Review Disposition (5926918378, 5927283775, 5927679222, 5928388244)

  • Suppress token deltas when trial counts differ & omit (—) parens (concern:20a8bfd5cbc920a4, concern:267d8afed58d808a, reviews 5926918378, 5928388244): Fixed in 3835217 and 4295c26 (.github/scripts/eval-report.mjs, .github/scripts/eval-report.test.mjs).
  • Include execution errors in the Re-run verdict & derive Outcome labels from overallVerdict (concern:5897f81e0137b7f7, concern:471233ba35a1f174, review 5926918378): Fixed in 3835217 (.github/scripts/eval-report.mjs, .github/scripts/eval-report.test.mjs).
  • Encode backticks as &#96; and preserve 📋 Unchanged evals grouping on minor token noise (review 5928388244): Fixed in 4295c26 (.github/scripts/eval-report.mjs, docs/checks-and-evals.md).
  • Optional multi-trial/added-scenario/multi-model branches (concern:e58d7092141b8c49, concern:2a918f7f102f2c96, concern:b41fe07b5a49957b, reviews 5927283775, 5927679222, 5928388244): Intentionally omitted in e6130c1 to keep .github/scripts/eval-report.mjs minimal.

@jongio
jongio requested an adaptive review from Copilot November 16, 2026 20:37
Comment thread .github/scripts/eval-report.mjs Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The report mislabels missing or added scenarios and hides confirmed behavioral failures behind a re-run status, producing misleading results and inconsistent shipping recommendations.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Medium severity Preserve behavioral failures when aggregating execution errors

.github/​scripts/​eval-report.mjs:574

When a repeated scenario has both a behavioral failure and an execution error, summarizeRun collapses its status to ERROR / NOT EVALUATED. This check then gives the skill ⚠️ Re-run, hiding the confirmed failure and contradicting the 🛑 No ship headline derived from summary.failedCount. Preserve a per-case behavioral-failure flag or count during aggregation and consult it here so the skill also receives 🛑 No ship.

Medium severity Distinguish missing candidates and added scenarios in report details

.github/​scripts/​eval-report.mjs:676

A baseline-only scenario renders as ❌ Failed with an Unchanged explanation, although no candidate trial ran and the missing result triggers Re-run. Added passing scenarios are also described as unchanged. Handle missing candidates and Added transitions explicitly so the details distinguish missing coverage and new scenarios from behavioral failures and unchanged results.

3 resolved since last review

Review effort: Balanced (adaptive)

…bel missing/added cases

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The report hides known candidate output-token counts when baseline telemetry is missing and incorrectly labels per-skill results using the run’s first model.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Medium severity Preserve candidate token counts when baseline telemetry is missing

.github/​scripts/​eval-report.mjs:701

For an added scenario, or when only the baseline lacks telemetry, this fallback prints — even when the candidate has recorded output tokens. Preserve the known candidate value (for example, — → 432) and suppress only the unavailable delta.

Medium severity Use each skill's recorded models in report labels

.github/​scripts/​eval-report.mjs:768

summary.models is the run-wide union, so its first entry is not necessarily the model used by another skill. With alpha on model-a, beta on model-b, and gamma on model-c, the generated report says beta's unchanged scenarios passed on model-a and labels gamma's pass rate with model-a, despite retaining the correct models in their case rows. Derive these labels from each skill's recorded models (and display all applicable models) rather than reusing the first model of the run.

Review effort: Balanced (adaptive)

jongio and others added 2 commits November 16, 2026 13:15
… and use per-skill models

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread .github/scripts/eval-report.mjs Resolved

@wbreza wbreza left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Nice work designing a clear, verdict-first comparison layout (## Evals: ✅ Ship / 🛑 No ship / ⚠️ Re-run), separating actionable <details> sections above ---, and keeping all inputs grounded in Vally results.jsonl.

Leaving a Comment review with 5 Medium and 1 Low findings:

  • Medium (.github/scripts/eval-report.mjs:12-17) — Backtick escaping in escapeCell breaks GFM code spans: .replaceAll("", "\") does not escape backticks inside CommonMark/GFM single-backtick code spans (`...`), allowing untrusted stimulus/evalName/model values containing ` to break out of table code spans while rendering literal \ inside <summary><strong>...</strong></summary>.
  • Medium (.github/scripts/eval-report.mjs:569-571) — 📋 Unchanged evals & run configuration bypassed on ±1 token variance: Any non-zero token difference triggers "Reduced/Increased tokens at parity" (transition !== "Unchanged"), setting group.changed = true and ejecting unchanged passing skills into standalone <details> blocks in real multi-skill LLM runs.
  • Medium (.github/scripts/eval-report.mjs:577-582) — Multi-trial failure + execution error contradiction (reverted in e6130c1): When a repeated scenario has both a behavioral failure and an execution error, the headline reports 🛑 No ship (summary.failedCount > 0) while the skill itself is labeled ⚠️ Re-run. (Note: #issuecomment-6928610849 lists this as fixed in 6766cef, but e6130c1 removed group.failedCount.)
  • Medium (.github/scripts/eval-report.mjs:662-681) — Missing-candidate, added-scenario, and mismatched-trial row rendering (reverted in e6130c1): e6130c1 also reverted the Removed/Added/token-fallback branches from 6766cef and ab402b68 (still cited in #issuecomment-6928610849), causing baseline-only scenarios to render as ❌ Failed / Not scored / Unchanged, added scenarios to hide candidate output tokens and say Unchanged / Still failing, and mismatched-trial rows to print (—).
  • Medium (.github/scripts/eval-report.mjs:785-789) — Per-skill model misattribution (reverted in e6130c1): renderSkillDetailsBlock footers and Unchanged evals rows reuse summary.models[0] instead of each skill's own recorded model(s).
  • Low (docs/checks-and-evals.md:308-316) — Companion action pin alignment: Step 3 describes the Ship / No-ship sticky comment layout from github/evals-action#35 while the version note right below (and workflow pin) still points to github/evals-action#34 (3c55133).

Comment thread .github/scripts/eval-report.mjs Resolved
Comment thread .github/scripts/eval-report.mjs Resolved
Comment thread .github/scripts/eval-report.mjs Resolved
Comment thread .github/scripts/eval-report.mjs Resolved
Comment thread .github/scripts/eval-report.mjs Resolved
Comment thread docs/checks-and-evals.md Resolved
…ing on token noise, and omit undefined delta parens

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@wbreza wbreza left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Approved at 4295c26 — the follow-up changes cleanly resolve the blocking items from the previous pass:

  • Markdown/HTML escaping (escapeCell): Encoding ` as &#96; (after & → &amp;) and wrapping scenario/model cells in <code>...</code> prevents inline code-span breakout and literal backslash rendering.
  • Unchanged skill grouping: Gating group.changed on behavioral/score/scenario transitions or >= 10% output-token delta keeps parity skills with minor LLM token variance grouped under 📋 Unchanged evals & run configuration.
  • Delta formatting & docs: Mismatched-trial rows no longer emit a trailing (—), and docs/checks-and-evals.md now notes companion PR github/evals-action#35.

Optional / Low follow-ups (non-blocking):

  • Consider adding unit assertions in .github/scripts/eval-report.test.mjs covering &#96; escaping in escapeCell and < 10% output-token variance remaining grouped under Unchanged evals.
  • Note: single-trial (runs: 1), matched-scenario, single-model-per-run assumptions for per-skill detail labels are noted as intentional scope boundaries for this starter's Vally setup.

@jongio
jongio merged commit 46de5d0 into main Nov 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants