You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Implements the Ship / No-ship / Re-run verdict summary table and collapsible <details> layout in .github/scripts/eval-report.mjs (and updates docs/checks-and-evals.md to match the companion github/evals-action PR):
Top-level Verdict Table: Leads comparison reports with ## Evals: ✅ Ship / 🛑 No ship / ⚠️ Re-run and a severity-ordered | Skill | Verdict | Why | table derived strictly from Vally results.jsonl.
Actionable vs. Informational <details> layout: Places 🛑 No ship and ⚠️ Re-run per-skill scenario <details> above ---, and places ✅ Ship, 📋 Unchanged evals & run configuration, and 📊 Telemetry, token deltas & run evidence below ---.
Consistent Aggregate Outcome & Totals: Aligns the Outcome row in Baseline vs. Candidate Comparison with the headline verdict and sums all evaluated scenarios/skills.
Changes
.github/scripts/eval-report.mjs: Added summarizeSkillVerdicts, describeCaseChange, renderSkillDetailsBlock, and verdict-first collapsible layout in renderReport.
.github/scripts/eval-report.test.mjs: Added unit tests verifying single-skill ✅ Ship and multi-skill mixed-verdict ordering around ---.
…mockup
- Order actionable No-ship / Re-run sections above --- and passing/telemetry sections below
- Keep all fields strictly derived from Vally results.jsonl (no git-diff or repo-scan heuristics)
- Reconcile 6-skill / 16-scenario aggregate totals and Outcome with the No-ship headline
- Replace # placeholder links with real workflow artifact URLs
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… report sections
- Add `summarizeSkillVerdicts` and collapsible `<details>` sections to
`renderReport` in `.github/scripts/eval-report.mjs` when comparing a
baseline and candidate run.
- Order `🛑 No ship` and `⚠️ Re-run` skill sections above `---`, and
place `✅ Ship`, `📋 Unchanged evals & run configuration`, and
`📊 Telemetry, token deltas & run evidence` below `---`.
- Align aggregate `Outcome` with the headline verdict, remove temporary
RFC mockup doc, and add unit tests in `eval-report.test.mjs`.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
jongio
changed the title
docs(evals): RFC mockups for Ship / No-ship skill eval PR comments
feat(evals): add Ship / No-ship verdict summary and collapsible comparison sections
Nov 16, 2026
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The report can misstate token savings when trial counts differ and present successful outcomes despite execution errors, incomplete comparisons, or regressions.
Updates the local eval comparison reporter with a verdict-first layout, making actionable results easier to find while keeping detailed evidence available.
Changes:
Adds severity-ordered Ship / No ship / Re-run verdicts and per-skill explanations.
Groups scenario results and telemetry into collapsible sections, with actionable skills first.
Adds verdict and layout tests and updates comparison-report documentation.
File
Description
docs/checks-and-evals.md
Describes the verdict summary and collapsible report layout.
.github/scripts/eval-report.test.mjs
Adds single-skill and mixed-verdict rendering assertions.
.github/scripts/eval-report.mjs
Adds skill verdicts, grader counts, collapsible sections, and an aggregate Outcome row.
Suppress token deltas when trial counts differ & omit (—) parens (concern:20a8bfd5cbc920a4, concern:267d8afed58d808a, reviews 5926918378, 5928388244): Fixed in 3835217 and 4295c26 (.github/scripts/eval-report.mjs, .github/scripts/eval-report.test.mjs).
Include execution errors in the Re-run verdict & derive Outcome labels from overallVerdict (concern:5897f81e0137b7f7, concern:471233ba35a1f174, review 5926918378): Fixed in 3835217 (.github/scripts/eval-report.mjs, .github/scripts/eval-report.test.mjs).
Encode backticks as ` and preserve 📋 Unchanged evals grouping on minor token noise (review 5928388244): Fixed in 4295c26 (.github/scripts/eval-report.mjs, docs/checks-and-evals.md).
Optional multi-trial/added-scenario/multi-model branches (concern:e58d7092141b8c49, concern:2a918f7f102f2c96, concern:b41fe07b5a49957b, reviews 5927283775, 5927679222, 5928388244): Intentionally omitted in e6130c1 to keep .github/scripts/eval-report.mjs minimal.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The report mislabels missing or added scenarios and hides confirmed behavioral failures behind a re-run status, producing misleading results and inconsistent shipping recommendations.
No open issues 🎉
2 previously missed issues
In code that hasn't changed since last review
Preserve behavioral failures when aggregating execution errors
.github/scripts/eval-report.mjs:574
When a repeated scenario has both a behavioral failure and an execution error, summarizeRun collapses its status to ERROR / NOT EVALUATED. This check then gives the skill ⚠️ Re-run, hiding the confirmed failure and contradicting the 🛑 No ship headline derived from summary.failedCount. Preserve a per-case behavioral-failure flag or count during aggregation and consult it here so the skill also receives 🛑 No ship.
Distinguish missing candidates and added scenarios in report details
.github/scripts/eval-report.mjs:676
A baseline-only scenario renders as ❌ Failed with an Unchanged explanation, although no candidate trial ran and the missing result triggers Re-run. Added passing scenarios are also described as unchanged. Handle missing candidates and Added transitions explicitly so the details distinguish missing coverage and new scenarios from behavioral failures and unchanged results.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The report hides known candidate output-token counts when baseline telemetry is missing and incorrectly labels per-skill results using the run’s first model.
No open issues 🎉
2 previously missed issues
In code that hasn't changed since last review
Preserve candidate token counts when baseline telemetry is missing
.github/scripts/eval-report.mjs:701
For an added scenario, or when only the baseline lacks telemetry, this fallback prints — even when the candidate has recorded output tokens. Preserve the known candidate value (for example, — → 432) and suppress only the unavailable delta.
Use each skill's recorded models in report labels
.github/scripts/eval-report.mjs:768
summary.models is the run-wide union, so its first entry is not necessarily the model used by another skill. With alpha on model-a, beta on model-b, and gamma on model-c, the generated report says beta's unchanged scenarios passed on model-a and labels gamma's pass rate with model-a, despite retaining the correct models in their case rows. Derive these labels from each skill's recorded models (and display all applicable models) rather than reusing the first model of the run.
The reason will be displayed to describe this comment to others. Learn more.
Review Summary
Nice work designing a clear, verdict-first comparison layout (## Evals: ✅ Ship / 🛑 No ship / ⚠️ Re-run), separating actionable <details> sections above ---, and keeping all inputs grounded in Vally results.jsonl.
Leaving a Comment review with 5 Medium and 1 Low findings:
Medium (.github/scripts/eval-report.mjs:12-17) — Backtick escaping in escapeCell breaks GFM code spans:.replaceAll("", "\") does not escape backticks inside CommonMark/GFM single-backtick code spans (`...`), allowing untrusted stimulus/evalName/model values containing ` to break out of table code spans while rendering literal \ inside <summary><strong>...</strong></summary>.
Medium (.github/scripts/eval-report.mjs:569-571) — 📋 Unchanged evals & run configuration bypassed on ±1 token variance: Any non-zero token difference triggers "Reduced/Increased tokens at parity" (transition !== "Unchanged"), setting group.changed = true and ejecting unchanged passing skills into standalone <details> blocks in real multi-skill LLM runs.
Medium (.github/scripts/eval-report.mjs:577-582) — Multi-trial failure + execution error contradiction (reverted in e6130c1): When a repeated scenario has both a behavioral failure and an execution error, the headline reports 🛑 No ship (summary.failedCount > 0) while the skill itself is labeled ⚠️ Re-run. (Note: #issuecomment-6928610849 lists this as fixed in 6766cef, but e6130c1 removed group.failedCount.)
Medium (.github/scripts/eval-report.mjs:662-681) — Missing-candidate, added-scenario, and mismatched-trial row rendering (reverted in e6130c1):e6130c1 also reverted the Removed/Added/token-fallback branches from 6766cef and ab402b68 (still cited in #issuecomment-6928610849), causing baseline-only scenarios to render as ❌ Failed / Not scored / Unchanged, added scenarios to hide candidate output tokens and say Unchanged / Still failing, and mismatched-trial rows to print (—).
Medium (.github/scripts/eval-report.mjs:785-789) — Per-skill model misattribution (reverted in e6130c1):renderSkillDetailsBlock footers and Unchanged evals rows reuse summary.models[0] instead of each skill's own recorded model(s).
Low (docs/checks-and-evals.md:308-316) — Companion action pin alignment: Step 3 describes the Ship / No-ship sticky comment layout from github/evals-action#35 while the version note right below (and workflow pin) still points to github/evals-action#34 (3c55133).
The reason will be displayed to describe this comment to others. Learn more.
Review Summary
Approved at 4295c26 — the follow-up changes cleanly resolve the blocking items from the previous pass:
Markdown/HTML escaping (escapeCell): Encoding ` as ` (after & → &) and wrapping scenario/model cells in <code>...</code> prevents inline code-span breakout and literal backslash rendering.
Unchanged skill grouping: Gating group.changed on behavioral/score/scenario transitions or >= 10% output-token delta keeps parity skills with minor LLM token variance grouped under 📋 Unchanged evals & run configuration.
Delta formatting & docs: Mismatched-trial rows no longer emit a trailing (—), and docs/checks-and-evals.md now notes companion PR github/evals-action#35.
Optional / Low follow-ups (non-blocking):
Consider adding unit assertions in .github/scripts/eval-report.test.mjs covering ` escaping in escapeCell and < 10% output-token variance remaining grouped under Unchanged evals.
Note: single-trial (runs: 1), matched-scenario, single-model-per-run assumptions for per-skill detail labels are noted as intentional scope boundaries for this starter's Vally setup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the Ship / No-ship / Re-run verdict summary table and collapsible
<details>layout in.github/scripts/eval-report.mjs(and updatesdocs/checks-and-evals.mdto match the companiongithub/evals-actionPR):## Evals: ✅ Ship/🛑 No ship/⚠️ Re-runand a severity-ordered| Skill | Verdict | Why |table derived strictly from Vallyresults.jsonl.<details>layout: Places🛑 No shipand⚠️ Re-runper-skill scenario<details>above---, and places✅ Ship,📋 Unchanged evals & run configuration, and📊 Telemetry, token deltas & run evidencebelow---.Outcomerow inBaseline vs. Candidate Comparisonwith the headline verdict and sums all evaluated scenarios/skills.Changes
.github/scripts/eval-report.mjs: AddedsummarizeSkillVerdicts,describeCaseChange,renderSkillDetailsBlock, and verdict-first collapsible layout inrenderReport..github/scripts/eval-report.test.mjs: Added unit tests verifying single-skill✅ Shipand multi-skill mixed-verdict ordering around---.docs/checks-and-evals.md: Updated PR comparison documentation.docs/pr-comment-verdict-mockups.mdRFC file (rendered preview remains in#issuecomment-6924338386).Verification
node --test --experimental-test-coverage .github/scripts/eval-report.test.mjs .github/scripts/pr-description.test.mjs: passed (7/7 tests, 96.23% line / 84.16% branch coverage)