Repository navigation
feat(report): add Ship / No-ship verdict table and collapsible PR comment sections - #35
Conversation
…ment sections - Add `classifyGroupVerdict` and `renderVerdictSummaryTable` to surface per-eval `✅ Ship`, `🛑 No ship`, and `⚠️ Re-run` verdicts at the top of comparison PR comments. - Order actionable `🛑 No ship` and `⚠️ Re-run` `<details>` sections above `---`, and place passing `<details>`, unchanged eval coverage, and full `Baseline vs. Candidate` / `Quality & token summary` telemetry below `---`. - Align the aggregate `Outcome` row in `renderComparisonSection` with the headline verdict and preserve comment byte-budget notices. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Evals: passedSuite: copilot-smoke Lint: validated (no model execution) - Evidence Run: passed - 1 trial - Evidence
Quality & token summary
|
There was a problem hiding this comment.
🟡 Changes recommended
Mandatory report tables can exceed the 60 KB limit and block sticky-comment updates, while coverage summaries misrepresent skipped scenarios and zero-trial handling obscures executed candidate failures.
3 open issues
What changed in this PR
Updates baseline comparison comments with a prominent Ship / No-ship / Re-run verdict and collapsible details, making evaluation results easier to scan.
Changes:
- Adds severity-ordered verdict summaries derived from native group evidence.
- Separates actionable results from passing results and telemetry.
- Aligns comparison outcomes with the headline and adds rendering tests and documentation.
| File | Description |
|---|---|
tests/reporter.test.ts |
Tests verdict rendering, section ordering, and outcome alignment. |
src/reporting.ts |
Adds group verdict classification and summary-table rendering. |
src/reporter.ts |
Organizes comparison comments into verdicts and collapsible sections. |
README.md |
Documents the updated comparison comment layout. |
Review effort: Balanced (adaptive)
…ssify zero-trial baselines - Bound unchanged coverage rows and telemetry Eval/Trials/Result rows against the 60 KB PR comment limit. - Sum stimuliTotal separately in unchanged coverage summaries and report skipped scenario counts accurately. - Let failing candidate groups with zero-trial baselines reach the No-ship classification. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Review Disposition (
|
There was a problem hiding this comment.
🟡 Changes recommended
The aggregate outcome row can incorrectly report an unverified baseline failure as fixed when its candidate eval is removed or runs zero trials, contradicting the Re-run headline.
1 open issue
3 resolved since last review
Review effort: Balanced (adaptive)
…ixed in Outcome Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… failure delta Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… score regressions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ment Keep only the top-level Ship / No-ship verdict table above the fold and a single collapsible Telemetry, token deltas & run evidence section. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
wbreza
left a comment
There was a problem hiding this comment.
PR Review Summary
Reviewed c7b2abb (feat(report): add Ship / No-ship verdict table and collapsible PR comment sections).
Removing the duplicated per-group <details> blocks in c7b2abb simplified the sticky PR comment layout significantly, and CI checks pass. However, that refactor re-introduced a 60 KB comment-budget overflow path and left a couple of verdict-classification edge cases and stale docs/tests that should be addressed before shipping:
| Severity | Location | Summary |
|---|---|---|
| High | src/reporter.ts:202-218 |
Unbounded verdict summary + 100-row detailsBody table can exceed postReport's 60 KB limit (report-comment-too-large) on valid reports that previously succeeded. |
| Medium | src/reporting.ts:304-329 |
Incomplete (error / cancelled) runs produce ✅ **Ship** rows and false "New eval passing" reasons underneath a # Evals: ⚠️ Re-run headline. |
| Medium | src/reporting.ts:255-265 |
Persistent failures (!baseline.passed && !candidate.passed) with token or score deltas report "Reduced/Increased token usage at parity" or "Improved score" without indicating the eval is still failing. |
| Low | README.md:257-261 |
Documentation still describes per-eval <details> blocks, --- ordering, and unchanged-coverage sections removed in c7b2abb. |
| Low | tests/reporter.test.ts:259-266 |
Refactor removed test coverage for 🛑 No ship, ⚠️ Re-run, score regressions, zero-trial/incomplete baselines, and the 60 KB comment budget. |
…plete runs, and byte limits Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
wbreza
left a comment
There was a problem hiding this comment.
LGTM — all findings from the previous review pass have been cleanly addressed in 1a46e84:
renderVerdictSummaryTableis now UTF-8 byte-bounded (maxBytes = 16_000) and comparisonEval | Trials | Resultrows are capped to25, keeping 100-group escaped reports comfortably withinpostReport's 60 KB comment ceiling.- Incomplete (
error/not-run/cancelled) baseline and candidate runs propagate⚠️ Re-runto group rows while preserving🛑 No shipprecedence for executed candidate failures. - Persistent failures (
!candidate.passed) now lead withStill failingacross score and token delta transitions instead of reporting parity/improvement reasons. README.mdandtests/reporter.test.tsmatch the updated layout and cover mixed verdicts and comment byte limits.
(Minor non-blocking note: consider updating the PR description summary to reflect removing the per-group <details> / --- split in c7b2abb before squash-merging.)


Summary
Updates the sticky PR comparison comment (
command: reportwithbaseline-report-id/baseline-artifact-id) to lead with a clear Ship / No-ship / Re-run verdict summary table at the top while keeping detailed scenario, quality, and token telemetry inside collapsible<details>sections:# Evals: ✅ Ship/🛑 No ship/⚠️ Re-runand| Skill | Verdict | Why |ordered by severity (🛑 No ship→⚠️ Re-run→✅ Ship), derived strictly from Vallyresults.jsonlgroup evidence.<details>separation: Places🛑 No shipand⚠️ Re-run<details>sections above---, and places passing<details>,📋 Unchanged evals & run configuration, and📊 Telemetry, token deltas & run evidencebelow---.Outcomerow inrenderComparisonSectionwith the computed headline verdict and preserves overflow notices when group counts exceed table or comment byte budgets.Changes
src/reporting.ts: AddedGroupVerdictEntry,classifyGroupVerdict,renderVerdictSummaryTable, andoverallVerdictoutcome alignment inrenderComparisonSection.src/reporter.ts: UpdatedrenderCommentwhenbaselineis provided to render the verdict summary table and collapsible<details>sections ordered around---.tests/reporter.test.ts: Added unit tests covering single-skill✅ Shipcomparison comments, multi-skill mixed verdict ordering around---, andOutcomealignment.README.md&dist/index.cjs: Updated comparison documentation and rebuilt the bundled Action entrypoint.Verification
node node_modules/typescript/bin/tsc --noEmit: passednode node_modules/oxlint/bin/oxlint --deny-warnings src tests: passed (0 warnings, 0 errors)node node_modules/vitest/vitest.mjs run --no-file-parallelism tests/reporting.test.ts tests/reporter.test.ts tests/inputs.test.ts tests/action.test.ts: passed (112/112 tests)node tooling/build.mjs --check: passednode --test tooling/packaged.test.mjs: passed (6/6 tests)