Skip to content

feat(report): add Ship / No-ship verdict table and collapsible PR comment sections - #35

Merged
jongio merged 7 commits into
mainfrom
feat/ship-verdict-summary
Nov 16, 2026
Merged

jongio merged 7 commits into
mainfrom
feat/ship-verdict-summary

Conversation

@jongio

@jongio jongio commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Updates the sticky PR comparison comment (command: report with baseline-report-id / baseline-artifact-id) to lead with a clear Ship / No-ship / Re-run verdict summary table at the top while keeping detailed scenario, quality, and token telemetry inside collapsible <details> sections:

  • Top-level Verdict Table (visible by default): Renders # Evals: ✅ Ship / 🛑 No ship / ⚠️ Re-run and | Skill | Verdict | Why | ordered by severity (🛑 No ship → ⚠️ Re-run → ✅ Ship), derived strictly from Vally results.jsonl group evidence.
  • Actionable vs. Informational <details> separation: Places 🛑 No ship and ⚠️ Re-run <details> sections above ---, and places passing <details>, 📋 Unchanged evals & run configuration, and 📊 Telemetry, token deltas & run evidence below ---.
  • Headline & Aggregate Outcome Alignment: Aligns the aggregate Outcome row in renderComparisonSection with the computed headline verdict and preserves overflow notices when group counts exceed table or comment byte budgets.

Changes

  • src/reporting.ts: Added GroupVerdictEntry, classifyGroupVerdict, renderVerdictSummaryTable, and overallVerdict outcome alignment in renderComparisonSection.
  • src/reporter.ts: Updated renderComment when baseline is provided to render the verdict summary table and collapsible <details> sections ordered around ---.
  • tests/reporter.test.ts: Added unit tests covering single-skill ✅ Ship comparison comments, multi-skill mixed verdict ordering around ---, and Outcome alignment.
  • README.md & dist/index.cjs: Updated comparison documentation and rebuilt the bundled Action entrypoint.

Verification

  • node node_modules/typescript/bin/tsc --noEmit: passed
  • node node_modules/oxlint/bin/oxlint --deny-warnings src tests: passed (0 warnings, 0 errors)
  • node node_modules/vitest/vitest.mjs run --no-file-parallelism tests/reporting.test.ts tests/reporter.test.ts tests/inputs.test.ts tests/action.test.ts: passed (112/112 tests)
  • node tooling/build.mjs --check: passed
  • node --test tooling/packaged.test.mjs: passed (6/6 tests)

…ment sections

- Add `classifyGroupVerdict` and `renderVerdictSummaryTable` to surface
  per-eval `✅ Ship`, `🛑 No ship`, and `⚠️ Re-run` verdicts at the top
  of comparison PR comments.
- Order actionable `🛑 No ship` and `⚠️ Re-run` `<details>` sections
  above `---`, and place passing `<details>`, unchanged eval coverage,
  and full `Baseline vs. Candidate` / `Quality & token summary`
  telemetry below `---`.
- Align the aggregate `Outcome` row in `renderComparisonSection` with
  the headline verdict and preserve comment byte-budget notices.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio
jongio requested a review from a team as a code owner November 16, 2026 20:14
Copilot AI adaptive review requested due to automatic review settings November 16, 2026 20:14
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Evals: passed

Suite: copilot-smoke

Lint: validated (no model execution) - Evidence

Run: passed - 1 trial - Evidence

Eval Trials Result
copilot-skill-smoke 1 passed

Quality & token summary

Eval Model Verdict Score Turns Duration Tokens
copilot-skill-smoke - passed 1.00 (threshold 1.00) 2 2515ms 32966 tokens (32893 in / 73 out)

Exact revision | Workflow run

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Mandatory report tables can exceed the 60 KB limit and block sticky-comment updates, while coverage summaries misrepresent skipped scenarios and zero-trial handling obscures executed candidate failures.

3 open issues
What changed in this PR

Updates baseline comparison comments with a prominent Ship / No-ship / Re-run verdict and collapsible details, making evaluation results easier to scan.

Changes:

  • Adds severity-ordered verdict summaries derived from native group evidence.
  • Separates actionable results from passing results and telemetry.
  • Aligns comparison outcomes with the headline and adds rendering tests and documentation.
File Description
tests/​reporter.test.ts Tests verdict rendering, section ordering, and outcome alignment.
src/​reporting.ts Adds group verdict classification and summary-table rendering.
src/​reporter.ts Organizes comparison comments into verdicts and collapsible sections.
README.md Documents the updated comparison comment layout.

Review effort: Balanced (adaptive)

Comment thread src/reporter.ts Resolved
Comment thread src/reporter.ts Resolved
Comment thread src/reporting.ts Resolved
…ssify zero-trial baselines

- Bound unchanged coverage rows and telemetry Eval/Trials/Result rows
  against the 60 KB PR comment limit.
- Sum stimuliTotal separately in unchanged coverage summaries and report
  skipped scenario counts accurately.
- Let failing candidate groups with zero-trial baselines reach the No-ship
  classification.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio

jongio commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Review Disposition (5926920926, 5927251389, 5927573412, 5927756159, 5928349526)

  • Bound verdict and telemetry rows to the 60 KB comment size limit (concern:b8776bb14730032f, concern:c807b32c49c45f82, reviews 5926920926, 5928349526): Fixed in 1a46e84 (src/reporting.ts, src/reporter.ts, tests/reporter.test.ts). renderVerdictSummaryTable is byte-bounded (maxBytes = 16_000) and the Eval | Trials | Result table is capped to 25 rows when baseline is present.
  • Classify candidate failures and incomplete baselines accurately (concern:baec76165cb75a81, concern:4428e02ea0cee2e1, concern:e7aaa203dee25745, concern:c0cdab338bb41087, reviews 5926920926, 5927756159, 5928349526): Fixed in ff42091 and 1a46e84 (src/reporting.ts). Persistent failures (!candidate.passed) report Still failing rather than ... at parity, and incomplete runs (error / not-run / cancelled) propagate ⚠️ Re-run to non-failing group rows.
  • Align aggregate Outcome row and README/tests (concern:7b904842295edbd6, concern:6bb75365abc00d68, concern:852ef144de4c6d91, concern:8f3564514c2e9915, concern:cfd70de557316ea9, reviews 5927251389, 5927573412, 5928349526): Fixed in efc76b1, 65a0a17, and 1a46e84 (src/reporting.ts, README.md, tests/reporter.test.ts).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The aggregate outcome row can incorrectly report an unverified baseline failure as fixed when its candidate eval is removed or runs zero trials, contradicting the Re-run headline.

1 open issue
3 resolved since last review

Review effort: Balanced (adaptive)

Comment thread src/reporting.ts Resolved
…ixed in Outcome

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The report mislabels unchanged or improved failing outcomes as regressions and shows baseline model data as candidate configuration for removed groups.

2 open issues
1 resolved since last review

Review effort: Balanced (adaptive)

Comment thread src/reporter.ts Resolved
Comment thread src/reporting.ts Resolved
… failure delta

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The report can incorrectly recommend shipping without usable baseline results and mislabel a score-only decline as a pass-to-fail transition.

2 open issues
2 resolved since last review

Review effort: Balanced (adaptive)

Comment thread src/reporting.ts Resolved
Comment thread src/reporting.ts Resolved
jongio and others added 2 commits November 16, 2026 13:15
… score regressions

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ment

Keep only the top-level Ship / No-ship verdict table above the fold and
a single collapsible Telemetry, token deltas & run evidence section.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@wbreza wbreza left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review Summary

Reviewed c7b2abb (feat(report): add Ship / No-ship verdict table and collapsible PR comment sections).

Removing the duplicated per-group <details> blocks in c7b2abb simplified the sticky PR comment layout significantly, and CI checks pass. However, that refactor re-introduced a 60 KB comment-budget overflow path and left a couple of verdict-classification edge cases and stale docs/tests that should be addressed before shipping:

Severity Location Summary
High src/reporter.ts:202-218 Unbounded verdict summary + 100-row detailsBody table can exceed postReport's 60 KB limit (report-comment-too-large) on valid reports that previously succeeded.
Medium src/reporting.ts:304-329 Incomplete (error / cancelled) runs produce ✅ **Ship** rows and false "New eval passing" reasons underneath a # Evals: ⚠️ Re-run headline.
Medium src/reporting.ts:255-265 Persistent failures (!baseline.passed && !candidate.passed) with token or score deltas report "Reduced/Increased token usage at parity" or "Improved score" without indicating the eval is still failing.
Low README.md:257-261 Documentation still describes per-eval <details> blocks, --- ordering, and unchanged-coverage sections removed in c7b2abb.
Low tests/reporter.test.ts:259-266 Refactor removed test coverage for 🛑 No ship, ⚠️ Re-run, score regressions, zero-trial/incomplete baselines, and the 60 KB comment budget.

Comment thread src/reporter.ts Resolved
Comment thread src/reporting.ts Resolved
Comment thread src/reporting.ts Resolved
Comment thread README.md Resolved
Comment thread tests/reporter.test.ts Resolved
…plete runs, and byte limits

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@wbreza wbreza left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — all findings from the previous review pass have been cleanly addressed in 1a46e84:

  • renderVerdictSummaryTable is now UTF-8 byte-bounded (maxBytes = 16_000) and comparison Eval | Trials | Result rows are capped to 25, keeping 100-group escaped reports comfortably within postReport's 60 KB comment ceiling.
  • Incomplete (error / not-run / cancelled) baseline and candidate runs propagate ⚠️ Re-run to group rows while preserving 🛑 No ship precedence for executed candidate failures.
  • Persistent failures (!candidate.passed) now lead with Still failing across score and token delta transitions instead of reporting parity/improvement reasons.
  • README.md and tests/reporter.test.ts match the updated layout and cover mixed verdicts and comment byte limits.

(Minor non-blocking note: consider updating the PR description summary to reflect removing the per-group <details> / --- split in c7b2abb before squash-merging.)

@jongio
jongio merged commit 0480c3e into main Nov 16, 2026
10 checks passed
@jongio
jongio deleted the feat/ship-verdict-summary branch November 16, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants