Repository navigation
feat(report): add Baseline vs. Candidate comparison reporting - #34
Conversation
Add optional `baseline-report-id`, `baseline-artifact-id`, and `baseline-artifact-digest` inputs to `report` mode and render a Baseline vs. Candidate comparison table (outcome, passed/failed counts, score delta, duration delta, and token in/out + % delta) in the sticky PR comment and job summary. Relates to #26 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Evals: passedSuite: copilot-smoke Lint: validated (no model execution) - Evidence Run: passed - 1 trial - Evidence
Quality & token summary
|
There was a problem hiding this comment.
🟡 Changes recommended
Reporting can produce false regressions and misleading performance deltas due to flawed group matching, insufficient workload-equivalence checks, and incorrect score and token comparisons.
5 open issues
Apply score tolerance before classifying score changes · New Classify total-token changes before output-token changes · New Require matching per-group workloads for aggregate deltas · New Match native group identities before using display names · New Document baseline workflow constraints and run-baseline setup · New
What changed in this PR
Adds optional Baseline vs. Candidate comparisons to the root Action’s report command, using verified native run artifacts from the same workflow run, attempt, and checkout revision.
Changes:
- Adds baseline report identity and artifact inputs with evidence validation.
- Renders comparison metrics and group transitions in sticky PR comments and job summaries.
- Extends tests and documents baseline workflow configuration.
| File | Description |
|---|---|
tests/reporting.test.ts |
Tests transitions, metric deltas, and rendering budgets. |
tests/reporter.test.ts |
Tests comparison comments and inconsistent baseline evidence. |
tests/inputs.test.ts |
Tests baseline input parsing and validation. |
tests/action.test.ts |
Tests baseline discovery, explicit downloads, and summary integration. |
src/reporting.ts |
Adds comparison rendering, transition classification, and delta formatting. |
src/reporter.ts |
Validates baseline evidence and adds bounded comment comparisons. |
src/inputs.ts |
Parses and validates baseline identities and artifact pairs. |
src/action.ts |
Loads baseline evidence and passes it to reporting. |
README.md |
Documents comparison setup and baseline inputs. |
action.yml |
Exposes the three optional baseline inputs. |
Review effort: Max (dynamic)
… findings - Pass through native `turnCount` (`turns`) and `model` from Vally `results.jsonl` into `Report` / `ReportGroup`, `renderQualitySection`, `renderSummary`, and `renderComparisonSection`. - Address Copilot review findings on score-delta tolerance (`0.0005`), total-token precedence in `classifyGroupTransition`, direct `group.identity` matching, per-group `trials` equality in `workloadsMatch`, and `README.md` baseline workflow notes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…son recipe - Evaluate `totalTokens` (`<` / `>`) before `outputTokens` in `classifyTransition` and use `outputTokens` as a tie-breaker only when `totalTokens` is equal. - Pin the `docs/checks-and-evals.md` comparison recipe to `github/evals-action@fcc3794a85b0e248241991add59e2eb9c87fcd1d` (`github/evals-action#34`), remove unused `COPILOT_CLI_PATH` / `npm ci` from the Action steps, and set `continue-on-error: true` on the baseline run step. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Missing report groups can produce unsupported added/removed results, small score changes display zero-looking deltas, and native CLI tests do not verify the new model and turn behavior.
3 open issues
5 resolved since last review
Review effort: Max (dynamic)
…and docs - Format non-zero score deltas `< 0.005` with 3 decimal places in `formatScoreDelta` so `>= 0.0005` transitions never display `0.00`. - Omit group-level comparison rows when either report lacks `groups`. - Assert projected group `model` and `turns` in `tests/native.test.ts` and update `README.md` (`doc-doctor`). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Keep automatic same-repository PR comparison free of a required `environment: evals` gate and align `README.md` and `docs/checks-and-evals.md` (`doc-doctor`). - Only stage `SKILL.verbose-fixture.md` in `skill-eval-comparison.yml` on `workflow_dispatch` or when the fixture itself changed in the PR. - Update the pinned `github/evals-action#34` commit SHA to `3c55133`. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Comparison rows for distinct models or variants share the same eval label, making opposite transitions indistinguishable and preventing readers from identifying the affected group.
1 open issue
3 resolved since last review
Review effort: Balanced (dynamic)
There was a problem hiding this comment.
🟡 Changes recommended
Absolute checkout paths produce inconsistent evaluation identities across jobs, causing comparisons to report false removals/additions instead of actual transitions and suppress aggregate deltas.
4 open issues
1 resolved since last review
Review effort: Max (dynamic)
| repository: requiredEnv('GITHUB_REPOSITORY'), runId: positiveEnv('GITHUB_RUN_ID'), | ||
| attempt: positiveEnv('GITHUB_RUN_ATTEMPT'), reportId: inputs.baselineReportId, | ||
| }, false); | ||
| if (!discoveredBaseline.run) throw new NativeError('report-artifacts-not-found', 'evidence'); |
| (evaluation ? `**Run:** ${evaluation.outcome} - ${evaluation.executed ?? 'unknown'} ${evaluation.executed === 1 ? 'trial' : 'trials'}` + | ||
| (artifactId !== undefined ? ` - [Evidence](${run}/artifacts/${artifactId})` : '') + '\n\n' + | ||
| (baseline ? `**Baseline:** ${baseline.report.outcome} - ${baseline.report.executed ?? 'unknown'} ${baseline.report.executed === 1 ? 'trial' : 'trials'} - [Evidence](${run}/artifacts/${baseline.artifactId})\n\n` : '') + |
| const baseMap = new Map(baseGroups.map((group) => [group.identity, group])); | ||
| const candMap = new Map(candGroups.map((group) => [group.identity, group])); |
| are shown when both arms share the same selection and per-eval trial counts. | ||
| You can also pass `baseline-artifact-id` and `baseline-artifact-digest` | ||
| explicitly instead of discovering the baseline artifact by `baseline-report-id` | ||
| (matching `baseline-report-id`, or `<report-id>-baseline` when omitted). |
…as, and PR comparison docs (#21) * feat(evals): add real Vally 0.21.0 pr-description runs, output-token deltas, and PR comparison docs - Update `evals/pr-description/skills/pr-description/eval.yaml` and `SKILL.verbose-fixture.md` so live `claude-haiku-5.5` runs reliably activate `pr-description` (`1/2` FAIL on `v0` -> `2/2` PASS on `v1`, `-56.2%` positive-case output tokens). - Replace synthetic `baseline-results.jsonl` and `candidate-results.jsonl` with real Vally 0.21.0 (`claude-haiku-5.5`) runs (`63a64377` and `ab6a6e4d`). - Add `Output Tokens` delta reporting in `.github/scripts/eval-report.mjs` and document automatic PR Baseline vs. Candidate skill comparison via `github/evals-action@v0` (`baseline-report-id: evals-baseline`). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * feat(report): include Model, Skills Invoked, Turns, and Agent Wall Time in eval-report Pass through native Vally `results.jsonl` fields (`model`, `trajectory.metrics.skillActivationBreakdown`, `trajectory.metrics.turnCount`, and `trajectory.metrics.wallTimeMs`) in single-run and Baseline vs. Candidate comparison tables. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(evals): prioritize total-token parity transitions and pin comparison recipe - Evaluate `totalTokens` (`<` / `>`) before `outputTokens` in `classifyTransition` and use `outputTokens` as a tie-breaker only when `totalTokens` is equal. - Pin the `docs/checks-and-evals.md` comparison recipe to `github/evals-action@fcc3794a85b0e248241991add59e2eb9c87fcd1d` (`github/evals-action#34`), remove unused `COPILOT_CLI_PATH` / `npm ci` from the Action steps, and set `continue-on-error: true` on the baseline run step. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * ci(evals): add automatic PR skill comparison workflow pinned to evals-action#34 Run Baseline (`v0` / base skill) vs. Candidate (`v1` / PR skill) via `github/evals-action@fcc3794a85b0e248241991add59e2eb9c87fcd1d` and post the sticky PR comparison comment automatically on PRs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(evals): align PR comparison workflow and docs (doc-doctor) - Keep automatic same-repository PR comparison free of a required `environment: evals` gate and align `README.md` and `docs/checks-and-evals.md` (`doc-doctor`). - Only stage `SKILL.verbose-fixture.md` in `skill-eval-comparison.yml` on `workflow_dispatch` or when the fixture itself changed in the PR. - Update the pinned `github/evals-action#34` commit SHA to `3c55133`. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(ci): restore baseline skill sources strictly from BASE_SHA on pull_request Reserve `SKILL.verbose-fixture.md` staging for `workflow_dispatch` and always restore `$BASE_SHA:$dir` on `pull_request`. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>


Summary
Link the tracking issue and describe the observable behavior.
Relates to #26.
Adds thin Baseline vs. Candidate comparison reporting directly to the root Action's
reportcommand without introducing a separate subaction or custom checkout/selection schema:action.yml&src/inputs.ts: Adds optionalbaseline-report-id,baseline-artifact-id, andbaseline-artifact-digestinputs forcommand: report.src/action.ts&src/reporter.ts: Discovers or downloads the verified baselinerunartifact inreportmode (preserving lint-only fallback when no candidate run artifact exists) and validates matching repository, revision, workflow run ID, attempt, and distinct artifact/report identities.src/reporting.ts: AddsrenderComparisonSection,classifyGroupTransition,formatSignedCount,formatScoreDelta,formatDurationDelta, andformatTokenDelta(including output-token delta breakdown) in the sticky PR comment (### Baseline vs. Candidate comparison) and job summary (## Baseline vs. Candidate comparison), bounded within the 60 KB comment budget.README.md: Documents thebaseline-report-idandbaseline-artifact-id/baseline-artifact-digestworkflow setup.Verification
distmatch source.Exact commands run and results:
node node_modules/typescript/bin/tsc --noEmit— passed (0errors)node node_modules/oxlint/bin/oxlint --deny-warnings src tests— passed (0warnings,0errors)node node_modules/vitest/vitest.mjs run tests/inputs.test.ts tests/reporting.test.ts tests/reporter.test.ts tests/action.test.ts— passed (4test files,112/112tests passed)node tooling/build.mjs && node tooling/build.mjs --check— passed (dist/index.cjsup to date)node --test tooling/packaged.test.mjs— passed (6/6tests passed)Limitations
List missing acceptance and dependency gates. Keep incomplete integration draft.