Skip to content

feat(report): add Baseline vs. Candidate comparison reporting - #34

Merged
jongio merged 3 commits into
mainfrom
feat/baseline-comparison-report
Nov 16, 2026
Merged

jongio merged 3 commits into
mainfrom
feat/baseline-comparison-report

Conversation

@jongio

@jongio jongio commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Link the tracking issue and describe the observable behavior.

Relates to #26.

Adds thin Baseline vs. Candidate comparison reporting directly to the root Action's report command without introducing a separate subaction or custom checkout/selection schema:

  • action.yml & src/inputs.ts: Adds optional baseline-report-id, baseline-artifact-id, and baseline-artifact-digest inputs for command: report.
  • src/action.ts & src/reporter.ts: Discovers or downloads the verified baseline run artifact in report mode (preserving lint-only fallback when no candidate run artifact exists) and validates matching repository, revision, workflow run ID, attempt, and distinct artifact/report identities.
  • src/reporting.ts: Adds renderComparisonSection, classifyGroupTransition, formatSignedCount, formatScoreDelta, formatDurationDelta, and formatTokenDelta (including output-token delta breakdown) in the sticky PR comment (### Baseline vs. Candidate comparison) and job summary (## Baseline vs. Candidate comparison), bounded within the 60 KB comment budget.
  • README.md: Documents the baseline-report-id and baseline-artifact-id / baseline-artifact-digest workflow setup.

Verification

  • Model-free checks and source-to-bundle verification pass.
  • Native integration evidence is distinguished from synthetic controls.
  • Credential, process and artifact boundaries were reviewed.
  • Documentation and packaged dist match source.

Exact commands run and results:

  • node node_modules/typescript/bin/tsc --noEmit — passed (0 errors)
  • node node_modules/oxlint/bin/oxlint --deny-warnings src tests — passed (0 warnings, 0 errors)
  • node node_modules/vitest/vitest.mjs run tests/inputs.test.ts tests/reporting.test.ts tests/reporter.test.ts tests/action.test.ts — passed (4 test files, 112/112 tests passed)
  • node tooling/build.mjs && node tooling/build.mjs --check — passed (dist/index.cjs up to date)
  • node --test tooling/packaged.test.mjs — passed (6/6 tests passed)

Limitations

List missing acceptance and dependency gates. Keep incomplete integration draft.

  • Baseline and candidate run artifacts must belong to the same workflow run, attempt, and checkout revision; cross-run artifact comparison is intentionally out of scope.

Add optional `baseline-report-id`, `baseline-artifact-id`, and
`baseline-artifact-digest` inputs to `report` mode and render a
Baseline vs. Candidate comparison table (outcome, passed/failed counts,
score delta, duration delta, and token in/out + % delta) in the sticky
PR comment and job summary.

Relates to #26

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio
jongio requested a review from a team as a code owner November 16, 2026 07:12
Copilot AI adaptive review requested due to automatic review settings November 16, 2026 07:12
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Evals: passed

Suite: copilot-smoke

Lint: validated (no model execution) - Evidence

Run: passed - 1 trial - Evidence

Eval Trials Result
copilot-skill-smoke 1 passed

Quality & token summary

Eval Model Verdict Score Turns Duration Tokens
copilot-skill-smoke - passed 1.00 (threshold 1.00) 2 1750ms 33128 tokens (33055 in / 73 out)

Exact revision | Workflow run

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Reporting can produce false regressions and misleading performance deltas due to flawed group matching, insufficient workload-equivalence checks, and incorrect score and token comparisons.

5 open issues
What changed in this PR

Adds optional Baseline vs. Candidate comparisons to the root Action’s report command, using verified native run artifacts from the same workflow run, attempt, and checkout revision.

Changes:

  • Adds baseline report identity and artifact inputs with evidence validation.
  • Renders comparison metrics and group transitions in sticky PR comments and job summaries.
  • Extends tests and documents baseline workflow configuration.
File Description
tests/​reporting.test.ts Tests transitions, metric deltas, and rendering budgets.
tests/​reporter.test.ts Tests comparison comments and inconsistent baseline evidence.
tests/​inputs.test.ts Tests baseline input parsing and validation.
tests/​action.test.ts Tests baseline discovery, explicit downloads, and summary integration.
src/​reporting.ts Adds comparison rendering, transition classification, and delta formatting.
src/​reporter.ts Validates baseline evidence and adds bounded comment comparisons.
src/​inputs.ts Parses and validates baseline identities and artifact pairs.
src/​action.ts Loads baseline evidence and passes it to reporting.
README.md Documents comparison setup and baseline inputs.
action.yml Exposes the three optional baseline inputs.

Review effort: Max (dynamic)

Comment thread src/reporting.ts Resolved
Comment thread src/reporting.ts Resolved
Comment thread src/reporting.ts Resolved
Comment thread src/reporting.ts Resolved
Comment thread README.md Resolved
… findings

- Pass through native `turnCount` (`turns`) and `model` from Vally
  `results.jsonl` into `Report` / `ReportGroup`, `renderQualitySection`,
  `renderSummary`, and `renderComparisonSection`.
- Address Copilot review findings on score-delta tolerance (`0.0005`),
  total-token precedence in `classifyGroupTransition`, direct
  `group.identity` matching, per-group `trials` equality in
  `workloadsMatch`, and `README.md` baseline workflow notes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio
jongio requested an adaptive review from Copilot November 16, 2026 07:43
jongio added a commit to github/plugin-marketplace-starter that referenced this pull request Nov 16, 2026
…son recipe

- Evaluate `totalTokens` (`<` / `>`) before `outputTokens` in
  `classifyTransition` and use `outputTokens` as a tie-breaker only when
  `totalTokens` is equal.
- Pin the `docs/checks-and-evals.md` comparison recipe to
  `github/evals-action@fcc3794a85b0e248241991add59e2eb9c87fcd1d`
  (`github/evals-action#34`), remove unused `COPILOT_CLI_PATH` / `npm ci`
  from the Action steps, and set `continue-on-error: true` on the
  baseline run step.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Missing report groups can produce unsupported added/removed results, small score changes display zero-looking deltas, and native CLI tests do not verify the new model and turn behavior.

3 open issues
5 resolved since last review

Review effort: Max (dynamic)

Comment thread src/reporting.ts Resolved
Comment thread src/reporting.ts Resolved
Comment thread tests/native.test.ts Resolved
@jongio
jongio requested a review from wbreza November 16, 2026 08:09
…and docs

- Format non-zero score deltas `< 0.005` with 3 decimal places in
  `formatScoreDelta` so `>= 0.0005` transitions never display `0.00`.
- Omit group-level comparison rows when either report lacks `groups`.
- Assert projected group `model` and `turns` in `tests/native.test.ts`
  and update `README.md` (`doc-doctor`).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
jongio added a commit to github/plugin-marketplace-starter that referenced this pull request Nov 16, 2026
- Keep automatic same-repository PR comparison free of a required
  `environment: evals` gate and align `README.md` and
  `docs/checks-and-evals.md` (`doc-doctor`).
- Only stage `SKILL.verbose-fixture.md` in `skill-eval-comparison.yml`
  on `workflow_dispatch` or when the fixture itself changed in the PR.
- Update the pinned `github/evals-action#34` commit SHA to `3c55133`.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio
jongio requested an adaptive review from Copilot November 16, 2026 08:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Comparison rows for distinct models or variants share the same eval label, making opposite transitions indistinguishable and preventing readers from identifying the affected group.

1 open issue
3 resolved since last review

Review effort: Balanced (dynamic)

Comment thread src/reporting.ts Resolved
@jongio
jongio requested an adaptive review from Copilot November 16, 2026 08:27
@jongio
jongio merged commit 711f0b7 into main Nov 16, 2026
12 checks passed
@jongio
jongio deleted the feat/baseline-comparison-report branch November 16, 2026 08:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Absolute checkout paths produce inconsistent evaluation identities across jobs, causing comparisons to report false removals/additions instead of actual transitions and suppress aggregate deltas.

4 open issues
1 resolved since last review

Review effort: Max (dynamic)

Comment thread src/action.ts
repository: requiredEnv('GITHUB_REPOSITORY'), runId: positiveEnv('GITHUB_RUN_ID'),
attempt: positiveEnv('GITHUB_RUN_ATTEMPT'), reportId: inputs.baselineReportId,
}, false);
if (!discoveredBaseline.run) throw new NativeError('report-artifacts-not-found', 'evidence');
Comment thread src/reporter.ts
Comment on lines 178 to +180
(evaluation ? `**Run:** ${evaluation.outcome} - ${evaluation.executed ?? 'unknown'} ${evaluation.executed === 1 ? 'trial' : 'trials'}` +
(artifactId !== undefined ? ` - [Evidence](${run}/artifacts/${artifactId})` : '') + '\n\n' +
(baseline ? `**Baseline:** ${baseline.report.outcome} - ${baseline.report.executed ?? 'unknown'} ${baseline.report.executed === 1 ? 'trial' : 'trials'} - [Evidence](${run}/artifacts/${baseline.artifactId})\n\n` : '') +
Comment thread src/reporting.ts
Comment on lines +223 to +224
const baseMap = new Map(baseGroups.map((group) => [group.identity, group]));
const candMap = new Map(candGroups.map((group) => [group.identity, group]));
Comment thread README.md
are shown when both arms share the same selection and per-eval trial counts.
You can also pass `baseline-artifact-id` and `baseline-artifact-digest`
explicitly instead of discovering the baseline artifact by `baseline-report-id`
(matching `baseline-report-id`, or `<report-id>-baseline` when omitted).
jongio added a commit to github/plugin-marketplace-starter that referenced this pull request Nov 16, 2026
…as, and PR comparison docs (#21)

* feat(evals): add real Vally 0.21.0 pr-description runs, output-token deltas, and PR comparison docs

- Update `evals/pr-description/skills/pr-description/eval.yaml` and
  `SKILL.verbose-fixture.md` so live `claude-haiku-5.5` runs reliably
  activate `pr-description` (`1/2` FAIL on `v0` -> `2/2` PASS on `v1`,
  `-56.2%` positive-case output tokens).
- Replace synthetic `baseline-results.jsonl` and `candidate-results.jsonl`
  with real Vally 0.21.0 (`claude-haiku-5.5`) runs (`63a64377` and
  `ab6a6e4d`).
- Add `Output Tokens` delta reporting in `.github/scripts/eval-report.mjs`
  and document automatic PR Baseline vs. Candidate skill comparison via
  `github/evals-action@v0` (`baseline-report-id: evals-baseline`).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* feat(report): include Model, Skills Invoked, Turns, and Agent Wall Time in eval-report

Pass through native Vally `results.jsonl` fields (`model`,
`trajectory.metrics.skillActivationBreakdown`, `trajectory.metrics.turnCount`,
and `trajectory.metrics.wallTimeMs`) in single-run and Baseline vs.
Candidate comparison tables.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(evals): prioritize total-token parity transitions and pin comparison recipe

- Evaluate `totalTokens` (`<` / `>`) before `outputTokens` in
  `classifyTransition` and use `outputTokens` as a tie-breaker only when
  `totalTokens` is equal.
- Pin the `docs/checks-and-evals.md` comparison recipe to
  `github/evals-action@fcc3794a85b0e248241991add59e2eb9c87fcd1d`
  (`github/evals-action#34`), remove unused `COPILOT_CLI_PATH` / `npm ci`
  from the Action steps, and set `continue-on-error: true` on the
  baseline run step.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* ci(evals): add automatic PR skill comparison workflow pinned to evals-action#34

Run Baseline (`v0` / base skill) vs. Candidate (`v1` / PR skill) via
`github/evals-action@fcc3794a85b0e248241991add59e2eb9c87fcd1d` and post
the sticky PR comparison comment automatically on PRs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(evals): align PR comparison workflow and docs (doc-doctor)

- Keep automatic same-repository PR comparison free of a required
  `environment: evals` gate and align `README.md` and
  `docs/checks-and-evals.md` (`doc-doctor`).
- Only stage `SKILL.verbose-fixture.md` in `skill-eval-comparison.yml`
  on `workflow_dispatch` or when the fixture itself changed in the PR.
- Update the pinned `github/evals-action#34` commit SHA to `3c55133`.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(ci): restore baseline skill sources strictly from BASE_SHA on pull_request

Reserve `SKILL.verbose-fixture.md` staging for `workflow_dispatch` and
always restore `$BASE_SHA:$dir` on `pull_request`.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants