Skip to content

feat(evals): real Vally 0.21.0 pr-description runs, output-token deltas, and PR comparison docs - #21

Merged
jongio merged 6 commits into
mainfrom
feat/skill-eval-comparison
Nov 16, 2026
Merged

jongio merged 6 commits into
mainfrom
feat/skill-eval-comparison

Conversation

@jongio

@jongio jongio commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Updates the pr-description evaluation golden path with real Vally 0.21.0 (claude-haiku-5.5) baseline and candidate model runs, adds Model, Skills Invoked, Turns, Output Tokens, and Agent Wall Time pass-through reporting to eval-report.mjs, and documents automatic PR Baseline vs. Candidate skill comparison using github/evals-action@v0 (baseline-report-id: evals-baseline, see github/evals-action#34):

  • evals/pr-description/skills/pr-description/eval.yaml & evals/_shared/fixtures/pr-description/SKILL.verbose-fixture.md: Tightens the stimulus prompt and v0 pre-token-diet fixture so live claude-haiku-5.5 trials via copilot-sdk reliably activate pr-description on the positive case (1/2 FAIL on v0 -> 2/2 PASS on v1, cutting positive-case output tokens by 56.2% from 918 to 402 and agent wall time by 36.2% from 11.1s to 7.1s).
  • evals/_shared/fixtures/pr-description/{baseline,candidate}-results.jsonl: Replaces the synthetic baseline-run-001 fixtures with normalized real Vally 0.21.0 (claude-haiku-5.5) run records (63a64377-917f-44df-8515-81db5f3c9b31 and ab6a6e4d-091c-4a1a-ac94-952db5ab00ef).
  • .github/scripts/eval-report.mjs & .github/scripts/eval-report.test.mjs: Passes through native Vally results.jsonl fields: Model (claude-haiku-5.5), Skills Invoked (pr-description vs none), Turns / Turn Delta, Output Tokens (1,447 -> 991, -456 (-31.5%)), and Agent Wall Time / Wall Time Delta (-4.0s (-36.2%) on the positive case).
  • .github/workflows/evals.yml, README.md, & docs/checks-and-evals.md: Documents automatic PR Baseline vs. Candidate skill comparison (plugins/<plugin>/skills/<skill>/ <-> evals/<plugin>/skills/<skill>/eval.yaml) and local npm run eval:compare.

Behavioral eval report

Saved run: evals/_shared/fixtures/pr-description/candidate-results.jsonl | Model: claude-haiku-5.5

Passed Failed Avg Score Total Turns Total Tokens Agent Wall Time Total Duration
2 0 100.0% 3 73,008 (in: 72,017, out: 991) 16.4s 34.1s
Case Verdict Score Skills Invoked Turns Tokens Agent Wall Time Duration
pr-description-drafts-structured-pr-body PASS 100.0% pr-description 2 48,724 (in: 48,322, out: 402) 7.1s 18.4s
pr-description-ignores-bug-and-review-question PASS 100.0% none 1 24,284 (in: 23,695, out: 589) 9.3s 15.8s

Baseline vs. Candidate Comparison

Baseline: evals/_shared/fixtures/pr-description/baseline-results.jsonl | Candidate: evals/_shared/fixtures/pr-description/candidate-results.jsonl

Metric Baseline Candidate Delta
Passed 1 2 +1
Failed 1 0 -1
Avg Score 93.8% 100.0% +6.3%
Total Turns 3 3 0
Total Tokens 73,199 (in: 71,752, out: 1,447) 73,008 (in: 72,017, out: 991) -191 (-0.3%)
Output Tokens 1,447 991 -456 (-31.5%)
Agent Wall Time 17.5s 16.4s -1.1s
Total Duration 38.5s 34.1s -4.4s
Case Baseline Candidate Score Delta Turn Delta Token Delta Wall Time Delta Transition
pr-description / pr-description-drafts-structured-pr-body BEHAVIORAL FAILURE (87.5%) PASS (100.0%) +12.5% 0 -308 (-0.6%) -4.0s (-36.2%) Fixed
pr-description / pr-description-ignores-bug-and-review-question PASS (100.0%) PASS (100.0%) 0.0% 0 +117 (+0.5%) +2.9s (+44.2%) Increased tokens at parity

Note: Single-run score, token, and duration deltas are observed signals for review rather than multi-trial statistical proof.


Validation

  • node --test .github/scripts/eval-report.test.mjs .github/scripts/pr-description.test.mjs .github/scripts/lint.test.mjs .github/scripts/validation.test.mjs — passed (34/34 tests passing)
  • node .github/scripts/eval-report.mjs --baseline evals/_shared/fixtures/pr-description/baseline-results.jsonl --candidate evals/_shared/fixtures/pr-description/candidate-results.jsonl — passed (exit code 0)

…deltas, and PR comparison docs

- Update `evals/pr-description/skills/pr-description/eval.yaml` and
  `SKILL.verbose-fixture.md` so live `claude-haiku-5.5` runs reliably
  activate `pr-description` (`1/2` FAIL on `v0` -> `2/2` PASS on `v1`,
  `-56.2%` positive-case output tokens).
- Replace synthetic `baseline-results.jsonl` and `candidate-results.jsonl`
  with real Vally 0.21.0 (`claude-haiku-5.5`) runs (`63a64377` and
  `ab6a6e4d`).
- Add `Output Tokens` delta reporting in `.github/scripts/eval-report.mjs`
  and document automatic PR Baseline vs. Candidate skill comparison via
  `github/evals-action@v0` (`baseline-report-id: evals-baseline`).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio
jongio requested a review from a team as a code owner November 16, 2026 07:24
Copilot AI adaptive review requested due to automatic review settings November 16, 2026 07:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The report can mislabel increased total-token usage as a reduction, and the documented comparison recipe relies on an unsupported action input.

2 open issues
What changed in this PR

Updates the marketplace’s pr-description evaluation example with recorded Vally runs, output-token comparisons, and guidance for opt-in PR evaluation workflows.

Changes:

  • Replaces synthetic results with recorded baseline and candidate runs and refines evaluation prompts.
  • Adds output-token deltas and separates skill-specific tests from report tests.
  • Documents local comparisons and an automatic PR workflow template while keeping the existing workflow manual-only.
File Description
README.md Introduces local and opt-in PR comparisons.
evals/​pr-description/​skills/​pr-description/​eval.yaml Refines positive and negative prompts.
evals/​_shared/​fixtures/​pr-description/​SKILL.verbose-fixture.md Strengthens the verbose baseline fixture.
evals/​_shared/​fixtures/​pr-description/​candidate-results.jsonl Records candidate run results.
evals/​_shared/​fixtures/​pr-description/​baseline-results.jsonl Records baseline run results.
docs/​checks-and-evals.md Adds recorded metrics and a PR comparison workflow template.
.github/​workflows/​evals.yml Adds explanatory comments; execution behavior is unchanged.
.github/​scripts/​pr-description.test.mjs Extracts skill and grader tests into a dedicated file.
.github/​scripts/​eval-report.test.mjs Updates token-delta and recorded-fixture assertions.
.github/​scripts/​eval-report.mjs Adds output-token deltas and changes transition classification.

Review effort: Balanced (dynamic)

Comment thread .github/scripts/eval-report.mjs Resolved
Comment thread docs/checks-and-evals.md Resolved
…me in eval-report

Pass through native Vally `results.jsonl` fields (`model`,
`trajectory.metrics.skillActivationBreakdown`, `trajectory.metrics.turnCount`,
and `trajectory.metrics.wallTimeMs`) in single-run and Baseline vs.
Candidate comparison tables.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Token classification can report reductions in both comparison directions, and the documented workflow relies on unsupported baseline-comparison functionality.

2 open issues
3 previously missed issues

In code that hasn't changed since last review

Low severity Require a released action with baseline-report-id support

README.md:216

github/evals-action@v0 does not support baseline-report-id; the implementation in github/evals-action#34 is still unmerged, so this setup cannot yet produce the promised comparison. Mark it as pending a release containing that feature before users enable paid baseline runs. Same issue in .github/workflows/evals.yml:6–7.

Low severity Forward COPILOT_CLI_PATH to both eval action runs

docs/​checks-and-evals.md:391

github/evals-action@v0 strips COPILOT_CLI_PATH when constructing Vally's environment, so setting it in env does not select the repository-pinned launcher. Add credential-env: COPILOT_CLI_PATH to both run steps to forward it explicitly. Same issue in the candidate step at docs/checks-and-evals.md:411.

Low severity Keep baseline behavior failures from failing the workflow

docs/​checks-and-evals.md:392

A behaviorally failing baseline makes this step fail, leaving the workflow red even when the candidate fixes every baseline failure. Make baseline behavioral failures non-blocking, while still failing on execution, configuration, or artifact-upload errors and leaving candidate failures blocking.

Review effort: Balanced (dynamic)

…son recipe

- Evaluate `totalTokens` (`<` / `>`) before `outputTokens` in
  `classifyTransition` and use `outputTokens` as a tie-breaker only when
  `totalTokens` is equal.
- Pin the `docs/checks-and-evals.md` comparison recipe to
  `github/evals-action@fcc3794a85b0e248241991add59e2eb9c87fcd1d`
  (`github/evals-action#34`), remove unused `COPILOT_CLI_PATH` / `npm ci`
  from the Action steps, and set `continue-on-error: true` on the
  baseline run step.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio

jongio commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Copilot Review Disposition (5916409495, 5916544515, 5916631230, 5916722588, 5916915322, 5916977593)

  • Tie parity transition to total-token delta (concern:38303302210355d4, concern:b0d387c90524e472, concern:9dd1dd35da7cf580, reviews 5916409495, 5916544515): Fixed in 1995814 (.github/scripts/eval-report.mjs, .github/scripts/eval-report.test.mjs). classifyTransition evaluates totalTokens first and uses outputTokens only as a tie-breaker when totalTokens is equal.
  • Qualify eval recipe until comparison support is released / Require a released action with baseline-report-id support (concern:a272c8e5feb33682, concern:3daecd9bf87b67d9, reviews 5916409495, 5916544515): Fixed in 1995814 and b66a309 (docs/checks-and-evals.md, README.md, .github/workflows/evals.yml).
  • Forward COPILOT_CLI_PATH to both eval action runs (concern:4b70f8c16fdad34d, review 5916544515): Fixed in 1995814 (docs/checks-and-evals.md).
  • Keep baseline behavior failures from failing the workflow (concern:97295b4affc0ae1b, review 5916544515): Fixed in 1995814 (docs/checks-and-evals.md).
  • Automatic PR comparison trust & execution boundaries (concern:be8b1df85c490e20, concern:2f0e9ed699573fc1, concern:aa087579132df9f2, concern:f9cbd9b2a7a96682, concern:12a43ca1f0115bf6, concern:861ba6b297e620e8, concern:97eb6ce865a31688, reviews 5916631230, 5916722588, 5916915322, 5916977593): Addressed in b66a309 and 8f8a7cd (.github/workflows/skill-eval-comparison.yml, docs/checks-and-evals.md, README.md): fork PRs are excluded (head.repo.full_name == github.repository), pull_request runs strictly restore $BASE_SHA:$dir, and reporting runs in a separate pull-requests: write job with SHA-256 artifact digest verification.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Coordinating paid PR runs, baseline source restoration, and write-enabled cross-job artifact reporting through an unreleased action revision creates interacting execution and trust boundaries that require deeper review.

No open issues 🎉

2 resolved since last review

Review effort: Balanced (dynamic)

…-action#34

Run Baseline (`v0` / base skill) vs. Candidate (`v1` / PR skill) via
`github/evals-action@fcc3794a85b0e248241991add59e2eb9c87fcd1d` and post
the sticky PR comparison comment automatically on PRs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio
jongio requested an adaptive review from Copilot November 16, 2026 07:59
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Evals: passed

Selection: Native eval paths or tags

Run: passed - 2 trials - Evidence

Baseline: failed - 2 trials - Evidence

Eval Trials Result
pr-description 2 passed

Quality & token summary

Eval Model Verdict Score Turns Duration Tokens
pr-description claude-haiku-5.5 passed 1.00 (threshold 1.00) 3 10735ms 58209 tokens (57165 in / 1044 out)

Baseline vs. Candidate comparison

Metric Baseline Candidate Delta
Outcome failed passed fixed
Passed 0 1 +1
Failed 1 0 -1
Turns 3 3 0
Duration 12920ms 10735ms -2185ms (-16.9%)
Tokens 58374 tokens (56954 in / 1420 out) 58209 tokens (57165 in / 1044 out) -165 tokens (-0.3%; out: -376 / -26.5%)
Eval Baseline Candidate Score Delta Turns Delta Duration Delta Token Delta Transition
pr-description failed (0.96) passed (1.00) +0.04 0 -2185ms (-16.9%) -165 tokens (-0.3%; out: -376 / -26.5%) fixed

Exact revision | Workflow run

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Both paid evaluation jobs execute PR-controlled setup and graders with a Copilot token without the evals environment gate, bypassing configured reviewer protections.

3 open issues

Review effort: Balanced (dynamic)

@jongio
jongio requested a review from wbreza November 16, 2026 08:09
- Keep automatic same-repository PR comparison free of a required
  `environment: evals` gate and align `README.md` and
  `docs/checks-and-evals.md` (`doc-doctor`).
- Only stage `SKILL.verbose-fixture.md` in `skill-eval-comparison.yml`
  on `workflow_dispatch` or when the fixture itself changed in the PR.
- Update the pinned `github/evals-action#34` commit SHA to `3c55133`.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…l_request

Reserve `SKILL.verbose-fixture.md` staging for `workflow_dispatch` and
always restore `$BASE_SHA:$dir` on `pull_request`.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Automatic PR-controlled evaluations use Copilot credentials and paid resources, while baseline source substitution and cross-job artifact reporting create interacting execution boundaries that require human review.

No open issues 🎉

3 resolved since last review

Review effort: Balanced (dynamic)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Automatic paid PR evaluations combine credential-bearing execution, baseline restoration, and cross-job artifact reporting through an unreleased action revision, requiring substantive integration review.

No open issues 🎉

Review effort: Balanced (dynamic)

@jongio
jongio merged commit b6a495f into main Nov 16, 2026
10 checks passed
@jongio
jongio deleted the feat/skill-eval-comparison branch November 16, 2026 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants