You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(evals): real Vally 0.21.0 pr-description runs, output-token deltas, and PR comparison docs - #21
Updates the pr-description evaluation golden path with real Vally 0.21.0 (claude-haiku-5.5) baseline and candidate model runs, adds Model, Skills Invoked, Turns, Output Tokens, and Agent Wall Time pass-through reporting to eval-report.mjs, and documents automatic PR Baseline vs. Candidate skill comparison using github/evals-action@v0 (baseline-report-id: evals-baseline, see github/evals-action#34):
evals/pr-description/skills/pr-description/eval.yaml & evals/_shared/fixtures/pr-description/SKILL.verbose-fixture.md: Tightens the stimulus prompt and v0 pre-token-diet fixture so live claude-haiku-5.5 trials via copilot-sdk reliably activate pr-description on the positive case (1/2 FAIL on v0 -> 2/2 PASS on v1, cutting positive-case output tokens by 56.2% from 918 to 402 and agent wall time by 36.2% from 11.1s to 7.1s).
evals/_shared/fixtures/pr-description/{baseline,candidate}-results.jsonl: Replaces the synthetic baseline-run-001 fixtures with normalized real Vally 0.21.0 (claude-haiku-5.5) run records (63a64377-917f-44df-8515-81db5f3c9b31 and ab6a6e4d-091c-4a1a-ac94-952db5ab00ef).
.github/scripts/eval-report.mjs & .github/scripts/eval-report.test.mjs: Passes through native Vally results.jsonl fields: Model (claude-haiku-5.5), Skills Invoked (pr-description vs none), Turns / Turn Delta, Output Tokens (1,447 -> 991, -456 (-31.5%)), and Agent Wall Time / Wall Time Delta (-4.0s (-36.2%) on the positive case).
.github/workflows/evals.yml, README.md, & docs/checks-and-evals.md: Documents automatic PR Baseline vs. Candidate skill comparison (plugins/<plugin>/skills/<skill>/ <-> evals/<plugin>/skills/<skill>/eval.yaml) and local npm run eval:compare.
…deltas, and PR comparison docs
- Update `evals/pr-description/skills/pr-description/eval.yaml` and
`SKILL.verbose-fixture.md` so live `claude-haiku-5.5` runs reliably
activate `pr-description` (`1/2` FAIL on `v0` -> `2/2` PASS on `v1`,
`-56.2%` positive-case output tokens).
- Replace synthetic `baseline-results.jsonl` and `candidate-results.jsonl`
with real Vally 0.21.0 (`claude-haiku-5.5`) runs (`63a64377` and
`ab6a6e4d`).
- Add `Output Tokens` delta reporting in `.github/scripts/eval-report.mjs`
and document automatic PR Baseline vs. Candidate skill comparison via
`github/evals-action@v0` (`baseline-report-id: evals-baseline`).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Updates the marketplace’s pr-description evaluation example with recorded Vally runs, output-token comparisons, and guidance for opt-in PR evaluation workflows.
Changes:
Replaces synthetic results with recorded baseline and candidate runs and refines evaluation prompts.
Adds output-token deltas and separates skill-specific tests from report tests.
Documents local comparisons and an automatic PR workflow template while keeping the existing workflow manual-only.
…me in eval-report
Pass through native Vally `results.jsonl` fields (`model`,
`trajectory.metrics.skillActivationBreakdown`, `trajectory.metrics.turnCount`,
and `trajectory.metrics.wallTimeMs`) in single-run and Baseline vs.
Candidate comparison tables.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Token classification can report reductions in both comparison directions, and the documented workflow relies on unsupported baseline-comparison functionality.
Require a released action with baseline-report-id support
README.md:216
github/evals-action@v0 does not support baseline-report-id; the implementation in github/evals-action#34 is still unmerged, so this setup cannot yet produce the promised comparison. Mark it as pending a release containing that feature before users enable paid baseline runs. Same issue in .github/workflows/evals.yml:6–7.
Forward COPILOT_CLI_PATH to both eval action runs
docs/checks-and-evals.md:391
github/evals-action@v0 strips COPILOT_CLI_PATH when constructing Vally's environment, so setting it in env does not select the repository-pinned launcher. Add credential-env: COPILOT_CLI_PATH to both run steps to forward it explicitly. Same issue in the candidate step at docs/checks-and-evals.md:411.
Keep baseline behavior failures from failing the workflow
docs/checks-and-evals.md:392
A behaviorally failing baseline makes this step fail, leaving the workflow red even when the candidate fixes every baseline failure. Make baseline behavioral failures non-blocking, while still failing on execution, configuration, or artifact-upload errors and leaving candidate failures blocking.
…son recipe
- Evaluate `totalTokens` (`<` / `>`) before `outputTokens` in
`classifyTransition` and use `outputTokens` as a tie-breaker only when
`totalTokens` is equal.
- Pin the `docs/checks-and-evals.md` comparison recipe to
`github/evals-action@fcc3794a85b0e248241991add59e2eb9c87fcd1d`
(`github/evals-action#34`), remove unused `COPILOT_CLI_PATH` / `npm ci`
from the Action steps, and set `continue-on-error: true` on the
baseline run step.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Tie parity transition to total-token delta (concern:38303302210355d4, concern:b0d387c90524e472, concern:9dd1dd35da7cf580, reviews 5916409495, 5916544515): Fixed in 1995814 (.github/scripts/eval-report.mjs, .github/scripts/eval-report.test.mjs). classifyTransition evaluates totalTokens first and uses outputTokens only as a tie-breaker when totalTokens is equal.
Qualify eval recipe until comparison support is released / Require a released action with baseline-report-id support (concern:a272c8e5feb33682, concern:3daecd9bf87b67d9, reviews 5916409495, 5916544515): Fixed in 1995814 and b66a309 (docs/checks-and-evals.md, README.md, .github/workflows/evals.yml).
Forward COPILOT_CLI_PATH to both eval action runs (concern:4b70f8c16fdad34d, review 5916544515): Fixed in 1995814 (docs/checks-and-evals.md).
Keep baseline behavior failures from failing the workflow (concern:97295b4affc0ae1b, review 5916544515): Fixed in 1995814 (docs/checks-and-evals.md).
Automatic PR comparison trust & execution boundaries (concern:be8b1df85c490e20, concern:2f0e9ed699573fc1, concern:aa087579132df9f2, concern:f9cbd9b2a7a96682, concern:12a43ca1f0115bf6, concern:861ba6b297e620e8, concern:97eb6ce865a31688, reviews 5916631230, 5916722588, 5916915322, 5916977593): Addressed in b66a309 and 8f8a7cd (.github/workflows/skill-eval-comparison.yml, docs/checks-and-evals.md, README.md): fork PRs are excluded (head.repo.full_name == github.repository), pull_request runs strictly restore $BASE_SHA:$dir, and reporting runs in a separate pull-requests: write job with SHA-256 artifact digest verification.
…-action#34
Run Baseline (`v0` / base skill) vs. Candidate (`v1` / PR skill) via
`github/evals-action@fcc3794a85b0e248241991add59e2eb9c87fcd1d` and post
the sticky PR comparison comment automatically on PRs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Both paid evaluation jobs execute PR-controlled setup and graders with a Copilot token without the evals environment gate, bypassing configured reviewer protections.
- Keep automatic same-repository PR comparison free of a required
`environment: evals` gate and align `README.md` and
`docs/checks-and-evals.md` (`doc-doctor`).
- Only stage `SKILL.verbose-fixture.md` in `skill-eval-comparison.yml`
on `workflow_dispatch` or when the fixture itself changed in the PR.
- Update the pinned `github/evals-action#34` commit SHA to `3c55133`.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Automatic PR-controlled evaluations use Copilot credentials and paid resources, while baseline source substitution and cross-job artifact reporting create interacting execution boundaries that require human review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates the
pr-descriptionevaluation golden path with real Vally 0.21.0 (claude-haiku-5.5) baseline and candidate model runs, addsModel,Skills Invoked,Turns,Output Tokens, andAgent Wall Timepass-through reporting toeval-report.mjs, and documents automatic PR Baseline vs. Candidate skill comparison usinggithub/evals-action@v0(baseline-report-id: evals-baseline, seegithub/evals-action#34):evals/pr-description/skills/pr-description/eval.yaml&evals/_shared/fixtures/pr-description/SKILL.verbose-fixture.md: Tightens the stimulus prompt andv0pre-token-diet fixture so liveclaude-haiku-5.5trials viacopilot-sdkreliably activatepr-descriptionon the positive case (1/2FAIL onv0->2/2PASS onv1, cutting positive-case output tokens by 56.2% from918to402and agent wall time by 36.2% from11.1sto7.1s).evals/_shared/fixtures/pr-description/{baseline,candidate}-results.jsonl: Replaces the syntheticbaseline-run-001fixtures with normalized real Vally 0.21.0 (claude-haiku-5.5) run records (63a64377-917f-44df-8515-81db5f3c9b31andab6a6e4d-091c-4a1a-ac94-952db5ab00ef)..github/scripts/eval-report.mjs&.github/scripts/eval-report.test.mjs: Passes through native Vallyresults.jsonlfields:Model(claude-haiku-5.5),Skills Invoked(pr-descriptionvsnone),Turns/Turn Delta,Output Tokens(1,447->991,-456 (-31.5%)), andAgent Wall Time/Wall Time Delta(-4.0s (-36.2%)on the positive case)..github/workflows/evals.yml,README.md, &docs/checks-and-evals.md: Documents automatic PR Baseline vs. Candidate skill comparison (plugins/<plugin>/skills/<skill>/<->evals/<plugin>/skills/<skill>/eval.yaml) and localnpm run eval:compare.Behavioral eval report
Saved run:
evals/_shared/fixtures/pr-description/candidate-results.jsonl| Model:claude-haiku-5.5Baseline vs. Candidate Comparison
Baseline:
evals/_shared/fixtures/pr-description/baseline-results.jsonl| Candidate:evals/_shared/fixtures/pr-description/candidate-results.jsonlValidation
node --test .github/scripts/eval-report.test.mjs .github/scripts/pr-description.test.mjs .github/scripts/lint.test.mjs .github/scripts/validation.test.mjs— passed (34/34tests passing)node .github/scripts/eval-report.mjs --baseline evals/_shared/fixtures/pr-description/baseline-results.jsonl --candidate evals/_shared/fixtures/pr-description/candidate-results.jsonl— passed (exit code0)