Repository navigation
chore(marketplace): update eval remote pin to copilot-evals-v0.24.41 - #24
Conversation
…and remove version churn Update marketplace.json's eval remote plugin pin to copilot-evals-v0.24.41 (9edf65253f9ced590401bd99618d52230fee88b2) and make .vally.yaml, README.md, docs/checks-and-evals.md, and backend-defaults.test.mjs read or link without hardcoded version/SHA churn so future pin updates only touch marketplace.json. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The new source URL in docs/checks-and-evals.md returns not found, leaving the project-configuration compatibility claim without a working source reference.
1 open issue
What changed in this PR
Updates the optional GitHub Copilot Evals plugin to 0.24.41 and centralizes its release pin in marketplace.json to reduce future update churn.
Changes:
- Updates the remote plugin’s version, release ref, and commit SHA.
- Makes eval setup read the catalog pin dynamically and clean up temporary staging files.
- Removes duplicated version references from documentation links and a test comment.
| File | Description |
|---|---|
README.md |
Points upstream documentation and license links to main. |
marketplace.json |
Pins Evals 0.24.41 and its release commit. |
docs/checks-and-evals.md |
References the catalog pin instead of repeating its version. |
.vally.yaml |
Reads and validates the catalog pin before staging upstream skills. |
.github/scripts/backend-defaults.test.mjs |
Removes a version-specific comment; test behavior is unchanged. |
Review effort: Balanced (adaptive)
…er doc ref Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The Evals upgrade from 0.24.6 to 0.24.41 changes executable runtime code alongside per-trial provisioning, requiring deeper compatibility and data-handling review beyond confirming the tag, SHA, and manifest match.
No open issues 🎉
1 resolved since last review
Review effort: Balanced (adaptive)
wbreza
left a comment
There was a problem hiding this comment.
Review Summary
Verified the eval remote pin update against github/evals:
marketplace.jsonpin (0.24.41):copilot-evals-v0.24.41resolves to9edf65253f9ced590401bd99618d52230fee88b2(latest non-draft prerelease on thecopilot-evalschannel).packages/plugin/plugin.jsonat that commit matchesname(eval),version(0.24.41),description, and Agent Plugins schema1.0.0.- Upstream changes (
0.24.6→0.24.41):mcp.json,LICENSE, the six skill directories underpackages/plugin/skills/, and the.vally.yamlroot-key allowlist (environments,params,metadata,paths,suites) remain compatible with.github/scripts/backend-defaults.test.mjs. README.mdlinks:blob/main/README.md,blob/main/README.md#install, andblob/main/LICENSEall resolve ongithub/evalsmain.
Leaving Comment (per autopilot review policy) for two Medium issues in the churn-reduction changes:
.vally.yaml(Medium): Dynamicevalpin loading breaks local evals if a template adopter removes the optionalevalplugin frommarketplace.json, omits upstreamsource/repo/pathallowlisting before cloning into.github/skills, and throwsTypeErrorwhen optionalsource.refis omitted (or fails checkout when a branch ref advances pastsource.sha).docs/checks-and-evals.md(Medium):https://github.com/github/evals/blob/main/packages/plugin/dist/mcp/server.mjsreturns404 Not Found(mainhas nopackages/tree), and in0.24.41the.vally.yamlroot-key reader moved fromdist/mcp/server.mjsintopackages/plugin/com.github.copilot/extensions/copilot-evals/workbench.mjs.
wbreza
left a comment
There was a problem hiding this comment.
Review Summary (Re-review at e7d966a)
Commit e7d966a resolves both Medium issues from e0c6df3:
.vally.yaml: Enforcespin.repo === 'github/evals',pin.path === 'packages/plugin',copilot-evals-vX.Y.Ztag format, and 40-char hexpin.shaagainsthttps://github.com/github/evals.git, eliminating the untrusted-repo clone risk,pin.pathdrift, and optional-refcoercion failure.docs/checks-and-evals.md: Replaces the brokenblob/main/packages/...link withgithub/evals/releasesand references the packaged MCP entrypoint viamarketplace.json'ssource.sha.- Upstream
0.24.6→0.24.41verification: Confirmedcopilot-evals-v0.24.41(9edf65253f9ced590401bd99618d52230fee88b2),packages/plugin/plugin.json,mcp.json,LICENSE,TELEMETRY.md(no root/SDK hooks added), skill updates, and the.vally.yamlroot-key allowlist (environments,params,metadata,paths,suites) remain compatible. All CI checks pass.
Optional Low / non-blocking notes:
- In
.vally.yaml, if a downstream template adopter removes the optionalevalentry frommarketplace.json, returning early insidetry / finally(rather than throwing when the entry is absent) would let local plugin evals run without editing.vally.yaml. - In
0.24.41,packages/plugin/dist/mcp/server.mjsdelegates the.vally.yamlroot-key reader check topackages/plugin/com.github.copilot/extensions/copilot-evals/workbench.mjs.
…bench.mjs doc ref Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
wbreza
left a comment
There was a problem hiding this comment.
Review Summary (Re-review at d347d75)
LGTM! Commit d347d75 cleanly incorporates the remaining follow-ups:
.vally.yaml: Wraps pin inspection intry / finally, skips upstream staging cleanly when the optionalevalentry is removed frommarketplace.json, enforcespin.source === 'github' && pin.repo === 'github/evals' && pin.path === 'packages/plugin'plus optionalpin.refvalidation, and sparse-fetchespin.shadirectly before removing.vally-eval-sourceand.vally-eval-pin.json.docs/checks-and-evals.md: Accurately citespackages/plugin/com.github.copilot/extensions/copilot-evals/workbench.mjs(loaded bypackages/plugin/dist/mcp/server.mjs) at the pinnedsource.ref/source.shainmarketplace.json.marketplace.json,README.md, and CI:copilot-evals-v0.24.41(9edf65253f9ced590401bd99618d52230fee88b2) matches upstreampackages/plugin/plugin.json, andlint,CodeQL, andgithub-advanced-securityall pass.
There was a problem hiding this comment.
🔵 Needs a closer look
The executable Evals MCP/CLI upgrade and catalog-driven skill staging require human review of runtime behavior, permissions, and workflow compatibility beyond the verified tag and manifest checks.
No open issues 🎉
Review effort: Balanced (adaptive)

Summary
epo/
ef/sha dynamically into .github/skills, and remove both temporary paths before the trial baseline is captured.
Validation
ode .github/scripts/check-remote-pins.mjs (�val: github/evals@9edf652 (packages/plugin/plugin.json) ok)
ode .github/scripts/check-local-catalog.mjs
pm run lint
ode --test .github/scripts/backend-defaults.test.mjs .github/scripts/eval-report.test.mjs .github/scripts/pr-description.test.mjs