Skip to content

chore(marketplace): update eval remote pin to copilot-evals-v0.24.41 - #24

Merged
jongio merged 3 commits into
mainfrom
repo-maintain/automated
Nov 17, 2026
Merged

jongio merged 3 commits into
mainfrom
repo-maintain/automated

Conversation

@jongio

@jongio jongio commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Update marketplace.json's �val remote pin to copilot-evals-v0.24.41 (9edf65253f9ced590401bd99618d52230fee88b2).
  • Eliminate version/SHA churn outside marketplace.json so future �val pin updates only touch marketplace.json:
    • .vally.yaml: stage marketplace.json as .vally-eval-pin.json in �nvironments.marketplace, validate and sparse-clone the pinned
      epo/
      ef/sha dynamically into .github/skills, and remove both temporary paths before the trial baseline is captured.
    • README.md & docs/checks-and-evals.md: link upstream github/evals docs and license via �lob/main/... while keeping marketplace.json as the audited commit pin.
    • .github/scripts/backend-defaults.test.mjs: remove redundant hardcoded version comment.

Validation

ode .github/scripts/check-remote-pins.mjs (�val: github/evals@9edf652 (packages/plugin/plugin.json) ok)

ode .github/scripts/check-local-catalog.mjs

  • �ash .github/scripts/validate-schema.sh

pm run lint

ode --test .github/scripts/backend-defaults.test.mjs .github/scripts/eval-report.test.mjs .github/scripts/pr-description.test.mjs

  • Temp-directory smoke test of .vally.yaml's �nvironments.marketplace setup command (6 upstream skills staged cleanly; .vally-eval-source and .vally-eval-pin.json removed)

…and remove version churn

Update marketplace.json's eval remote plugin pin to copilot-evals-v0.24.41
(9edf65253f9ced590401bd99618d52230fee88b2) and make .vally.yaml, README.md,
docs/checks-and-evals.md, and backend-defaults.test.mjs read or link without
hardcoded version/SHA churn so future pin updates only touch marketplace.json.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jongio
jongio requested a review from a team as a code owner November 16, 2026 22:51
@jongio jongio self-assigned this Nov 16, 2026
Copilot AI adaptive review requested due to automatic review settings November 16, 2026 22:51
Comment thread .vally.yaml Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new source URL in docs/checks-and-evals.md returns not found, leaving the project-configuration compatibility claim without a working source reference.

1 open issue
What changed in this PR

Updates the optional GitHub Copilot Evals plugin to 0.24.41 and centralizes its release pin in marketplace.json to reduce future update churn.

Changes:

  • Updates the remote plugin’s version, release ref, and commit SHA.
  • Makes eval setup read the catalog pin dynamically and clean up temporary staging files.
  • Removes duplicated version references from documentation links and a test comment.
File Description
README.md Points upstream documentation and license links to main.
marketplace.json Pins Evals 0.24.41 and its release commit.
docs/​checks-and-evals.md References the catalog pin instead of repeating its version.
.vally.yaml Reads and validates the catalog pin before staging upstream skills.
.github/​scripts/​backend-defaults.test.mjs Removes a version-specific comment; test behavior is unchanged.

Review effort: Balanced (adaptive)

Comment thread docs/checks-and-evals.md Resolved
…er doc ref

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The Evals upgrade from 0.24.6 to 0.24.41 changes executable runtime code alongside per-trial provisioning, requiring deeper compatibility and data-handling review beyond confirming the tag, SHA, and manifest match.

No open issues 🎉

1 resolved since last review

Review effort: Balanced (adaptive)

@wbreza wbreza left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Verified the eval remote pin update against github/evals:

  • marketplace.json pin (0.24.41): copilot-evals-v0.24.41 resolves to 9edf65253f9ced590401bd99618d52230fee88b2 (latest non-draft prerelease on the copilot-evals channel). packages/plugin/plugin.json at that commit matches name (eval), version (0.24.41), description, and Agent Plugins schema 1.0.0.
  • Upstream changes (0.24.6 → 0.24.41): mcp.json, LICENSE, the six skill directories under packages/plugin/skills/, and the .vally.yaml root-key allowlist (environments, params, metadata, paths, suites) remain compatible with .github/scripts/backend-defaults.test.mjs.
  • README.md links: blob/main/README.md, blob/main/README.md#install, and blob/main/LICENSE all resolve on github/evals main.

Leaving Comment (per autopilot review policy) for two Medium issues in the churn-reduction changes:

  1. .vally.yaml (Medium): Dynamic eval pin loading breaks local evals if a template adopter removes the optional eval plugin from marketplace.json, omits upstream source/repo/path allowlisting before cloning into .github/skills, and throws TypeError when optional source.ref is omitted (or fails checkout when a branch ref advances past source.sha).
  2. docs/checks-and-evals.md (Medium): https://github.com/github/evals/blob/main/packages/plugin/dist/mcp/server.mjs returns 404 Not Found (main has no packages/ tree), and in 0.24.41 the .vally.yaml root-key reader moved from dist/mcp/server.mjs into packages/plugin/com.github.copilot/extensions/copilot-evals/workbench.mjs.

Comment thread .vally.yaml Resolved
Comment thread docs/checks-and-evals.md Resolved

@wbreza wbreza left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary (Re-review at e7d966a)

Commit e7d966a resolves both Medium issues from e0c6df3:

  • .vally.yaml: Enforces pin.repo === 'github/evals', pin.path === 'packages/plugin', copilot-evals-vX.Y.Z tag format, and 40-char hex pin.sha against https://github.com/github/evals.git, eliminating the untrusted-repo clone risk, pin.path drift, and optional-ref coercion failure.
  • docs/checks-and-evals.md: Replaces the broken blob/main/packages/... link with github/evals/releases and references the packaged MCP entrypoint via marketplace.json's source.sha.
  • Upstream 0.24.6 → 0.24.41 verification: Confirmed copilot-evals-v0.24.41 (9edf65253f9ced590401bd99618d52230fee88b2), packages/plugin/plugin.json, mcp.json, LICENSE, TELEMETRY.md (no root/SDK hooks added), skill updates, and the .vally.yaml root-key allowlist (environments, params, metadata, paths, suites) remain compatible. All CI checks pass.

Optional Low / non-blocking notes:

  • In .vally.yaml, if a downstream template adopter removes the optional eval entry from marketplace.json, returning early inside try / finally (rather than throwing when the entry is absent) would let local plugin evals run without editing .vally.yaml.
  • In 0.24.41, packages/plugin/dist/mcp/server.mjs delegates the .vally.yaml root-key reader check to packages/plugin/com.github.copilot/extensions/copilot-evals/workbench.mjs.

…bench.mjs doc ref

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@wbreza wbreza left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary (Re-review at d347d75)

LGTM! Commit d347d75 cleanly incorporates the remaining follow-ups:

  • .vally.yaml: Wraps pin inspection in try / finally, skips upstream staging cleanly when the optional eval entry is removed from marketplace.json, enforces pin.source === 'github' && pin.repo === 'github/evals' && pin.path === 'packages/plugin' plus optional pin.ref validation, and sparse-fetches pin.sha directly before removing .vally-eval-source and .vally-eval-pin.json.
  • docs/checks-and-evals.md: Accurately cites packages/plugin/com.github.copilot/extensions/copilot-evals/workbench.mjs (loaded by packages/plugin/dist/mcp/server.mjs) at the pinned source.ref / source.sha in marketplace.json.
  • marketplace.json, README.md, and CI: copilot-evals-v0.24.41 (9edf65253f9ced590401bd99618d52230fee88b2) matches upstream packages/plugin/plugin.json, and lint, CodeQL, and github-advanced-security all pass.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The executable Evals MCP/CLI upgrade and catalog-driven skill staging require human review of runtime behavior, permissions, and workflow compatibility beyond the verified tag and manifest checks.

No open issues 🎉

Review effort: Balanced (adaptive)

@jongio
jongio merged commit 7027b31 into main Nov 17, 2026
6 checks passed
@jongio
jongio deleted the repo-maintain/automated branch November 17, 2026 06:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants