Skip to content

Remove the GitHub Copilot Evals plugin and use Vally only for evals - #3

Merged
ronniegeraghty merged 3 commits into
mainfrom
ronniegeraghty/remove-evals-plugin
Oct 30, 2026
Merged

ronniegeraghty merged 3 commits into
mainfrom
ronniegeraghty/remove-evals-plugin

Conversation

@ronniegeraghty

Copy link
Copy Markdown
Collaborator

What and why

Follow-up to #1 for the OSPO release, github/open-source-releases#744. The catalog pointed at the GitHub Copilot Evals plugin in github/evals, and the evals workflow used github/evals-action by default. Neither repo is public, and github/evals has a different license, so this MIT template shouldn't depend on them. Eval guidance now relies only on Vally, which is public as the @microsoft/vally-cli npm package (MIT) with docs at https://microsoft.github.io/vally/.

Three commits:

  • Remove the GitHub Copilot Evals plugin from the catalog. Removed the remote eval entry with npm run remove-plugin -- eval, which also dropped its README setup step and its section in docs/remote-plugins.md. Removed the other github/evals mentions in docs/evals.md, docs/customization-types.md, .vally.yaml, and the tests. The remote plugin docs and checks stay, with the existing anarlog example.
  • Run CI evals with the pinned Vally CLI only. evals.yml now has one job that runs the same launcher as local runs (node scripts/evals.mjs eval --suite <suite> --require-pass). Removed the EVALS_RUNNER switch, the action-params command and its gitignored params files, and the upload-transcripts warning, since they only existed for the Action. Updated README.md, docs/ci.md, docs/security.md, and AGENTS.md to match, and they link to Vally's public docs.
  • Rebuild the JSON configuration server with fast-uri 3.1.9. The push reported 4 open Dependabot alerts on main, all in fast-uri, which is bundled into the shipped json-configuration MCP server. Updated it, rebuilt dist/ and its notices, and bumped the plugin to 0.1.7. Drop this commit if you'd rather let Dependabot do it.

Every external link in the repo now resolves publicly, except placeholders such as github.com/me/plugins and this repo's own issues page, which works once it's public. A scan found no other references to private repos, internal hosts, or internal tools.

Checklist

  • For each changed plugin, bumped version in plugin.json and marketplace.json, and added a CHANGELOG.md entry. json-configuration is now 0.1.7.
  • Kept the catalog description identical to the manifest, and the plugin enabled in .github/copilot/settings.json. N/A for the removed remote entry, which was never enabled.
  • Updated .vally.yaml paths.skills and params for added, renamed, or removed plugins. N/A: no local plugins added or removed.
  • Added or updated evals with invocation and outcome assertions, smoke coverage, and a negative control. N/A: only a comment in evals/customization-authoring/eval.yaml changed.
  • Updated the plugin's README and any other docs that describe what changed.
  • Ran npm run build if sources of generated files changed, and committed the dist/ output.
  • npm run verify passes.
  • Used only synthetic data; no secrets, tokens, or private code.

Risk

Question Answer
What runs, and when: hooks and their events, MCP servers, extensions, scripts No new code. The evals workflow runs one job instead of two, and no longer downloads github/evals-action.
Files it reads or writes, and how it limits them CI no longer writes eval.params.yaml files.
Network destinations none
Credentials or tokens it uses, and where they come from The eval job's token keeps contents: read and copilot-requests: write. pull-requests: write is gone with the Action job.
Data it keeps, where, and for how long No change. Results upload only when EVALS_UPLOAD_RESULTS is true.
Data it sends to the model none
New dependencies, their licenses, and install scripts none. fast-uri moves from 3.1.8 to 3.1.9 (BSD-3-Clause).
Remote sources: repository, path, and pinned sha Removed github/evals packages/plugin at acd4b50. No remote entries remain.

Paid evals

None.

Tests:

  • npm run verify: passes, 147 pass and 1 skipped. One test went away with action-params.
  • npm run check-remotes: passes, "marketplace.json has no remote plugins to check."
  • npm run eval:list -- --suite smoke: works.

ronniegeraghty and others added 3 commits October 30, 2026 10:00
github/evals isn't public yet and uses a different license, so this MIT
template shouldn't point at it. Removed the remote eval entry with
npm run remove-plugin, and dropped the docs, comments, and tests that
named github/evals. The eval file name guidance now cites Vally's
defaults.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github/evals-action isn't public, so the evals workflow now has one job
that runs the same launcher as local runs, with the Vally CLI pinned in
package.json. Removed the EVALS_RUNNER switch, the action-params command
and its gitignored params files, and the upload-transcripts warning,
which only applied to the Action. Docs now point at Vally's public docs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The bundled MCP server includes fast-uri, which had four open Dependabot
advisories (two moderate, two low). Updated it in the lockfile, rebuilt
the bundle and its notices, and bumped json-configuration to 0.1.7.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the removal of private evaluation dependencies and consolidation on Vally.

No open issues 🎉

What changed in this PR

Removes dependencies on the non-public GitHub Copilot Evals repositories for the open-source release, while retaining Vally-based evaluation tooling for the plugin marketplace.

Changes:

  • Removes the remote eval plugin and its documentation references.
  • Consolidates CI evals onto the pinned Vally launcher and removes Action-specific helpers and tests.
  • Updates bundled fast-uri to 3.1.9 and releases json-configuration 0.1.7.
File Description
test/​marketplace.test.mjs Removes the Action-specific transcript warning assertion.
test/​eval-setup.test.mjs Updates CI assertions and removes Action params tests.
test/​catalog.test.mjs Replaces eval-plugin-specific checks with generic remote validation.
test/​adoption.test.mjs Checks Vally params directly after plugin removal.
scripts/​validate.mjs Removes the Action-specific transcript warning.
scripts/​evals.mjs Removes the action-params command and generator.
scripts/​catalog.mjs Removes the params filename helper and updates example text.
README.md Removes remote eval setup and documents Vally-only CI.
plugins/​json-configuration/​plugin.json Bumps the plugin version to 0.1.7.
plugins/​json-configuration/​dist/​THIRD_PARTY_NOTICES.txt Updates the bundled fast-uri notice.
plugins/​json-configuration/​dist/​server.mjs Rebuilds with fast-uri fixes and the new plugin version.
plugins/​json-configuration/​CHANGELOG.md Records the dependency security update.
package-lock.json Locks fast-uri to 3.1.9.
marketplace.json Removes the remote eval entry and updates the local plugin version.
evals/​customization-authoring/​eval.yaml Updates only the comment explaining the Vally path argument.
docs/​security.md Describes data handling for the single eval job.
docs/​remote-plugins.md Removes the GitHub Copilot Evals installation section.
docs/​evals.md Replaces private-repository references with Vally guidance.
docs/​customization-types.md Removes the external eval plugin example.
docs/​ci.md Documents the single Vally job and removes runner selection.
AGENTS.md Updates maintenance guidance for Vally-only CI.
.vally.yaml Updates filename-convention comments without changing configuration.
.gitignore Removes obsolete generated params exclusions.
.github/​workflows/​evals.yml Uses one Vally launcher job and removes Action-specific permissions.

Review effort: Balanced (auto)

@ronniegeraghty
ronniegeraghty merged commit 668bea8 into main Oct 30, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants