Repository navigation
Remove the GitHub Copilot Evals plugin and use Vally only for evals - #3
Merged
Merged
Conversation
github/evals isn't public yet and uses a different license, so this MIT template shouldn't point at it. Removed the remote eval entry with npm run remove-plugin, and dropped the docs, comments, and tests that named github/evals. The eval file name guidance now cites Vally's defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github/evals-action isn't public, so the evals workflow now has one job that runs the same launcher as local runs, with the Vally CLI pinned in package.json. Removed the EVALS_RUNNER switch, the action-params command and its gitignored params files, and the upload-transcripts warning, which only applied to the Action. Docs now point at Vally's public docs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The bundled MCP server includes fast-uri, which had four open Dependabot advisories (two moderate, two low). Updated it in the lockfile, rebuilt the bundle and its notices, and bumped json-configuration to 0.1.7. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are recorded for the removal of private evaluation dependencies and consolidation on Vally.
No open issues 🎉
What changed in this PR
Removes dependencies on the non-public GitHub Copilot Evals repositories for the open-source release, while retaining Vally-based evaluation tooling for the plugin marketplace.
Changes:
- Removes the remote
evalplugin and its documentation references. - Consolidates CI evals onto the pinned Vally launcher and removes Action-specific helpers and tests.
- Updates bundled
fast-urito 3.1.9 and releasesjson-configuration0.1.7.
| File | Description |
|---|---|
test/marketplace.test.mjs |
Removes the Action-specific transcript warning assertion. |
test/eval-setup.test.mjs |
Updates CI assertions and removes Action params tests. |
test/catalog.test.mjs |
Replaces eval-plugin-specific checks with generic remote validation. |
test/adoption.test.mjs |
Checks Vally params directly after plugin removal. |
scripts/validate.mjs |
Removes the Action-specific transcript warning. |
scripts/evals.mjs |
Removes the action-params command and generator. |
scripts/catalog.mjs |
Removes the params filename helper and updates example text. |
README.md |
Removes remote eval setup and documents Vally-only CI. |
plugins/json-configuration/plugin.json |
Bumps the plugin version to 0.1.7. |
plugins/json-configuration/dist/THIRD_PARTY_NOTICES.txt |
Updates the bundled fast-uri notice. |
plugins/json-configuration/dist/server.mjs |
Rebuilds with fast-uri fixes and the new plugin version. |
plugins/json-configuration/CHANGELOG.md |
Records the dependency security update. |
package-lock.json |
Locks fast-uri to 3.1.9. |
marketplace.json |
Removes the remote eval entry and updates the local plugin version. |
evals/customization-authoring/eval.yaml |
Updates only the comment explaining the Vally path argument. |
docs/security.md |
Describes data handling for the single eval job. |
docs/remote-plugins.md |
Removes the GitHub Copilot Evals installation section. |
docs/evals.md |
Replaces private-repository references with Vally guidance. |
docs/customization-types.md |
Removes the external eval plugin example. |
docs/ci.md |
Documents the single Vally job and removes runner selection. |
AGENTS.md |
Updates maintenance guidance for Vally-only CI. |
.vally.yaml |
Updates filename-convention comments without changing configuration. |
.gitignore |
Removes obsolete generated params exclusions. |
.github/workflows/evals.yml |
Uses one Vally launcher job and removes Action-specific permissions. |
Review effort: Balanced (auto)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Follow-up to #1 for the OSPO release, github/open-source-releases#744. The catalog pointed at the GitHub Copilot Evals plugin in
github/evals, and the evals workflow usedgithub/evals-actionby default. Neither repo is public, andgithub/evalshas a different license, so this MIT template shouldn't depend on them. Eval guidance now relies only on Vally, which is public as the@microsoft/vally-clinpm package (MIT) with docs at https://microsoft.github.io/vally/.Three commits:
evalentry withnpm run remove-plugin -- eval, which also dropped its README setup step and its section indocs/remote-plugins.md. Removed the othergithub/evalsmentions indocs/evals.md,docs/customization-types.md,.vally.yaml, and the tests. The remote plugin docs and checks stay, with the existinganarlogexample.evals.ymlnow has one job that runs the same launcher as local runs (node scripts/evals.mjs eval --suite <suite> --require-pass). Removed theEVALS_RUNNERswitch, theaction-paramscommand and its gitignored params files, and theupload-transcriptswarning, since they only existed for the Action. UpdatedREADME.md,docs/ci.md,docs/security.md, andAGENTS.mdto match, and they link to Vally's public docs.main, all infast-uri, which is bundled into the shipped json-configuration MCP server. Updated it, rebuiltdist/and its notices, and bumped the plugin to 0.1.7. Drop this commit if you'd rather let Dependabot do it.Every external link in the repo now resolves publicly, except placeholders such as
github.com/me/pluginsand this repo's own issues page, which works once it's public. A scan found no other references to private repos, internal hosts, or internal tools.Checklist
versioninplugin.jsonandmarketplace.json, and added aCHANGELOG.mdentry. json-configuration is now 0.1.7..github/copilot/settings.json. N/A for the removed remote entry, which was never enabled..vally.yamlpaths.skillsandparamsfor added, renamed, or removed plugins. N/A: no local plugins added or removed.evals/customization-authoring/eval.yamlchanged.npm run buildif sources of generated files changed, and committed thedist/output.npm run verifypasses.Risk
github/evals-action.eval.params.yamlfiles.contents: readandcopilot-requests: write.pull-requests: writeis gone with the Action job.EVALS_UPLOAD_RESULTSistrue.fast-urimoves from 3.1.8 to 3.1.9 (BSD-3-Clause).github/evalspackages/pluginatacd4b50. No remote entries remain.Paid evals
None.
Tests:
npm run verify: passes, 147 pass and 1 skipped. One test went away withaction-params.npm run check-remotes: passes, "marketplace.json has no remote plugins to check."npm run eval:list -- --suite smoke: works.