Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 7 additions & 85 deletions .github/workflows/evals.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
name: Behavioral evals
# Off by default, because evals run real Copilot sessions and spend AI credits.
# Pull requests run evals only after you set the repository variable
# ENABLE_EVALS to true. Manual runs work either way. The repository variable
# EVALS_RUNNER picks the job: unset or "action" uses github/evals-action;
# "npm" runs the pinned Vally CLI from package.json. Any other value fails. Both eval
# jobs use the "evals" environment, so you can require a reviewer to approve
# each run before it gets a token. See docs/ci.md.
# ENABLE_EVALS to true. Manual runs work either way. The job runs the pinned
# Vally CLI from package.json through the same launcher contributors use
# locally. It uses the "evals" environment, so you can require a reviewer to
# approve each run before it gets a token. See docs/ci.md.
on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
Expand All @@ -28,91 +27,14 @@ on:
permissions:
contents: read
jobs:
# Fails fast on a mistyped EVALS_RUNNER instead of silently picking a runner.
check-runner:
if: vars.EVALS_RUNNER != '' && vars.EVALS_RUNNER != 'action' && vars.EVALS_RUNNER != 'npm'
runs-on: ubuntu-latest
permissions: {}
steps:
- run: |
echo "::error::EVALS_RUNNER is '$EVALS_RUNNER'. Set it to action, npm, or nothing. See docs/ci.md."
exit 1
env:
EVALS_RUNNER: ${{ vars.EVALS_RUNNER }}

# Runner "action". A separate job, because a job downloads every action it
# names before it starts, even ones in skipped steps.
evals:
# Manual runs always run. Pull requests run only when ENABLE_EVALS is true,
# the branch is in this repository (never a fork), and the PR isn't a draft.
if: >-
(vars.EVALS_RUNNER == '' || vars.EVALS_RUNNER == 'action') && (github.event_name == 'workflow_dispatch' ||
(vars.ENABLE_EVALS == 'true' &&
github.event.pull_request.head.repo.full_name == github.repository &&
!github.event.pull_request.draft))
runs-on: ubuntu-latest
# Add required reviewers to this environment to approve each run before it starts.
environment: evals
timeout-minutes: 30
env:
VALLY_TELEMETRY_OPTOUT: '1'
permissions:
contents: read
copilot-requests: write
pull-requests: write
concurrency:
group: evals-${{ github.workflow_ref }}
cancel-in-progress: false
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: 24
cache: npm
# The Action installs only Vally. Graders, eval MCP hosts, and the pinned
# full Copilot runtime for custom-agent delegation come from npm ci.
- run: npm ci --no-audit --no-fund
# The Action can't pass --param, so write gitignored eval-level params files
# with this runner's absolute paths from .vally.yaml params.
- run: node scripts/evals.mjs action-params
# Fail fast if the runtime bridge the Action forwards can't start the pinned
# runtime, instead of reporting confusing eval failures later. It prints
# the pinned package version and the version that actually runs.
- name: Check the runtime bridge
run: node "$COPILOT_CLI_PATH" --version
env:
COPILOT_CLI_PATH: ${{ github.workspace }}/scripts/eval-runtime.js
- uses: github/evals-action@83fa12332c3e44ad899f347eaf153a0d9b6432dc # v0.1.0
env:
# Vally's SDK executor launches this bridge, which runs the pinned full runtime.
COPILOT_CLI_PATH: ${{ github.workspace }}/scripts/eval-runtime.js
with:
working-directory: .
suite: ${{ inputs.suite || 'smoke' }}
vally-version: '0.19.0'
copilot-token: ${{ github.token }}
# Workaround: the Action forwards only an allowlist of variables to Vally,
# and credential-env is its only way to pass another one, during the run
# phase only. COPILOT_CLI_PATH is a non-secret path, not a credential.
# Naming anything here turns off automatic token injection, so
# copilot-token is set explicitly above.
# VALLY_TELEMETRY_OPTOUT turns off Vally's telemetry during the run phase.
credential-env: |
COPILOT_CLI_PATH
VALLY_TELEMETRY_OPTOUT
workers: 1
max-retries: 1
upload-transcripts: false

# Runner "npm": the same launcher contributors use locally, with the job token.
evals-npm:
if: >-
vars.EVALS_RUNNER == 'npm' && (github.event_name == 'workflow_dispatch' ||
github.event_name == 'workflow_dispatch' ||
(vars.ENABLE_EVALS == 'true' &&
github.event.pull_request.head.repo.full_name == github.repository &&
!github.event.pull_request.draft))
!github.event.pull_request.draft)
runs-on: ubuntu-latest
# Add required reviewers to this environment to approve each run before it starts.
environment: evals
Expand All @@ -137,7 +59,7 @@ jobs:
- run: npm ci --no-audit --no-fund
- name: Check the runtime bridge
run: node scripts/eval-runtime.js --version
# --require-pass fails the job on a failed verdict, like the Action does.
# --require-pass fails the job on a failed verdict.
# The launcher uses one worker and one bounded retry.
- name: Run evals
run: node scripts/evals.mjs eval --suite "$SUITE" --require-pass
Expand Down
4 changes: 0 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,3 @@ vally-results/
.vally/
# Isolated Copilot homes from local install checks.
.copilot-home/
# Written by `node scripts/evals.mjs action-params` in CI. To commit a params file
# you wrote yourself, add it with `git add -f`; action-params won't overwrite it.
evals/**/eval.params.yaml
evals/**/*.eval.params.yaml
4 changes: 2 additions & 2 deletions .vally.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ paths:
- plugins/json-configuration/skills
evals:
- evals
# eval.yaml is this repository's convention; *.eval.yaml is the upstream
# github/evals convention. Both work.
# eval.yaml is Vally's default. *.eval.yaml lets one folder hold several
# specs, such as release.eval.yaml. Both work.
evalFilenames:
- eval.yaml
- "*.eval.yaml"
Expand Down
14 changes: 7 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,12 @@ place; don't duplicate them here.
- `npm run eval:smoke`, `npm run eval`, and the evals workflow spend AI
credits. Don't run them unless the user asks, and don't turn on
`ENABLE_EVALS` for anyone. Static checks don't establish behavioral quality.
- When upgrading Vally or Copilot, update the pins, the lockfile, and the
workflow's `vally-version` together, and verify real custom-agent delegation
and MCP behavior, not only schema validity. When you touch the workflows,
refresh every action's commit SHA (with its version comment), keep the
evals workflow on a Node major that `validate.yml` tests, and keep both eval
runner jobs (`action` and `npm`) in step.
- When upgrading Vally or Copilot, update the pins in `package.json` and the
lockfile together, and verify real custom-agent delegation and MCP behavior,
not only schema validity. CI runs evals through the same launcher as local
runs. When you touch the workflows, refresh every action's commit SHA (with
its version comment), and keep the evals workflow on a Node major that
`validate.yml` tests.
- Keep remote plugins opt-in. The repository settings overlay must never enable
a remote plugin; that's a deliberate security rule, so developers install
remote plugins themselves. A remote entry's `name` must match the upstream
Expand Down Expand Up @@ -105,7 +105,7 @@ Consult the relevant contract before changing package or catalog fields:
- [Creating a team marketplace](https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/plugins-marketplace).
- [CLI plugin and catalog reference](https://docs.github.com/en/copilot/reference/copilot-cli-reference/cli-plugin-reference).
- [Vally eval schema](https://microsoft.github.io/vally/reference/eval-spec/) and
[Evals Action](https://github.com/github/evals-action).
[Vally CI guide](https://microsoft.github.io/vally/get-started/add-to-ci/).

The README's [references](README.md#references) cover skills, hooks, VS Code,
and this repository's guides.
30 changes: 10 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,6 @@ Two terms come up a lot:
| [code-review](plugins/code-review/) | "Use adversarial-review to review this diff without changing it" | A skill that delegates to a read-only custom agent |
| [json-configuration](plugins/json-configuration/) | 'Use json-configuration to validate {"port":8443} and look up /port' | A skill backed by a bundled local MCP server |

<!-- plugin:eval -->
The catalog also has an optional reference to the GitHub Copilot Evals plugin,
which lives in another repository. See [Remote plugins](docs/remote-plugins.md).
<!-- /plugin:eval -->
<!-- examples -->
[Examples](docs/examples.md) explains what each example shows and which files
it owns.
Expand Down Expand Up @@ -104,12 +100,7 @@ You need Node.js 22.14 or newer.
4. Add your first plugin with `npm run new-plugin -- <name>`. See
[Add a plugin](#add-a-plugin). `npm run verify` passes even with no plugins
at all.
<!-- plugin:eval -->
5. Decide whether to keep the remote `eval` plugin in the catalog. See
[Remote plugins](docs/remote-plugins.md). Remove it with
`npm run remove-plugin -- eval`.
<!-- /plugin:eval -->
6. Replace what's meant for you before you ship:
5. Replace what's meant for you before you ship:
- Your own copyright line in [LICENSE](LICENSE), beside the template's. That
one file covers the whole repository, including every plugin. See
[Licensing](#licensing).
Expand All @@ -123,15 +114,15 @@ You need Node.js 22.14 or newer.
- Your maintenance rules in
[.github/copilot-instructions.md](.github/copilot-instructions.md) and
[AGENTS.md](AGENTS.md).
7. Run `npm run verify`. It fails on anything inconsistent and warns about
6. Run `npm run verify`. It fails on anything inconsistent and warns about
leftovers, such as the template's name or introduction, a `LICENSE` that
doesn't name you yet, scaffold text you haven't replaced, or a mention of a
plugin you removed.
8. Push, then turn on branch protection or a ruleset for your default branch.
7. Push, then turn on branch protection or a ruleset for your default branch.
Rulesets are available for public repositories on every plan. Behavioral
evals stay off in CI until you turn them on; see
[Continuous integration](#continuous-integration).
9. [Test installation](docs/local-development.md#test-installation-without-publishing)
8. [Test installation](docs/local-development.md#test-installation-without-publishing)
locally, then from your GitHub repository, before you share it.

For agent-assisted setup and maintenance, see [AGENTS.md](AGENTS.md).
Expand Down Expand Up @@ -259,11 +250,9 @@ marketplace consistent.
[evals.yml](.github/workflows/evals.yml) runs behavioral evals. It's off for
pull requests until you set the repository variable `ENABLE_EVALS` to `true`,
and you can always run it by hand. Each run can wait for a reviewer's approval
through the `evals` environment. By default it uses
[`github/evals-action`](https://github.com/github/evals-action). Set the
repository variable `EVALS_RUNNER` to `npm` to run
the same evals with the pinned Vally CLI. On a personal account, usage is
billed to your own Copilot plan. [Continuous integration](docs/ci.md) covers
through the `evals` environment. It runs the same launcher you use locally,
with the pinned Vally CLI. On a personal account, usage is billed to your own
Copilot plan. [Continuous integration](docs/ci.md) covers
setup, personal accounts, billing, cost, and security.

## Repository layout
Expand Down Expand Up @@ -354,5 +343,6 @@ to everyone who's contributed since.
- [Hooks reference](https://docs.github.com/en/copilot/reference/hooks-reference)
- [VS Code agent plugins](https://code.visualstudio.com/docs/agent-customization/agent-plugins)
- [Agent Skills specification](https://agentskills.io/specification)
- [Vally](https://microsoft.github.io/vally/) and the
[Evals Action](https://github.com/github/evals-action)
- [Vally](https://microsoft.github.io/vally/), its
[eval schema](https://microsoft.github.io/vally/reference/eval-spec/), and
[CI guide](https://microsoft.github.io/vally/get-started/add-to-ci/)
Loading
Loading