Repository navigation
Prepare repository for open source release - #1
Conversation
Add the community files OSPO asks for and remove links to private repositories from what the template copies into new repositories. - Add CODE_OF_CONDUCT.md and SUPPORT.md from the OSPO templates. - Add GitHub's coordinated disclosure channel to SECURITY.md in a template-only block, so adopters keep their own placeholder. - Name @github/code-ai-evals in CODEOWNERS, keeping the adopter examples. - Add status, Background, Requirements, Maintainers, Support, and Acknowledgement to the README. GitHub-specific sections are template-only. - Add the license and Code of Conduct lines to CONTRIBUTING.md. - Repoint the remote Evals plugin from github/evals to github/evals-dist (eval, copilot-evals-v0.20.3) and update its docs and tests. - Make the npm runner the default in evals.yml; the Evals Action is opt-in with EVALS_RUNNER=action until it's public. - Link the public GitHub Copilot Evals docs instead of private repos. - Warn in verify, after a rename, when CODEOWNERS or CODE_OF_CONDUCT.md still names the template's maintainers or contact. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The docs site isn't public, so a public repository can't link to it. Drop the aka.ms/evals/site links from the README, AGENTS.md, SUPPORT.md, the evals and remote plugin docs, .vally.yaml, and the remote eval plugin's catalog homepage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The validator still matches the default enforcement address inside an HTML comment, causing a false warning even after the actual contact address is replaced.
1 open issue
What changed in this PR
This draft prepares the marketplace template for open-source release by adding community policies, clarifying ownership, and reducing reliance on private resources while preserving adoption guidance.
Changes:
- Add support and conduct policies, security-reporting guidance, and maintainer ownership.
- Repoint the optional Evals plugin to
github/evals-distand make npm the default CI eval runner. - Add adoption warnings and regression checks, and remove private documentation links.
| File | Description |
|---|---|
test/marketplace.test.mjs |
Tests warnings for unchanged template owners and conduct contact. |
test/eval-setup.test.mjs |
Updates assertions for the npm-default runner selection. |
test/catalog.test.mjs |
Updates expectations for the renamed remote Evals plugin. |
test/adoption.test.mjs |
Checks removal of the updated remote-plugin guidance. |
SUPPORT.md |
Adds support channels and policy. |
SECURITY.md |
Adds GitHub disclosure and Safe Harbor guidance. |
scripts/validate.mjs |
Warns adopters about retained template owners and contact. |
scripts/catalog.mjs |
Uses a neutral repository example in validation errors. |
README.md |
Adds release-readiness sections and updates adoption and eval guidance. |
marketplace.json |
Repoints and upgrades the optional Evals plugin. |
docs/security.md |
Clarifies the Evals Action’s publication status. |
docs/remote-plugins.md |
Updates installation instructions and records the publication prerequisite. |
docs/evals.md |
Removes private upstream links while retaining convention guidance. |
docs/ci.md |
Documents npm-default evals and Action opt-in. |
CONTRIBUTING.md |
Adds licensing, conduct, and contributor-agreement guidance. |
CODE_OF_CONDUCT.md |
Adds the Contributor Covenant and enforcement contact. |
AGENTS.md |
Removes the private Evals Action reference. |
.vally.yaml |
Updates a comment describing eval filename conventions. |
.github/workflows/evals.yml |
Makes npm the default and the Evals Action opt-in. |
.github/CODEOWNERS |
Assigns template ownership while retaining adopter examples. |
Review effort: Max
github/evals-dist is being renamed to github/evals, which will be the public distribution repo, so restore the original github-copilot catalog entry and its docs and tests instead of pointing at github/evals-dist. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github/evals and github/evals-action open alongside this repository, so drop the "isn't public yet" wording and TODOs from the eval workflow, README, CI, remote plugin, and security docs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Revert changes no Prepare Project item requires: the npm default runner in evals.yml, the verify warnings for the template's code owners and conduct contact, the CODEOWNERS header rewording, the CLA note in CONTRIBUTING.md, the extra adopter steps in the README, the catalog homepage anchor, and link and wording edits for github/evals and github/evals-action, which open alongside this repository. Use the OSPO SECURITY.md, SUPPORT.md, CODE_OF_CONDUCT.md and CONTRIBUTING.md template text without extras. Keep only the minimum edits that drop the not-public-yet disclaimers and the internal docs links. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github/evals was renamed, and the old pin (copilot-evals-v0.16.0 at dd4c501) isn't a commit in the new github/evals, so npm run check-remotes failed. Pin the latest release, copilot-evals-v0.20.3 at acd4b50. Its plugin.json is named "eval", and check-remotes requires the catalog entry name to match, so rename the entry from github-copilot to eval and update the markers, remove-plugin and install commands, docs, and tests that use the old name. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
jongio
left a comment
There was a problem hiding this comment.
Reviewed all 14 changed files at this head in an isolated worktree. Two actionable findings, both reproduced through the production renameMarketplace and staleReferenceWarnings functions on a disposable copy:
[P2] Preserve the adoption warning when the template team remains CODEOWNER
.github/CODEOWNERS:14-15.
Adding the active * @github/code-ai-evals rule makes the existing missing-owner warning stop working for template adopters: adoptionWarnings only warns when every CODEOWNERS rule is commented out, while renameMarketplace does not rewrite this file. Renaming to a different owner/repository therefore retains the template's actual team and emits no CODEOWNERS warning, unlike the previous commented-only template. The README's manual owner-replacement step is useful, but the validator no longer catches a skipped step. Warn when an adopted marketplace still names the template's team, or require an explicit owner replacement, and cover this in the adoption regression.
[P2] Include the new conduct-reporting contact in template adoption
CODE_OF_CONDUCT.md:55-58.
The new enforcement address is not inside a template-only block and is neither rewritten nor flagged during adoption. The README's replacement checklist mentions SECURITY.md and CODEOWNERS but not this new conduct contact. After the documented rename, a new project's policy still directs complaints to opensource@github.com rather than its own maintainers, potentially misrouting sensitive reports. Add an explicit adopter replacement/placeholder and a corresponding leftover warning, while keeping the real address for this template repository.
Review boundary: no upstream code was fetched or installed, and no paid evals were run. The changed github/evals pin and its upstream installation behavior remain unverified because that repository is blocked; this review does not certify those external changes.

Gets this template ready for the OSPO open source release, with only the changes the Prepare Project checklist asks for. OSPO request: github/open-source-releases#744.
Removed the not-public-yet disclaimers (feedback from @jongio).
Prepare Project checklist
github/evalsandgithub/evals-action. No other internal links, paths, codenames, telemetry or emails.@github/employeeswith at least read@github/employees(Read) and@github/code-ai-evals(Maintain).v1.0.0after this merges. No packages.Copilot-Sessiontrailers.LICENSEis MIT,Copyright (c) 2026 GitHub, Inc.Not touched.* @github/code-ai-evals.GitHub-only text sits in
<!-- template-only -->blocks, sonpm run rename-marketplacestrips it from repositories made from this template.Also in this PR
github-copilotcatalog entry togithub/evalscopilot-evals-v0.20.3(acd4b50) and renamed it toeval, the name in that release'splugin.json, whichcheck-remotesrequires. The old pin pointed to a commit that no longer exists after the repo rename, sonpm run check-remotesfailed. It passes now. Ronnie asked for this; it isn't a checklist item.Changes in this PR
README.md: item 9 (new sections), item 1 (drops the private wording and the "currently private" note)..github/CODEOWNERS: item 10.CONTRIBUTING.md: item 11.SUPPORT.md: item 12.CODE_OF_CONDUCT.md: item 13.SECURITY.md: item 14.docs/remote-plugins.md: item 1 (drops the internal install guide link and the private wording).docs/ci.md,docs/security.md,.github/workflows/evals.yml,test/eval-setup.test.mjs: item 1 (drops the private wording only).Tests
npm run verify: passes, 148 pass and 1 skipped, same asmain.npm run check-remotes: passes. It still fails onmain, where the pinnedgithub-copilotSHA (dd4c501) isn't a commit in the renamedgithub/evals.Out of scope follow-ups
github/evals-actionisn't public when this repo goes public, make the npm runner the default inevals.yml.#installed-skillshomepage anchor and thegithub-copilot@evalsinstall command against the renamedgithub/evals.