Skip to content

Prepare repository for open source release - #1

Merged
ronniegeraghty merged 6 commits into
mainfrom
ronniegeraghty/ospo-release-prep
Oct 30, 2026
Merged

ronniegeraghty merged 6 commits into
mainfrom
ronniegeraghty/ospo-release-prep

Conversation

@ronniegeraghty

@ronniegeraghty ronniegeraghty commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Gets this template ready for the OSPO open source release, with only the changes the Prepare Project checklist asks for. OSPO request: github/open-source-releases#744.

Removed the not-public-yet disclaimers (feedback from @jongio).

Prepare Project checklist

# Item Status Details or owner
1 Review and address items in the release guidelines 🔧 Done in this PR Removed the link to the internal install guide and the "private" and "not public yet" wording about github/evals and github/evals-action. No other internal links, paths, codenames, telemetry or emails.
2 Add @github/employees with at least read ✅ Already met @github/employees (Read) and @github/code-ai-evals (Maintain).
3 Secret scanning, Dependabot security updates, CodeQL ✅ Already met Secret scanning and push protection on, Dependabot security updates on, CodeQL default setup on.
4 Check with #rai for AI related code 👤 Manual, outside this PR Ronnie.
5 Description, tags, release, packages, link 👤 Manual, outside this PR Description and topics are set. Ronnie cuts v1.0.0 after this merges. No packages.
6 Tag the product managers in the OSPO issue ✅ Already met @RyanHecht, @pierceboggan and @digitarald are tagged in #744.
7 Remove sensitive materials in history, issues, PRs 👤 Manual, outside this PR Ronnie. Nothing sensitive found; two commits carry Copilot-Session trailers.
8 Add LICENSE (MIT) ✅ Already met LICENSE is MIT, Copyright (c) 2026 GitHub, Inc. Not touched.
9 README sections from the template 🔧 Done in this PR Adds status, Background, Requirements, Maintainers, Support and Acknowledgement. License was already there as Licensing.
10 CODEOWNERS lists maintainers 🔧 Done in this PR * @github/code-ai-evals.
11 Add and update CONTRIBUTING.md 🔧 Done in this PR Adds the template's license and Code of Conduct lines.
12 Add and update SUPPORT.md 🔧 Done in this PR New, from the template.
13 Add CODE_OF_CONDUCT.md 🔧 Done in this PR New, the template as is.
14 Add SECURITY.md 🔧 Done in this PR Adds the template's reporting channel (opensource-security[@]github.com).
15 Branch protection: PR and 1 approval from CODEOWNERS ✅ Already met Ruleset "Protect default branch": PR, 1 approval, code owner review.

GitHub-only text sits in <!-- template-only --> blocks, so npm run rename-marketplace strips it from repositories made from this template.

Also in this PR

  • Re-pinned the github-copilot catalog entry to github/evals copilot-evals-v0.20.3 (acd4b50) and renamed it to eval, the name in that release's plugin.json, which check-remotes requires. The old pin pointed to a commit that no longer exists after the repo rename, so npm run check-remotes failed. It passes now. Ronnie asked for this; it isn't a checklist item.

Changes in this PR

  • README.md: item 9 (new sections), item 1 (drops the private wording and the "currently private" note).
  • .github/CODEOWNERS: item 10.
  • CONTRIBUTING.md: item 11.
  • SUPPORT.md: item 12.
  • CODE_OF_CONDUCT.md: item 13.
  • SECURITY.md: item 14.
  • docs/remote-plugins.md: item 1 (drops the internal install guide link and the private wording).
  • docs/ci.md, docs/security.md, .github/workflows/evals.yml, test/eval-setup.test.mjs: item 1 (drops the private wording only).

Tests

  • npm run verify: passes, 148 pass and 1 skipped, same as main.
  • npm run check-remotes: passes. It still fails on main, where the pinned github-copilot SHA (dd4c501) isn't a commit in the renamed github/evals.

Out of scope follow-ups

  • If github/evals-action isn't public when this repo goes public, make the npm runner the default in evals.yml.
  • Tell adopters to replace the template's CODEOWNERS team, Code of Conduct contact and SUPPORT.md.
  • Check the #installed-skills homepage anchor and the github-copilot@evals install command against the renamed github/evals.

ronniegeraghty and others added 2 commits October 27, 2026 13:53
Add the community files OSPO asks for and remove links to private
repositories from what the template copies into new repositories.

- Add CODE_OF_CONDUCT.md and SUPPORT.md from the OSPO templates.
- Add GitHub's coordinated disclosure channel to SECURITY.md in a
  template-only block, so adopters keep their own placeholder.
- Name @github/code-ai-evals in CODEOWNERS, keeping the adopter examples.
- Add status, Background, Requirements, Maintainers, Support, and
  Acknowledgement to the README. GitHub-specific sections are template-only.
- Add the license and Code of Conduct lines to CONTRIBUTING.md.
- Repoint the remote Evals plugin from github/evals to github/evals-dist
  (eval, copilot-evals-v0.20.3) and update its docs and tests.
- Make the npm runner the default in evals.yml; the Evals Action is
  opt-in with EVALS_RUNNER=action until it's public.
- Link the public GitHub Copilot Evals docs instead of private repos.
- Warn in verify, after a rename, when CODEOWNERS or CODE_OF_CONDUCT.md
  still names the template's maintainers or contact.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The docs site isn't public, so a public repository can't link to it.
Drop the aka.ms/evals/site links from the README, AGENTS.md, SUPPORT.md,
the evals and remote plugin docs, .vally.yaml, and the remote eval
plugin's catalog homepage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@ronniegeraghty
ronniegeraghty marked this pull request as ready for review October 27, 2026 21:28
Copilot AI max review requested due to automatic review settings October 27, 2026 21:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The validator still matches the default enforcement address inside an HTML comment, causing a false warning even after the actual contact address is replaced.

1 open issue
What changed in this PR

This draft prepares the marketplace template for open-source release by adding community policies, clarifying ownership, and reducing reliance on private resources while preserving adoption guidance.

Changes:

  • Add support and conduct policies, security-reporting guidance, and maintainer ownership.
  • Repoint the optional Evals plugin to github/evals-dist and make npm the default CI eval runner.
  • Add adoption warnings and regression checks, and remove private documentation links.
File Description
test/​marketplace.test.mjs Tests warnings for unchanged template owners and conduct contact.
test/​eval-setup.test.mjs Updates assertions for the npm-default runner selection.
test/​catalog.test.mjs Updates expectations for the renamed remote Evals plugin.
test/​adoption.test.mjs Checks removal of the updated remote-plugin guidance.
SUPPORT.md Adds support channels and policy.
SECURITY.md Adds GitHub disclosure and Safe Harbor guidance.
scripts/​validate.mjs Warns adopters about retained template owners and contact.
scripts/​catalog.mjs Uses a neutral repository example in validation errors.
README.md Adds release-readiness sections and updates adoption and eval guidance.
marketplace.json Repoints and upgrades the optional Evals plugin.
docs/​security.md Clarifies the Evals Action’s publication status.
docs/​remote-plugins.md Updates installation instructions and records the publication prerequisite.
docs/​evals.md Removes private upstream links while retaining convention guidance.
docs/​ci.md Documents npm-default evals and Action opt-in.
CONTRIBUTING.md Adds licensing, conduct, and contributor-agreement guidance.
CODE_OF_CONDUCT.md Adds the Contributor Covenant and enforcement contact.
AGENTS.md Removes the private Evals Action reference.
.vally.yaml Updates a comment describing eval filename conventions.
.github/​workflows/​evals.yml Makes npm the default and the Evals Action opt-in.
.github/​CODEOWNERS Assigns template ownership while retaining adopter examples.

Review effort: Max

Comment thread scripts/validate.mjs Resolved
ronniegeraghty and others added 4 commits October 27, 2026 14:39
github/evals-dist is being renamed to github/evals, which will be the
public distribution repo, so restore the original github-copilot catalog
entry and its docs and tests instead of pointing at github/evals-dist.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github/evals and github/evals-action open alongside this repository, so
drop the "isn't public yet" wording and TODOs from the eval workflow,
README, CI, remote plugin, and security docs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Revert changes no Prepare Project item requires: the npm default runner in evals.yml, the verify warnings for the template's code owners and conduct contact, the CODEOWNERS header rewording, the CLA note in CONTRIBUTING.md, the extra adopter steps in the README, the catalog homepage anchor, and link and wording edits for github/evals and github/evals-action, which open alongside this repository.

Use the OSPO SECURITY.md, SUPPORT.md, CODE_OF_CONDUCT.md and CONTRIBUTING.md template text without extras. Keep only the minimum edits that drop the not-public-yet disclaimers and the internal docs links.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github/evals was renamed, and the old pin (copilot-evals-v0.16.0 at
dd4c501) isn't a commit in the new github/evals, so npm run check-remotes
failed. Pin the latest release, copilot-evals-v0.20.3 at acd4b50.

Its plugin.json is named "eval", and check-remotes requires the catalog
entry name to match, so rename the entry from github-copilot to eval and
update the markers, remove-plugin and install commands, docs, and tests
that use the old name.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@jongio jongio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 14 changed files at this head in an isolated worktree. Two actionable findings, both reproduced through the production renameMarketplace and staleReferenceWarnings functions on a disposable copy:

[P2] Preserve the adoption warning when the template team remains CODEOWNER

.github/CODEOWNERS:14-15.

Adding the active * @github/code-ai-evals rule makes the existing missing-owner warning stop working for template adopters: adoptionWarnings only warns when every CODEOWNERS rule is commented out, while renameMarketplace does not rewrite this file. Renaming to a different owner/repository therefore retains the template's actual team and emits no CODEOWNERS warning, unlike the previous commented-only template. The README's manual owner-replacement step is useful, but the validator no longer catches a skipped step. Warn when an adopted marketplace still names the template's team, or require an explicit owner replacement, and cover this in the adoption regression.

[P2] Include the new conduct-reporting contact in template adoption

CODE_OF_CONDUCT.md:55-58.

The new enforcement address is not inside a template-only block and is neither rewritten nor flagged during adoption. The README's replacement checklist mentions SECURITY.md and CODEOWNERS but not this new conduct contact. After the documented rename, a new project's policy still directs complaints to opensource@github.com rather than its own maintainers, potentially misrouting sensitive reports. Add an explicit adopter replacement/placeholder and a corresponding leftover warning, while keeping the real address for this template repository.

Review boundary: no upstream code was fetched or installed, and no paid evals were run. The changed github/evals pin and its upstream installation behavior remain unverified because that repository is blocked; this review does not certify those external changes.

@ronniegeraghty
ronniegeraghty merged commit feac137 into main Oct 30, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants