Repository navigation
Commit cfedc89
authored
feat(ci): cap and triage pull requests from outside contributors (#2947)
Two changes to how pull requests from outside contributors are triaged, aimed at the influx of agent-generated PRs we've seen recently.
**Volume:** outside contributors can have one PR open at a time, three once they're on the vouch list. Anything beyond that is closed with a note asking them to finish what they already have open. One contributor opened five PRs in two days — nobody can review that properly, and in practice none of them land.
**Authenticity:** a PR from someone not on the list stays open but picks up a `needs-vouch` label and a comment asking them to introduce themselves, so reviewers can see at a glance what has and hasn't been vetted.
Nothing else is closed automatically, and issues stay open to everyone. Closes HDX-5122.
## What changed
- **A concurrent-PR cap for outside contributors.** One open PR, or three if they're vouched. Drafts don't count — they're not in the review queue yet. Over the cap, the newest PR is closed with a comment naming the ones already open; their existing work is untouched and they can reopen once one lands.
- **Unvouched PRs are flagged, not blocked.** `needs-vouch` label plus one comment pointing at the introduction issue template. The PR stays open.
- **Denounced authors are the only other auto-close.** `/denounce` is the escalation for repeat spam or bad faith.
- **Bots and anyone with write access are never touched**, so nothing changes for the team.
- **Issues are not gated**, on purpose. Bug reports from strangers are worth having, and an issue is how someone introduces themselves to get vouched.
- **Maintainers manage the list from comments** — `/vouch @user`, `/unvouch @user`, `/denounce @user` on any issue or PR. The bot opens a PR against the list; merging it applies the change.
- **The list is seeded with 47 handles**: the `hyperdxio` org plus every external contributor with two or more merged PRs.
- **Two issue templates**, `introduce-yourself` (the on-ramp the comment links to) and `bug-report`. The repo had none before, so `/issues/new` now shows a chooser; blank issues remain enabled.
## Key decisions
- **The cap is enforced by closing; the vouch check isn't.** These look inconsistent but they're different asks. Closing an excess PR doesn't reject the person or their work — it asks them to serialise, and their other PRs are untouched. A label alone wouldn't limit anything, which is the entire point of the cap. Rejecting someone *as a contributor* is the expensive mistake, so that path only labels.
- **The vouch list is the reputation signal for the higher allowance.** Jaeger's model is that contributing more earns more concurrent PRs. Rather than build a second notion of standing, being on the list is what moves you from one to three.
- **Two merged PRs as the seed threshold, not one.** One merged PR includes the drive-by contributions this is meant to filter. The 41 single-PR authors can each be vouched with one comment if they return.
- **`GITHUB_TOKEN` with PR mode, not a GitHub App.** An App would let the bot commit straight to protected `main`, but needs org admin and two secrets. PR mode needs neither and gives a human a look at every list change. The tradeoff is that the bot's PR needs an approval instead of a green CI run — GitHub doesn't run workflows on PRs it creates. The changesets release PR already works this way.
- **Actions pinned to a commit SHA** rather than the floating `@v1`, since the manage job hands the action a `contents: write` token on a trigger any commenter can reach.
- **Out of scope:** gating contributions to issues we've explicitly labelled `accepting-prs`. That needs us to own the issue backlog and agree a labelling convention first, so it belongs in its own PR.
## Background
[Vouch](https://github.com/mitchellh/vouch) is a small trust-list system by Mitchell Hashimoto, currently used by Ghostty. The list is a flat text file — one handle per line, `-handle` to block someone — and a set of GitHub Actions that check authors against it. It carries no policy of its own; who gets vouched and what that permits are entirely ours. It's not a security control: being vouched only affects how a PR is triaged, and review and merge permissions are unchanged.
## Impact
- An outside contributor's first PR gets a label and a comment. A second concurrent one gets closed with an explanation. Everyone already contributing regularly is on the seed list.
- **The `introduction` label doesn't exist yet.** GitHub silently drops unknown labels from issue templates, so introduction issues arrive unlabelled until someone creates it. (`needs-vouch` is created by the workflow on first use.)
- **Known gap:** `claude-code-review.yml` fires on the same `pull_request_target` event with `allowed_non_write_users: '*'`, so an unvouched fork PR still gets a full Claude review. Pre-existing and the flag looks deliberate, so it's left alone here — worth a follow-up to gate it on vouch status.
- Currently-open external PRs are untouched; the check only fires on open, reopen and ready-for-review. Note that an existing contributor with several PRs already open will have the next one closed.
- `external-contributor-alerts.yml` still posts to Slack and applies its own `external` label, independently. The two don't interfere.
<details>
<summary>Implementation detail</summary>
Both jobs live in one workflow with per-job permissions, since `check-pr` needs less than `manage` does.
Points worth a look:
- **`require-vouch: false` with `auto-close: true`** produces the split behaviour: the action blocks only denounced authors and returns `allowed` for unvouched ones. No checkout is needed at all — it reads `VOUCHED.td` from the default branch over the API.
- **One `github-script` step handles both checks.** The action's `vouched` status covers collaborators *and* listed outsiders, so it can't tell us who is internal; the step redoes the `author_association` plus collaborator-permission check that `external-contributor-alerts.yml` uses, including the slow path for org members with concealed membership. Sharing the step also means an author who is both unvouched and over the cap gets one comment, not two.
- **Open PRs are counted by paginating `pulls.list`**, not the search API, which lags its index and would let someone slip past the cap by opening two in quick succession.
- **The label is the "already asked" marker**, so a reopen or ready-for-review doesn't post the vouch comment twice. Same trick `external-contributor-alerts.yml` uses with `external`.
- **Drafts are skipped and re-checked on `ready_for_review`**, matching `external-contributor-alerts.yml`.
- **The `manage` job is role-gated in the `if:`** before the runner starts. The action re-checks the commenter's permission authoritatively; the workflow-level check just stops anyone on the internet from starting a write-token runner. `contains()` rather than `startsWith()` so a leading space doesn't silently drop a real command.
- **Concurrency is keyed per comment.** A shared group lets a newly queued run cancel a pending one, which would drop a vouch with no feedback anywhere.
- **A `failure()` step comments back on the issue.** The action is silent on every failure path and `issue_comment` runs aren't surfaced on the issue timeline.
- **The list is seeded lower-case.** Vouch lower-cases handles when it rewrites the file, so mixed case would turn the first bot edit into a diff that looks like it renamed six people.
- **`copilot` is on the list.** Vouch only auto-exempts logins ending in `[bot]`; GitHub's Copilot coding agent has the login `Copilot` (confirmed on #2500).
Reviewed with four parallel reviewers covering Actions security, action wiring, adversarial abuse paths and repo standards. All 47 seeded handles checked against the GitHub API. No changeset — `AGENTS.md` exempts CI and docs changes.
</details>1 parent 828b277 commit cfedc89
6 files changed
Lines changed: 419 additions & 0 deletions
File tree
- .github
- ISSUE_TEMPLATE
- workflows
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
0 commit comments