Skip to content

Commit cfedc89

Browse files
feat(ci): cap and triage pull requests from outside contributors (#2947)
Two changes to how pull requests from outside contributors are triaged, aimed at the influx of agent-generated PRs we've seen recently. **Volume:** outside contributors can have one PR open at a time, three once they're on the vouch list. Anything beyond that is closed with a note asking them to finish what they already have open. One contributor opened five PRs in two days — nobody can review that properly, and in practice none of them land. **Authenticity:** a PR from someone not on the list stays open but picks up a `needs-vouch` label and a comment asking them to introduce themselves, so reviewers can see at a glance what has and hasn't been vetted. Nothing else is closed automatically, and issues stay open to everyone. Closes HDX-5122. ## What changed - **A concurrent-PR cap for outside contributors.** One open PR, or three if they're vouched. Drafts don't count — they're not in the review queue yet. Over the cap, the newest PR is closed with a comment naming the ones already open; their existing work is untouched and they can reopen once one lands. - **Unvouched PRs are flagged, not blocked.** `needs-vouch` label plus one comment pointing at the introduction issue template. The PR stays open. - **Denounced authors are the only other auto-close.** `/denounce` is the escalation for repeat spam or bad faith. - **Bots and anyone with write access are never touched**, so nothing changes for the team. - **Issues are not gated**, on purpose. Bug reports from strangers are worth having, and an issue is how someone introduces themselves to get vouched. - **Maintainers manage the list from comments** — `/vouch @user`, `/unvouch @user`, `/denounce @user` on any issue or PR. The bot opens a PR against the list; merging it applies the change. - **The list is seeded with 47 handles**: the `hyperdxio` org plus every external contributor with two or more merged PRs. - **Two issue templates**, `introduce-yourself` (the on-ramp the comment links to) and `bug-report`. The repo had none before, so `/issues/new` now shows a chooser; blank issues remain enabled. ## Key decisions - **The cap is enforced by closing; the vouch check isn't.** These look inconsistent but they're different asks. Closing an excess PR doesn't reject the person or their work — it asks them to serialise, and their other PRs are untouched. A label alone wouldn't limit anything, which is the entire point of the cap. Rejecting someone *as a contributor* is the expensive mistake, so that path only labels. - **The vouch list is the reputation signal for the higher allowance.** Jaeger's model is that contributing more earns more concurrent PRs. Rather than build a second notion of standing, being on the list is what moves you from one to three. - **Two merged PRs as the seed threshold, not one.** One merged PR includes the drive-by contributions this is meant to filter. The 41 single-PR authors can each be vouched with one comment if they return. - **`GITHUB_TOKEN` with PR mode, not a GitHub App.** An App would let the bot commit straight to protected `main`, but needs org admin and two secrets. PR mode needs neither and gives a human a look at every list change. The tradeoff is that the bot's PR needs an approval instead of a green CI run — GitHub doesn't run workflows on PRs it creates. The changesets release PR already works this way. - **Actions pinned to a commit SHA** rather than the floating `@v1`, since the manage job hands the action a `contents: write` token on a trigger any commenter can reach. - **Out of scope:** gating contributions to issues we've explicitly labelled `accepting-prs`. That needs us to own the issue backlog and agree a labelling convention first, so it belongs in its own PR. ## Background [Vouch](https://github.com/mitchellh/vouch) is a small trust-list system by Mitchell Hashimoto, currently used by Ghostty. The list is a flat text file — one handle per line, `-handle` to block someone — and a set of GitHub Actions that check authors against it. It carries no policy of its own; who gets vouched and what that permits are entirely ours. It's not a security control: being vouched only affects how a PR is triaged, and review and merge permissions are unchanged. ## Impact - An outside contributor's first PR gets a label and a comment. A second concurrent one gets closed with an explanation. Everyone already contributing regularly is on the seed list. - **The `introduction` label doesn't exist yet.** GitHub silently drops unknown labels from issue templates, so introduction issues arrive unlabelled until someone creates it. (`needs-vouch` is created by the workflow on first use.) - **Known gap:** `claude-code-review.yml` fires on the same `pull_request_target` event with `allowed_non_write_users: '*'`, so an unvouched fork PR still gets a full Claude review. Pre-existing and the flag looks deliberate, so it's left alone here — worth a follow-up to gate it on vouch status. - Currently-open external PRs are untouched; the check only fires on open, reopen and ready-for-review. Note that an existing contributor with several PRs already open will have the next one closed. - `external-contributor-alerts.yml` still posts to Slack and applies its own `external` label, independently. The two don't interfere. <details> <summary>Implementation detail</summary> Both jobs live in one workflow with per-job permissions, since `check-pr` needs less than `manage` does. Points worth a look: - **`require-vouch: false` with `auto-close: true`** produces the split behaviour: the action blocks only denounced authors and returns `allowed` for unvouched ones. No checkout is needed at all — it reads `VOUCHED.td` from the default branch over the API. - **One `github-script` step handles both checks.** The action's `vouched` status covers collaborators *and* listed outsiders, so it can't tell us who is internal; the step redoes the `author_association` plus collaborator-permission check that `external-contributor-alerts.yml` uses, including the slow path for org members with concealed membership. Sharing the step also means an author who is both unvouched and over the cap gets one comment, not two. - **Open PRs are counted by paginating `pulls.list`**, not the search API, which lags its index and would let someone slip past the cap by opening two in quick succession. - **The label is the "already asked" marker**, so a reopen or ready-for-review doesn't post the vouch comment twice. Same trick `external-contributor-alerts.yml` uses with `external`. - **Drafts are skipped and re-checked on `ready_for_review`**, matching `external-contributor-alerts.yml`. - **The `manage` job is role-gated in the `if:`** before the runner starts. The action re-checks the commenter's permission authoritatively; the workflow-level check just stops anyone on the internet from starting a write-token runner. `contains()` rather than `startsWith()` so a leading space doesn't silently drop a real command. - **Concurrency is keyed per comment.** A shared group lets a newly queued run cancel a pending one, which would drop a vouch with no feedback anywhere. - **A `failure()` step comments back on the issue.** The action is silent on every failure path and `issue_comment` runs aren't surfaced on the issue timeline. - **The list is seeded lower-case.** Vouch lower-cases handles when it rewrites the file, so mixed case would turn the first bot edit into a diff that looks like it renamed six people. - **`copilot` is on the list.** Vouch only auto-exempts logins ending in `[bot]`; GitHub's Copilot coding agent has the login `Copilot` (confirmed on #2500). Reviewed with four parallel reviewers covering Actions security, action wiring, adversarial abuse paths and repo standards. All 47 seeded handles checked against the GitHub API. No changeset — `AGENTS.md` exempts CI and docs changes. </details>
1 parent 828b277 commit cfedc89

6 files changed

Lines changed: 419 additions & 0 deletions

File tree

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
---
2+
name: Bug report
3+
about: Something in HyperDX is broken or behaving unexpectedly
4+
title: ''
5+
labels: bug
6+
---
7+
8+
**What happened?**
9+
10+
<!-- What you saw, and what you expected instead. -->
11+
12+
**Steps to reproduce**
13+
14+
1.
15+
2.
16+
3.
17+
18+
**How are you running HyperDX?**
19+
20+
<!--
21+
The all-in-one Docker image, Docker Compose, Helm, ClickHouse Cloud, or a local
22+
dev stack — and which version or image tag. If you brought your own ClickHouse,
23+
say which version.
24+
-->
25+
26+
**Where does it show up?**
27+
28+
<!--
29+
The UI (which page, and which browser), the API, the OpenTelemetry collector, or
30+
the CLI. A screenshot or a short video helps a lot for UI bugs.
31+
-->
32+
33+
**Logs**
34+
35+
<!--
36+
Anything relevant from the browser console, the API container, or the collector.
37+
Please redact API keys and connection strings.
38+
-->
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
---
2+
name: Introduce yourself
3+
about: Say hello and get vouched so you can open pull requests
4+
title: 'Hello, I am <your GitHub handle>'
5+
labels: introduction
6+
---
7+
8+
Tell us a bit about yourself before your first pull request. A couple of
9+
sentences is plenty.
10+
11+
**Who are you?**
12+
13+
**What would you like to work on?**
14+
15+
<!--
16+
A bug that's biting you, something from
17+
https://github.com/hyperdxio/hyperdx/labels/good%20first%20issue, or your own
18+
idea. Not sure? Say so and we'll help you pick.
19+
-->
20+
21+
**Anything we can help with?**
22+
23+
<!--
24+
Dev setup, which package a change belongs in, or a steer on the approach before
25+
you write code. Ask here, or on Discord: https://discord.gg/FErRRKU78j
26+
-->
27+
28+
---
29+
30+
A maintainer will reply and vouch for you, usually within a day or two. Then you
31+
can open pull requests as normal. CONTRIBUTING.md has the dev setup if you want
32+
to get started in the meantime.

‎.github/VOUCHED.td‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# Vouched contributors for hyperdxio/hyperdx.
2+
#
3+
# Pull requests from GitHub users who are not in this list are closed
4+
# automatically by .github/workflows/vouch.yml. See CONTRIBUTING.md for how to
5+
# get vouched — it is deliberately easy.
6+
#
7+
# Org members and anyone else with write access are allowed regardless of this
8+
# list. The team is listed anyway so the file reads as a complete roster.
9+
#
10+
# Maintainers manage the list by commenting on any issue or PR:
11+
# /vouch @user [reason]
12+
# /denounce @user [reason]
13+
# /unvouch @user
14+
#
15+
# Syntax (https://github.com/mitchellh/vouch):
16+
# - One handle per line, without @, sorted alphabetically, lower case.
17+
# Vouch lower-cases handles when it rewrites the file, and matching is
18+
# case-insensitive, so seeding lower-case keeps bot edits to one line.
19+
# - Denounce with a leading minus: -username [reason]
20+
# - Optional platform prefix: github:username
21+
#
22+
# Seeded from the hyperdxio org plus every external contributor with two or
23+
# more merged pull requests as of August 2026.
24+
#
25+
# copilot is listed because vouch only auto-exempts logins ending in "[bot]".
26+
# GitHub's Copilot coding agent has the login "Copilot", so without an entry
27+
# its PRs would be closed.
28+
adityapimpalkar
29+
alex-fedotyev
30+
alok87
31+
aryainguz
32+
brandon-pereira
33+
colehpage
34+
copilot
35+
dhable
36+
dkong16
37+
elizabetdev
38+
ernestii
39+
fallintoplace
40+
fleon
41+
gingerwizard
42+
gyancr7
43+
jaggederest
44+
jeyrathnam
45+
jordan-simonovski
46+
juliojimenez
47+
jwhitaker-gridcog
48+
karl-power
49+
knudtty
50+
kyle-mally-ch
51+
leticiawebb
52+
lyhuang
53+
mark-omarov
54+
martincholuj
55+
michael-anastasakis
56+
mikeshi42
57+
mkaesz
58+
motsc
59+
niladrix719
60+
peter-leonov-ch
61+
pulpdrew
62+
rawatprince
63+
sanjams2
64+
shuvamk
65+
spencertorres
66+
teeohhem
67+
tommyzli
68+
tsushanth
69+
tylerhannan
70+
vineethasok
71+
vinzee
72+
wrn14897
73+
zeynelkoca
74+
zlareb1

‎.github/workflows/vouch.yml‎

Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
name: Vouch
2+
3+
# Trust gate for outside contributions, built on https://github.com/mitchellh/vouch.
4+
#
5+
# Anyone with write access (org members, collaborators) and bots are always
6+
# allowed. For everyone else this does two things:
7+
#
8+
# 1. Volume. Outside contributors get one open PR at a time, three once they
9+
# are on the vouch list. Anything over that is closed, asking them to
10+
# finish what they have open first. Drafts don't count.
11+
# 2. Authenticity. A PR from someone not in .github/VOUCHED.td stays open but
12+
# gets a `needs-vouch` label and a comment asking them to introduce
13+
# themselves.
14+
#
15+
# Authors explicitly denounced in the list are always closed. Issues are NOT
16+
# gated — opening one is how a new contributor introduces themselves.
17+
#
18+
# Maintainers manage the list from any issue or PR comment. The keyword has to
19+
# be first on the first line:
20+
# /vouch @user [reason] /denounce @user [reason] /unvouch @user
21+
#
22+
# The list edit arrives as a pull request because `main` is protected and
23+
# GITHUB_TOKEN cannot bypass that. Note the bot's PR does not trigger `Main` —
24+
# GitHub does not start workflow runs for events raised by GITHUB_TOKEN — so it
25+
# needs a maintainer approval rather than a green CI run. This is the same path
26+
# the changesets release PR already takes.
27+
28+
on:
29+
pull_request_target:
30+
types: [opened, reopened, ready_for_review]
31+
issue_comment:
32+
types: [created]
33+
34+
permissions: {}
35+
36+
jobs:
37+
check-pr:
38+
# Drafts are skipped and re-checked on ready_for_review, matching
39+
# external-contributor-alerts.yml so the two agree on what counts.
40+
if: >-
41+
github.event_name == 'pull_request_target' &&
42+
!github.event.pull_request.draft
43+
runs-on: ubuntu-latest
44+
timeout-minutes: 10
45+
permissions:
46+
contents: read
47+
issues: write
48+
pull-requests: write
49+
steps:
50+
# require-vouch: false means only denounced authors are blocked. Everyone
51+
# unvouched comes back as `allowed`, which the next step picks up. No
52+
# checkout needed — the action reads VOUCHED.td from the default branch
53+
# over the API, and the denounce message is upstream's, not a file here.
54+
- id: vouch
55+
uses: mitchellh/vouch/action/check-pr@d66fa29a64600490892131ad87597c30c91fcac4 # v1.5.0
56+
with:
57+
pr-number: ${{ github.event.pull_request.number }}
58+
require-vouch: false
59+
auto-close: true
60+
env:
61+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
62+
63+
# One step covers both checks so an author who is unvouched *and* over
64+
# the PR limit gets a single comment rather than two.
65+
- uses: actions/github-script@v9
66+
env:
67+
VOUCH_STATUS: ${{ steps.vouch.outputs.status }}
68+
with:
69+
script: |
70+
const LABEL = 'needs-vouch';
71+
const { owner, repo } = context.repo;
72+
const pr = context.payload.pull_request;
73+
const status = process.env.VOUCH_STATUS;
74+
75+
// skipped = bot. closed = denounced, already handled upstream.
76+
if (status === 'skipped' || status === 'closed') return;
77+
78+
// `vouched` covers both collaborators and listed outsiders, so it
79+
// cannot tell us who is internal. Same check as
80+
// external-contributor-alerts.yml, including the slow path that
81+
// catches org members with concealed membership.
82+
let internal = ['OWNER', 'MEMBER', 'COLLABORATOR'].includes(pr.author_association);
83+
if (!internal) {
84+
try {
85+
const { data } = await github.rest.repos.getCollaboratorPermissionLevel({
86+
owner, repo, username: pr.user.login,
87+
});
88+
const p = data.user?.permissions ?? {};
89+
internal = !!(p.admin || p.maintain || p.push || p.triage);
90+
} catch (e) {
91+
if (e.status !== 404) throw e; // 404 = not a collaborator => external
92+
}
93+
}
94+
if (internal) { core.info(`#${pr.number} is internal; nothing to do.`); return; }
95+
96+
// Volume cap, per Jaeger's model: contributing more earns more
97+
// concurrent PRs. Being on the vouch list is the reputation signal.
98+
const allowance = status === 'vouched' ? 3 : 1;
99+
const open = await github.paginate(github.rest.pulls.list, {
100+
owner, repo, state: 'open', per_page: 100,
101+
});
102+
// Drafts do not count — they are not asking for review yet, which
103+
// is the queue we are actually protecting.
104+
const theirs = open.filter(p =>
105+
p.user?.login === pr.user.login && p.number !== pr.number && !p.draft);
106+
107+
if (theirs.length >= allowance) {
108+
const list = theirs.map(p => `#${p.number}`).join(', ');
109+
await github.rest.issues.createComment({
110+
owner, repo, issue_number: pr.number,
111+
body: [
112+
`Hi @${pr.user.login}, thanks for this.`,
113+
``,
114+
`We limit outside contributors to ${allowance} open pull request${allowance === 1 ? '' : 's'} at a time, and you already have ${list} open. Reviewing several changes from one author at once is more than we can give proper attention to, and it tends to mean none of them land.`,
115+
``,
116+
`I have closed this one to keep the queue honest — **your branch and your work are untouched**. Once ${theirs.length === 1 ? 'that PR' : 'one of those'} is merged or closed, reopen this and it will go through.`,
117+
].join('\n'),
118+
});
119+
await github.rest.pulls.update({ owner, repo, pull_number: pr.number, state: 'closed' });
120+
core.info(`Closed #${pr.number}: ${theirs.length} already open, allowance ${allowance}.`);
121+
return;
122+
}
123+
124+
if (status !== 'allowed') return; // vouched and within the limit
125+
126+
// The label doubles as the "already asked" marker so a reopen or a
127+
// ready_for_review doesn't post the same comment a second time.
128+
if (pr.labels?.some(l => l.name === LABEL)) {
129+
core.info(`#${pr.number} already labelled; not commenting again.`);
130+
return;
131+
}
132+
133+
try {
134+
await github.rest.issues.createLabel({
135+
owner, repo, name: LABEL, color: 'FBCA04',
136+
description: 'Author needs a maintainer to vouch for them',
137+
});
138+
} catch (e) {
139+
if (e.status !== 422) throw e; // 422 = already exists
140+
}
141+
await github.rest.issues.addLabels({
142+
owner, repo, issue_number: pr.number, labels: [LABEL],
143+
});
144+
145+
const body = [
146+
`Hi @${pr.user.login}, thanks for the pull request!`,
147+
``,
148+
`Before we review code from a first-time contributor we ask that a maintainer vouches for you, and you're not on our list yet. **This PR stays open** — it just isn't in the review queue until someone vouches.`,
149+
``,
150+
`To get vouched, open an issue saying hello and what you're working on:`,
151+
``,
152+
`https://github.com/${owner}/${repo}/issues/new?template=introduce-yourself.md`,
153+
``,
154+
`A maintainer will usually reply within a day or two, and then this PR gets picked up as normal. More detail in [our contributing guide](https://github.com/${owner}/${repo}/blob/${pr.base.repo.default_branch}/CONTRIBUTING.md#getting-vouched).`,
155+
].join('\n');
156+
await github.rest.issues.createComment({
157+
owner, repo, issue_number: pr.number, body,
158+
});
159+
core.info(`Labelled and commented on #${pr.number}.`);
160+
161+
manage:
162+
# Role-gated before the runner starts. The action re-checks the commenter's
163+
# permission authoritatively; this only stops anyone on the internet from
164+
# spinning up a write-token runner by typing /vouch. `contains` rather than
165+
# `startsWith` so a leading space doesn't silently drop a real command.
166+
if: >-
167+
github.event_name == 'issue_comment' &&
168+
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'),
169+
github.event.comment.author_association) &&
170+
(contains(github.event.comment.body, '/vouch') ||
171+
contains(github.event.comment.body, '/unvouch') ||
172+
contains(github.event.comment.body, '/denounce'))
173+
runs-on: ubuntu-latest
174+
timeout-minutes: 10
175+
# Keyed per comment. A shared group would let a newly queued run cancel a
176+
# pending one, silently dropping a vouch; each run cuts its own branch, so
177+
# there is nothing to serialise.
178+
concurrency:
179+
group: vouch-manage-${{ github.event.comment.id }}
180+
cancel-in-progress: false
181+
permissions:
182+
contents: write
183+
issues: write
184+
pull-requests: write
185+
steps:
186+
- uses: actions/checkout@v6
187+
188+
- uses: mitchellh/vouch/action/manage-by-issue@d66fa29a64600490892131ad87597c30c91fcac4 # v1.5.0
189+
with:
190+
issue-id: ${{ github.event.issue.number }}
191+
comment-id: ${{ github.event.comment.id }}
192+
vouch-keyword: /vouch
193+
denounce-keyword: /denounce
194+
unvouch-keyword: /unvouch
195+
pull-request: true
196+
env:
197+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
198+
199+
# The action is silent on every failure path — no comment, no reaction —
200+
# and issue_comment runs are not surfaced on the issue timeline, so a
201+
# maintainer would otherwise assume a failed /vouch worked.
202+
- if: failure()
203+
uses: actions/github-script@v9
204+
with:
205+
script: |
206+
await github.rest.issues.createComment({
207+
...context.repo,
208+
issue_number: context.payload.issue.number,
209+
body: `The vouch command failed. [See the run](${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}) — the list was not changed.`,
210+
});

0 commit comments

Comments
 (0)