Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions .github/workflows/pr-title.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
# Validate every pull request title (and, when available, its commits) against
# the Conventional Commits shape release-please reads. Runs on every pull
# request but gates nothing; the same script is runnable locally, so a
# contributor sees the identical failure before pushing.
# Validate every pull request title (and, when available, its commits)
# against the Conventional Commits shape release-please reads, and check the
# pull request body and every commit message in it for agent attribution: a
# Co-Authored-By trailer naming a coding agent, a "generated with" banner
# naming one, or a link back to an agent session. Runs on every pull request
# but gates nothing; the same script is runnable locally, so a contributor
# sees the identical failure before pushing.
'name': 'PR Title Validation'

'on':
Expand Down Expand Up @@ -35,14 +38,16 @@
- 'name': 'Install'
'run': 'npm ci'

- 'name': 'Check PR title and commits'
- 'name': 'Check PR title, commits and attribution'
'shell': 'bash'
'env':
'PR_TITLE': '${{ github.event.pull_request.title }}'
'PR_BODY': '${{ github.event.pull_request.body }}'
'run': |
set -euo pipefail
upstream=refs/remotes/origin/${GITHUB_BASE_REF:-main}
base_sha=$(git merge-base "${upstream}" "${GITHUB_SHA}")
node scripts/check-pr-title.ts \
--title "${PR_TITLE}" \
--range "${base_sha}..${GITHUB_SHA}" || exit 1
--range "${base_sha}..${GITHUB_SHA}" \
--body "${PR_BODY}" || exit 1
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ npm run verify # lint, format, typecheck, ADR contract, tests + coverage
`npm run lint:adrs` alone runs the decision-record contract, and `npm test`
runs the suite without enforcing coverage. `npm run test:coverage` (part of
`npm run verify`) enforces the ratchet in `vitest.config.ts`: statements
97.71%, branches 89.78%, functions 100%, lines 97.51%.
97.88%, branches 90.27%, functions 100%, lines 97.69%.

## Conventions

Expand Down
3 changes: 2 additions & 1 deletion docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ test file and holds at least one test; ids are unique; the count this
document states matches the number of rows it holds; and every id cited
anywhere in the tracked tree resolves to a row here.

This ledger holds **11** rows. The compiler's behaviours join it as they land.
This ledger holds **12** rows. The compiler's behaviours join it as they land.

| id | a contributor or a consumer can rely on | proved by |
|---|---|---|
Expand All @@ -35,3 +35,4 @@ This ledger holds **11** rows. The compiler's behaviours join it as they land.
| REQ-009 | The npm package ships nothing outside `docs/adr/` and `spec/`, checked against what npm would really pack rather than the advisory `files` field | [test/package-contents-contract.test.ts](../test/package-contents-contract.test.ts) |
| REQ-010 | A gate's npm script and the CI job that runs it land in the same pull request, so neither can drift from the other unnoticed | [test/pipeline-wiring.test.ts](../test/pipeline-wiring.test.ts) |
| REQ-011 | Every script, path, coverage number and Node version README.md and CONTRIBUTING.md name matches the repository they describe | [test/docs-contract.test.ts](../test/docs-contract.test.ts) |
| REQ-012 | A pull request's title, body and every commit in it carry no agent attribution: no Co-Authored-By trailer naming a coding agent, no "generated with" banner naming one, no link back to an agent session | [test/pr-title-contract.test.ts](../test/pr-title-contract.test.ts) |
179 changes: 172 additions & 7 deletions scripts/check-pr-title.ts
Original file line number Diff line number Diff line change
@@ -1,14 +1,18 @@
// Validate a pull request title, or every commit in a range, against the
// Conventional Commits shape release-please reads, and explain the expected
// shape when one does not match.
// shape when one does not match. The same pass also scans the pull request
// body and every commit message in the range for agent attribution: a
// Co-Authored-By trailer naming a coding agent rather than a person, a
// "generated with" banner naming one, or a link back to an agent session.
//
// The set of accepted types is this script's single source of truth, and a
// test keeps it equal to the set the vendored commit-msg hook accepts. The
// workflow runs this on every pull request, and a contributor runs the same
// command before pushing, so the workflow enforces nothing that cannot be
// reproduced on a laptop:
//
// node scripts/check-pr-title.ts --title "feat(ci): add a workflow" [--range a..b]
// node scripts/check-pr-title.ts --title "feat(ci): add a workflow" \
// [--range a..b] [--body "pull request body text"]
import { execFileSync } from "node:child_process";
import { isEntrypoint } from "./lib/entrypoint.ts";
import { processOutput, type GateOutput } from "./lib/output.ts";
Expand Down Expand Up @@ -51,20 +55,160 @@ export function failureReason(subject: string): string | null {
);
}

// Coding-agent identifiers seen in real trailers, banners and links. Brand
// terms only: no bare human first name, because Cody, Devin and Jules are
// all names real people carry, so matching them alone would fail a genuine
// human co-author on a coincidence. Where an agent's own commits identify it
// through a distinctive bot account (`devin-ai-integration`,
// `google-labs-jules[bot]`) that account name is the marker instead of the
// plain name.
const AGENT_MARKERS: readonly string[] = [
"claude",
"anthropic",
"copilot",
"chatgpt",
"openai",
"codex",
"cursor",
"windsurf",
"aider",
"codeium",
"tabnine",
"devin-ai-integration",
"devin.ai",
"google-labs-jules",
"jules[bot]",
"amazon-q-developer",
"codewhisperer",
];

/** The first agent marker `text` contains, case-insensitively, or undefined. */
function agentMarker(text: string): string | undefined {
const lower = text.toLowerCase();
return AGENT_MARKERS.find((marker) => lower.includes(marker));
}

const CO_AUTHOR_LINE = /^co-authored-by:\s*(.+)$/i;
const BANNER_LINE = /\bgenerated\s+(with|by|using)\b/i;

/**
* Escape every regex metacharacter in `text` (including backslash itself),
* so it can be dropped into a `RegExp` and only ever match itself. The
* canonical escape-string-regexp shape: escaping only some characters (a
* dot, say) leaves the rest live, which is exactly what let a bare
* `chat.openai.com` also match `chatXopenai.com`.
*/
function escapeRegExp(text: string): string {
return text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}

const SESSION_HOSTS: readonly string[] = [
"claude.ai",
"chatgpt.com",
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
"chat.openai.com",
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
"devin.ai",
"cursor.sh",
"windsurf.com",
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
"windsurf.ai",
"aider.chat",
"jules.google.com",
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
];

// A host matches only at a real host boundary: right after `scheme://`, an
// optional run of `label.` subdomain segments (so `app.devin.ai` still
// matches `devin.ai`), the escaped host itself, and then `/`, `:`,
// whitespace or the end of the string. Unanchored, a bare host substring
// also matches a banned host sitting in another host's path
// (`https://evil.com/claude.ai/x`), which is a spoof in one direction, and
// an unescaped dot lets a host match one character off
// (`chatXopenai.com` for `chat.openai.com`), a spoof in the other.
const SESSION_LINK = new RegExp(
"https?://(?:[a-z0-9-]+\\.)*(?:" +
SESSION_HOSTS.map(escapeRegExp).join("|") +
")(?=[/:\\s]|$)",
"i",
);

/** One line of a pull request body or commit message that carries agent attribution. */
export interface AttributionFinding {
readonly line: string;
readonly reason: string;
}

/**
* Every failure in a pull request title and, when a range is given, in each
* commit subject in that range of the repository at `cwd`. Empty means clean.
* Every line in `text` that carries agent attribution: a Co-Authored-By
* trailer naming a coding agent, a "generated with" banner naming one, or a
* link back to an agent session. A human co-author, an ordinary banner-free
* message, and a link to anything else all produce nothing here.
*/
export function attributionFindings(text: string): AttributionFinding[] {
const findings: AttributionFinding[] = [];
for (const rawLine of text.split("\n")) {
const line = rawLine.trim();
if (line === "") continue;

const coAuthor = CO_AUTHOR_LINE.exec(line);
if (coAuthor) {
// The `.+` group cannot be empty when the outer regex has matched.
const marker = agentMarker(coAuthor[1] as string);
if (marker) {
findings.push({
line,
reason:
`a Co-Authored-By trailer names a coding agent ("${marker}"); ` +
`credit a person instead, or remove the trailer`,
});
continue;
}
}

if (BANNER_LINE.test(line)) {
const marker = agentMarker(line);
if (marker) {
findings.push({
line,
reason:
`a "generated with" banner names a coding agent ("${marker}"); ` +
`remove the banner`,
});
continue;
}
}

if (SESSION_LINK.test(line))
findings.push({
line,
reason: "a link back to a coding agent session; remove the link",
});
}
return findings;
}

/** Every attribution finding in `text`, each named as coming from `context`. */
function attributionFailures(context: string, text: string): string[] {
return attributionFindings(text).map(
(finding) => `${context}: ${finding.reason} ("${finding.line}")`,
);
}

/**
* Every failure in a pull request title and body and, when a range is given,
* in each commit subject and full message in that range of the repository at
* `cwd`. Empty means clean.
*/
export function check(
title: string,
range?: string,
cwd: string = process.cwd(),
body?: string,
): string[] {
const failures: string[] = [];
const titleFailure = failureReason(title);
if (titleFailure !== null)
failures.push(`pull request title: ${titleFailure}`);

if (body) failures.push(...attributionFailures("pull request body", body));

if (range) {
const subjects = execFileSync("git", ["log", "--format=%s", range], {
cwd,
Expand All @@ -76,11 +220,31 @@ export function check(
const why = failureReason(subject);
if (why !== null) failures.push(`commit "${subject}": ${why}`);
}

// %B carries the full, unwrapped commit message (subject and body), so a
// trailer or banner in the body is only visible here. \x00 separates
// records, since a commit message may itself hold a blank line.
const messages = execFileSync("git", ["log", "--format=%B%x00", range], {
cwd,
encoding: "utf8",
})
.split("\0")
.map((message) => message.replace(/\n+$/, ""))
.filter((message) => message !== "");
for (const message of messages) {
// `messages` is filtered to non-empty strings, so splitting one always
// yields at least one element.
const subject = message.split("\n")[0] as string;
failures.push(...attributionFailures(`commit "${subject}"`, message));
}
}
return failures;
}

/** Check `--title` and an optional `--range`; exit 0 when clean, 1 when not, 2 on bad usage. */
/**
* Check `--title`, an optional `--range` and an optional `--body`; exit 0
* when clean, 1 when not, 2 on bad usage.
*/
export function main(
argv: readonly string[],
output: GateOutput = processOutput,
Expand All @@ -92,11 +256,12 @@ export function main(
const title = flag("--title");
if (!title) {
output.err(
'usage: node scripts/check-pr-title.ts --title "..." [--range a..b]\n',
'usage: node scripts/check-pr-title.ts --title "..." ' +
'[--range a..b] [--body "..."]\n',
);
return 2;
}
const failures = check(title, flag("--range"));
const failures = check(title, flag("--range"), process.cwd(), flag("--body"));
if (failures.length > 0) {
output.err(failures.map((failure) => `${failure}\n`).join(""));
return 1;
Expand Down
Loading
Loading