[habdelra]

Some of the card definitions that boxel's host and realm-server tests depend on, such as the realm policy cards, live in cardstack/boxel-catalog, not in this repo. packages/catalog/test-subset.json pins the catalog commit those tests load them from. The Catalog Test Subset workflow runs sync-test-subset.ts --check-pin to make sure that pin is a commit on catalog main. Deployments serve catalog main, so a pin anywhere else means our tests run against definitions that no deployment actually has.

The check fails in two ways, and neither message used to say which catalog PR is involved, so whoever read it had to work backwards from a bare SHA or file path.

The pin isn't on catalog main at all. This usually happens on purpose: you change a catalog definition and the boxel code that relies on it at the same time, and while the catalog PR is still open, the boxel PR pins that PR's commit. The message now links the PR, for example:

catalog test subset: 84fee4d216f820f77baa33782188e152fe20b402 is not on cardstack/boxel-catalog main (compare status "diverged"). It is in cardstack/boxel-catalog#772. Merge the cardstack/boxel-catalog change first, then re-pin to a commit on cardstack/boxel-catalog main (pnpm catalog:test-subset --bump).

To find the PR, the script asks GitHub which pull requests contain the pinned commit (GET /repos/{repo}/commits/{sha}/pulls). For a commit that isn't on catalog main, that answer has two possible shapes, and the message follows each:

• An open PR carries the commit. This is the normal paired-change state: the catalog change hasn't merged yet. The message links the PR (or PRs) and still says to merge the catalog change first.
• The commit is the head of a PR that was squash-merged. A squash merge puts a new commit on catalog main, so the PR's own head commit never lands there and still compares diverged. That is exactly what the pin looks like right after the catalog side of a paired change merges. GitHub returns that merged PR, and the message says so, gives the commit it merged as, and says to re-pin with --bump. Telling the reader to "merge the catalog change first" at that point would send them to do something already done.

A PR that was closed without merging isn't returned at all, so a commit that only lives on such a branch gets the original message with no link. The lookup reuses the GITHUB_TOKEN the check already sends, and boxel-catalog is public, so no new permissions are needed.

The pin is on catalog main, but main has since changed a subset file. This is what every boxel PR touching the subset sees in the window after a catalog definition change merges and before some boxel PR re-pins to it. The message now names the catalog PRs that made the change, for example:

catalog test subset: cardstack/boxel-catalog main has changed realm-policy/realm-policy.gts since 27003bc9cd072d8f8a9de5cc789326e3aa91f3ea (in cardstack/boxel-catalog#772), so boxel's tests would run against definitions deployments no longer serve. #6371, #6380 already pin a revision that matches cardstack/boxel-catalog main: once one merges, merge cardstack/boxel main into this branch. To move the pin without them, re-pin to cardstack/boxel-catalog main (pnpm catalog:test-subset --bump), run the manifest's tests against it, and commit the new pin.

To find them, the script takes the commits catalog main has that the pin lacks (GET /repos/{repo}/compare/{pin}...main), keeps the ones in each changed file's history on catalog main (GET /repos/{repo}/commits?sha=main&path=…), and asks GitHub for each kept commit's pull request with the same commits/{sha}/pulls lookup as above. It drops duplicate PRs, so a PR with several commits touching the file is listed once, and every PR that touched a changed file since the pin appears.

The message also says where a fix is already waiting, because that changes what you should do. A catalog definition change usually needs matching boxel code, so re-pinning a bystander PR to the new definition would likely break its tests. The better step is to wait for the pin that another branch already carries, then merge boxel main into this branch. The script reads that pin directly rather than guessing from branch names. One GraphQL query (a page per hundred open PRs) returns packages/catalog/test-subset.json from boxel main and from the head of every open boxel PR. For each pin other than this branch's, it compares every subset file at that pin with catalog main, which is the same condition this check passes on. Then:

• If boxel main already has such a pin, the message just says to merge boxel main into this branch. That's the case where a branch is failing only because it's behind boxel main.
• Otherwise, it names every open PR whose pin matches, however its branch is named. Today that's Declare explain on RealmPolicy itself and render its answer on the policy card #6371, the boxel side of Let a policy card explain what it decides boxel-catalog#772, and Pin that every batch entry is gated and one refusal writes nothing #6380, which re-pinned to the same revision. PRs still on an older catalog revision because their branch is behind boxel main don't match, so they aren't named.
• It never names the branch under check (GITHUB_HEAD_REF in CI, the current branch locally), so Declare explain on RealmPolicy itself and render its answer on the policy card #6371's own run names only Pin that every batch entry is gated and one refusal writes nothing #6380.

Every message names the repo of each main it mentions: cardstack/boxel-catalog main for the pin and re-pin, and cardstack/boxel main for the branch to merge into this one. A reader never has to guess which main "merge main" means.

GraphQL doesn't accept anonymous requests. Without a GITHUB_TOKEN, which is how the check usually runs locally, this part is skipped and the message ends with the plain re-pin advice. CI always has the token.

These lookups only add detail to a message the check is already failing with, so they can't change the check's result. If a request errors, gets a non-OK answer, or finds no PR, the message is the same as before.

I ran --check-pin locally with the manifest pinned to several catalog commits. The head of open cardstack/boxel-catalog#767 links it and says to merge first. The head of squash-merged cardstack/boxel-catalog#771 says it was merged as 9f467f3f21 and to re-pin. The head of cardstack/boxel-catalog#689, which was closed without merging, gives the original message. The current pin 27003bc9 names cardstack/boxel-catalog#772 as the change to realm-policy.gts, and #6371 and #6380 as the PRs already carrying a matching pin. Run as #6371's branch (GITHUB_HEAD_REF set to it), it names only #6380. Without a token, it gives the plain re-pin advice. With the script's boxel main lookup pointed at #6371's head, to stand in for main having re-pinned, it says to merge boxel main into this branch.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-29T13:39:29.243237Z	a2c0866	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[lucaslyl]

[Claude Code 🤖] This review checked the PR lookup against how GitHub's commits/{sha}/pulls endpoint actually answers for boxel-catalog, and followed each answer through to the failure message it produces.

One blocking issue: after a catalog PR is squash-merged, the check links that already-merged PR and still says to merge it first. boxel-catalog allows squash merges, so this happens in the normal paired-branch flow. The details and a fix are in the inline thread on pullRequestsFor.

[lucaslyl]

[Claude Code 🤖] The claim that GitHub lists only open PRs doesn't hold when the catalog PR was squash-merged. In that case the endpoint also returns the closed, merged PR. boxel-catalog allows squash merges. I checked five of its recently merged PRs whose head commits are diverged from main, and for each head commits/{sha}/pulls returned the PR with state: "closed" and merged_at set. That covers the usual paired flow: the boxel PR pins the catalog PR's head, the catalog PR squash-merges, and the next --check-pin fails with a link to the already-merged PR and "Merge the catalog change first". The right step at that point is just --bump.

Fix: read merged_at and merge_commit_sha, both in this response, and split the message. For example:

let prs = (await response.json()) as { html_url: string; merged_at: string | null; merge_commit_sha: string | null }[];

If a PR has merged_at set, say " was merged as <merge_commit_sha>; re-pin to a commit on main (pnpm catalog:test-subset --bump)". Keep "It is in . Merge the catalog change first…" for open PRs only. Also update the comment above to match what the endpoint actually returns.

Regression, since the message is new in this PR. Blocking, because in the post-merge case it sends the reader to the wrong action.

[habdelra]

[Claude Code 🤖] Confirmed and fixed in fbb9f6a. I reproduced it with cardstack/boxel-catalog#771: that PR was squash-merged, its head 567630f676 compares diverged with main, and the lookup returns it with merged_at set.

When the lookup returns a merged PR, the message now reads "It is from , which was merged as <merge_commit_sha>. Re-pin to a commit on main (pnpm catalog:test-subset --bump)." "It is in . Merge the catalog change first…" is now used only when every returned PR is open. The comment on pullRequestsFor now describes both kinds of answer.

I also checked the third case. For catalog PRs closed without merging (cardstack/boxel-catalog#689 and cardstack/boxel-catalog#679), the endpoint returns no PRs at all, so the message falls back to the original wording with no link. I ran --check-pin locally against the heads of cardstack/boxel-catalog#771, the open cardstack/boxel-catalog#767, and cardstack/boxel-catalog#689, and each produced the matching message.

[habdelra]

[Claude Code 🤖] @lucaslyl Thanks. The blocking issue in your review is fixed in fbb9f6a, with details in the inline thread on pullRequestsFor. When the pinned commit is the head of a squash-merged catalog PR, the check now links that PR, gives the commit it merged as, and says to re-pin with --bump. "Merge the catalog change first" appears only when every returned PR is still open. The PR description now describes both cases.