Skip to content

Document how we review adapter PRs - #128

Open
audreina17 wants to merge 6 commits into
mainfrom
docs/adapter-reviewer-guide
Open

audreina17 wants to merge 6 commits into
mainfrom
docs/adapter-reviewer-guide

Conversation

@audreina17

Copy link
Copy Markdown
Collaborator

We already have a contributor guide and an automated adapter review. What I wanted to make clearer is what happens after the bot leaves its report. A reviewer still needs to follow a task back to upstream, generate it from the submitted adapter, and check how the verifier scores an actual submission. That matters especially when the dataset and the adapter were updated at different times.

The guide walks through those checks, when to agree on parity scope with the author, and what the second reviewer needs to see before signing off. It also covers two cases that come up often: checking a subset when a full oracle run is expensive, and regrading saved submissions when only the verifier changed.

The README and contributor walkthrough both link to the guide. I checked the new links locally.

@audreina17
audreina17 marked this pull request as ready for review November 21, 2026 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant