fix(ci): make refresh dispatch explicit about evals and experiments

[mattrossman]

Dispatching a refresh for specific experiments has failed several times since July (Jul 16, Jul 24, Jul 28, Aug 7, Aug 14, Aug 26, Sep 9, Sep 30), and keeps coming up on Slack. This causes false alarms on runs and wasted sandboxes.

Manual refresh dispatches now run only what you ask for:

• Explicit experiments run only in suites they belong to. Each listed experiment used to be planned in every suite the eval allows. A mixed list now plans just the 2 valid pairs.
• No benchmark default. suite and experiment_suite start blank, and the run fails fast unless you name evals (suite or eval) and experiments (experiment_suite or experiments). The default made sense when refreshes were mainly handled by AI team and mostly for the public benchmark, but now the benchmark is rarely refreshed and multiple teams work in the repo. A dispatch with only eval set and suite blank still plans its pairs.
• One refresh branch per manual run. create-pull-request resets an existing branch, so a shared name let a later run wipe an earlier run's results. Scheduled runs still share one branch. To keep adding to one refresh PR, dispatch on its branch with commit_to_branch=true.
• merge now defaults to true. This is the most common way we dispatch refreshes manually, and merging is necessary for the recommended workflow of accumulating multiple refreshes on a single branch with commit_to_branch=true

Closes AI-1005

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Oct 6, 2026 1:28pm UTC

[Rodriguespn]

Approving with two comments. Please read them before merging. Thank you for working on this!

[mattrossman]

Updated!