From 9c3fa0b3acf15959be710fce4a99853438aa7551 Mon Sep 17 00:00:00 2001 From: OpenAI Codex Date: Thu, 24 Sep 2026 16:08:28 +0100 Subject: [PATCH 1/4] docs: add eval review guidance --- CONTRIBUTING.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3d8739fe..00235a01 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -22,7 +22,7 @@ If your scenario contains anything not self-explanatory, consider adding a `READ ## Eval criteria -Every new scenario needs a `motivation:` defined in `PROMPT.md` frontmatter that cites some evidence for the scenario being a part of the Supabase user journey, ideally a pain point. Examples include support tickets, GitHub issues, Linear issues, or social media threads. +Every new scenario needs a `motivation:` defined in `PROMPT.md` frontmatter that cites concrete evidence for the scenario being part of the Supabase user journey, ideally a pain point. Examples include Supabase troubleshooting guides, user or support reports, GitHub or Linear issues, Slack threads, or social posts. For new **benchmark** scenarios, we need to see at least one agent, ideally more, failing the new scenario to ensure we're getting signal from results. If agents are already acing your scenario, consider hardening it with a more ambiguous or misleading prompt, unusual seed data, or subtle footgun. Run locally and review agent failures to ensure they're legitimate reasoning mistakes, not eval framework limitations. We also want to keep benchmarks representative of the user journey. Review the [Evals coverage table](https://app.hex.tech/supabase/app/Evals-033abDlwqlTbW5ktgwFffU/latest) and make sure you're not over-indexing on a niche use case. @@ -34,9 +34,9 @@ Instead of spoonfeeding agents in the prompt, move details into seed data to let ## Writing scorers -Prefer deterministic checks where possible for stability and efficiency. Avoid being overly prescriptive with the process an agent takes to reach a solution (unless critical to the scenario), prefer checking the end state by inspecting the project or filesystem. +Prefer deterministic checks where possible because they are cheaper, faster, repeatable, and easier to debug. Avoid being overly prescriptive with the process an agent takes to reach a solution (unless critical to the scenario), prefer checking the end state by inspecting the project or filesystem. -If deterministic checks are too inflexible or convoluted, use an LLM-as-a-judge check via `judge()` to check semantic correctness. +Reserve LLM-as-a-judge checks via `judge()` for semantic or free-form outcomes where multiple valid forms make exact checks brittle. Prefer building checks declaratively and returning the list in one place instead of accumulating checks within branching logic, so the list remains stable if one path fails. @@ -60,6 +60,18 @@ You have a few options to run evals in CI: Include refreshed results for PRs with new/changed evals so a reviewer can see results directly from your PR or Vercel preview build. +## Reviewing an eval + +Start with [Eval criteria](#eval-criteria) and [Writing prompts](#writing-prompts). The `motivation:` should point to a real user problem, not just an interesting edge case. The prompt should have one clear, observable goal and withhold commands, schema details, or rubric language that would leak the intended solution. + +Review [Writing scorers](#writing-scorers) check by check. For each assertion, identify the user requirement it represents and ask how it could falsely pass a wrong answer or falsely fail a valid one. Prefer behavior-level evidence over one implementation path. Keep true pass/fail expectations as assertions; use metrics for diagnostics only. If a judge checks something an exact or fixture-based check can prove, ask for the deterministic version. + +Use the [Submitting evals for review](#submitting-evals-for-review) workflow to require refreshed CI results before approval. Inspect at least one success run and every distinct failure shape. Confirm the experiment environment matches the eval's intended suite, skills, runtime, hosted or local state, and any CLI or Docker assumptions. + +Classify failures before requesting changes: agent gap, product gap, scorer bug, or harness/runtime failure. A failing run can be a valid product or agent gap, so an all-green result is not required. Exercise at least one intentionally wrong solution or counterexample to prove the scorer does not accept the failure mode the eval is meant to catch. + +Approve when the evidence is real, the prompt and scorer measure the same outcome, no known false pass or false fail remains, and the intended CI setup produced trustworthy results. Review findings should name the exact file and line, describe the concrete false pass or false fail, and propose the smallest correction. Tag the AI team only when the PR changes eval framework behavior, runtime setup, experiment definitions, result schemas, or contracts; scenario wording and scorer tuning should stay with the owning team. + ## Docs evals The docs team owns `evals/docs/` and its results. Docs evals run without skills on a single experiment (`codex-gpt-6-luna-no-skills`). From 51f57a057308ef6cb9329c7f7288cf1cc7244114 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 25 Sep 2026 14:35:08 +0000 Subject: [PATCH 2/4] docs: trim duplicated eval review guidance Co-authored-by: Rodriguespn <44656907+Rodriguespn@users.noreply.github.com> --- CONTRIBUTING.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 00235a01..beb48818 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -62,15 +62,13 @@ Include refreshed results for PRs with new/changed evals so a reviewer can see r ## Reviewing an eval -Start with [Eval criteria](#eval-criteria) and [Writing prompts](#writing-prompts). The `motivation:` should point to a real user problem, not just an interesting edge case. The prompt should have one clear, observable goal and withhold commands, schema details, or rubric language that would leak the intended solution. +Start with [Eval criteria](#eval-criteria) and [Writing prompts](#writing-prompts): confirm the `motivation:` cites a real user problem and the prompt has one clear goal without leaking commands, schema details, or rubric language. -Review [Writing scorers](#writing-scorers) check by check. For each assertion, identify the user requirement it represents and ask how it could falsely pass a wrong answer or falsely fail a valid one. Prefer behavior-level evidence over one implementation path. Keep true pass/fail expectations as assertions; use metrics for diagnostics only. If a judge checks something an exact or fixture-based check can prove, ask for the deterministic version. +Then review [Writing scorers](#writing-scorers) assertion by assertion. Map each check to a user requirement, look for false passes and false fails, prefer behavior-level evidence over one implementation path, keep metrics diagnostic only, and ask for a deterministic or fixture-based check whenever it can replace `judge()`. -Use the [Submitting evals for review](#submitting-evals-for-review) workflow to require refreshed CI results before approval. Inspect at least one success run and every distinct failure shape. Confirm the experiment environment matches the eval's intended suite, skills, runtime, hosted or local state, and any CLI or Docker assumptions. +Before approval, use the [Submitting evals for review](#submitting-evals-for-review) workflow to require refreshed CI results. Inspect at least one success run and each distinct failure shape, verify the experiment matches the intended suite, skills, runtime, state, and any CLI or Docker assumptions, and classify failures as agent gap, product gap, scorer bug, or harness/runtime failure. An all-green run is not required, but at least one known-bad solution or counterexample should fail. -Classify failures before requesting changes: agent gap, product gap, scorer bug, or harness/runtime failure. A failing run can be a valid product or agent gap, so an all-green result is not required. Exercise at least one intentionally wrong solution or counterexample to prove the scorer does not accept the failure mode the eval is meant to catch. - -Approve when the evidence is real, the prompt and scorer measure the same outcome, no known false pass or false fail remains, and the intended CI setup produced trustworthy results. Review findings should name the exact file and line, describe the concrete false pass or false fail, and propose the smallest correction. Tag the AI team only when the PR changes eval framework behavior, runtime setup, experiment definitions, result schemas, or contracts; scenario wording and scorer tuning should stay with the owning team. +Approve when the prompt, scorer, and CI evidence all support the same outcome and no known false pass or false fail remains. Review findings should cite the exact file and line with the concrete issue and smallest fix. Tag the AI team only for framework, runtime, experiment, result schema, or contract changes; scenario wording and scorer tuning should stay with the owning team. ## Docs evals From 1216bf3fa5e387df1d9020f60afbe9e33220042e Mon Sep 17 00:00:00 2001 From: Pedro Rodrigues Date: Mon, 28 Sep 2026 10:57:06 +0100 Subject: [PATCH 3/4] fix contributing.md --- CONTRIBUTING.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index beb48818..139e93f0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -62,13 +62,17 @@ Include refreshed results for PRs with new/changed evals so a reviewer can see r ## Reviewing an eval -Start with [Eval criteria](#eval-criteria) and [Writing prompts](#writing-prompts): confirm the `motivation:` cites a real user problem and the prompt has one clear goal without leaking commands, schema details, or rubric language. +Start with [Eval criteria](#eval-criteria) and [Writing prompts](#writing-prompts): confirm the `motivation:` cites a real user problem and the prompt has one clear, observable goal without leaking commands, schema details, or rubric language. -Then review [Writing scorers](#writing-scorers) assertion by assertion. Map each check to a user requirement, look for false passes and false fails, prefer behavior-level evidence over one implementation path, keep metrics diagnostic only, and ask for a deterministic or fixture-based check whenever it can replace `judge()`. +Then review [Writing scorers](#writing-scorers) assertion by assertion. Each check should map to a user requirement, avoid known false passes and false fails, measure behavior instead of one implementation path, and keep diagnostics as metrics rather than pass/fail gates. Ask for a deterministic or fixture-based check whenever it can replace `judge()`. -Before approval, use the [Submitting evals for review](#submitting-evals-for-review) workflow to require refreshed CI results. Inspect at least one success run and each distinct failure shape, verify the experiment matches the intended suite, skills, runtime, state, and any CLI or Docker assumptions, and classify failures as agent gap, product gap, scorer bug, or harness/runtime failure. An all-green run is not required, but at least one known-bad solution or counterexample should fail. +Use the [Submitting evals for review](#submitting-evals-for-review) workflow to require refreshed CI results before approval. The refreshed eval CI job should be green, which means the requested evals ran and results were recorded; it does not mean every agent passed the eval. After CI completes, inspect at least one passing result and each distinct failure shape, and confirm the experiment matches the intended suite, skills, runtime, hosted or local state, and any CLI or Docker assumptions. -Approve when the prompt, scorer, and CI evidence all support the same outcome and no known false pass or false fail remains. Review findings should cite the exact file and line with the concrete issue and smallest fix. Tag the AI team only for framework, runtime, experiment, result schema, or contract changes; scenario wording and scorer tuning should stay with the owning team. +Classify failing eval results before requesting changes: agent gap, product gap, scorer bug, or harness/runtime failure. A failing eval can be valid signal, but at least one known-bad solution or counterexample should fail to prove the scorer catches the intended failure mode. + +Approve when the evidence, prompt, scorer, CI setup, and inspected results support the same outcome, and no known false pass or false fail remains. Findings should cite the exact file and line, name the concrete issue, and propose the smallest correction. + +Tag the AI team only for framework, runtime, experiment, result schema, or contract changes; scenario wording and scorer tuning should stay with the owning team. ## Docs evals From 4f6e89f3312a0ce6db9fd87cbc29799cae43ef94 Mon Sep 17 00:00:00 2001 From: Pedro Rodrigues Date: Mon, 28 Sep 2026 22:40:45 +0100 Subject: [PATCH 4/4] docs: clarify eval review checklist --- CONTRIBUTING.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 139e93f0..59c346c8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -62,17 +62,15 @@ Include refreshed results for PRs with new/changed evals so a reviewer can see r ## Reviewing an eval -Start with [Eval criteria](#eval-criteria) and [Writing prompts](#writing-prompts): confirm the `motivation:` cites a real user problem and the prompt has one clear, observable goal without leaking commands, schema details, or rubric language. +Use this checklist after the author has followed [Submitting evals for review](#submitting-evals-for-review). A failing run can be useful signal, so all-green results are not required. -Then review [Writing scorers](#writing-scorers) assertion by assertion. Each check should map to a user requirement, avoid known false passes and false fails, measure behavior instead of one implementation path, and keep diagnostics as metrics rather than pass/fail gates. Ask for a deterministic or fixture-based check whenever it can replace `judge()`. +- Read [Eval criteria](#eval-criteria) and [Writing prompts](#writing-prompts). The `motivation:` should cite a real user problem, with the respective external link or Linear issue ID, and the prompt should have one clear, observable goal without leaking commands, schema details, or rubric language. +- Review [Writing scorers](#writing-scorers) assertion by assertion. Each check should map to a user requirement, avoid known false passes and false fails, and measure behavior instead of one implementation path. Don't turn incidental diagnostics into pass/fail checks. +- Ask for a deterministic or fixture-based check whenever it can replace `judge()`. Reserve judges for semantic or free-form outcomes where exact checks would be brittle. +- Require refreshed CI results, inspect at least one success run and each distinct failure shape, and confirm the experiment matches the intended suite, skills, runtime, hosted or local state, and any CLI or Docker assumptions. +- Classify each failure before requesting changes: agent gap, product gap, scorer bug, or harness/runtime failure. Ideally, show that the scorer rejects at least one known-bad solution or counterexample with a scorer unit test or a failing run linked in the PR. -Use the [Submitting evals for review](#submitting-evals-for-review) workflow to require refreshed CI results before approval. The refreshed eval CI job should be green, which means the requested evals ran and results were recorded; it does not mean every agent passed the eval. After CI completes, inspect at least one passing result and each distinct failure shape, and confirm the experiment matches the intended suite, skills, runtime, hosted or local state, and any CLI or Docker assumptions. - -Classify failing eval results before requesting changes: agent gap, product gap, scorer bug, or harness/runtime failure. A failing eval can be valid signal, but at least one known-bad solution or counterexample should fail to prove the scorer catches the intended failure mode. - -Approve when the evidence, prompt, scorer, CI setup, and inspected results support the same outcome, and no known false pass or false fail remains. Findings should cite the exact file and line, name the concrete issue, and propose the smallest correction. - -Tag the AI team only for framework, runtime, experiment, result schema, or contract changes; scenario wording and scorer tuning should stay with the owning team. +Tag #team-ai on Slack only for framework, runtime, experiment, result schema, or contract changes; scenario wording and scorer tuning should stay with the owning team. ## Docs evals