Skip to content

feat(rubrics_verifier): fail_task_when_not_passing fails the task on a verdict that isn't passing - #640

Merged
earakely-scale merged 2 commits into
mainfrom
edgararakelyan/fail-task-when-not-passing
Nov 13, 2026
Merged

earakely-scale merged 2 commits into
mainfrom
edgararakelyan/fail-task-when-not-passing

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

Some tasks run a verifier that checks whether the run was a fair test at all: whether the scripted events fired, whether colleagues stayed on script, whether the solver stayed out of the harness. Today that verifier records its verdict, but nothing acts on it, so a run that fails the check still reports its quality score.

What changes

  • A new option on rubrics_verifier. fail_task_when_not_passing: true fails the step with VerifierNotPassingError when the recorded verdict isn't passing, once the verdict has been written.
    • "Passing" means a score of 1.0.
    • The score is computed exactly as before, with any score_aggregator.
    • The error names the rows that weren't met.
  • It must stop the task. The option is refused together with fail_task_on_error: false, because a tolerated failure would let the steps after it run and score the run.
  • No re-grading. The scheduler doesn't retry a VerifierNotPassingError under retry_config: grading the same run again could pass it.
  • Where the error lives. VerifierNotPassingError sits in agent_env.task_step.context, next to StepContextError, and is exported from agent_env.task_step.

With the option set, and the scoring step depending on the verifier, a run that fails the check ends FAILED before anything scores it. Consumers that already drop failed runs then drop it too:

{"id": "validity", "type": "rubrics_verifier", "fail_task_when_not_passing": true, "...": "..."},
{"id": "final-score", "type": "aggregate_verifiers", "verifier_ids": ["rubric-judge"],
 "depends_on": ["rubric-judge", "validity"]}

Docs. The step reference should document fail_task_when_not_passing: what passing means, the fail_task_on_error requirement, and that the scoring step should depend on the verifier.

How it was tested

Unit tests: tst/unit/task_step/verifiers/test_rubrics_verifier_fail_task_when_not_passing.py (10 tests) covers:

  • a passing verdict;
  • an unmet row, with the verdict kept on the context;
  • a partial weighted score;
  • a penalized negative row;
  • a run the judge couldn't grade;
  • the option off;
  • the refusal with fail_task_on_error: false;
  • storing the option only when it's set;
  • a task that fails at the verifier and never runs its scoring step;
  • retry_config not grading the run again: three gradings without the scheduler change, one with it.

tst/unit/task_step, tst/unit/task, tst/unit/bundle and tst/unit/runner pass: 4,038 passed, 11 skipped.

Full suite: CI's unit and integration-local shards pass.

End to end. A benchmark task with a scripted world (timed events and colleague agents), run as a bundle with agent-env run from this branch on a Linux VM. The solver ran on claude-sonnet-5-5 and the judges on their pinned model. Each copy sets fail_task_when_not_passing: true and fail_task_on_error: true on the task's validity verifier and adds the verifier to the scoring step's depends_on.

World Validity verdict Result
intact 1.0 The scoring step ran after validity and scored 1.0; the task passed.
broken: one scheduled email delayed past the run, plus a validity row that checks for it that row false, score 0.0 The step failed with VerifierNotPassingError: Verifier 'validity' isn't passing (score 0.0); rows not met: ['v-world-mail']. The run ended FAILED at that step on its first attempt, the scoring step never ran, and every sandbox was torn down.

In the broken run both quality judges scored 1.0, so without the option it would have reported 1.0.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge, with the earlier documentation suggestion still open.

Fix All in CursorFindings

  1. P2 Validity setup is undocumented ▶
Fix with agent prompt
### Issue 1
src/agent_env/task_step/task_steps/verifiers/rubrics_verifier.py:undefined-468
The new `fail_task_when_not_passing` option has no reference documentation, despite the PR description requesting it. Document that passing means an aggregate score of 1.0, that `fail_task_on_error` must be true, and that the scoring step must depend on this verifier. Without that dependency guidance, users can enable the option while still letting scoring run alongside the validity check.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds an optional failure gate to rubrics_verifier.

  • Rubric verifiers can stop tasks when their score is below 1.0.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Grade run and record verdict] --> B{Failure gate enabled?}
    B -- No --> C[Step succeeds]
    B -- Yes --> D{Aggregate score is 1.0?}
    D -- Yes --> C
    D -- No --> E[Raise VerifierNotPassingError]
    E --> F[Fail task without retrying]
    F --> G[Dependent scoring step stays blocked]
Loading

Reviews (7) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile

…a verdict that isn't passing

A rubrics_verifier with fail_task_when_not_passing set fails its step with VerifierNotPassingError when its
recorded verdict scores below 1.0, after the verdict is written; the score itself is computed as before. The
option is refused with fail_task_on_error false, since a tolerated failure lets the steps after it score the
run. The scheduler does not retry a VerifierNotPassingError under retry_config: grading the same run again
could pass it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 12, 2026 17:31
judge_batch_concurrency: Optional[int] = None,
depends_on: Optional[list[TaskStepDependency]] = None,
fail_task_on_error: bool = True,
fail_task_when_not_passing: bool = False,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Validity setup is undocumented

The new fail_task_when_not_passing option has no reference documentation, despite the PR description requesting it. Document that passing means an aggregate score of 1.0, that fail_task_on_error must be true, and that the scoring step must depend on this verifier. Without that dependency guidance, users can enable the option while still letting scoring run alongside the validity check.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/verifiers/rubrics_verifier.py
Line: 465

Comment:
**Validity setup is undocumented**

The new `fail_task_when_not_passing` option has no reference documentation, despite the PR description requesting it. Document that passing means an aggregate score of 1.0, that `fail_task_on_error` must be true, and that the scoring step must depend on this verifier. Without that dependency guidance, users can enable the option while still letting scoring run alongside the validity check.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor Fix in Claude Code Fix in Codex

…ask-when-not-passing

# Conflicts:
#	src/agent_env/task/task.py
#	src/agent_env/task_step/task_steps/verifiers/rubrics_verifier.py
@earakely-scale
earakely-scale merged commit 40e7d7b into main Nov 13, 2026
34 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fail-task-when-not-passing branch November 13, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant