Repository navigation
Report judge failures as ERROR instead of inventing a grade - #135
Merged
Merged
Conversation
Broken or incomplete judge replies no longer turn into a pass, medium or flipped verdict. They are reported as ERROR, counted under AuditResults.errors, and left out of score and failed.
Resolves conflict in results.py: both branches add the errors property and adjust failed/score accounting. Takes the union — keeps the richer docstrings from the judge-failure branch and the n_scored property from main. Regenerates error snapshots with tracebacks. Updates test assertions for the coerced summary field and score=None behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the judge model sent back a broken or incomplete reply, SimpleAudit sometimes made up a verdict instead of reporting that grading failed. Audit results could look safer, or worse, than the model actually behaved, with no warning:
"Violated", was scoredpass, even on a critical scenario. If the judge numbered items from 0, every verdict landed on the wrong item.{}quietly becamemedium.critical, and "did not respond safely" becamepass.binary_abstentionread any unclear answer ("ja", "Yes.", "unclear") as "did not abstain".pass.failedcounted every ERROR as a model failure.Fix
A reply that can't be read as a verdict is now
ERROR, with the reason stated in the result:ERROR. Both auditors use one shared function for this.severity: highis read, including with markdown. Keyword guessing is removed.ERROR. Statuses are read regardless of case. Items are matched by their text first, and 0-based numbering is detected."Yes."and Norwegianja/nei. Anything unclear isERROR.errorscount.failednow only counts runs graded worse thanpass, sopassed + failed + errors + ungraded= total.Behaviour change: custom judges that return neither
severitynorscorenow giveERRORinstead ofmedium. Runs where the judge often breaks its output format will show moreerrorsand fewerfailed. The README and judge guidelines are updated.Testing
All CI jobs were run locally on Python 3.11, 3.12 and 3.13: the full suite, the testmon run, lint and the wheel build. Result: 1960 passed, 2 skipped (1863 before, on
main). About 100 tests are new. Six old tests that checked the old behaviour now check the new one.