Repository navigation
fix: count a protected-file violation as a graded failure - #29
Merged
Merged
Conversation
A trial that edits a protected file is recorded with no functional verdict, because the verifier stops before testing it. Coverage then treated the empty verdict as missing grading, so one such trial left its harness without cost or time figures and the whole lane without a verdict. The protection failure is the verdict; the trial still counts as a failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
In coverage accounting, a trial whose protected state failed is no longer counted as
missing_gradingwhen its functional verdict is empty.Why
When a submission edits a protected file, the report deliberately records
correctness: nullwithprotected_state: false: the verifier short-circuits, so there is no functional verdict to invent._datasetthen counted that trial as missing grading, which marked the harness's coverage incomplete, blanked its cost and time means, and left the lane withinsufficient_evidence.This surfaced on GPT-6.1 Sol, where Nothing edited
src/Serializer.jsonnode-export-ambiguous. The trial was already a failure in_success; only the coverage count was wrong.No published report contains such a trial, so no published verdict changes.
Testing
New test: the violating harness keeps complete coverage, counts the trial as a failure, keeps its cost figures and is ineligible; a missing verdict with the protected state intact still counts as ungraded.
🤖 Generated with Claude Code