Repository navigation
Conversation
…tems as optional When a provider's safety filter blocks the target's reply (Anthropic or Gemini via any-llm, finish_reason="content_filter"), the content is empty and the refusal sits in message.refusal. The target call only read content, so the reply counted as empty and the scenario became an unscored ERROR. The harmful prompts that trigger the filter therefore dropped out of the score. The target call now returns the refusal text (or a placeholder) so it is judged. Judge and auditor calls are unchanged. The checklist judge only treated "Kan nevne (ikke påkrevd)" as optional, so 8 shipped items written "KAN NEVNE:", "Kan nevne at/hva" or "Kan peke på" counted as required. Any item starting with "Kan" is now optional, matching the English "May". Also drops a duplicate "errors" key in the repeated-run aggregate.
SushantGautam
force-pushed
the
fix/target-refusal-and-optional-items
branch
from
November 4, 2026 12:39
50090a6 to
ad7321d
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
1. Safety-filter refusals were dropped from the score. When a provider's safety filter blocks the target's reply (Anthropic or Gemini through any-llm), the reply comes back with empty
content,finish_reason="content_filter", and the refusal inmessage.refusal("Response blocked by Anthropic content filtering."). The target call only readcontent, so it saw an empty reply and the scenario became an unscored ERROR. The filter fires on the most harmful prompts, so exactly those scenarios dropped out and the score was computed on the rest.2. Optional checklist items counted as required. The checklist judge only treated items starting
Kan nevne (ikke påkrevd)as optional. Eight shipped items are writtenKAN NEVNE:,Kan nevne at/hvaorKan peke på(in arbeidstilsynet_arbeidstid, skatteetaten_legitimasjon, nb_kryss_ordning and lanekassen), so a model that skipped them was marked as violating them.Fix
[The provider's safety filter blocked this reply.]when there is no text, so the judge grades it. Judge and auditor calls are unchanged: a blocked judge reply is still ERROR and is not retried.Kanis optional, the same asMay. I checked every shipped pack: exactly those 8 items change, and no scenario is left with only optional items.Also removes a duplicate
"errors"key in the repeated-run aggregate. It had the same value, so nothing changes.Testing
pytest -n autoandpytest -p no:randomlyon 3.11, 3.12 and 3.13 (1990 passed), and a wheel build plus import.