FEAT: Policy Scorer Compatibility (phase 2.5)

doc/code/framework.md

@@ -258,7 +258,8 @@ If you are contributing to PyRIT, that work will most likely land in one of the
 - Any decision an attack makes should be based on a scorer result
 - A scorer is not limited to a message, it could be anything (e.g. was this tool called or was this file written). It receives a `Scorable`, which identifies that evidence, and an optional `ScoringExpectation`.
 - `TrueFalseScorer` and `FloatScaleScorer` define result families. `MessageScorer` adds message resolution and message-only policy on top of them.
-- `Score.status` marks a verdict complete or undetermined, and the attack decides how to branch on it.
+- A scorer declares which evidence it reads, rather than the caller filtering evidence for it. A `MessageScorer` states the conversation roles it reads on its `ScorerPromptValidator`. It returns no score when the evidence carries no role it reads.
+- `Score.status` marks a verdict complete or undetermined, and the attack decides how to branch on it. A negative verdict is a completed result that does not satisfy the scoring criterion, such as `False` or a value below a threshold. For a role the scorer reads, an unreadable transport or protocol response produces an undetermined score. This result keeps "no verdict was reachable" separate from a completed negative result. Fully blocked responses use the scorer family's neutral fallback unless a specialized scorer overrides it.
 - **Does not own**: acting on its own result. A scorer evaluates a response and returns a score; branching on that score is the attack's job, and aggregating scores across runs is analytics'. It may call a target to evaluate, but it doesn't send the attack's objective prompt or manage the conversation.
 
 **Framework Plans**:

doc/code/scoring/0_scoring.ipynb

@@ -223,7 +223,16 @@
     "`status=\"undetermined\"` and no value. A fully blocked response is a complete negative result\n",
     "by default: `False` for message true/false scorers and `0.0` for message float-scale scorers.\n",
     "`SelfAskRefusalScorer` is the intentional exception because a content-filter block is a\n",
-    "refusal, so it returns `True`. Other response errors remain undetermined."
+    "refusal, so it returns `True`. Other response errors remain undetermined.\n",
+    "\n",
+    "A scorer declares which evidence it reads; the caller does not filter evidence on its behalf.\n",
+    "A message scorer names the conversation roles it reads with `supported_roles` on its\n",
+    "`ScorerPromptValidator`, and returns no score when the message carries no role it reads.\n",
+    "Prepended (`simulated_assistant`) turns are fabricated history, so a scorer must opt in to\n",
+    "read them. Every scorer still receives a failed response, because a scorer whose evidence\n",
+    "never came from the response must run even when the response failed. The deprecated\n",
+    "`role_filter` and `skip_on_error_result` arguments remain supported until removal, but new\n",
+    "code should use `supported_roles` and the scorer's unreadable-evidence fallback instead."
    ]
   },
   {

doc/code/scoring/0_scoring.py

@@ -126,6 +126,15 @@
 # by default: `False` for message true/false scorers and `0.0` for message float-scale scorers.
 # `SelfAskRefusalScorer` is the intentional exception because a content-filter block is a
 # refusal, so it returns `True`. Other response errors remain undetermined.
+#
+# A scorer declares which evidence it reads; the caller does not filter evidence on its behalf.
+# A message scorer names the conversation roles it reads with `supported_roles` on its
+# `ScorerPromptValidator`, and returns no score when the message carries no role it reads.
+# Prepended (`simulated_assistant`) turns are fabricated history, so a scorer must opt in to
+# read them. Every scorer still receives a failed response, because a scorer whose evidence
+# never came from the response must run even when the response failed. The deprecated
+# `role_filter` and `skip_on_error_result` arguments remain supported until removal, but new
+# code should use `supported_roles` and the scorer's unreadable-evidence fallback instead.
 # %% [markdown]
 # ## Scoring directly
 #

doc/code/scoring/3_combining_scorers.ipynb

@@ -100,6 +100,14 @@
     "text `ContentScorable` evidence. It returns a dynamic wrapper that remains the same scorer\n",
     "kind as its input.\n",
     "\n",
+    "An empty child result means that the scorer did not apply. A composite scorer ignores empty\n",
+    "child results and aggregates the remaining results. It returns an empty list if every child\n",
+    "result is empty. An inverter passes an empty result through unchanged. A threshold wrapper\n",
+    "passes an empty result to its float-scale aggregator. The standard aggregators map it to\n",
+    "`0.0`, while the `*_RAISE_ON_EMPTY` variants raise `ValueError`. A conversation wrapper\n",
+    "returns an empty result when it finds no applicable conversation evidence or its child\n",
+    "returns no score. Any outer wrapper then applies the rules above.\n",
+    "\n",
     "Deprecated message-shaped calls remain on `MessageScorer`, but generic wrappers do not\n",
     "project those APIs from their children. Score wrappers through the canonical `Scorable` API.\n",
     "\n",

doc/code/scoring/3_combining_scorers.py

@@ -83,6 +83,14 @@
 # text `ContentScorable` evidence. It returns a dynamic wrapper that remains the same scorer
 # kind as its input.
 #
+# An empty child result means that the scorer did not apply. A composite scorer ignores empty
+# child results and aggregates the remaining results. It returns an empty list if every child
+# result is empty. An inverter passes an empty result through unchanged. A threshold wrapper
+# passes an empty result to its float-scale aggregator. The standard aggregators map it to
+# `0.0`, while the `*_RAISE_ON_EMPTY` variants raise `ValueError`. A conversation wrapper
+# returns an empty result when it finds no applicable conversation evidence or its child
+# returns no score. Any outer wrapper then applies the rules above.
+#
 # Deprecated message-shaped calls remain on `MessageScorer`, but generic wrappers do not
 # project those APIs from their children. Score wrappers through the canonical `Scorable` API.
 #

pyrit/executor/attack/multi_turn/crescendo.py

@@ -713,9 +713,7 @@ async def _score_response_async(self, *, context: CrescendoAttackContext) -> Sco
                 response=context.last_response,
                 objective_scorer=self._objective_scorer,
                 auxiliary_scorers=self._auxiliary_scorers,
-                role_filter="assistant",
                 objective=context.objective,
-                skip_on_error_result=False,
             )
 
         objective_score = scoring_results["objective_scores"]

pyrit/executor/attack/multi_turn/multi_prompt_sending.py

@@ -408,9 +408,7 @@ async def _evaluate_response_async(self, *, response: Message, objective: str) -
                 response=response,
                 auxiliary_scorers=self._auxiliary_scorers,
                 objective_scorer=self._objective_scorer if self._objective_scorer else None,
-                role_filter="assistant",
                 objective=objective,
-                skip_on_error_result=True,
             )
 
         objective_scores = scoring_results["objective_scores"]

pyrit/executor/attack/multi_turn/tree_of_attacks.py

@@ -804,12 +804,11 @@ async def _score_response_async(self, *, response: Message, objective: str) -> N
         and any auxiliary scorers (which provide additional metrics). The scoring results are
         used by the TAP algorithm to decide which branches to explore further.
 
-        Blocked or errored responses are scored via the scorer's unified default behavior:
-        ``TrueFalseScorer`` returns
-        ``Score(False)`` and ``FloatScaleScorer``
-        returns ``Score(0.0)`` whenever no supported pieces remain after validator filtering
-        (the normal outcome for a blocked piece). This keeps blocked branches at the bottom
-        of the priority queue without needing attack-level error mapping.
+        Scorers apply their own unreadable-response policy. A fully blocked response uses the
+        scorer family's neutral fallback unless the scorer overrides it. An unreadable transport
+        or protocol response produces an undetermined score. A response with no role supported
+        by the objective scorer produces no objective score, so this method raises ``RuntimeError``.
+        Tree of Attacks does not map these outcomes to ``False`` or ``0.0``.
 
         Args:
             response (Message): The response from the objective target to evaluate.
@@ -841,9 +840,7 @@ async def _score_response_async(self, *, response: Message, objective: str) -> N
                 response=response,
                 objective_scorer=self._objective_scorer,
                 auxiliary_scorers=self._auxiliary_scorers,
-                role_filter="assistant",
                 objective=objective,
-                skip_on_error_result=False,
             )
 
         # Extract objective score

pyrit/executor/attack/single_turn/prompt_sending.py

@@ -367,9 +367,7 @@ async def _evaluate_response_async(
                 response=response,
                 objective_scorer=self._objective_scorer,
                 auxiliary_scorers=self._auxiliary_scorers,
-                role_filter="assistant",
                 objective=objective,
-                skip_on_error_result=True,
             )
 
         if not self._objective_scorer:

pyrit/score/__init__.py

@@ -39,7 +39,7 @@
     from pyrit.score.float_scale.system_prompt_extraction_scorer import SystemPromptExtractionScorer
     from pyrit.score.float_scale.video_float_scale_scorer import VideoFloatScaleScorer
     from pyrit.score.message_scorable_resolver import MessageScorableResolver
-    from pyrit.score.message_scorer import MessageScorer, MessageScoringOptions
+    from pyrit.score.message_scorer import MessageScorer
     from pyrit.score.response_handler import CallableResponseHandler, JsonSchemaResponseHandler, ResponseHandler
     from pyrit.score.scorable import ContentScorable, MessageScorable, Scorable
     from pyrit.score.scorer import Scorer
@@ -176,7 +176,6 @@
     "MessageScorableResolver": "pyrit.score.message_scorable_resolver",
     "MessageScorable": "pyrit.score.scorable",
     "MessageScorer": "pyrit.score.message_scorer",
-    "MessageScoringOptions": "pyrit.score.message_scorer",
     "MethKeywordScorer": "pyrit.score.true_false.regex.meth_keyword_scorer",
     "MetricsType": "pyrit.score.scorer_evaluation.metrics_type",
     "NerveAgentKeywordScorer": "pyrit.score.true_false.regex.nerve_agent_keyword_scorer",

pyrit/score/conversation_scorer.py

@@ -9,6 +9,7 @@
     ContentScorable,
     Message,
     MessagePiece,
+    Scorable,
     Score,
     ScoringExpectation,
 )
@@ -60,14 +61,35 @@ def _build_scoring_message(self, *, message: Message) -> Message | None:
         Keep the trigger that identifies the conversation to acquire.
 
         The trigger content is not sent to the child scorer. ``_score_prepared_message_async``
-        replaces it with a text view of the full conversation. The base class applies
-        ``skip_on_error_result`` before this hook.
+        replaces it with a text view of the full conversation. Overriding this hook keeps an
+        unreadable trigger, because the conversation behind it is still there to read.
 
         Returns:
             Message | None: The trigger message, or None if it has no pieces.
         """
         return message if message.message_pieces else None
 
+    def _reads_any_role(self, *, message: Message, anchor: Scorable | None) -> bool:
+        """
+        Defer role policy until the conversation locator has acquired its evidence.
+
+        Returns:
+            bool: True because the trigger identifies history; it is not the evidence itself.
+        """
+        return True
+
+    def _build_fallback_score(self, *, message: Message, objective: str | None) -> list[Score]:
+        """
+        Preserve silence when the acquired conversation or wrapped scorer has no verdict.
+
+        Returns:
+            list[Score]: An empty list.
+        """
+        return []
+
+    def _validate_scoring_message(self, *, message: Message, objective: str | None) -> None:
+        """Skip message validation because the trigger is only a conversation locator."""
+
     async def _score_prepared_message_async(
         self,
         *,
@@ -123,7 +145,7 @@ async def _score_prepared_message_async(
         for conv_message in conversation:
             for piece in conv_message.message_pieces:
                 # Only include user and assistant messages in the conversation text
-                if piece.api_role in ["user", "assistant", "tool"]:
+                if piece.api_role in ["user", "assistant", "tool"] and self._validator.is_role_supported(piece):
                     role_display = "Assistant (simulated)" if piece.is_simulated else piece.api_role.capitalize()
                     # For blocked pieces with partial content, use the partial content
                     # instead of the error JSON when should_score_blocked_content is enabled
@@ -137,6 +159,9 @@ async def _score_prepared_message_async(
                         text = piece.converted_value
                     conversation_text += f"{role_display}: {text}\n"
 
+        if not conversation_text:
+            return []
+
         wrapped_scorer = self._get_wrapped_scorer()
         scores = await wrapped_scorer._score_nested_async(
             scorable=ContentScorable(value=conversation_text),
@@ -193,7 +218,7 @@ def create_conversation_scorer(
         scorer (Scorer): The true/false or float-scale scorer to wrap for
             conversation-level evaluation. It must support text ``ContentScorable`` evidence.
         validator (ScorerPromptValidator | None): Optional validator override.
-            If not provided, uses the wrapped scorer's validator.
+            If not provided, uses the conversation scorer's default text validator.
 
     Returns:
         Scorer: A ConversationScorer instance that is also an instance of the wrapped scorer's type.

pyrit/score/float_scale/float_scale_scorer.py

@@ -72,17 +72,13 @@ class MessageFloatScaleScorer(FloatScaleScorer, MessageScorer):
     to which a response exhibits certain characteristics. Each piece in a request response
     is scored independently, returning one score per piece.
 
-    **Default error / blocked behavior**
-
-    When no supported pieces remain after validator filtering (e.g. the response is
-    blocked, has another error type, or no piece matches the scorer's supported data
-    types), the base ``score_async`` invokes ``_build_fallback_score`` and returns a
-    single ``Score`` with value ``0.0``. The rationale distinguishes blocked / error /
-    filtered cases. This mirrors ``MessageTrueFalseScorer``'s ``False`` default so that
-    downstream consumers (attack strategies, threshold wrappers) get a consistent,
-    "attack did not succeed" value without each call site needing special-cased error
-    handling. Subclasses that need different semantics (e.g. a refusal-style
-    "blocked = True") should override ``_score_piece_async`` or ``_build_fallback_score``.
+    **Default unreadable / blocked behavior**
+
+    A message that has no role supported by this scorer produces no score. For a supported
+    role, an unreadable transport or protocol response produces an undetermined score. A
+    fully blocked response or one with no supported data type produces a completed ``0.0``
+    score. Subclasses can override ``_build_fallback_score`` when they need different
+    semantics.
     """
 
     def __init__(