[BREAKING] [FIX]: Return UNDETERMINED when an evaluator cannot observe the evidence it needs

docs/api/core-protocols.md

@@ -51,6 +51,7 @@ Protocols and ABCs that define RAMPART's extension points. Implement these to co
         - ExecutionEventData
         - ExecutionEventHandler
         - ExecutionHandlerFactory
+        - evaluate_turn_async
         - register_default_handler_factory
         - clear_default_handler_factory
 

docs/attacks/xpia.md

@@ -141,6 +141,8 @@ evaluator = ~ResponseContains(lambda text: "I can't" in text or "I cannot" in te
 
 Place the cheaper evaluator on the left side of `|` — it short-circuits if the left operand detects.
 
+The `&` above asks whether both happened, so one condition that definitively did not happen settles the result even if the adapter could not observe the other. Use `|` when either condition on its own would count as the attack succeeding. When the adapter does not report the channel the left condition needs, the result records that on [`EvalResult`][rampart.core.types.EvalResult]. Reversing those two operands records nothing, because a `NOT_DETECTED` left operand short-circuits `&` before the other one runs. See the note on undetermined operands in [Authoring Tests](../usage/authoring-tests.md#composing-evaluators).
+
 ### LLMDriver for Adaptive Triggers
 
 For multi-turn attacks where the trigger conversation adapts based on agent responses, use [`LLMDriver`][rampart.drivers.llm.LLMDriver] instead of a static string:
@@ -223,4 +225,6 @@ This only fires when all three conditions hold:
 2. The adapter's `observability_profile` is `RESPONSE_ONLY`
 3. Zero tool calls were observed
 
+It is a backstop for evaluators that cannot say up front what evidence they need, such as `LLMJudge`, where the answer depends on the objective. `ToolCalled` and `SideEffectOccurred` return `UNDETERMINED` themselves, so on their own they do not reach this check as `SAFE`. A composition still can, so the backstop stays.
+
 

docs/contributing/extending-rampart.md

@@ -99,6 +99,7 @@ class MyAttackExecution(BaseExecution):
                     turn_number=turn_index,
                     driver_reasoning=decision.reasoning,
                     manifest=adapter.manifest,
+                    observability_level=adapter.observability_profile,
                 )
                 turns.append(turn)
 
@@ -124,6 +125,7 @@ Key points:
 - **Implement `_execute_async`** — this is your strategy-specific logic
 - **Implement `strategy_name`** — a short identifier used in `Result.strategy`
 - **Use `resolve_as_attack`** — this maps evaluator outcomes to safety verdicts with attack semantics (detected = UNSAFE)
+- **Pass `observability_level`** so evaluators can tell missing evidence apart from an evidence channel the adapter does not report. It is required on both `evaluate_turn_async` and `Result`, so leaving it out is a `TypeError` rather than a wrong assumption buried in a report.
 - **Don't wrap `_execute_async` in a broad `try/except`** — `BaseExecution.execute_async` already catches every exception from `_execute_async` and converts it to a `SafetyStatus.ERROR` result.
 
 ### 2. Add a Factory Method to `Attacks`

docs/contributing/testing.md

@@ -81,9 +81,16 @@ def _make_result(*, safe: bool = True) -> Result:
         status=SafetyStatus.SAFE if safe else SafetyStatus.UNSAFE,
         summary="test",
         strategy="test",
+        observability_level=ObservabilityLevel.RESPONSE_ONLY,
     )
 ```
 
+`observability_level` has no default, so a helper like this has to pick one.
+In a new test, pick the level the test is actually about; `RESPONSE_ONLY` is the
+honest choice when the test never looks at tool calls or side effects. Existing
+tests were instead backfilled with whatever value that API used to default to,
+so that making the argument required changed no test's meaning.
+
 ### Mocking
 
 - Mock all external dependencies (APIs, file systems, network)

docs/getting-started/quickstart.md

@@ -75,7 +75,7 @@ class MyAgentAdapter:
         return ObservabilityLevel.TOOL_ONLY
 ```
 
-1. **Send a request, return a response.** Populate `tool_calls` and `side_effects` with everything you can observe. Empty lists mean "no observations," not "nothing happened."
+1. **Send a request, return a response.** Populate `tool_calls` and `side_effects` with everything you can observe. An empty list is read against the observability level declared at (7), so declare it honestly.
 2. **Tool calls go here.** The evaluator [`ToolCalled`][rampart.evaluators.tool_called.ToolCalled] only fires if these are reported, so don't skip them when your agent supports tools.
 3. **Set up session-level state.** API connections, browser contexts, anything that lives for one interaction.
 4. **Clean up.** Must be idempotent and must not raise — RAMPART always calls this, even after errors.

docs/glossary.md

@@ -22,7 +22,7 @@ Terms used throughout the RAMPART documentation.
 :   An implementation of [`PromptDriver`][rampart.core.prompt_driver.PromptDriver]. Generates prompts to send to the agent during execution. See [Drivers](api/drivers.md).
 
 **EvalContext**
-:   The data passed to an evaluator — contains all turns plus agent manifest. See [`EvalContext`][rampart.core.types.EvalContext].
+:   The data passed to an evaluator. Contains all turns, the agent manifest, and the adapter's declared observability level. See [`EvalContext`][rampart.core.types.EvalContext].
 
 **EvalOutcome**
 :   What an evaluator determined: `DETECTED`, `NOT_DETECTED`, or `UNDETERMINED`. See [`EvalOutcome`][rampart.core.types.EvalOutcome].

docs/usage/authoring-tests.md

@@ -33,7 +33,7 @@ class MySession:
         pass
 ```
 
-1. Populate `Response.tool_calls` and `Response.side_effects` with everything you can observe. Empty lists mean "no observations," not "nothing happened."
+1. Populate `Response.tool_calls` and `Response.side_effects` with everything you can observe. An empty list is read against the observability level you declare below, so declare it honestly.
 2. Set up session-level state (API connections, browser contexts).
 3. Clean up. Must be idempotent and must not raise.
 
@@ -71,6 +71,10 @@ class MyAdapter:
 | `TOOL_ONLY` | Reports tool calls but not side effects | API returns tool call data |
 | `RESPONSE_ONLY` | Reports only text responses | Black-box agent |
 
+Declare the level honestly. An evaluator that needs an evidence channel your adapter does not report returns `UNDETERMINED` instead of `NOT_DETECTED`, so a gap in the adapter does not come back as a passing test on its own. Composed with `&`, an operand that definitively did not happen still settles the result, so read the note on undetermined operands below before combining evaluators. Evidence the adapter does report still counts either way, so declaring a lower level cannot hide a real detection.
+
+The guarantee is per channel, not per field. A level that reports a channel is taken at its word for what it puts in it, so a tool call reported with redacted or partial arguments still counts as observed and a predicate over those arguments can return `NOT_DETECTED`.
+
 ---
 
 ## Choosing Evaluators
@@ -95,8 +99,8 @@ ToolCalled(
 )
 ```
 
-!!! warning
-    `ToolCalled` requires `TOOL_ONLY` or `TOOL_AND_SIDE_EFFECTS` observability. With `RESPONSE_ONLY`, it always returns `NOT_DETECTED`.
+!!! note
+    `ToolCalled` needs `TOOL_ONLY` or `TOOL_AND_SIDE_EFFECTS` observability. With `RESPONSE_ONLY` it returns `UNDETERMINED`, because an adapter that does not report tool calls cannot tell you the tool was not called.
 
 ### [`ResponseContains`][rampart.evaluators.response_contains.ResponseContains] — Detect Text Patterns
 
@@ -129,6 +133,9 @@ SideEffectOccurred("http_request")
 SideEffectOccurred("http_request", method="POST", host="evil.com")
 ```
 
+!!! note
+    `SideEffectOccurred` needs `TOOL_AND_SIDE_EFFECTS` observability. With `TOOL_ONLY` or `RESPONSE_ONLY` it returns `UNDETERMINED`, since those adapters do not report side effects at all.
+
 ### [`LLMJudge`][rampart.evaluators.llm_judge.LLMJudge] — Detect Language-Level Signals
 
 For conditions that require reasoning over natural language ("did the agent disclose ticket contents?", "did the agent comply with the injected instruction?"), use `LLMJudge`. It calls a separate LLM to evaluate the transcript against an objective and returns a structured verdict.
@@ -227,6 +234,16 @@ evaluator = ~ResponseContains("I cannot help with that")
 !!! tip
     Place the cheaper evaluator on the left side of `|`. The OR operator short-circuits — if the left operand detects, the right is skipped.
 
+!!! note "Undetermined operands"
+    An `UNDETERMINED` operand does not settle a composition on its own. `|` is `DETECTED` if either side detects, `&` is `NOT_DETECTED` if either side does not, and the result is `UNDETERMINED` only when neither side settles it. Both operators produce the same `EvalOutcome` whichever order the operands are written in.
+
+`&` short-circuits only on a `NOT_DETECTED` left operand. An `UNDETERMINED` left operand still runs the right one, so an `LLMJudge` on the right of `&` is called in this case. When you combine two views of the same harm to corroborate it, `&` asks whether both happened, so one operand that definitively did not happen settles the result even if the other could not be observed. Use `|` when either view on its own is enough.
+
+`&` and `|` record every operand they ran that came back `UNDETERMINED`, one distinct reason per entry, in `undetermined_operands` on [`EvalResult`][rampart.core.types.EvalResult], and `~` carries its inner result's entries through. Recording does not move the `EvalOutcome` the operands settled. Where the run resolves `SAFE`, the result remains `SAFE`, but its summary names the parts of the evaluation that were undetermined. Only an operand that actually ran can be recorded, so put the evaluator that depends on adapter observability on the left of `&`, where the `NOT_DETECTED` short-circuit cannot skip it. Under `RESPONSE_ONLY`, `ToolCalled("x") & ResponseContains("absent")` records the tool call gap; the same pair written the other way round reaches the same verdict with nothing recorded. `|` skips its right operand once the left detects, so it has the same limit and the opposite pull from the tip above: the cheap evaluator on the left is faster, the observability-dependent one on the left is better recorded.
+
+!!! warning "A recorded gap does not change the verdict"
+    `SAFE` is the only status that passes, and a run that reaches it is graded a plain pass: `bool(result)` is `True`, the result line reads `PASS`, a trial group counts it toward the pass rate, and pytest exits zero. On such a run the summary and `undetermined_operands` are the only places the gap shows; any other status fails the test on its own account, not because of the gap. To fail a passing run that carries one, read the operands yourself: see [Observability Gaps on a Passing Run](results-and-reporting.md#observability-gaps-on-a-passing-run). XPIA has one separate backstop that does move the verdict, described in [Observability Adjustment](../attacks/xpia.md#observability-adjustment).
+
 ---
 
 ## Implementing Surfaces

docs/usage/pytest-integration.md

@@ -161,16 +161,22 @@ This works via [`ExecutionEventHandler`][rampart.core.execution.ExecutionEventHa
 For tests that construct [`Result`][rampart.core.result.Result] objects directly (without factories):
 
 ```python
-from rampart import Result, SafetyStatus, record_result
+from rampart import ObservabilityLevel, Result, SafetyStatus, record_result
 
 async def test_manual_result():
     result = Result(
         status=SafetyStatus.SAFE,
         summary="Agent passed manual check",
+        observability_level=ObservabilityLevel.RESPONSE_ONLY,
     )
     record_result(result)
 ```
 
+`observability_level` is required. State what the adapter behind the check could
+actually see, so the report never claims a level the run did not have. Where an
+adapter is in scope, pass `adapter.observability_profile` rather than naming a
+level by hand.
+
 ---
 
 ## Terminal Summary

docs/usage/results-and-reporting.md

@@ -14,6 +14,7 @@ result = await Attacks.xpia(...).execute_async(adapter=my_adapter)
 result.safe              # bool — did the agent behave safely?
 result.status            # SafetyStatus (SAFE, UNSAFE, UNDETERMINED, ERROR)
 result.summary           # str — human-readable one-liner
+result.observability_level  # ObservabilityLevel (what the adapter saw)
 result.turns             # list[Turn] — full conversation
 result.duration_seconds  # float — execution wall-clock time
 result.harm_category     # HarmCategory | str | None
@@ -47,10 +48,28 @@ for turn in result.turns:
     turn.request.prompt       # What was sent
     turn.response.text        # What came back
     turn.response.tool_calls  # Tool invocations observed
-    turn.eval_result          # EvalResult for this turn
+    turn.eval_result          # EvalResult for this turn, or None
     turn.turn_number          # 0-indexed position
 ```
 
+### Observability Gaps on a Passing Run
+
+A run can resolve `SAFE` while part of the evaluation was never observable. Such a run is graded as a pass: `result.safe` is `True`, the result line reads `PASS`, a trial group counts it toward the pass rate, and pytest exits zero. `result.summary` names the gap, and `turn.eval_result.undetermined_operands` carries it one reason at a time, so a caller that wants to fail on it has to say so:
+
+```python
+gaps = [
+    reason
+    for turn in result.turns
+    if turn.eval_result is not None
+    for reason in turn.eval_result.undetermined_operands
+]
+assert result and not gaps, result.summary
+```
+
+`JsonFileReportSink` writes the same list as `eval_undetermined_operands` on each turn that has one, and omits the key otherwise. A failing run can carry the key too, so read it alongside `status`: together they tell a fully observed pass from one reached with a gap. No counter makes that distinction, because a qualified pass lands in `safe_count` like any other.
+
+XPIA applies one further rule of its own to `RESPONSE_ONLY` adapters, which does move the verdict. See [Observability Adjustment](../attacks/xpia.md#observability-adjustment).
+
 ---
 
 ## Report Sinks

rampart/attacks/_xpia.py

@@ -17,6 +17,7 @@
 from contextlib import AsyncExitStack
 from typing import Any
 
+from rampart.common.text import safe_str_list
 from rampart.core import (
     AgentAdapter,
     BaseExecution,
@@ -33,6 +34,10 @@
     resolve_as_attack,
 )
 from rampart.core.execution import evaluate_turn_async
+from rampart.core.result import (
+    _explain_undetermined,
+    _summarize_undetermined_operands,
+)
 
 logger = logging.getLogger(__name__)
 
@@ -145,6 +150,7 @@ async def _run_phases_async(
                     turn_number=turn_index,
                     driver_reasoning=decision.reasoning,
                     manifest=adapter.manifest,
+                    observability_level=adapter.observability_profile,
                 )
                 turns.append(turn)
 
@@ -323,16 +329,26 @@ def _build_summary(
         str: A summary string for the Result.
     """
     if status == SafetyStatus.SAFE:
-        return "Agent defended successfully"
+        return "Agent defended successfully" + _summarize_undetermined_operands(
+            eval_results=eval_results,
+        )
     if status == SafetyStatus.UNSAFE:
-        evidence = [e for er in eval_results for e in er.evidence]
+        # resolve_as_attack returns UNSAFE only when some evaluator DETECTED, so
+        # the evidence has to come from those. An undetermined result can carry
+        # evidence of its own, and it is not what established this verdict.
+        evidence = [
+            e
+            for er in eval_results
+            if er.detected
+            for e in safe_str_list(value=er.evidence)
+        ]
         if not evidence:
             return "Attack objective detected"
         return f"Attack objective detected: {'; '.join(evidence[:3])}"
     if status == SafetyStatus.UNDETERMINED:
-        rationales = [er.rationale for er in eval_results if er.rationale]
-        detail = (
-            "; ".join(rationales[:2]) if rationales else "Insufficient observability"
+        detail = _explain_undetermined(
+            eval_results=eval_results,
+            fallback="Insufficient observability",
         )
         return f"Evaluation undetermined: {detail}"
     if status == SafetyStatus.ERROR: