Refuse a string UTF-8 cannot encode, in words rather than as a UnicodeEncodeError

docs/CLAIMS.md

@@ -53,7 +53,7 @@ catalogue, `GUARANTEES` (`verify/guarantees.py:39`).
 | "An action the policy does not list is denied" | `Policy.evaluate` — `policy.py:678` | `test_T6_unknown_action_is_denied_with_reason_unknown_action` |
 | "Authority first ... then policy" / "authority first" | `Control.execute` evaluates authority before policy and a denial appends `AUTHORITY_DENIED` and never `POLICY_EVALUATED` — `control.py:542` | `test_T74_a_denial_leaves_no_pending_approval_request` |
 | "Neither axis reads the agent's instructions" | `Policy.evaluate` — `policy.py:678` — sees the action's name and arguments; `Authority.evaluate` — `authority.py:807` — sees the action and the principal; neither is handed a prompt, a message or a tool result | `test_T6_an_action_name_is_matched_exactly`, `test_T67_a_principal_with_no_grant_is_denied` |
-| "canonical arguments (sorted keys, no floats) ... Its SHA-256 is the action hash" | `canonicalize` / `action_hash` — `action.py`; `float` refused at any depth — `action.py:47` | `test_T7_canonical_form_is_exactly_the_specified_serialization`, `test_T7_nested_dicts_are_sorted_recursively` |
+| "canonical arguments (sorted keys, no floats) ... Its SHA-256 is the action hash" | `canonicalize` / `action_hash` — `action.py`; `float` refused at any depth — `action.py:71` | `test_T7_canonical_form_is_exactly_the_specified_serialization`, `test_T7_nested_dicts_are_sorted_recursively` |
 | "The approval is single-use, expires, and matches nothing but that exact action." | `_authorize_and_reserve` — `state.py:780` — checks expiry at consumption | `test_T5_expiry_is_checked_at_consumption_not_only_at_grant`, `test_T4_replaying_the_approval_raises_ApprovalMismatch_with_reason_consumed` |
 | "Only `NotExecuted`, raised by you, means `FAILED`." | `control.py:1036`; `NotExecuted` — `errors.py:157` | `test_T1_a_lost_response_leaves_the_effect_ambiguous` |
 | "the hash of the policy that decided it, chained to the receipt before it" | `Policy.policy_hash` — `policy.py:559`; `prev_hash`, `GENESIS_HASH` for the first — `receipt.py:284` | `test_T172_every_receipt_carries_the_hash_and_the_declared_version`, `test_T164_an_altered_receipt_is_content_altered_at_its_seq` |
@@ -84,7 +84,7 @@ catalogue, `GUARANTEES` (`verify/guarantees.py:39`).
 |---|---|---|
 | "cheap to undo is autonomous, anything that leaves the building needs a human, money is by amount with both ends bound" | `Decision` — `policy.py:234` — is exactly `allow`, `approve`, `deny`; rules match first-wins over `Condition` (`policy.py:298`) with the operators `eq`, `neq`, `in`, `lt`, `lte`, `gt`, `gte` — `_OPERATORS` — `policy.py:83` | `test_T6_an_action_name_is_matched_exactly`, `test_T176_the_operators_behave_as_they_do_everywhere_else` |
 | "Unknown actions are denied; there is no default-allow." | `Policy.evaluate` — `policy.py:678` | `test_T6_unknown_action_is_denied_with_reason_unknown_action` |
-| "Amounts are integer minor units; floats are rejected outright" | `float` refused at any depth — `action.py:47` | `test_T7_canonical_form_is_exactly_the_specified_serialization` |
+| "Amounts are integer minor units; floats are rejected outright" | `float` refused at any depth — `action.py:71` | `test_T7_canonical_form_is_exactly_the_specified_serialization` |
 | "The policy cannot see who is asking — deliberately, since v0.1" | `Policy.evaluate` still takes only the action's name and arguments; `RESERVED_ARGUMENTS` — `policy.py:150` — refuses `agent_eq` and every other principal-addressing condition at load, in a document of **every** schema version | `test_T74b_a_reserved_name_in_a_policy_rule_is_a_load_error`, `test_T74b_a_reserved_name_in_a_grant_constraint_is_a_load_error` |
 | "the second axis, `authority:`" | `Authority.evaluate` — `authority.py:807`; `Control._authority_result` — `control.py:408` | `test_T67_a_principal_with_no_grant_is_denied` |
 | "opt-in, and then fail-closed" | `_optional_authority` returns `None` for a document with no section — `control.py`; `Control.authority is None` is v0.2 behaviour exactly | `test_T66_a_document_with_no_authority_section_leaves_control_authority_none`, `test_T66_no_authority_event_is_appended_without_a_section`, and T66's session-wide guard in `tests/conftest.py` |
@@ -189,7 +189,7 @@ restating the code; the ones that are new to the site carry their own code and p
 | `get-started/install` | "importing `ctrlrun` imports nothing from an extra" | `test_T30_a_subprocess_importing_ctrlrun_pulls_in_no_module_from_an_extra` |
 | `get-started/install` | "raises `MissingDependency` with the install command in the message" | `test_a_missing_extra_raises_MissingDependency_naming_the_install_command` |
 | `get-started/quickstart` | every block on the page, and the outputs shown | the blocks are `runnable` and pass `tools/docs_audit/snippets.py` in one temporary directory, in order; the outputs are pasted from one run of the same blocks |
-| `concepts/action-and-hash` | "The action hash is the SHA-256 of that canonical form"; sorted keys, no whitespace, UTF-8, `float` rejected; `action_id` excluded | `canonicalize` / `action_hash` — `action.py`; `float` refused — `action.py:47`; `test_T7_canonical_form_is_exactly_the_specified_serialization`, `test_T7_nested_dicts_are_sorted_recursively`, `test_T60_claims_do_not_change_the_action_hash` |
+| `concepts/action-and-hash` | "The action hash is the SHA-256 of that canonical form"; sorted keys, no whitespace, UTF-8, `float` rejected; `action_id` excluded | `canonicalize` / `action_hash` — `action.py`; `float` refused — `action.py:71`; `test_T7_canonical_form_is_exactly_the_specified_serialization`, `test_T7_nested_dicts_are_sorted_recursively`, `test_T60_claims_do_not_change_the_action_hash` |
 | `concepts/decisions` | three decisions, first match wins, unknown denied, principal-addressing conditions refused at load | the "Write down what the agent may do" rows above |
 | `concepts/approval-binding` | A1–A4, the mismatch leaving the approval granted, one core provider writing every grant | the matrix row "An approval is bound to the exact action…", the "Three ways to use it" adapter rows, and `test_T2_a_mutated_action_leaves_the_approval_granted` |
 | `concepts/approval-binding` | the `DENY` and `ALLOW` rows when the policy changed between grant and consumption | "the approval is re-checked against the policy in force at execution" above |

docs/reference/api/Action.mdx

@@ -5,7 +5,7 @@ description: "A proposed agent action: what, with which arguments, by whom, on w
 
 {/* generated by tools/docs_audit/render_api.py from the docstrings — edit the docstring, never this page */}
 
-`ctrlrun.Action` — class, defined at `src/ctrlrun/action.py:151`
+`ctrlrun.Action` — class, defined at `src/ctrlrun/action.py:178`
 
 ```python
 from ctrlrun import Action

docs/reference/api/Principal.mdx

@@ -5,7 +5,7 @@ description: "Who is acting: an agent, optionally on behalf of a human."
 
 {/* generated by tools/docs_audit/render_api.py from the docstrings — edit the docstring, never this page */}
 
-`ctrlrun.Principal` — class, defined at `src/ctrlrun/action.py:112`
+`ctrlrun.Principal` — class, defined at `src/ctrlrun/action.py:139`
 
 ```python
 from ctrlrun import Principal

docs/reference/api/action_hash.mdx

@@ -5,7 +5,7 @@ description: "Return the action hash used to bind approvals to an exact action (
 
 {/* generated by tools/docs_audit/render_api.py from the docstrings — edit the docstring, never this page */}
 
-`ctrlrun.action_hash` — function, defined at `src/ctrlrun/action.py:283`
+`ctrlrun.action_hash` — function, defined at `src/ctrlrun/action.py:327`
 
 ```python
 from ctrlrun import action_hash

docs/reference/api/canonical_bytes.mdx

@@ -5,7 +5,7 @@ description: "The canonical form of an arbitrary mapping: UTF-8 JSON, sorted key
 
 {/* generated by tools/docs_audit/render_api.py from the docstrings — edit the docstring, never this page */}
 
-`ctrlrun.canonical_bytes` — function, defined at `src/ctrlrun/action.py:201`
+`ctrlrun.canonical_bytes` — function, defined at `src/ctrlrun/action.py:228`
 
 ```python
 from ctrlrun import canonical_bytes

docs/reference/api/canonicalize.mdx

@@ -5,7 +5,7 @@ description: "Return the canonical form of an Action: UTF-8 JSON, sorted keys, n
 
 {/* generated by tools/docs_audit/render_api.py from the docstrings — edit the docstring, never this page */}
 
-`ctrlrun.canonicalize` — function, defined at `src/ctrlrun/action.py:264`
+`ctrlrun.canonicalize` — function, defined at `src/ctrlrun/action.py:308`
 
 ```python
 from ctrlrun import canonicalize

fuzz/README.md

@@ -70,23 +70,34 @@ and requires the check to notice. A property that cannot fail proves nothing.
 are recorded rather than worked around in the decoder, because a fuzzer whose corpus is pruned
 to avoid its own findings reports zero forever.
 
-`test_the_known_findings_still_reproduce` asserts each one **still happens**. The day it is
-fixed that test goes red and the entry must be deleted — which is the point. A recorded limit
-that quietly starts passing is the failure mode this directory is about.
+`test_the_seed_corpus_reproduces_every_known_finding` compares the dict against what the corpus
+actually produces, in **both** directions: an entry cannot be added without a reproducer, and a
+reproducer cannot quietly stop reproducing.
 
-### `lone-surrogate-in-a-string`
+**It is currently empty, and that is a result rather than a default.**
 
-`canonical_bytes` raises `UnicodeEncodeError`, not `InvalidArgument`, for a string holding an
-unpaired UTF-16 surrogate.
+### `lone-surrogate-in-a-string` — found here, fixed in `action.py`
 
-Reachable: `json.loads('"\ud800"')` produces one, so an MCP tool call can carry it into the
-action path. `Action(...)` accepts it and `action_hash` is where it fails.
+`canonical_bytes` raised `UnicodeEncodeError`, not `InvalidArgument`, for a string holding an
+unpaired UTF-16 surrogate. Reachable the ordinary way: `json.loads` produces one from a
+six-character escape, so an MCP tool call carried it into the action path; `Action(...)`
+accepted it and `action_hash` was where it failed.
 
-Fail-closed **holds** — both the gateway and `Control` wrap the action path in
-`except Exception` — so this is a contract violation and a crash, not an authorization bypass.
-What it breaks is the closed error set in `errors.py` and `InvalidArgument`'s documented
-promise: a caller catching `CTRLRunError` does not catch this.
+Fail-closed **held** — the gateway and `Control` both wrap the action path in
+`except Exception` — so it was a contract violation and a crash, not an authorization bypass.
+What it broke was the closed error set in `errors.py`: a caller catching `CTRLRunError` did not
+catch it.
 
-The fix is narrow and belongs to whoever owns `src/`: reject unencodable strings in
-`_no_floats` alongside the float and non-string-key checks. It changes no hash that previously
-succeeded, so `v0.1 §2.3`'s "old hashes still verify" rule is satisfied without a schema bump.
+`_encodable` in `action.py` now refuses it at construction, beside the `float` and
+non-string-key checks, and `canonical_bytes` refuses it again at the encode. A refusal and not
+a repair: `errors="replace"` would map two distinct arguments onto one canonical form, which is
+the collision §2.3 exists to prevent. A *paired* surrogate is unaffected, and a negative control
+asserts that, because a check keyed on "contains a surrogate code point" rather than on
+encodability would reject an ordinary emoji and pass every other test here.
+
+No hash that previously succeeded changes, so §2.3's "old hashes still verify" rule holds
+without a schema bump.
+
+**The entry did not have to be noticed and removed by hand.** Its own test asserted the finding
+*still reproduced*, so fixing the bug turned that test red — which is what forced the entry out
+and the reproducer in `corpus/canonical/seed-10-surrogate` into a regression input.

fuzz/properties.py

@@ -37,32 +37,20 @@ def policy_from_yaml(text: str, *, source: str = "<string>") -> Policy:
 
 # --- known findings ---------------------------------------------------------------------------
 
-#: A finding that is real, reported, and not yet fixed. It is recorded here rather than worked
-#: around in the decoder, because a fuzzer whose corpus is pruned to avoid its own findings is
-#: a fuzzer that reports zero forever.
+#: Findings that are real, reported, and not yet fixed -- recorded here rather than worked
+#: around in the decoder, because a fuzzer whose corpus is pruned to avoid its own findings
+#: reports zero forever.
 #:
-#: `test_the_known_findings_still_reproduce` asserts each of these **still happens**. The day
-#: one is fixed that test goes red and the entry must be deleted -- which is the point. A
-#: documented limit that quietly starts passing is the failure mode this whole file is about.
-KNOWN_FINDINGS: dict[str, str] = {
-    "lone-surrogate-in-a-string": (
-        "canonical_bytes raises UnicodeEncodeError, not InvalidArgument, for a string holding "
-        "an unpaired UTF-16 surrogate. Reachable: json.loads('\"\\\\ud800\"') produces one, so "
-        "an MCP tool call can carry it. Fail-closed holds -- the gateway and Control both wrap "
-        "the action path in `except Exception` -- but the error escapes the closed set in "
-        "errors.py and violates InvalidArgument's documented contract."
-    ),
-}
-
-
-def _has_surrogate(value: object) -> bool:
-    if isinstance(value, str):
-        return any(0xD800 <= ord(char) <= 0xDFFF for char in value)
-    if isinstance(value, dict):
-        return any(_has_surrogate(k) or _has_surrogate(v) for k, v in value.items())
-    if isinstance(value, list | tuple):
-        return any(_has_surrogate(item) for item in value)
-    return False
+#: **Empty, and the machinery stays.** It held `lone-surrogate-in-a-string` --
+#: `canonical_bytes` raising `UnicodeEncodeError` instead of `InvalidArgument` for an unpaired
+#: UTF-16 surrogate -- until that was fixed in `action.py`. The entry's own test asserted the
+#: finding *still reproduced*, so the fix turned it red and the entry had to go. That is the
+#: mechanism working, and it is why the dict is left in place for the next one.
+#:
+#: `test_the_seed_corpus_reproduces_every_known_finding` compares this to what the corpus
+#: actually produces, in both directions, so an entry cannot be added without a reproducer and
+#: a reproducer cannot start failing unnoticed.
+KNOWN_FINDINGS: dict[str, str] = {}
 
 
 # --- decoders ---------------------------------------------------------------------------------
@@ -211,10 +199,14 @@ def check_canonical(document: dict[Any, Any]) -> str | None:
         first = _encode(document, "the first pass")
     except InvalidArgument:
         return None  # a refusal in words is the contract working
-    except UnicodeEncodeError:
-        if _has_surrogate(document):
-            return "lone-surrogate-in-a-string"
-        raise AssertionError("UnicodeEncodeError with no surrogate in the input") from None
+    except UnicodeEncodeError as exc:
+        # Was `KNOWN_FINDINGS["lone-surrogate-in-a-string"]`. `action.py` refuses an
+        # unencodable string with `InvalidArgument` now, so this escaping again is a
+        # regression and not a recorded limit.
+        raise AssertionError(
+            f"canonicalization raised UnicodeEncodeError, which is outside the closed error "
+            f"set in errors.py: {exc}"
+        ) from exc
 
     assert not must_refuse, "canonicalization accepted a document holding " + " and ".join(
         must_refuse

src/ctrlrun/action.py

@@ -42,14 +42,40 @@ def _new_action_id() -> str:
     return f"act_{secrets.token_hex(_ID_HEX_BYTES)}"
 
 
+def _encodable(text: str, path: str) -> str:
+    """Refuse a `str` that UTF-8 cannot represent (SPEC-v0.1 §2.3).
+
+    A lone UTF-16 surrogate is such a string: Python accepts it, UTF-8 has no encoding for it,
+    and it arrives the ordinary way -- `json.loads('"\\ud800"')` produces one, so an MCP tool
+    call carries it into the action path. It is refused here for the reason `float` is: an
+    argument that cannot be canonicalized cannot be hashed, and an `Action` that can be built
+    but never hashed is a trap set at construction and sprung somewhere else.
+
+    A refusal and not a repair. `errors="replace"` would map two distinct arguments to one
+    canonical form, which is the collision §2.3 exists to prevent. A *paired* surrogate is not
+    affected: Python has already decoded it to the character it denotes.
+    """
+    try:
+        text.encode("utf-8")
+    except UnicodeEncodeError as exc:
+        raise InvalidArgument(
+            f"{path} holds a string UTF-8 cannot encode at position {exc.start}: {exc.reason}. "
+            "A lone surrogate has no canonical form, and substituting one would give two "
+            "distinct actions the same hash (v0.1 §2.3)"
+        ) from exc
+    return text
+
+
 def _frozen_value(value: object, path: str) -> FrozenValue:
     """Validate an argument value and return a deep-frozen copy of it."""
     if isinstance(value, float):
         raise InvalidArgument(
             f"float is not an allowed argument type at {path}: "
             "use integer minor units (amount=200000) or a decimal string ('2000.00')"
         )
-    if value is None or isinstance(value, str | int):  # bool is a subclass of int
+    if isinstance(value, str):
+        return _encodable(value, path)
+    if value is None or isinstance(value, int):  # bool is a subclass of int
         return value
     if isinstance(value, Mapping):
         return _frozen_mapping(value, path)
@@ -65,6 +91,7 @@ def _frozen_mapping(value: Mapping[Any, Any], path: str) -> Mapping[str, FrozenV
     for key, item in value.items():
         if not isinstance(key, str):
             raise InvalidArgument(f"argument keys must be str, got {type(key).__name__} at {path}")
+        _encodable(key, f"{path} key {key!r}")
         result[key] = _frozen_value(item, f"{path}.{key}")
     return MappingProxyType(result)
 
@@ -214,13 +241,30 @@ def canonical_bytes(payload: Mapping[str, Any]) -> bytes:
 
     `ctrlrun.action/v1` is unchanged by this promotion and T164b is the corpus that proves it.
     """
-    return json.dumps(
+    encoded = json.dumps(
         _no_floats(payload, "payload"),
         sort_keys=True,
         separators=(",", ":"),
         ensure_ascii=False,
         allow_nan=False,
-    ).encode("utf-8")
+    )
+    try:
+        return encoded.encode("utf-8")
+    except UnicodeEncodeError as exc:
+        # A lone UTF-16 surrogate: a `str` Python accepts and UTF-8 cannot represent. It
+        # arrives the ordinary way -- `json.loads('"\ud800"')` produces one -- so an MCP tool
+        # call carries it here, and this used to escape as `UnicodeEncodeError`, outside the
+        # closed set in `errors.py`. A caller catching `CTRLRunError` did not catch it.
+        #
+        # Checked here rather than per string in `_no_floats`: the encode already walks every
+        # character, so this costs nothing on the path that succeeds. It is a refusal, not a
+        # repair -- `errors="replace"` would map two distinct arguments to one canonical form,
+        # which is the collision the whole of §2.3 exists to prevent.
+        raise InvalidArgument(
+            f"an argument holds a string that UTF-8 cannot encode at position {exc.start}: "
+            f"{exc.reason}. A lone surrogate has no canonical form, and substituting one "
+            "would give two distinct actions the same hash (v0.1 §2.3)"
+        ) from exc
 
 
 def _no_floats(value: object, path: str) -> PlainValue: