From a70696a0161c507ad1de662ecd874068f625227f Mon Sep 17 00:00:00 2001 From: arpan Date: Tue, 15 Sep 2026 18:48:50 +0530 Subject: [PATCH 1/3] The landscape page, regenerated against 0.12.1, and the test it never had A submission to the OWASP Agentic Solutions Landscape is filled in from `docs/OWASP-SOLUTIONS-LANDSCAPE.md` and from nothing else. The page said it is regenerated when the guarantee catalogue changes and when a version it names is tagged. Both happened and it was not: it still read *written against v1.0, guarantees G1-G24* against a catalogue at v7 with thirty-two guarantees, hedged every row as a design row waiting on a 1.0 that has not shipped, and cited G28 anyway. Regenerated against 0.12.1, which is what a reader can install: - the header names 0.12.1, `G1`-`G32` and `ctrlrun.guarantees/v7`, and the design-row hedge is gone rather than left to discount rows that ship; - G27 is named in the connector-contracts row, with `upstream_mismatch` and `upstream_unverified`; G29, G30 and G32 are named in the immutable-logs row, because what retention does to a chain is the first thing a reviewer of that box will ask; - the Agentic Top 10 summary is regenerated from the mapping it cites, so each row's guarantees are exactly what `docs/OWASP-AGENTIC-TOP10.md` maps to that entry. **`ASI04` was the one that mattered.** The summary ticked it *Partly* on upstream identity pinning; the reading it points at says `ASI04` is out of scope, that no guarantee maps to it, and that the pinning is G27, which belongs under `ASI02` and `ASI07`, "not here". A form filled in from the summary would have ticked a box the page behind it refutes, which is the one failure this page exists to prevent. It is now `No`. The Top 10 reading had a stale row of its own. Its `ASI09` half said CTRLRun "does not authenticate the approver, does not model separation of duties" while G17, G18 and G19 sat in the table above it mapped to that entry. Rewritten to say what v0.8 closed and what it did not: the kernel matches a recorded entitlement and does not decide who is entitled, and a persuaded human still gives a valid approval. Its footer named `v2` and now names `v7`. `tests/test_owasp_landscape.py` is why this will not happen again. The mapping page kept up because `test_owasp_mapping.py` reads it; the page that leaves the project had no test at all. Twenty-two checks: every guarantee cited exists in the registry, the catalogue named is the one `verify` reports, the header's range spans the catalogue, no `Since` column names a version the changelog has not released (with 0.3.0's TestPyPI-only publication exempted by name and stated on the page), every status is one of the three words, and each entry's guarantees equal what the mapping maps to it, in both directions. --- docs/OWASP-AGENTIC-TOP10.md | 4 +- docs/OWASP-SOLUTIONS-LANDSCAPE.md | 57 +++--- tests/test_owasp_landscape.py | 287 ++++++++++++++++++++++++++++++ 3 files changed, 321 insertions(+), 27 deletions(-) create mode 100644 tests/test_owasp_landscape.py diff --git a/docs/OWASP-AGENTIC-TOP10.md b/docs/OWASP-AGENTIC-TOP10.md index 2fc177e..0c83b85 100644 --- a/docs/OWASP-AGENTIC-TOP10.md +++ b/docs/OWASP-AGENTIC-TOP10.md @@ -116,7 +116,7 @@ stated here rather than left implied: |---|---|---|---| | `ASI06:2026` | Memory & Context Poisoning | G6 and G1 constrain what an agent acting on a poisoned context can *do*: the action must still be named in the policy, so a belief an attacker planted cannot reach a tool the agent was never entitled to use, and an approval granted for one action cannot be spent on another. This is the same downstream constraint that makes `ASI01` partial, and it is here for the same reason. | CTRLRun never reads a model's memory, its context or its prompt, so it neither detects nor prevents the poisoning. And the shape poisoning most often takes is the one the kernel has least to say about: **corrupted arguments to an action the agent is entitled to take** — the right operation against the wrong record. Policy conditions, resource patterns and v0.6 data scope bite on part of that, and since v0.9 **G23 bites on the identifier itself**: where a deployment configures a scope provider, the kernel asks its system of record whether `customer:90210` is this principal's before reserving anything. That closes the sharpest version of this and not the category. The provider is the operator's own code reading the operator's own data, so a poisoned source answers wrongly with the kernel none the wiser, and a deployment that configures no provider is exactly where it was. | | `ASI01:2026` | Agent Goal Hijack | G1 and G6 constrain what a hijacked agent can *do*: it still meets the policy, and it still cannot present an approval granted for a different action. | CTRLRun does not detect or prevent the hijack. It never sees the prompt, the plan or the reasoning, so an agent whose goal was replaced proposes actions exactly as a healthy one would — and every action inside its policy and its grants will run. | -| `ASI09:2026` | Human-Agent Trust Exploitation | G1 and G2 close the shape where an approval a human gave for one action is spent on another, or spent twice. | CTRLRun does not authenticate the *approver*, does not model separation of duties, and has no opinion on whether the human was misled into approving. A human persuaded to approve the right action for the wrong reason gets a valid approval, and the receipt records it as one. | +| `ASI09:2026` | Human-Agent Trust Exploitation | G1 and G2 close the shape where an approval a human gave for one action is spent on another, or spent twice. Since 0.8.0 the approver is a resolved principal rather than a string: G17 refuses an approval whose recorded entitlement does not cover the role the request pinned, G18 refuses the requester approving their own action, compared on the principal and never on the approver string, and G19 counts distinct principals, so one person cannot be several of an M-of-N. | CTRLRun does not decide **who is entitled**. What entitled an approver was decided where the credential was verified, which is the operator MCP server or an embedding application; the kernel matches a recorded claim byte for byte, does not check that the role exists anywhere, and a cited control naming no role gates nobody. It has no opinion on whether the human was misled: a person persuaded to approve the right action for the wrong reason gives a valid approval, and the receipt records it as one. | --- @@ -135,5 +135,5 @@ every report as `descends_from`. --- This document is regenerated when the guarantee catalogue changes, and when OWASP publishes a -new edition. It was written against `ctrlrun.guarantees/v2` and the **2026** edition of the +new edition. It was written against `ctrlrun.guarantees/v7` and the **2026** edition of the OWASP Top 10 for Agentic Applications. diff --git a/docs/OWASP-SOLUTIONS-LANDSCAPE.md b/docs/OWASP-SOLUTIONS-LANDSCAPE.md index f4e3c14..f446840 100644 --- a/docs/OWASP-SOLUTIONS-LANDSCAPE.md +++ b/docs/OWASP-SOLUTIONS-LANDSCAPE.md @@ -1,6 +1,6 @@ --- title: "OWASP Agentic Solutions Landscape" -description: "A reading of the OWASP Agentic Solutions Landscape checklist against what CTRLRun v1.0 ships, with the boxes it does not tick named." +description: "A reading of the OWASP Agentic Solutions Landscape checklist against what CTRLRun 0.12.1 ships, with the boxes it does not tick named." sidebarTitle: "OWASP Solutions Landscape" --- @@ -26,14 +26,16 @@ credible. | **Publisher** | OWASP GenAI Security Project, OWASP Foundation | | **Form** | [https://genai.owasp.org/solution-submission-agentic/](https://genai.owasp.org/solution-submission-agentic/) | | **Read on** | 2026-09-10 | -| **Written against** | CTRLRun **v1.0**, guarantees `G1`–`G24` | +| **Written against** | CTRLRun **0.12.1**, guarantees `G1`–`G32`, catalogue `ctrlrun.guarantees/v7` | -**Written against v1.0, and the version is printed on every row that needs one.** The -guarantee catalogue this page reads is the one v1.0 freezes: `G1`–`G11` ship today, and -`G12`–`G24` arrive with v0.7 through v0.9 on the chain the [roadmap](/docs/ROADMAP) sets out. -A row whose *Since* column names a version that is not yet tagged is a **(design)** row until -it is, and a reader on an earlier release should discount it. Nothing here is ticked on the -strength of a version that the roadmap does not already commit to before 1.0. +**Written against what is tagged, and the version is printed on every row that needs one.** +Every guarantee cited below is in the catalogue `ctrlrun verify` runs today, `G1` through +`G32`, and every one of them is in the 0.12.1 wheel. **There are no design rows.** A *Since* +column names the milestone that added the row rather than a separate download: 0.3.0 was +published to TestPyPI alone, and what it added reached PyPI inside 0.4.0. This page was first written against an unreleased 1.0 and marked the rows that waited +on it; none of them waits any longer, and the hedge came out rather than being left to read +as a disclaimer on rows that ship. Nothing here is ticked on the strength of something +unreleased, and a row whose guarantee loses its test goes back to *No* in the same commit. The form's entry codes are `ASI01:26` through `ASI10:26`, the same 2026 edition the [OWASP Agentic Top 10 reading](/docs/OWASP-AGENTIC-TOP10) was written against; that page @@ -61,7 +63,7 @@ stages from that one point rather than covering each in its own right. | Release | Yes | The `ctrlrun verify` GitHub Action and badge as a release gate; the badge means *declared guarantees pass*, never "this agent is secure by inspection". | v0.4 | | Deploy | Yes | The MCP gateway in front of an existing tool server; framework adapters; observe mode to roll out without refusing anything yet. | v0.2, v0.3 | | Operate | Yes | Exact-action approvals; `ctrlrun resolve` for an `AMBIGUOUS` effect; the operator MCP server; break-glass as a recorded grant; `ctrlrun revoke --by` / `--under`. | v0.1, v0.8 | -| Monitor | Yes | Receipts on a hash chain with an external anchor; OpenTelemetry export; `ctrlrun stats`; enforcement coverage from events already written. | v0.6, v0.11 | +| Monitor | Yes | Receipts on a hash chain with an external anchor; OpenTelemetry export; `ctrlrun stats`; enforcement coverage from events already written; a retention pass that refuses rather than warns, `ctrlrun prune` and `ctrlrun hold place`. | v0.6, v0.11 | | Govern | Yes | Authority grants and delegation that cannot escalate; approver entitlement; consequence budgets; task-bound authority; a policy change as a protected action. Action governance, in the one sense the [roadmap](/docs/ROADMAP) permits: what an action is allowed to do, not how an organisation runs its agents. | v0.3, v0.8, v0.9 | --- @@ -90,7 +92,7 @@ example is theirs and is not a claim that CTRLRun uses it. |---|---|---|---| | Perform SAST/DAST on agent planning code, tool wrappers, & plugin interfaces | Partly | `ctrlrun scan` reads a Python tree and reports the consequential call sites and policy entries CTRLRun is *not* covering, and since v0.11 `--coverage` adds the runtime half: what the store shows was declared and never exercised. It is a coverage scanner, not a vulnerability scanner, and its report says what it misses by construction on every run. **Neither half produces a number**: a policy entry nothing exercised may be correctly unused, and saying otherwise would be grading the operator's document. | v0.6, v0.11 | | Harden agent loop logic against infinite loops, unsafe function routing, & unauthorized self-modification | Partly | A retry loop cannot turn one intended effect into several (`G3`, `G5`), an unknown action is refused (`G6`), and renewal after `FAILED` has an operator-set ceiling (`G15`). Nothing here inspects loop logic or prevents self-modification. | v0.1, v0.7 | -| Validate connector (e.g., MCP) contracts (input/output schemas & permissions) | Partly | The gateway maps every MCP tool call onto a policy decision, and a policy entry may pin the hash of an upstream's advertised tool schema so a schema that moved under an approved name is a `deny`. It does not validate schemas in general. | v0.2, v0.10 | +| Validate connector (e.g., MCP) contracts (input/output schemas & permissions) | Partly | The gateway maps every MCP tool call onto a policy decision, and a policy entry may pin the upstream it authorises, by the SHA-256 of the server's leaf certificate or by the hash of a tool's advertised schema: a tool whose schema moved under an approved action name is refused `upstream_mismatch`, and an upstream nothing observed is refused `upstream_unverified` and never admitted (`G27`). Enforced by `ctrlrun gateway`, the surface that holds the connection; in-process there is no upstream to observe. It does not validate schemas in general. | v0.2, v0.10 | | Implement policy enforcement hooks in Frameworks (e.g. LangGraph, CrewAI, Others) | Yes | `@protect` for anything in-process; the adapter contract with OpenAI Agents SDK and LangGraph reference adapters, each routing an `approve` through the framework's own interrupt; the OWASP Agent Control Standard adapter ([ACS](/docs/ACS)). | v0.1, v0.2, v0.5 | ### Augment & Fine Tune Data @@ -136,7 +138,7 @@ example is theirs and is not a claim that CTRLRun uses it. | Checkbox | Status | What it means here | Since | |---|---|---|---| | Audit reflection accuracy by comparing stated & observed planning outcomes | No | CTRLRun never sees a plan or a reflection. | none | -| Use immutable logs (e.g., Sigstore, Immudb) for forensic readiness | Partly | Tamper-evident, not immutable. Each receipt hashes the one before it, so alteration, deletion and reordering are reported by name and by `seq` (`G11`); the head is anchored outside the database at an interval, so a suffix erased or appended between anchors is detected. Receipts are not signed, and the chain says nothing about who wrote it. | v0.6, v0.11 | +| Use immutable logs (e.g., Sigstore, Immudb) for forensic readiness | Partly | Tamper-evident, not immutable. Each receipt hashes the one before it, so alteration, deletion and reordering are reported by name and by `seq` (`G11`); the head is anchored outside the database at an interval, so a suffix erased below an anchor is detected (`G28`). Retention does not quietly break it: `ctrlrun prune` removes a prefix and leaves a checkpoint the reader seeds from, so the chain verifies across the gap, it refuses rather than warns and has no `--force` (`G29`), an honest prune leaves the anchors reproducing (`G32`), and a range under `ctrlrun hold place` refuses to prune at all, with the hold named (`G30`). Receipts are not signed, and the chain says nothing about who wrote it. | v0.6, v0.11 | | Alert on anomalies; e.g., goal reversal, unexpected plan depth, adversarial-input, excessive tool usage, or rapid inter-agent chatter | Partly | Excessive use is refused rather than alerted on: a consequence budget on a grant is consumed on reserve and held until an `AMBIGUOUS` effect is reconciled (`G22`), and every refusal is an event on the sink your alerting reads. CTRLRun ships no alerting and sees no plan or prompt. | v0.2, v0.9 | | Correlate telemetry from agent step tracing, tool execution, & message logs | Yes | The OpenTelemetry sink opens one span per action carrying `ctrlrun.action_id`, `ctrlrun.effect_key` and `ctrlrun.approval_id`, and the receipt carries the same `action_id`, so a tool-execution span joins an agent trace and a receipt on one identifier ([export guide](/docs/guides/export-to-opentelemetry)). | v0.2 | @@ -166,21 +168,22 @@ example is theirs and is not a claim that CTRLRun uses it. The form asks for `ASI01:26` through `ASI10:26` as ten checkboxes. The [Agentic Top 10 reading](/docs/OWASP-AGENTIC-TOP10) carries the row-by-row mapping and the -sentence for each entry saying what is *not* covered; this table is the summary at v1.0, with -the version that moved each entry. +sentence for each entry saying what is *not* covered; this table is the summary at 0.12.1, +with the version that moved each entry. Each row's guarantees are exactly what that reading +maps to the entry, which is a test and not an intention. | Entry | Status | Guarantees | What stays out | Since | |---|---|---|---|---| -| `ASI01:26` Agent Goal Hijack | Partly | `G1`, `G6`, `G24` | The hijack itself. A task-bound grant shrinks what a hijacked agent can do; nothing reads the hijack. | v0.1, v0.9 | -| `ASI02:26` Tool Misuse | Yes | `G3`, `G6`, `G16`, `G22` | Misuse that stays inside the policy, the grant and the budget. | v0.1, v0.7, v0.9 | -| `ASI03:26` Identity & Privilege Abuse | Yes | `G7`, `G8`, `G9`, `G17`–`G20` | Issuing identity. CTRLRun verifies what it is handed. | v0.3, v0.8 | -| `ASI04:26` Agentic Supply Chain Vulnerabilities | Partly | upstream identity pinning | Provenance at large. A pinned TLS key or tool-schema hash refuses a swapped upstream; no package, model or registry is ever inspected. | v0.10 | +| `ASI01:26` Agent Goal Hijack | Partly | `G1`, `G6`, `G16`, `G21`, `G22`, `G23`, `G24` | The hijack itself. Task binding, a budget and a scope provider shrink what a hijacked agent can do; nothing reads the hijack. | v0.1, v0.7, v0.8, v0.9 | +| `ASI02:26` Tool Misuse | Yes | `G3`, `G6`, `G23`, `G27` | Misuse that stays inside the policy, the grant and the budget. | v0.1, v0.9, v0.10 | +| `ASI03:26` Identity & Privilege Abuse | Yes | `G7`, `G8`, `G9`, `G17`, `G18`, `G19`, `G20`, `G21`, `G24`, `G25`, `G26` | Issuing identity. CTRLRun verifies what it is handed. | v0.1, v0.3, v0.8, v0.9, v0.10 | +| `ASI04:26` Agentic Supply Chain Vulnerabilities | No | none | Out of scope, and no guarantee maps to it: no package, model, build, registry or signature chain is ever inspected. Upstream pinning binds one connection and one tool schema, and it is mapped under `ASI02:26` and `ASI07:26`, where binding a peer belongs, rather than here. | none | | `ASI05:26` Unexpected Code Execution | No | none | Out of scope: nothing here sandboxes an interpreter. | none | -| `ASI06:26` Memory & Context Poisoning | Partly | `G1`, `G6`, `G16`, `G23` | The poisoning. A scope provider now bites on an identifier an attacker chose, fetched before the reservation and fail-closed; the recheck still cannot run inside the atomic write, and that residual gap is stated wherever the feature is. | v0.1, v0.7, v0.9 | -| `ASI07:26` Insecure Inter-Agent Communication | Partly | hop propagation | Transport security. Authority across an A2A hop is a subset of the sender's, checked at the hop and on every evaluation; the channel itself is not CTRLRun's. | v0.10 | -| `ASI08:26` Cascading Failures | Yes | `G3`, `G4`, `G5`, `G10`, `G12`, `G15`, `G22` | A failure that never reaches a consequential action. | v0.1, v0.7, v0.9 | -| `ASI09:26` Human-Agent Trust Exploitation | Partly | `G1`, `G2`, `G17`, `G18`, `G19` | Persuasion. A human misled into approving the right action for the wrong reason gives a valid approval, and the receipt records it as one. | v0.1, v0.8 | -| `ASI10:26` Rogue Agents | Partly | `G8`, `G9`, `G20`, `G22`, `G24` | Detection. A rogue agent is bounded, expired and revoked; it is not recognised as rogue. | v0.3, v0.8, v0.9 | +| `ASI06:26` Memory & Context Poisoning | Partly | `G1`, `G6`, `G23` | The poisoning. A scope provider bites on an identifier an attacker chose, fetched before the reservation and fail-closed; the recheck still cannot run inside the atomic write, and that residual gap is stated wherever the feature is. | v0.1, v0.9 | +| `ASI07:26` Insecure Inter-Agent Communication | Partly | `G25`, `G26`, `G27` | Transport security. Authority across an A2A hop is a subset of the sender's, checked at the hop and on every evaluation, and a pinned upstream refuses a swapped peer; the channel itself is not CTRLRun's. | v0.10 | +| `ASI08:26` Cascading Failures | Yes | `G3`, `G4`, `G5`, `G10`, `G12`, `G13`, `G14`, `G15`, `G22` | A failure that never reaches a consequential action. | v0.1, v0.7, v0.9 | +| `ASI09:26` Human-Agent Trust Exploitation | Partly | `G1`, `G2`, `G11`, `G12`, `G16`, `G17`, `G18`, `G19`, `G28`, `G29`, `G30`, `G31`, `G32` | Persuasion. A human misled into approving the right action for the wrong reason gives a valid approval, and the receipt records it as one. | v0.1, v0.6, v0.7, v0.8, v0.11 | +| `ASI10:26` Rogue Agents | Partly | `G8`, `G9`, `G15`, `G20`, `G21`, `G22`, `G24`, `G25`, `G26` | Detection. A rogue agent is bounded, expired and revoked; it is not recognised as rogue. | v0.3, v0.7, v0.8, v0.9, v0.10 | --- @@ -193,9 +196,13 @@ ticked on the form and its row here says *No*, the form is wrong. If a guarantee [Agentic Top 10 reading](/docs/OWASP-AGENTIC-TOP10) already follows. This page is regenerated when the guarantee catalogue changes, when a version named in a -*Since* column is tagged, and when OWASP revises the form. It was written against -`ctrlrun.guarantees/v5` as the [roadmap](/docs/ROADMAP) defines it for v0.9 and the form as -read on 2026-09-10. +*Since* column is tagged, and when OWASP revises the form. The first two both happened without +it, so the rule is now a test rather than a sentence: `tests/test_owasp_landscape.py` checks +that every guarantee cited here exists in the registry, that the catalogue named is the one +`ctrlrun verify` reports, that no *Since* column names a version the changelog has not +released, and that each `ASI` row's guarantees are exactly what the +[Agentic Top 10 reading](/docs/OWASP-AGENTIC-TOP10) maps to that entry. It was written against +`ctrlrun.guarantees/v7`, the catalogue 0.12.1 ships, and the form as read on 2026-09-10. ## Next diff --git a/tests/test_owasp_landscape.py b/tests/test_owasp_landscape.py new file mode 100644 index 0000000..ce676f9 --- /dev/null +++ b/tests/test_owasp_landscape.py @@ -0,0 +1,287 @@ +"""`docs/OWASP-SOLUTIONS-LANDSCAPE.md`. The page a landscape submission is filled in from. + +The page says a submission to the OWASP Agentic Solutions Landscape is filled in from it and +from nothing else, and that it is regenerated when the guarantee catalogue changes, when a +version in a *Since* column is tagged, and when OWASP revises the form. Two of those three +happened without it: the catalogue moved from `v5` to `v7` and v0.10 through v0.12 were tagged +while the page still read "written against v1.0, guarantees G1-G24" and still ticked `ASI04` +on a guarantee the mapping it cites deliberately files elsewhere. + +Nothing caught either, because the page had no test. `test_owasp_mapping.py` reads the Top 10 +reading beside it, which is why that document kept up. This is the same idea pointed at the +page that leaves the project: a form is filled in from rows, so the rows are checked against +the registry, against the changelog and against the mapping they claim to summarise. +""" + +from __future__ import annotations + +import re +import tomllib +from pathlib import Path + +import pytest +from ctrlrun.verify import guarantees as reg + +from _core import CORE_ROOT + +REPO_ROOT = Path(__file__).resolve().parents[1] +LANDSCAPE = REPO_ROOT / "docs" / "OWASP-SOLUTIONS-LANDSCAPE.md" +MAPPING = REPO_ROOT / "docs" / "OWASP-AGENTIC-TOP10.md" + +#: The form writes its entry codes `ASI01:26`, the Top 10 document writes them `ASI01:2026`. +#: Each page uses the spelling of the document it is quoting, so the two differ on purpose and +#: a test that normalised them would hide a code drifting between the pages. +FORM_CODE = re.compile(r"ASI(\d{2}):26\b") +MAPPING_CODE = re.compile(r"ASI(\d{2}):2026\b") + +#: `G7` and `G17` are different guarantees, so the digits are anchored on both sides. +GUARANTEE = re.compile(r"\bG(\d+)\b") + +#: A `Since` column names a minor version, `v0.11`, never a patch. +SINCE = re.compile(r"\bv(\d+)\.(\d+)\b") + +#: The three words the page defines for itself, and the only ones any status column may hold. +STATUS = {"Yes", "Partly", "No"} + +#: The nine the form lists, in the form's own spelling. `Augm & Fine Tune Data` is abbreviated +#: on the form and written out on the page, which is the one deviation. +STAGES = ( + "Scope & Plan", + "Develop & Experiment", + "Augment & Fine Tune Data", + "Test & Evaluate", + "Release", + "Deploy", + "Monitor", + "Operate", + "Govern", +) + + +def _document() -> str: + return LANDSCAPE.read_text(encoding="utf-8") + + +def _flat() -> str: + return " ".join(_document().split()) + + +def _rows(text: str, header: str) -> list[list[str]]: + """The rows of the one table whose header line is `header`, as stripped cells. + + Keyed on the header rather than on a line number so that adding a row above a table does + not silently move which table is read. + """ + lines = text.splitlines() + start = next(i for i, line in enumerate(lines) if line.strip() == header.strip()) + rows = [] + for line in lines[start + 2 :]: # the header, then its `|---|` rule + if not line.startswith("|"): + break + rows.append([cell.strip() for cell in line.strip().strip("|").split("|")]) + return rows + + +def _summary() -> dict[str, list[str]]: + """The `Agentic Top 10 coverage` table, by entry code: `[status, guarantees, since]`.""" + header = "| Entry | Status | Guarantees | What stays out | Since |" + summary = {} + for cells in _rows(_document(), header): + code = FORM_CODE.search(cells[0]) + assert code, f"a row of the summary names no entry: {cells[0]}" + summary[code.group(1)] = [cells[1], cells[2], cells[4]] + return summary + + +def _mapped() -> dict[str, set[str]]: + """Entry number -> the guarantees the Top 10 reading maps to it, from its own table.""" + header = "| Guarantee | Invariant | Entries | How |" + mapped: dict[str, set[str]] = {} + for cells in _rows(MAPPING.read_text(encoding="utf-8"), header): + guarantee = GUARANTEE.search(cells[0]) + assert guarantee, f"a guarantee row names no guarantee: {cells[0]}" + for entry in MAPPING_CODE.findall(cells[2]): + mapped.setdefault(entry, set()).add(f"G{guarantee.group(1)}") + return mapped + + +#: The one minor version a `Since` column names that was never published, with the reason. +#: `0.3.0` went to TestPyPI as `0.3.0rc1` and no further; `src/ctrlrun/authority.py` is in the +#: `v0.4.0` tag, so everything it added reached PyPI one milestone later. A row may say +#: *since v0.3* because that is the milestone that added it, and the page says so in as many +#: words rather than leaving a reader to find an 0.3.0 on PyPI that is not there. +SUPERSEDED = {(0, 3): "0.3.0 was published to TestPyPI alone; 0.4.0 carried it to PyPI"} + + +def _changelog() -> str: + return (CORE_ROOT / "CHANGELOG.md").read_text(encoding="utf-8") + + +def _releases() -> list[tuple[int, int, int]]: + """Every version the library's changelog records as released, newest first. + + Not `pyproject.toml`: that moves at the start of a milestone and the changelog heading for + it reads `unreleased` until the end of one, so a page checked against `pyproject.toml` + could name a version nobody can install. `test_docs_production.py` learned the same thing + from a README that said a version was on PyPI while PyPI held the one before it. + """ + found = [] + for line in _changelog().splitlines(): + heading = re.match(r"^## \[(\d+)\.(\d+)\.(\d+)\]", line) + if heading and "unreleased" not in line.lower(): + found.append(tuple(int(part) for part in heading.groups())) + assert found, "the changelog records no released version" + return found + + +def _released() -> tuple[int, int, int]: + """The newest of them, which is the version the page is written against.""" + return _releases()[0] + + +# --- the rows point at guarantees that exist, in the catalogue that ships --------------------- + + +def test_every_guarantee_the_page_cites_exists_in_the_registry(): + """A row citing `G33` would read as a promise and be a typo.""" + cited = {f"G{number}" for number in GUARANTEE.findall(_document())} + + assert cited, "the page cites no guarantee at all" + assert cited <= set(reg.BY_ID), sorted(cited - set(reg.BY_ID)) + + +def test_the_catalogue_the_page_names_is_the_one_verify_reports(): + """The page named `v5` for two releases after the catalogue reached `v7`.""" + assert reg.CATALOGUE in _document(), reg.CATALOGUE + + stale = {f"ctrlrun.guarantees/v{n}" for n in range(1, 20)} - {reg.CATALOGUE} + assert not [name for name in stale if name in _document()] + + +def test_the_guarantee_range_in_the_header_spans_the_whole_registry(): + """`G1`-`G32` is a claim about the catalogue's size, so it is read as one.""" + first, last = reg.GUARANTEES[0].id, reg.GUARANTEES[-1].id + + assert f"guarantees `{first}`–`{last}`" in _flat(), f"{first}-{last}" + + +def test_the_version_the_page_is_written_against_is_released(): + major, minor, patch = _released() + + assert f"CTRLRun **{major}.{minor}.{patch}**" in _document() + + +def test_no_since_column_names_a_version_that_is_not_out(): + """The page's rule is that it ticks nothing on an unreleased version. + + It was broken the other way round for five releases: every row was hedged as `(design)` + against a 1.0 that had not shipped, while the versions the rows actually named had. Either + direction is the same defect, a page describing a version other than the one a reader can + install, and this catches both because every number has to be one the changelog released. + """ + released = {version[:2] for version in _releases()} + + for row in _document().splitlines(): + for found in SINCE.finditer(row): + version = (int(found.group(1)), int(found.group(2))) + if version in SUPERSEDED: + continue + assert version in released, f"{found.group(0)} in: {row[:80]}" + + +def test_the_page_says_which_version_a_since_column_names_that_was_never_published(): + """A reader who checks `v0.3` against PyPI finds nothing there, so the page says why. + + The exemption above is what makes this necessary: a test that skipped `v0.3` silently + would be a test agreeing to a number a reader cannot verify. + """ + flat = _flat() + + for (major, minor), reason in SUPERSEDED.items(): + assert f"{major}.{minor}.0 was published to TestPyPI alone" in flat, reason + + +def test_every_status_is_one_of_the_three_words_the_page_defines(): + """*Yes*, *Partly*, *No*. A fourth word is a hedge the page has no definition for.""" + for table in ("| Stage | Status | What CTRLRun has there | Since |", + "| Checkbox | Status | What it means here | Since |"): + for cells in _rows(_document(), table): + assert cells[1] in STATUS, cells + + +def test_every_entry_status_is_one_of_the_three_words(): + for entry, (status, _, _) in _summary().items(): + assert status in STATUS, (entry, status) + + +# --- the summary is the mapping, not a second opinion of it ---------------------------------- + + +@pytest.mark.parametrize("entry", sorted(f"{n:02d}" for n in range(1, 11))) +def test_each_entry_names_exactly_the_guarantees_the_mapping_maps_to_it(entry): + """The two pages disagreed on `ASI04` and the form would have carried the disagreement. + + The summary ticked it *Partly* on upstream identity pinning; the reading it cites says + `ASI04` is out of scope, that no guarantee maps to it, and that the pinning is G27 which + belongs under `ASI02` and `ASI07` — "not here", in as many words. A submission filled in + from the summary would have ticked a box the page behind it refutes, which is the one + failure the page exists to prevent. + """ + summary = _summary() + assert entry in summary, entry + + listed = {f"G{number}" for number in GUARANTEE.findall(summary[entry][1])} + assert listed == _mapped().get(entry, set()), entry + + +def test_an_entry_no_guarantee_maps_to_is_not_ticked(): + """`none` in the column, `No` in the status. Not a blank, which reads as an oversight.""" + mapped = _mapped() + + for entry, (status, guarantees, since) in _summary().items(): + if entry in mapped: + continue + assert status == "No", (entry, status) + assert guarantees == "none", (entry, guarantees) + assert since == "none", (entry, since) + + +def test_every_entry_the_mapping_covers_is_ticked_in_the_summary(): + """The other direction: an entry with guarantees behind it cannot be left at `No`.""" + summary = _summary() + + for entry in _mapped(): + assert summary[entry][0] in {"Yes", "Partly"}, entry + + +# --- what the form asks for, and what the page refuses to claim ------------------------------ + + +def test_all_ten_entries_and_all_nine_lifecycle_stages_appear(): + document = _document() + + found = {f"{int(number):02d}" for number in FORM_CODE.findall(document)} + assert found == {f"{n:02d}" for n in range(1, 11)}, sorted(found) + + # The nine, and no tenth. Not their order: the page groups Operate beside Deploy and the + # form lists it after Monitor, which is a reading order and not a claim. + stages = {cells[0] for cells in _rows(document, "| Stage | Status | What CTRLRun has there | Since |")} + assert stages == set(STAGES), sorted(stages.symmetric_difference(STAGES)) + + +def test_the_page_claims_no_listing_and_no_endorsement(): + """OWASP endorses nothing, and a directory entry is a directory entry.""" + flat = _flat() + + assert "It is not a listing claim, a conformance claim, or an endorsement." in flat + assert "does\nnot endorse or recommend" in _document() + + +def test_the_page_says_the_form_is_filled_in_from_it_and_names_where_it_came_from(): + flat = _flat() + + assert "filled in from this page and from nothing else" in flat + assert "genai.owasp.org/solution-submission-agentic" in flat + assert "2026-09-10" in flat + # And the mapping it summarises, since half these tests read that document. + assert "OWASP-AGENTIC-TOP10" in flat From 560132859b32c462d8484754047c0d88f80d61cf Mon Sep 17 00:00:00 2001 From: arpan Date: Tue, 15 Sep 2026 18:51:32 +0530 Subject: [PATCH 2/3] ruff: the formatter's line breaks, an import the draft left behind, and the en dash spelled --- tests/test_owasp_landscape.py | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/tests/test_owasp_landscape.py b/tests/test_owasp_landscape.py index ce676f9..dd1f5ba 100644 --- a/tests/test_owasp_landscape.py +++ b/tests/test_owasp_landscape.py @@ -16,7 +16,6 @@ from __future__ import annotations import re -import tomllib from pathlib import Path import pytest @@ -159,10 +158,15 @@ def test_the_catalogue_the_page_names_is_the_one_verify_reports(): def test_the_guarantee_range_in_the_header_spans_the_whole_registry(): - """`G1`-`G32` is a claim about the catalogue's size, so it is read as one.""" + """`G1`-`G32` is a claim about the catalogue's size, so it is read as one. + + The page writes the range with an en dash, as the rest of its prose does, so the escape is + spelled rather than pasted: a literal one here is `RUF001`, and a hyphen would not match. + """ first, last = reg.GUARANTEES[0].id, reg.GUARANTEES[-1].id + en_dash = "\N{EN DASH}" - assert f"guarantees `{first}`–`{last}`" in _flat(), f"{first}-{last}" + assert f"guarantees `{first}`{en_dash}`{last}`" in _flat(), f"{first}-{last}" def test_the_version_the_page_is_written_against_is_released(): @@ -203,8 +207,10 @@ def test_the_page_says_which_version_a_since_column_names_that_was_never_publish def test_every_status_is_one_of_the_three_words_the_page_defines(): """*Yes*, *Partly*, *No*. A fourth word is a hedge the page has no definition for.""" - for table in ("| Stage | Status | What CTRLRun has there | Since |", - "| Checkbox | Status | What it means here | Since |"): + for table in ( + "| Stage | Status | What CTRLRun has there | Since |", + "| Checkbox | Status | What it means here | Since |", + ): for cells in _rows(_document(), table): assert cells[1] in STATUS, cells @@ -265,7 +271,9 @@ def test_all_ten_entries_and_all_nine_lifecycle_stages_appear(): # The nine, and no tenth. Not their order: the page groups Operate beside Deploy and the # form lists it after Monitor, which is a reading order and not a claim. - stages = {cells[0] for cells in _rows(document, "| Stage | Status | What CTRLRun has there | Since |")} + stages = { + cells[0] for cells in _rows(document, "| Stage | Status | What CTRLRun has there | Since |") + } assert stages == set(STAGES), sorted(stages.symmetric_difference(STAGES)) From 6e465f181392cdeeb6e7676aa55a9a77b56a574d Mon Sep 17 00:00:00 2001 From: arpan Date: Tue, 15 Sep 2026 19:05:19 +0530 Subject: [PATCH 3/3] Two qualifications a review earned: the approver checks are opt-in, and the anchor is yours MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both on rows a landscape reviewer reads, and both the same failure: a sentence true of a configured deployment, written as though it were true of every one. **The approver checks.** `control.py` returns before all three where `approver_identity` is `None`, so G17, G18 and G19 run only where a deployment names one, and the `approver` stays the string it was before 0.8.0 where it does not. The `ASI09` row said *since 0.8.0 the approver is a resolved principal rather than a string* with no condition on it, and the HITL row on the landscape page said the same thing in its own words. Both now carry the condition, and both name the one case that is not silent: a threshold above one is denied before a human is asked rather than counted against strings, which is §4.2's rule and the opposite of a quiet downgrade to one approval. **The anchor.** The immutable-logs row said the head is anchored outside the database, which reads as something the library does. CTRLRun ships no anchor provider, on purpose, because an RFC 3161 client is a network client. The row says whose provider it is, and that the exposure is the operator's own anchoring interval, which is the number to quote on that box instead of any sentence about tamper-evidence. --- docs/OWASP-AGENTIC-TOP10.md | 2 +- docs/OWASP-SOLUTIONS-LANDSCAPE.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/OWASP-AGENTIC-TOP10.md b/docs/OWASP-AGENTIC-TOP10.md index 0c83b85..379de78 100644 --- a/docs/OWASP-AGENTIC-TOP10.md +++ b/docs/OWASP-AGENTIC-TOP10.md @@ -116,7 +116,7 @@ stated here rather than left implied: |---|---|---|---| | `ASI06:2026` | Memory & Context Poisoning | G6 and G1 constrain what an agent acting on a poisoned context can *do*: the action must still be named in the policy, so a belief an attacker planted cannot reach a tool the agent was never entitled to use, and an approval granted for one action cannot be spent on another. This is the same downstream constraint that makes `ASI01` partial, and it is here for the same reason. | CTRLRun never reads a model's memory, its context or its prompt, so it neither detects nor prevents the poisoning. And the shape poisoning most often takes is the one the kernel has least to say about: **corrupted arguments to an action the agent is entitled to take** — the right operation against the wrong record. Policy conditions, resource patterns and v0.6 data scope bite on part of that, and since v0.9 **G23 bites on the identifier itself**: where a deployment configures a scope provider, the kernel asks its system of record whether `customer:90210` is this principal's before reserving anything. That closes the sharpest version of this and not the category. The provider is the operator's own code reading the operator's own data, so a poisoned source answers wrongly with the kernel none the wiser, and a deployment that configures no provider is exactly where it was. | | `ASI01:2026` | Agent Goal Hijack | G1 and G6 constrain what a hijacked agent can *do*: it still meets the policy, and it still cannot present an approval granted for a different action. | CTRLRun does not detect or prevent the hijack. It never sees the prompt, the plan or the reasoning, so an agent whose goal was replaced proposes actions exactly as a healthy one would — and every action inside its policy and its grants will run. | -| `ASI09:2026` | Human-Agent Trust Exploitation | G1 and G2 close the shape where an approval a human gave for one action is spent on another, or spent twice. Since 0.8.0 the approver is a resolved principal rather than a string: G17 refuses an approval whose recorded entitlement does not cover the role the request pinned, G18 refuses the requester approving their own action, compared on the principal and never on the approver string, and G19 counts distinct principals, so one person cannot be several of an M-of-N. | CTRLRun does not decide **who is entitled**. What entitled an approver was decided where the credential was verified, which is the operator MCP server or an embedding application; the kernel matches a recorded claim byte for byte, does not check that the role exists anywhere, and a cited control naming no role gates nobody. It has no opinion on whether the human was misled: a person persuaded to approve the right action for the wrong reason gives a valid approval, and the receipt records it as one. | +| `ASI09:2026` | Human-Agent Trust Exploitation | G1 and G2 close the shape where an approval a human gave for one action is spent on another, or spent twice. Since 0.8.0, **where the deployment configures an approver identity**, the approver is a resolved principal rather than a string: G17 refuses an approval whose recorded entitlement does not cover the role the request pinned, G18 refuses the requester approving their own action, compared on the principal and never on the approver string, and G19 counts distinct principals, so one person cannot be several of an M-of-N. | **The three checks above are opt-in**: a deployment that configures no approver identity runs none of them, and its `approver` is the string it was before 0.8.0. The one case that is not left silent is a threshold above one, which is denied before a human is asked rather than counted against strings. And CTRLRun does not decide **who is entitled**: what entitled an approver was decided where the credential was verified, which is the operator MCP server or an embedding application; the kernel matches a recorded claim byte for byte, does not check that the role exists anywhere, and a cited control naming no role gates nobody. It has no opinion on whether the human was misled: a person persuaded to approve the right action for the wrong reason gives a valid approval, and the receipt records it as one. | --- diff --git a/docs/OWASP-SOLUTIONS-LANDSCAPE.md b/docs/OWASP-SOLUTIONS-LANDSCAPE.md index f446840..3352963 100644 --- a/docs/OWASP-SOLUTIONS-LANDSCAPE.md +++ b/docs/OWASP-SOLUTIONS-LANDSCAPE.md @@ -138,7 +138,7 @@ example is theirs and is not a claim that CTRLRun uses it. | Checkbox | Status | What it means here | Since | |---|---|---|---| | Audit reflection accuracy by comparing stated & observed planning outcomes | No | CTRLRun never sees a plan or a reflection. | none | -| Use immutable logs (e.g., Sigstore, Immudb) for forensic readiness | Partly | Tamper-evident, not immutable. Each receipt hashes the one before it, so alteration, deletion and reordering are reported by name and by `seq` (`G11`); the head is anchored outside the database at an interval, so a suffix erased below an anchor is detected (`G28`). Retention does not quietly break it: `ctrlrun prune` removes a prefix and leaves a checkpoint the reader seeds from, so the chain verifies across the gap, it refuses rather than warns and has no `--force` (`G29`), an honest prune leaves the anchors reproducing (`G32`), and a range under `ctrlrun hold place` refuses to prune at all, with the hold named (`G30`). Receipts are not signed, and the chain says nothing about who wrote it. | v0.6, v0.11 | +| Use immutable logs (e.g., Sigstore, Immudb) for forensic readiness | Partly | Tamper-evident, not immutable. Each receipt hashes the one before it, so alteration, deletion and reordering are reported by name and by `seq` (`G11`); the head is anchored outside the database through a provider **you** supply, since CTRLRun ships none, so a suffix erased at or below an anchored `seq` is detected (`G28`) and the window you are exposed to is your own anchoring interval. Retention does not quietly break it: `ctrlrun prune` removes a prefix and leaves a checkpoint the reader seeds from, so the chain verifies across the gap, it refuses rather than warns and has no `--force` (`G29`), an honest prune leaves the anchors reproducing (`G32`), and a range under `ctrlrun hold place` refuses to prune at all, with the hold named (`G30`). Receipts are not signed, and the chain says nothing about who wrote it. | v0.6, v0.11 | | Alert on anomalies; e.g., goal reversal, unexpected plan depth, adversarial-input, excessive tool usage, or rapid inter-agent chatter | Partly | Excessive use is refused rather than alerted on: a consequence budget on a grant is consumed on reserve and held until an `AMBIGUOUS` effect is reconciled (`G22`), and every refusal is an event on the sink your alerting reads. CTRLRun ships no alerting and sees no plan or prompt. | v0.2, v0.9 | | Correlate telemetry from agent step tracing, tool execution, & message logs | Yes | The OpenTelemetry sink opens one span per action carrying `ctrlrun.action_id`, `ctrlrun.effect_key` and `ctrlrun.approval_id`, and the receipt carries the same `action_id`, so a tool-execution span joins an agent trace and a receipt on one identifier ([export guide](/docs/guides/export-to-opentelemetry)). | v0.2 | @@ -151,7 +151,7 @@ example is theirs and is not a claim that CTRLRun uses it. | Runtime guardrails & moderation; anomalous tool use | Partly | An action outside the policy or outside the grant is refused, by name. No moderation, and no model of what is anomalous beyond *not permitted*. | v0.1, v0.3 | | Monitor agent memory mutation patterns for drift | No | Out of scope. | none | | Detect task replay, infinite delegation, or hallucination loops | Partly | A replayed approval is refused (`G2`), a duplicate effect is refused (`G3`), delegation carries a depth limit across hops, and a task-bound grant is refused on a task it does not name (`G24`). Hallucination loops are not detected; their consequential actions are refused. | v0.1, v0.9, v0.10 | -| Enable human-in-the-loop (HITL) override thresholds on high-risk or ambiguous actions | Yes | An `approve` decision with a condition such as `amount_gt`; the approval bound to the exact action the human saw, single-use, expiring (`G1`, `G2`); an `AMBIGUOUS` effect that only a human or a reconciliation hook may leave (`G5`); the approver a verified principal who is entitled, not the requester, and counted once under M-of-N (`G17`–`G19`). | v0.1, v0.8 | +| Enable human-in-the-loop (HITL) override thresholds on high-risk or ambiguous actions | Yes | An `approve` decision with a condition such as `amount_gt`; the approval bound to the exact action the human saw, single-use, expiring (`G1`, `G2`); an `AMBIGUOUS` effect that only a human or a reconciliation hook may leave (`G5`); and where the deployment configures an approver identity, the approver a verified principal who is entitled, not the requester, and counted once under M-of-N (`G17`–`G19`), with a threshold above one denied outright where it configures none. | v0.1, v0.8 | ### Govern