diff --git a/.agents/skills/rook/references/headless-contract.md b/.agents/skills/rook/references/headless-contract.md index cb636f6..03cd955 100644 --- a/.agents/skills/rook/references/headless-contract.md +++ b/.agents/skills/rook/references/headless-contract.md @@ -48,6 +48,9 @@ Use `error`, or `reason` when present. Translate remedy tokens with - `halted: true` can retain a report and exit 0. Present it as incomplete. - Gate refusals on `report`, `status` and `ask`, and parser errors, may leave stdout empty. Use the stderr diagnostic rather than inventing an error object. +- A command that calls the model controller or opens an upstream job refuses + before anything runs or is charged when its session could not be registered. + Report it as no run and no spend, using the diagnostic it returns. ## Grants and accounting @@ -65,7 +68,7 @@ still use the network, change state or invoke the target. ## On disk -In 0.1.6, `.testmuai/rook/settings.json` keeps a project selection per +Since 0.1.6, `.testmuai/rook/settings.json` keeps a project selection per environment in `selections[environment]`. After a switch, rook uses the new environment's selection, which may be empty even when another environment has one. Run `rook doctor` to see the active environment and @@ -87,4 +90,6 @@ assuming either form. Under its `agents//` directory: | `runs//scenarios//` | `snapshot.yaml`, `request.json`, `response.json`, artifacts and `verdict.yaml` | Home state at `~/.testmuai/rook/` includes credentials, grants and session -transcripts. Keep it outside commits and shared evidence. +transcripts. Each session's `transcript.jsonl` records the raw input and output +of every role call, including refused, failed and interrupted calls. Keep it +outside commits and shared evidence. diff --git a/.agents/skills/rook/references/scenarios.md b/.agents/skills/rook/references/scenarios.md index cd34421..8756ca4 100644 --- a/.agents/skills/rook/references/scenarios.md +++ b/.agents/skills/rook/references/scenarios.md @@ -17,6 +17,8 @@ them, though existing scenario files that use them still load and run. Every scenario also carries free-form tags; `run --tag ` selects on them. +Red-team scenarios also carry OWASP LLM, OWASP Agentic (ASI) and MITRE ATLAS +references. ## Generating @@ -38,6 +40,11 @@ different feature, `generate` warns and saves that scenario excluded; review it in `rook scenarios list --json` and bring it back with `rook scenarios include ` if it belongs in runs. +`rook sync` stops before uploading anything when a scenario names a feature its +agent does not carry, and lists each one with its agent, the missing feature and +a fix. Apply the listed fix and sync again; do not run against upstream until +sync succeeds. + ## Listing and curating ```bash @@ -56,6 +63,16 @@ multiple turns without an observed conversation handle, or token-economy grading without reported usage, is skipped by `run` and counts toward the assurance gap, not toward Fail. File and image requirements are not rejected solely by their input kind: inspect what the profile script can actually send. +`generate` keeps a scenario it cannot make runnable instead of dropping it: +`executable: false` with a `skip_reason` starting `coverage gap:`, +`invalid scenario:` or `invalid red-team scenario:`. Attack patterns with no +scenario for a feature are listed in the agent's `coverage-gaps.yaml`. Report +both as gaps, not as failures. + +Some attack patterns are skipped until the profile has the hook they need, +and the skip reason names it: an `open` hook that seeds the fixture in +`$ROOK_STATE_DIR/attack-fixture.json`, a `collect` hook that reports the +observed control and outcome, or `open` and `close` hooks for a fresh session. ## Scoping a run @@ -70,6 +87,10 @@ rook run --run --phases collect,judge --json # collect evidence that la rook run --resume --json # carry finished work into a new run ``` +Without `--only`, a run selects at most two scenarios per attack pattern; the +rest are skipped as `pattern already has 2 candidates in this run`. Name +them with `--only` to run more. + `--only` takes one comma-separated argument; repeating the flag keeps only the last occurrence, and a space-separated second id is read as instruction text. Each scenario runs once; there is no repeat count. diff --git a/.agents/skills/rook/references/verdicts.md b/.agents/skills/rook/references/verdicts.md index 11fd55f..567610a 100644 --- a/.agents/skills/rook/references/verdicts.md +++ b/.agents/skills/rook/references/verdicts.md @@ -15,11 +15,16 @@ criterion details, read `runs//scenarios//verdict.yaml`. Statuses are **Pass**, **Fail**, and **Unable to Verify**. Unverifiable reasons: `agent_never_ran`, `not_observable`, or `undecidable`. A scenario that never ran -has no verdict; do not turn its absence into a failure. +has no verdict; do not turn its absence into a failure. A grounded partial +compromise counts as Fail. Reports keep refusals, interrupted runs and coverage +gaps; carry them into the summary rather than dropping them. `report --json` +can add `coverage_gaps`, `redteam_coverage` (OWASP coverage of the scenario +inventory) and `summary_claims`; an unmapped control is unverified, not passed. +A **Partial** stored by an older run reads as Unable to Verify. ## Pass rate -In 0.1.6, `totals.pass_rate` is `Pass / (Pass + Fail + Unable to Verify)`. +Since 0.1.6, `totals.pass_rate` is `Pass / (Pass + Fail + Unable to Verify)`. `totals.decided` still means `Pass + Fail`. Unable to Verify contributes to the pass-rate denominator as a recorded result, but remains a separate verdict; never describe it as Fail. If every recorded result is Unable to Verify, diff --git a/.claude/skills/rook/references/headless-contract.md b/.claude/skills/rook/references/headless-contract.md index cb636f6..03cd955 100644 --- a/.claude/skills/rook/references/headless-contract.md +++ b/.claude/skills/rook/references/headless-contract.md @@ -48,6 +48,9 @@ Use `error`, or `reason` when present. Translate remedy tokens with - `halted: true` can retain a report and exit 0. Present it as incomplete. - Gate refusals on `report`, `status` and `ask`, and parser errors, may leave stdout empty. Use the stderr diagnostic rather than inventing an error object. +- A command that calls the model controller or opens an upstream job refuses + before anything runs or is charged when its session could not be registered. + Report it as no run and no spend, using the diagnostic it returns. ## Grants and accounting @@ -65,7 +68,7 @@ still use the network, change state or invoke the target. ## On disk -In 0.1.6, `.testmuai/rook/settings.json` keeps a project selection per +Since 0.1.6, `.testmuai/rook/settings.json` keeps a project selection per environment in `selections[environment]`. After a switch, rook uses the new environment's selection, which may be empty even when another environment has one. Run `rook doctor` to see the active environment and @@ -87,4 +90,6 @@ assuming either form. Under its `agents//` directory: | `runs//scenarios//` | `snapshot.yaml`, `request.json`, `response.json`, artifacts and `verdict.yaml` | Home state at `~/.testmuai/rook/` includes credentials, grants and session -transcripts. Keep it outside commits and shared evidence. +transcripts. Each session's `transcript.jsonl` records the raw input and output +of every role call, including refused, failed and interrupted calls. Keep it +outside commits and shared evidence. diff --git a/.claude/skills/rook/references/scenarios.md b/.claude/skills/rook/references/scenarios.md index cd34421..8756ca4 100644 --- a/.claude/skills/rook/references/scenarios.md +++ b/.claude/skills/rook/references/scenarios.md @@ -17,6 +17,8 @@ them, though existing scenario files that use them still load and run. Every scenario also carries free-form tags; `run --tag ` selects on them. +Red-team scenarios also carry OWASP LLM, OWASP Agentic (ASI) and MITRE ATLAS +references. ## Generating @@ -38,6 +40,11 @@ different feature, `generate` warns and saves that scenario excluded; review it in `rook scenarios list --json` and bring it back with `rook scenarios include ` if it belongs in runs. +`rook sync` stops before uploading anything when a scenario names a feature its +agent does not carry, and lists each one with its agent, the missing feature and +a fix. Apply the listed fix and sync again; do not run against upstream until +sync succeeds. + ## Listing and curating ```bash @@ -56,6 +63,16 @@ multiple turns without an observed conversation handle, or token-economy grading without reported usage, is skipped by `run` and counts toward the assurance gap, not toward Fail. File and image requirements are not rejected solely by their input kind: inspect what the profile script can actually send. +`generate` keeps a scenario it cannot make runnable instead of dropping it: +`executable: false` with a `skip_reason` starting `coverage gap:`, +`invalid scenario:` or `invalid red-team scenario:`. Attack patterns with no +scenario for a feature are listed in the agent's `coverage-gaps.yaml`. Report +both as gaps, not as failures. + +Some attack patterns are skipped until the profile has the hook they need, +and the skip reason names it: an `open` hook that seeds the fixture in +`$ROOK_STATE_DIR/attack-fixture.json`, a `collect` hook that reports the +observed control and outcome, or `open` and `close` hooks for a fresh session. ## Scoping a run @@ -70,6 +87,10 @@ rook run --run --phases collect,judge --json # collect evidence that la rook run --resume --json # carry finished work into a new run ``` +Without `--only`, a run selects at most two scenarios per attack pattern; the +rest are skipped as `pattern already has 2 candidates in this run`. Name +them with `--only` to run more. + `--only` takes one comma-separated argument; repeating the flag keeps only the last occurrence, and a space-separated second id is read as instruction text. Each scenario runs once; there is no repeat count. diff --git a/.claude/skills/rook/references/verdicts.md b/.claude/skills/rook/references/verdicts.md index 11fd55f..567610a 100644 --- a/.claude/skills/rook/references/verdicts.md +++ b/.claude/skills/rook/references/verdicts.md @@ -15,11 +15,16 @@ criterion details, read `runs//scenarios//verdict.yaml`. Statuses are **Pass**, **Fail**, and **Unable to Verify**. Unverifiable reasons: `agent_never_ran`, `not_observable`, or `undecidable`. A scenario that never ran -has no verdict; do not turn its absence into a failure. +has no verdict; do not turn its absence into a failure. A grounded partial +compromise counts as Fail. Reports keep refusals, interrupted runs and coverage +gaps; carry them into the summary rather than dropping them. `report --json` +can add `coverage_gaps`, `redteam_coverage` (OWASP coverage of the scenario +inventory) and `summary_claims`; an unmapped control is unverified, not passed. +A **Partial** stored by an older run reads as Unable to Verify. ## Pass rate -In 0.1.6, `totals.pass_rate` is `Pass / (Pass + Fail + Unable to Verify)`. +Since 0.1.6, `totals.pass_rate` is `Pass / (Pass + Fail + Unable to Verify)`. `totals.decided` still means `Pass + Fail`. Unable to Verify contributes to the pass-rate denominator as a recorded result, but remains a separate verdict; never describe it as Fail. If every recorded result is Unable to Verify, diff --git a/skill-installer/skills/references/headless-contract.md b/skill-installer/skills/references/headless-contract.md index cb636f6..03cd955 100644 --- a/skill-installer/skills/references/headless-contract.md +++ b/skill-installer/skills/references/headless-contract.md @@ -48,6 +48,9 @@ Use `error`, or `reason` when present. Translate remedy tokens with - `halted: true` can retain a report and exit 0. Present it as incomplete. - Gate refusals on `report`, `status` and `ask`, and parser errors, may leave stdout empty. Use the stderr diagnostic rather than inventing an error object. +- A command that calls the model controller or opens an upstream job refuses + before anything runs or is charged when its session could not be registered. + Report it as no run and no spend, using the diagnostic it returns. ## Grants and accounting @@ -65,7 +68,7 @@ still use the network, change state or invoke the target. ## On disk -In 0.1.6, `.testmuai/rook/settings.json` keeps a project selection per +Since 0.1.6, `.testmuai/rook/settings.json` keeps a project selection per environment in `selections[environment]`. After a switch, rook uses the new environment's selection, which may be empty even when another environment has one. Run `rook doctor` to see the active environment and @@ -87,4 +90,6 @@ assuming either form. Under its `agents//` directory: | `runs//scenarios//` | `snapshot.yaml`, `request.json`, `response.json`, artifacts and `verdict.yaml` | Home state at `~/.testmuai/rook/` includes credentials, grants and session -transcripts. Keep it outside commits and shared evidence. +transcripts. Each session's `transcript.jsonl` records the raw input and output +of every role call, including refused, failed and interrupted calls. Keep it +outside commits and shared evidence. diff --git a/skill-installer/skills/references/scenarios.md b/skill-installer/skills/references/scenarios.md index cd34421..8756ca4 100644 --- a/skill-installer/skills/references/scenarios.md +++ b/skill-installer/skills/references/scenarios.md @@ -17,6 +17,8 @@ them, though existing scenario files that use them still load and run. Every scenario also carries free-form tags; `run --tag ` selects on them. +Red-team scenarios also carry OWASP LLM, OWASP Agentic (ASI) and MITRE ATLAS +references. ## Generating @@ -38,6 +40,11 @@ different feature, `generate` warns and saves that scenario excluded; review it in `rook scenarios list --json` and bring it back with `rook scenarios include ` if it belongs in runs. +`rook sync` stops before uploading anything when a scenario names a feature its +agent does not carry, and lists each one with its agent, the missing feature and +a fix. Apply the listed fix and sync again; do not run against upstream until +sync succeeds. + ## Listing and curating ```bash @@ -56,6 +63,16 @@ multiple turns without an observed conversation handle, or token-economy grading without reported usage, is skipped by `run` and counts toward the assurance gap, not toward Fail. File and image requirements are not rejected solely by their input kind: inspect what the profile script can actually send. +`generate` keeps a scenario it cannot make runnable instead of dropping it: +`executable: false` with a `skip_reason` starting `coverage gap:`, +`invalid scenario:` or `invalid red-team scenario:`. Attack patterns with no +scenario for a feature are listed in the agent's `coverage-gaps.yaml`. Report +both as gaps, not as failures. + +Some attack patterns are skipped until the profile has the hook they need, +and the skip reason names it: an `open` hook that seeds the fixture in +`$ROOK_STATE_DIR/attack-fixture.json`, a `collect` hook that reports the +observed control and outcome, or `open` and `close` hooks for a fresh session. ## Scoping a run @@ -70,6 +87,10 @@ rook run --run --phases collect,judge --json # collect evidence that la rook run --resume --json # carry finished work into a new run ``` +Without `--only`, a run selects at most two scenarios per attack pattern; the +rest are skipped as `pattern already has 2 candidates in this run`. Name +them with `--only` to run more. + `--only` takes one comma-separated argument; repeating the flag keeps only the last occurrence, and a space-separated second id is read as instruction text. Each scenario runs once; there is no repeat count. diff --git a/skill-installer/skills/references/verdicts.md b/skill-installer/skills/references/verdicts.md index 11fd55f..567610a 100644 --- a/skill-installer/skills/references/verdicts.md +++ b/skill-installer/skills/references/verdicts.md @@ -15,11 +15,16 @@ criterion details, read `runs//scenarios//verdict.yaml`. Statuses are **Pass**, **Fail**, and **Unable to Verify**. Unverifiable reasons: `agent_never_ran`, `not_observable`, or `undecidable`. A scenario that never ran -has no verdict; do not turn its absence into a failure. +has no verdict; do not turn its absence into a failure. A grounded partial +compromise counts as Fail. Reports keep refusals, interrupted runs and coverage +gaps; carry them into the summary rather than dropping them. `report --json` +can add `coverage_gaps`, `redteam_coverage` (OWASP coverage of the scenario +inventory) and `summary_claims`; an unmapped control is unverified, not passed. +A **Partial** stored by an older run reads as Unable to Verify. ## Pass rate -In 0.1.6, `totals.pass_rate` is `Pass / (Pass + Fail + Unable to Verify)`. +Since 0.1.6, `totals.pass_rate` is `Pass / (Pass + Fail + Unable to Verify)`. `totals.decided` still means `Pass + Fail`. Unable to Verify contributes to the pass-rate denominator as a recorded result, but remains a separate verdict; never describe it as Fail. If every recorded result is Unable to Verify,