From ae629ae4a033e6eaa16602cc91d1a2cb014f0a2e Mon Sep 17 00:00:00 2001 From: samyakLambda Date: Tue, 13 Oct 2026 17:28:07 +0530 Subject: [PATCH] skill: skip_reason prefixes come only from rook's own validation A signed-in 0.1.11 run of samples/triage-service showed checker-written skip reasons as plain sentences ("The scenario requires ticket T-1043 to exist..."), so the skill should read `executable`, not match a prefix. The same run's plain generate planned no attacks; say how to ask for them. Co-Authored-By: Claude Opus 5.5 (1M context) --- .agents/skills/rook/references/scenarios.md | 17 ++++++++++++----- .claude/skills/rook/references/scenarios.md | 17 ++++++++++++----- skill-installer/skills/references/scenarios.md | 17 ++++++++++++----- 3 files changed, 36 insertions(+), 15 deletions(-) diff --git a/.agents/skills/rook/references/scenarios.md b/.agents/skills/rook/references/scenarios.md index 8756ca4..fd0d683 100644 --- a/.agents/skills/rook/references/scenarios.md +++ b/.agents/skills/rook/references/scenarios.md @@ -23,7 +23,7 @@ references. ## Generating ```bash -rook generate --json # functional + adversarial, rook picks the count +rook generate --json # functional + adversarial planned, rook picks the count rook generate --total 20 --class functional --json rook generate --category prompt_injection,jailbreak --json rook generate -- focus on the refund limits and the identity check @@ -64,10 +64,17 @@ grading without reported usage, is skipped by `run` and counts toward the assurance gap, not toward Fail. File and image requirements are not rejected solely by their input kind: inspect what the profile script can actually send. `generate` keeps a scenario it cannot make runnable instead of dropping it: -`executable: false` with a `skip_reason` starting `coverage gap:`, -`invalid scenario:` or `invalid red-team scenario:`. Attack patterns with no -scenario for a feature are listed in the agent's `coverage-gaps.yaml`. Report -both as gaps, not as failures. +`executable: false` with a `skip_reason`. Reasons from rook's own validation +start with `coverage gap:`, `invalid scenario:` or `invalid red-team +scenario:`; the scenario checker writes plain sentences, often a precondition +the profile cannot establish. Do not match on the prefix to find them; read +`executable`. Attack patterns with no scenario for a feature are listed in the +agent's `coverage-gaps.yaml` and the report's coverage gaps. Report both as +gaps, not as failures. + +A plain `generate` can plan no attacks for a small agent. When the user asked +for red-team coverage and none was written, run +`rook generate --class adversarial --json` and say so. Some attack patterns are skipped until the profile has the hook they need, and the skip reason names it: an `open` hook that seeds the fixture in diff --git a/.claude/skills/rook/references/scenarios.md b/.claude/skills/rook/references/scenarios.md index 8756ca4..fd0d683 100644 --- a/.claude/skills/rook/references/scenarios.md +++ b/.claude/skills/rook/references/scenarios.md @@ -23,7 +23,7 @@ references. ## Generating ```bash -rook generate --json # functional + adversarial, rook picks the count +rook generate --json # functional + adversarial planned, rook picks the count rook generate --total 20 --class functional --json rook generate --category prompt_injection,jailbreak --json rook generate -- focus on the refund limits and the identity check @@ -64,10 +64,17 @@ grading without reported usage, is skipped by `run` and counts toward the assurance gap, not toward Fail. File and image requirements are not rejected solely by their input kind: inspect what the profile script can actually send. `generate` keeps a scenario it cannot make runnable instead of dropping it: -`executable: false` with a `skip_reason` starting `coverage gap:`, -`invalid scenario:` or `invalid red-team scenario:`. Attack patterns with no -scenario for a feature are listed in the agent's `coverage-gaps.yaml`. Report -both as gaps, not as failures. +`executable: false` with a `skip_reason`. Reasons from rook's own validation +start with `coverage gap:`, `invalid scenario:` or `invalid red-team +scenario:`; the scenario checker writes plain sentences, often a precondition +the profile cannot establish. Do not match on the prefix to find them; read +`executable`. Attack patterns with no scenario for a feature are listed in the +agent's `coverage-gaps.yaml` and the report's coverage gaps. Report both as +gaps, not as failures. + +A plain `generate` can plan no attacks for a small agent. When the user asked +for red-team coverage and none was written, run +`rook generate --class adversarial --json` and say so. Some attack patterns are skipped until the profile has the hook they need, and the skip reason names it: an `open` hook that seeds the fixture in diff --git a/skill-installer/skills/references/scenarios.md b/skill-installer/skills/references/scenarios.md index 8756ca4..fd0d683 100644 --- a/skill-installer/skills/references/scenarios.md +++ b/skill-installer/skills/references/scenarios.md @@ -23,7 +23,7 @@ references. ## Generating ```bash -rook generate --json # functional + adversarial, rook picks the count +rook generate --json # functional + adversarial planned, rook picks the count rook generate --total 20 --class functional --json rook generate --category prompt_injection,jailbreak --json rook generate -- focus on the refund limits and the identity check @@ -64,10 +64,17 @@ grading without reported usage, is skipped by `run` and counts toward the assurance gap, not toward Fail. File and image requirements are not rejected solely by their input kind: inspect what the profile script can actually send. `generate` keeps a scenario it cannot make runnable instead of dropping it: -`executable: false` with a `skip_reason` starting `coverage gap:`, -`invalid scenario:` or `invalid red-team scenario:`. Attack patterns with no -scenario for a feature are listed in the agent's `coverage-gaps.yaml`. Report -both as gaps, not as failures. +`executable: false` with a `skip_reason`. Reasons from rook's own validation +start with `coverage gap:`, `invalid scenario:` or `invalid red-team +scenario:`; the scenario checker writes plain sentences, often a precondition +the profile cannot establish. Do not match on the prefix to find them; read +`executable`. Attack patterns with no scenario for a feature are listed in the +agent's `coverage-gaps.yaml` and the report's coverage gaps. Report both as +gaps, not as failures. + +A plain `generate` can plan no attacks for a small agent. When the user asked +for red-team coverage and none was written, run +`rook generate --class adversarial --json` and say so. Some attack patterns are skipped until the profile has the hook they need, and the skip reason names it: an `open` hook that seeds the fixture in