diff --git a/.agents/skills/rook/references/headless-contract.md b/.agents/skills/rook/references/headless-contract.md index 73c70d0..cb636f6 100644 --- a/.agents/skills/rook/references/headless-contract.md +++ b/.agents/skills/rook/references/headless-contract.md @@ -20,7 +20,7 @@ Commands that emit JSON write one document to stdout and prose to stderr. | `plan` | `{ username, user_id, org_name, org_id, subscription, credits }`; null credits means the balance could not be read | | `run` | `{ ok, error?, run_id?, halted, reason?, discarded?, unrunnable?, flag_problems?, credits, report? }` | | `report` | `{ run_id, name?, dir, report }`; `dir` is the evidence folder | -| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, repeat, criteria }] }` | +| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, criteria }] }` | | `scenarios exclude/include/delete` | `{ ok: true, verb, ... }` | | `status` | `{ project_id, offline, agents: [{ local_id, name, tree, offline, features, scenarios, profiles, unfinished_runs, owed_runs }], runs? }` | | `mcp list` | `{ errors, servers: [{ name, origin, transport, state, source, changed_since_approval, shadowed_by }] }` | diff --git a/.agents/skills/rook/references/scenarios.md b/.agents/skills/rook/references/scenarios.md index af6de4f..cd34421 100644 --- a/.agents/skills/rook/references/scenarios.md +++ b/.agents/skills/rook/references/scenarios.md @@ -10,9 +10,12 @@ | Class | Categories | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | functional | `happy_path`, `negative`, `boundary`, `integration`, `state_context` | -| non_functional | `performance`, `token_economy`, `reliability`, `quality` | +| non_functional | `token_economy`, `quality` | | adversarial | `prompt_injection`, `jailbreak`, `data_exfiltration`, `pii_leakage`, `harmful_content`, `hallucination`, `hijacking`, `policy_violation`, `technical_injection` | +`performance` and `reliability` are no longer categories: `generate` refuses +them, though existing scenario files that use them still load and run. + Every scenario also carries free-form tags; `run --tag ` selects on them. ## Generating @@ -27,7 +30,13 @@ rook generate -- focus on the refund limits and the identity check `generate` derives scenarios per feature and pins them; a second `generate` only refreshes features whose pins are stale. `--force` re-derives everything. It may emit text despite `--json`; check the exit code, then read -`rook scenarios list --json`. +`rook scenarios list --json`. An unknown `--class` or `--category` makes it exit +1 before analysing anything, listing every valid value. + +Each scenario's feature is the one it was assigned. If the model named a +different feature, `generate` warns and saves that scenario excluded; review it +in `rook scenarios list --json` and bring it back with +`rook scenarios include ` if it belongs in runs. ## Listing and curating @@ -63,4 +72,4 @@ rook run --resume --json # carry finished work into `--only` takes one comma-separated argument; repeating the flag keeps only the last occurrence, and a space-separated second id is read as instruction -text. +text. Each scenario runs once; there is no repeat count. diff --git a/.claude/skills/rook/references/headless-contract.md b/.claude/skills/rook/references/headless-contract.md index 73c70d0..cb636f6 100644 --- a/.claude/skills/rook/references/headless-contract.md +++ b/.claude/skills/rook/references/headless-contract.md @@ -20,7 +20,7 @@ Commands that emit JSON write one document to stdout and prose to stderr. | `plan` | `{ username, user_id, org_name, org_id, subscription, credits }`; null credits means the balance could not be read | | `run` | `{ ok, error?, run_id?, halted, reason?, discarded?, unrunnable?, flag_problems?, credits, report? }` | | `report` | `{ run_id, name?, dir, report }`; `dir` is the evidence folder | -| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, repeat, criteria }] }` | +| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, criteria }] }` | | `scenarios exclude/include/delete` | `{ ok: true, verb, ... }` | | `status` | `{ project_id, offline, agents: [{ local_id, name, tree, offline, features, scenarios, profiles, unfinished_runs, owed_runs }], runs? }` | | `mcp list` | `{ errors, servers: [{ name, origin, transport, state, source, changed_since_approval, shadowed_by }] }` | diff --git a/.claude/skills/rook/references/scenarios.md b/.claude/skills/rook/references/scenarios.md index af6de4f..cd34421 100644 --- a/.claude/skills/rook/references/scenarios.md +++ b/.claude/skills/rook/references/scenarios.md @@ -10,9 +10,12 @@ | Class | Categories | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | functional | `happy_path`, `negative`, `boundary`, `integration`, `state_context` | -| non_functional | `performance`, `token_economy`, `reliability`, `quality` | +| non_functional | `token_economy`, `quality` | | adversarial | `prompt_injection`, `jailbreak`, `data_exfiltration`, `pii_leakage`, `harmful_content`, `hallucination`, `hijacking`, `policy_violation`, `technical_injection` | +`performance` and `reliability` are no longer categories: `generate` refuses +them, though existing scenario files that use them still load and run. + Every scenario also carries free-form tags; `run --tag ` selects on them. ## Generating @@ -27,7 +30,13 @@ rook generate -- focus on the refund limits and the identity check `generate` derives scenarios per feature and pins them; a second `generate` only refreshes features whose pins are stale. `--force` re-derives everything. It may emit text despite `--json`; check the exit code, then read -`rook scenarios list --json`. +`rook scenarios list --json`. An unknown `--class` or `--category` makes it exit +1 before analysing anything, listing every valid value. + +Each scenario's feature is the one it was assigned. If the model named a +different feature, `generate` warns and saves that scenario excluded; review it +in `rook scenarios list --json` and bring it back with +`rook scenarios include ` if it belongs in runs. ## Listing and curating @@ -63,4 +72,4 @@ rook run --resume --json # carry finished work into `--only` takes one comma-separated argument; repeating the flag keeps only the last occurrence, and a space-separated second id is read as instruction -text. +text. Each scenario runs once; there is no repeat count. diff --git a/skill-installer/skills/references/headless-contract.md b/skill-installer/skills/references/headless-contract.md index 73c70d0..cb636f6 100644 --- a/skill-installer/skills/references/headless-contract.md +++ b/skill-installer/skills/references/headless-contract.md @@ -20,7 +20,7 @@ Commands that emit JSON write one document to stdout and prose to stderr. | `plan` | `{ username, user_id, org_name, org_id, subscription, credits }`; null credits means the balance could not be read | | `run` | `{ ok, error?, run_id?, halted, reason?, discarded?, unrunnable?, flag_problems?, credits, report? }` | | `report` | `{ run_id, name?, dir, report }`; `dir` is the evidence folder | -| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, repeat, criteria }] }` | +| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, criteria }] }` | | `scenarios exclude/include/delete` | `{ ok: true, verb, ... }` | | `status` | `{ project_id, offline, agents: [{ local_id, name, tree, offline, features, scenarios, profiles, unfinished_runs, owed_runs }], runs? }` | | `mcp list` | `{ errors, servers: [{ name, origin, transport, state, source, changed_since_approval, shadowed_by }] }` | diff --git a/skill-installer/skills/references/scenarios.md b/skill-installer/skills/references/scenarios.md index af6de4f..cd34421 100644 --- a/skill-installer/skills/references/scenarios.md +++ b/skill-installer/skills/references/scenarios.md @@ -10,9 +10,12 @@ | Class | Categories | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | functional | `happy_path`, `negative`, `boundary`, `integration`, `state_context` | -| non_functional | `performance`, `token_economy`, `reliability`, `quality` | +| non_functional | `token_economy`, `quality` | | adversarial | `prompt_injection`, `jailbreak`, `data_exfiltration`, `pii_leakage`, `harmful_content`, `hallucination`, `hijacking`, `policy_violation`, `technical_injection` | +`performance` and `reliability` are no longer categories: `generate` refuses +them, though existing scenario files that use them still load and run. + Every scenario also carries free-form tags; `run --tag ` selects on them. ## Generating @@ -27,7 +30,13 @@ rook generate -- focus on the refund limits and the identity check `generate` derives scenarios per feature and pins them; a second `generate` only refreshes features whose pins are stale. `--force` re-derives everything. It may emit text despite `--json`; check the exit code, then read -`rook scenarios list --json`. +`rook scenarios list --json`. An unknown `--class` or `--category` makes it exit +1 before analysing anything, listing every valid value. + +Each scenario's feature is the one it was assigned. If the model named a +different feature, `generate` warns and saves that scenario excluded; review it +in `rook scenarios list --json` and bring it back with +`rook scenarios include ` if it belongs in runs. ## Listing and curating @@ -63,4 +72,4 @@ rook run --resume --json # carry finished work into `--only` takes one comma-separated argument; repeating the flag keeps only the last occurrence, and a space-separated second id is read as instruction -text. +text. Each scenario runs once; there is no repeat count.