Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/rook/references/headless-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Commands that emit JSON write one document to stdout and prose to stderr.
| `plan` | `{ username, user_id, org_name, org_id, subscription, credits }`; null credits means the balance could not be read |
| `run` | `{ ok, error?, run_id?, halted, reason?, discarded?, unrunnable?, flag_problems?, credits, report? }` |
| `report` | `{ run_id, name?, dir, report }`; `dir` is the evidence folder |
| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, repeat, criteria }] }` |
| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, criteria }] }` |
| `scenarios exclude/include/delete` | `{ ok: true, verb, ... }` |
| `status` | `{ project_id, offline, agents: [{ local_id, name, tree, offline, features, scenarios, profiles, unfinished_runs, owed_runs }], runs? }` |
| `mcp list` | `{ errors, servers: [{ name, origin, transport, state, source, changed_since_approval, shadowed_by }] }` |
Expand Down
15 changes: 12 additions & 3 deletions .agents/skills/rook/references/scenarios.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,12 @@
| Class | Categories |
| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| functional | `happy_path`, `negative`, `boundary`, `integration`, `state_context` |
| non_functional | `performance`, `token_economy`, `reliability`, `quality` |
| non_functional | `token_economy`, `quality` |
| adversarial | `prompt_injection`, `jailbreak`, `data_exfiltration`, `pii_leakage`, `harmful_content`, `hallucination`, `hijacking`, `policy_violation`, `technical_injection` |

`performance` and `reliability` are no longer categories: `generate` refuses
them, though existing scenario files that use them still load and run.

Every scenario also carries free-form tags; `run --tag <names>` selects on them.

## Generating
Expand All @@ -27,7 +30,13 @@ rook generate -- focus on the refund limits and the identity check
`generate` derives scenarios per feature and pins them; a second `generate`
only refreshes features whose pins are stale. `--force` re-derives everything.
It may emit text despite `--json`; check the exit code, then read
`rook scenarios list --json`.
`rook scenarios list --json`. An unknown `--class` or `--category` makes it exit
1 before analysing anything, listing every valid value.

Each scenario's feature is the one it was assigned. If the model named a
different feature, `generate` warns and saves that scenario excluded; review it
in `rook scenarios list --json` and bring it back with
`rook scenarios include <id>` if it belongs in runs.

## Listing and curating

Expand Down Expand Up @@ -63,4 +72,4 @@ rook run --resume <id> --json # carry finished work into

`--only` takes one comma-separated argument; repeating the flag keeps only
the last occurrence, and a space-separated second id is read as instruction
text.
text. Each scenario runs once; there is no repeat count.
2 changes: 1 addition & 1 deletion .claude/skills/rook/references/headless-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Commands that emit JSON write one document to stdout and prose to stderr.
| `plan` | `{ username, user_id, org_name, org_id, subscription, credits }`; null credits means the balance could not be read |
| `run` | `{ ok, error?, run_id?, halted, reason?, discarded?, unrunnable?, flag_problems?, credits, report? }` |
| `report` | `{ run_id, name?, dir, report }`; `dir` is the evidence folder |
| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, repeat, criteria }] }` |
| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, criteria }] }` |
| `scenarios exclude/include/delete` | `{ ok: true, verb, ... }` |
| `status` | `{ project_id, offline, agents: [{ local_id, name, tree, offline, features, scenarios, profiles, unfinished_runs, owed_runs }], runs? }` |
| `mcp list` | `{ errors, servers: [{ name, origin, transport, state, source, changed_since_approval, shadowed_by }] }` |
Expand Down
15 changes: 12 additions & 3 deletions .claude/skills/rook/references/scenarios.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,12 @@
| Class | Categories |
| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| functional | `happy_path`, `negative`, `boundary`, `integration`, `state_context` |
| non_functional | `performance`, `token_economy`, `reliability`, `quality` |
| non_functional | `token_economy`, `quality` |
| adversarial | `prompt_injection`, `jailbreak`, `data_exfiltration`, `pii_leakage`, `harmful_content`, `hallucination`, `hijacking`, `policy_violation`, `technical_injection` |

`performance` and `reliability` are no longer categories: `generate` refuses
them, though existing scenario files that use them still load and run.

Every scenario also carries free-form tags; `run --tag <names>` selects on them.

## Generating
Expand All @@ -27,7 +30,13 @@ rook generate -- focus on the refund limits and the identity check
`generate` derives scenarios per feature and pins them; a second `generate`
only refreshes features whose pins are stale. `--force` re-derives everything.
It may emit text despite `--json`; check the exit code, then read
`rook scenarios list --json`.
`rook scenarios list --json`. An unknown `--class` or `--category` makes it exit
1 before analysing anything, listing every valid value.

Each scenario's feature is the one it was assigned. If the model named a
different feature, `generate` warns and saves that scenario excluded; review it
in `rook scenarios list --json` and bring it back with
`rook scenarios include <id>` if it belongs in runs.

## Listing and curating

Expand Down Expand Up @@ -63,4 +72,4 @@ rook run --resume <id> --json # carry finished work into

`--only` takes one comma-separated argument; repeating the flag keeps only
the last occurrence, and a space-separated second id is read as instruction
text.
text. Each scenario runs once; there is no repeat count.
2 changes: 1 addition & 1 deletion skill-installer/skills/references/headless-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Commands that emit JSON write one document to stdout and prose to stderr.
| `plan` | `{ username, user_id, org_name, org_id, subscription, credits }`; null credits means the balance could not be read |
| `run` | `{ ok, error?, run_id?, halted, reason?, discarded?, unrunnable?, flag_problems?, credits, report? }` |
| `report` | `{ run_id, name?, dir, report }`; `dir` is the evidence folder |
| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, repeat, criteria }] }` |
| `scenarios list` | `{ agent_id, profile_id, total, runnable, scenarios: [{ scenario_id, title, feature_id, class, category?, state, excluded, unrunnable, multi_turn, criteria }] }` |
| `scenarios exclude/include/delete` | `{ ok: true, verb, ... }` |
| `status` | `{ project_id, offline, agents: [{ local_id, name, tree, offline, features, scenarios, profiles, unfinished_runs, owed_runs }], runs? }` |
| `mcp list` | `{ errors, servers: [{ name, origin, transport, state, source, changed_since_approval, shadowed_by }] }` |
Expand Down
15 changes: 12 additions & 3 deletions skill-installer/skills/references/scenarios.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,12 @@
| Class | Categories |
| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| functional | `happy_path`, `negative`, `boundary`, `integration`, `state_context` |
| non_functional | `performance`, `token_economy`, `reliability`, `quality` |
| non_functional | `token_economy`, `quality` |
| adversarial | `prompt_injection`, `jailbreak`, `data_exfiltration`, `pii_leakage`, `harmful_content`, `hallucination`, `hijacking`, `policy_violation`, `technical_injection` |

`performance` and `reliability` are no longer categories: `generate` refuses
them, though existing scenario files that use them still load and run.

Every scenario also carries free-form tags; `run --tag <names>` selects on them.

## Generating
Expand All @@ -27,7 +30,13 @@ rook generate -- focus on the refund limits and the identity check
`generate` derives scenarios per feature and pins them; a second `generate`
only refreshes features whose pins are stale. `--force` re-derives everything.
It may emit text despite `--json`; check the exit code, then read
`rook scenarios list --json`.
`rook scenarios list --json`. An unknown `--class` or `--category` makes it exit
1 before analysing anything, listing every valid value.

Each scenario's feature is the one it was assigned. If the model named a
different feature, `generate` warns and saves that scenario excluded; review it
in `rook scenarios list --json` and bring it back with
`rook scenarios include <id>` if it belongs in runs.

## Listing and curating

Expand Down Expand Up @@ -63,4 +72,4 @@ rook run --resume <id> --json # carry finished work into

`--only` takes one comma-separated argument; repeating the flag keeps only
the last occurrence, and a space-separated second id is read as instruction
text.
text. Each scenario runs once; there is no repeat count.
Loading