diff --git a/.agents/skills/rook/references/headless-contract.md b/.agents/skills/rook/references/headless-contract.md index e45508f..0323f6c 100644 --- a/.agents/skills/rook/references/headless-contract.md +++ b/.agents/skills/rook/references/headless-contract.md @@ -65,10 +65,17 @@ still use the network, change state or invoke the target. ## On disk -`.testmuai/rook/settings.json` selects the project. New project directories -usually use `.testmuai/rook/projects/--/`; existing -`` directories are reused. Find the directory by its project ID -rather than assuming either form. Under its `agents//` directory: +In 0.1.6, `.testmuai/rook/settings.json` keeps project selections separately for +`local`, `stage` and `prod` in `selections[environment]`. After a switch, rook +uses the new environment's selection, which may be empty even when another +environment has one. Run `rook doctor` to see the active environment and +`rook status --json` to read its `project_id`; if it is missing or wrong, run +`rook project use `. + +New project directories usually use +`.testmuai/rook/projects/--/`; existing `` +directories are reused. Find the directory by its project ID rather than +assuming either form. Under its `agents//` directory: | Path | Contents | | --- | --- | diff --git a/.agents/skills/rook/references/troubleshooting.md b/.agents/skills/rook/references/troubleshooting.md index e1221f1..47397fd 100644 --- a/.agents/skills/rook/references/troubleshooting.md +++ b/.agents/skills/rook/references/troubleshooting.md @@ -10,8 +10,8 @@ rook status --json ``` `doctor` prints version, node, workspace, environment, controller and api -reachability, identity, auth state, project, mode (`headless` or `tui`), -tty, and state. +reachability, identity, auth state, project, mode (`headless` or `tui`) +and tty. ## Failure documents @@ -59,14 +59,18 @@ are about something rook cannot see (a tool call the agent made off-record, an image); that is a gap to report, not a failure to fix. Inspect the saved raw response and hook records before concluding that the -target omitted evidence. Two limitations observed in 0.1.1 can hide evidence already saved: -an explicitly empty execute `calls` array is omitted from the hook record, and -`json_path` checks do not decode a JSON string in `raw_response` before traversing -it. A recorded `calls: []` or a field visible inside that string therefore does -not guarantee a conclusive check. Report the limitation and retain the verdict; -do not invent calls or change scenario criteria just to obtain a pass. On another -release, verify whether the limitation still applies before attributing the gap -to it. +target omitted evidence. In 0.1.1, an explicitly empty execute `calls` array +could be omitted from the hook record. It is not established whether later +releases still do this. If the saved raw response contains `calls: []` but the +hook record has no `calls`, report that gap and retain the recorded verdict; +do not invent calls or change scenario criteria just to obtain a pass. + +The JSON-string decoding limitation observed in 0.1.1 is fixed in 0.1.6: +deterministic `json_path` checks can traverse valid JSON saved as a string in +`raw_response`. Invalid JSON, a raw string over the 8 MiB decode limit, or a +path that does not resolve yields Unable to Verify; the criterion's `achieved` +field explains why. Report that explanation and retain the recorded verdict; +do not invent evidence to fill the path. ## A verdict you disagree with diff --git a/.agents/skills/rook/references/verdicts.md b/.agents/skills/rook/references/verdicts.md index 97267d6..11fd55f 100644 --- a/.agents/skills/rook/references/verdicts.md +++ b/.agents/skills/rook/references/verdicts.md @@ -17,6 +17,16 @@ Statuses are **Pass**, **Fail**, and **Unable to Verify**. Unverifiable reasons: `agent_never_ran`, `not_observable`, or `undecidable`. A scenario that never ran has no verdict; do not turn its absence into a failure. +## Pass rate + +In 0.1.6, `totals.pass_rate` is `Pass / (Pass + Fail + Unable to Verify)`. +`totals.decided` still means `Pass + Fail`. Unable to Verify contributes to the +pass-rate denominator as a recorded result, but remains a separate verdict; +never describe it as Fail. If every recorded result is Unable to Verify, +`pass_rate` is `0`; if there are no recorded Pass, Fail or Unable to Verify +results, it is `null`, not zero. +Unjudged, not-run and unrunnable work does not enter this denominator. + ## Presenting results Use a table like this, populated from the records: diff --git a/.claude/skills/rook/references/headless-contract.md b/.claude/skills/rook/references/headless-contract.md index e45508f..0323f6c 100644 --- a/.claude/skills/rook/references/headless-contract.md +++ b/.claude/skills/rook/references/headless-contract.md @@ -65,10 +65,17 @@ still use the network, change state or invoke the target. ## On disk -`.testmuai/rook/settings.json` selects the project. New project directories -usually use `.testmuai/rook/projects/--/`; existing -`` directories are reused. Find the directory by its project ID -rather than assuming either form. Under its `agents//` directory: +In 0.1.6, `.testmuai/rook/settings.json` keeps project selections separately for +`local`, `stage` and `prod` in `selections[environment]`. After a switch, rook +uses the new environment's selection, which may be empty even when another +environment has one. Run `rook doctor` to see the active environment and +`rook status --json` to read its `project_id`; if it is missing or wrong, run +`rook project use `. + +New project directories usually use +`.testmuai/rook/projects/--/`; existing `` +directories are reused. Find the directory by its project ID rather than +assuming either form. Under its `agents//` directory: | Path | Contents | | --- | --- | diff --git a/.claude/skills/rook/references/troubleshooting.md b/.claude/skills/rook/references/troubleshooting.md index e1221f1..47397fd 100644 --- a/.claude/skills/rook/references/troubleshooting.md +++ b/.claude/skills/rook/references/troubleshooting.md @@ -10,8 +10,8 @@ rook status --json ``` `doctor` prints version, node, workspace, environment, controller and api -reachability, identity, auth state, project, mode (`headless` or `tui`), -tty, and state. +reachability, identity, auth state, project, mode (`headless` or `tui`) +and tty. ## Failure documents @@ -59,14 +59,18 @@ are about something rook cannot see (a tool call the agent made off-record, an image); that is a gap to report, not a failure to fix. Inspect the saved raw response and hook records before concluding that the -target omitted evidence. Two limitations observed in 0.1.1 can hide evidence already saved: -an explicitly empty execute `calls` array is omitted from the hook record, and -`json_path` checks do not decode a JSON string in `raw_response` before traversing -it. A recorded `calls: []` or a field visible inside that string therefore does -not guarantee a conclusive check. Report the limitation and retain the verdict; -do not invent calls or change scenario criteria just to obtain a pass. On another -release, verify whether the limitation still applies before attributing the gap -to it. +target omitted evidence. In 0.1.1, an explicitly empty execute `calls` array +could be omitted from the hook record. It is not established whether later +releases still do this. If the saved raw response contains `calls: []` but the +hook record has no `calls`, report that gap and retain the recorded verdict; +do not invent calls or change scenario criteria just to obtain a pass. + +The JSON-string decoding limitation observed in 0.1.1 is fixed in 0.1.6: +deterministic `json_path` checks can traverse valid JSON saved as a string in +`raw_response`. Invalid JSON, a raw string over the 8 MiB decode limit, or a +path that does not resolve yields Unable to Verify; the criterion's `achieved` +field explains why. Report that explanation and retain the recorded verdict; +do not invent evidence to fill the path. ## A verdict you disagree with diff --git a/.claude/skills/rook/references/verdicts.md b/.claude/skills/rook/references/verdicts.md index 97267d6..11fd55f 100644 --- a/.claude/skills/rook/references/verdicts.md +++ b/.claude/skills/rook/references/verdicts.md @@ -17,6 +17,16 @@ Statuses are **Pass**, **Fail**, and **Unable to Verify**. Unverifiable reasons: `agent_never_ran`, `not_observable`, or `undecidable`. A scenario that never ran has no verdict; do not turn its absence into a failure. +## Pass rate + +In 0.1.6, `totals.pass_rate` is `Pass / (Pass + Fail + Unable to Verify)`. +`totals.decided` still means `Pass + Fail`. Unable to Verify contributes to the +pass-rate denominator as a recorded result, but remains a separate verdict; +never describe it as Fail. If every recorded result is Unable to Verify, +`pass_rate` is `0`; if there are no recorded Pass, Fail or Unable to Verify +results, it is `null`, not zero. +Unjudged, not-run and unrunnable work does not enter this denominator. + ## Presenting results Use a table like this, populated from the records: diff --git a/skill-installer/skills/references/headless-contract.md b/skill-installer/skills/references/headless-contract.md index e45508f..0323f6c 100644 --- a/skill-installer/skills/references/headless-contract.md +++ b/skill-installer/skills/references/headless-contract.md @@ -65,10 +65,17 @@ still use the network, change state or invoke the target. ## On disk -`.testmuai/rook/settings.json` selects the project. New project directories -usually use `.testmuai/rook/projects/--/`; existing -`` directories are reused. Find the directory by its project ID -rather than assuming either form. Under its `agents//` directory: +In 0.1.6, `.testmuai/rook/settings.json` keeps project selections separately for +`local`, `stage` and `prod` in `selections[environment]`. After a switch, rook +uses the new environment's selection, which may be empty even when another +environment has one. Run `rook doctor` to see the active environment and +`rook status --json` to read its `project_id`; if it is missing or wrong, run +`rook project use `. + +New project directories usually use +`.testmuai/rook/projects/--/`; existing `` +directories are reused. Find the directory by its project ID rather than +assuming either form. Under its `agents//` directory: | Path | Contents | | --- | --- | diff --git a/skill-installer/skills/references/troubleshooting.md b/skill-installer/skills/references/troubleshooting.md index e1221f1..47397fd 100644 --- a/skill-installer/skills/references/troubleshooting.md +++ b/skill-installer/skills/references/troubleshooting.md @@ -10,8 +10,8 @@ rook status --json ``` `doctor` prints version, node, workspace, environment, controller and api -reachability, identity, auth state, project, mode (`headless` or `tui`), -tty, and state. +reachability, identity, auth state, project, mode (`headless` or `tui`) +and tty. ## Failure documents @@ -59,14 +59,18 @@ are about something rook cannot see (a tool call the agent made off-record, an image); that is a gap to report, not a failure to fix. Inspect the saved raw response and hook records before concluding that the -target omitted evidence. Two limitations observed in 0.1.1 can hide evidence already saved: -an explicitly empty execute `calls` array is omitted from the hook record, and -`json_path` checks do not decode a JSON string in `raw_response` before traversing -it. A recorded `calls: []` or a field visible inside that string therefore does -not guarantee a conclusive check. Report the limitation and retain the verdict; -do not invent calls or change scenario criteria just to obtain a pass. On another -release, verify whether the limitation still applies before attributing the gap -to it. +target omitted evidence. In 0.1.1, an explicitly empty execute `calls` array +could be omitted from the hook record. It is not established whether later +releases still do this. If the saved raw response contains `calls: []` but the +hook record has no `calls`, report that gap and retain the recorded verdict; +do not invent calls or change scenario criteria just to obtain a pass. + +The JSON-string decoding limitation observed in 0.1.1 is fixed in 0.1.6: +deterministic `json_path` checks can traverse valid JSON saved as a string in +`raw_response`. Invalid JSON, a raw string over the 8 MiB decode limit, or a +path that does not resolve yields Unable to Verify; the criterion's `achieved` +field explains why. Report that explanation and retain the recorded verdict; +do not invent evidence to fill the path. ## A verdict you disagree with diff --git a/skill-installer/skills/references/verdicts.md b/skill-installer/skills/references/verdicts.md index 97267d6..11fd55f 100644 --- a/skill-installer/skills/references/verdicts.md +++ b/skill-installer/skills/references/verdicts.md @@ -17,6 +17,16 @@ Statuses are **Pass**, **Fail**, and **Unable to Verify**. Unverifiable reasons: `agent_never_ran`, `not_observable`, or `undecidable`. A scenario that never ran has no verdict; do not turn its absence into a failure. +## Pass rate + +In 0.1.6, `totals.pass_rate` is `Pass / (Pass + Fail + Unable to Verify)`. +`totals.decided` still means `Pass + Fail`. Unable to Verify contributes to the +pass-rate denominator as a recorded result, but remains a separate verdict; +never describe it as Fail. If every recorded result is Unable to Verify, +`pass_rate` is `0`; if there are no recorded Pass, Fail or Unable to Verify +results, it is `null`, not zero. +Unjudged, not-run and unrunnable work does not enter this denominator. + ## Presenting results Use a table like this, populated from the records: