Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .claude/skills/run-assert-eval/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,18 @@ around X behavior", "add a stratify dimension for Y", or **govern the failure wi
re-measure to prove the rate dropped** — see Step 8 and
`workflows/govern-and-remeasure.md`).

**Non-ACS fixes are the user's call, outside this skill.** Troubleshooting a failure
may lead a coding agent to a non-ACS fix — upgrading or swapping the target model,
rewriting the agent's system prompt, or otherwise changing the agent itself. Those
are legitimate but are *agent changes*, not ACS governance: ACS can constrain inputs,
outputs, and tool calls, but it cannot add a capability the agent lacks. **Never fold
one into the ACS-governed run.** Step 8's A/B is readable only because the governed
run differs from the baseline in nothing but `run:` and `target.callable`; change the
model or the prompt inside it and two variables moved at once, so the delta is
attributable to neither — relabelling the run does not recover it. Measure an agent
change as its own arm instead: branch from the same baseline, reuse its exact test
set, and name that run for the change (e.g. `model-upgrade`).

## Authoritative references

Team-maintained docs on `main`. Prefer linking these over restating their content here —
Expand Down
8 changes: 8 additions & 0 deletions .cursor/rules/assert.mdc
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,14 @@ read https://raw.githubusercontent.com/responsibleai/assert-ai-action/main/ONBOA
- **Suggested next step**: one concrete action (tighten the system prompt around X, add a stratify
dimension for Y, or govern the failure with ACS and re-measure to prove the rate dropped — see Step 8 and
`../../.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md`).
- **Non-ACS fixes are the user's call, outside this skill**: troubleshooting may lead a coding agent to a
non-ACS fix — upgrading or swapping the target model, rewriting the system prompt, or otherwise changing the
agent. These are *agent changes*, not ACS governance (ACS can constrain inputs, outputs, and tool calls, but
cannot add a capability the agent lacks). **Never fold one into the ACS-governed run** — the A/B is readable
only because the governed run differs from the baseline in nothing but `run:` and `target.callable`, so a
model or prompt change inside it moves two variables at once and the delta is attributable to neither;
relabelling does not recover it. Measure an agent change as its own arm: branch from the same baseline, reuse
its exact test set, and name that run for the change (e.g. `model-upgrade`).

### Authoritative references

Expand Down
2 changes: 2 additions & 0 deletions .github/prompts/run-assert-eval.prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,8 @@ For each failure:

**Suggested next step**: one concrete action (e.g. "tighten the system prompt around X behavior", "add a stratify dimension for Y", or **govern the failure with ACS and re-measure to prove the rate dropped** — see Step 8 and `../../.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md`).

**Non-ACS fixes are the user's call, outside this skill.** Troubleshooting a failure may lead a coding agent to a non-ACS fix — upgrading or swapping the target model, rewriting the agent's system prompt, or otherwise changing the agent itself. Those are legitimate but are *agent changes*, not ACS governance: ACS can constrain inputs, outputs, and tool calls, but it cannot add a capability the agent lacks. **Never fold one into the ACS-governed run.** Step 8's A/B is readable only because the governed run differs from the baseline in nothing but `run:` and `target.callable`; change the model or the prompt inside it and two variables moved at once, so the delta is attributable to neither — relabelling the run does not recover it. Measure an agent change as its own arm instead: branch from the same baseline, reuse its exact test set, and name that run for the change (e.g. `model-upgrade`).

## Authoritative references

Team-maintained docs under `docs/` on `main` — prefer them over restating product behavior here. `guides/create-evaluation.md` and `config/schema.md` (step 3), `targets/callable.md` and `targets/model-and-tools.md` (step 4), `guides/troubleshooting.md` (step 5), `guides/results.md` (step 6), `guides/use-local-viewer.md` (step 7), `guides/securing-agents-with-acs.md` (step 8). This skill owns the methodology — the Clarity → ASSERT → ACS → ASSERT loop; those docs own product behavior.
Expand Down
Loading