From f60a14fc83db9ed837c85c5ea39b53548f296f11 Mon Sep 17 00:00:00 2001 From: Alex Ngo Date: Wed, 19 Aug 2026 16:33:36 -0700 Subject: [PATCH 1/2] feat(skill): keep non-ACS fixes out of acs-governed run labels. --- .claude/skills/run-assert-eval/SKILL.md | 9 +++++++++ .cursor/rules/assert.mdc | 5 +++++ .github/prompts/run-assert-eval.prompt.md | 2 ++ 3 files changed, 16 insertions(+) diff --git a/.claude/skills/run-assert-eval/SKILL.md b/.claude/skills/run-assert-eval/SKILL.md index 919cfc229..b179d1580 100644 --- a/.claude/skills/run-assert-eval/SKILL.md +++ b/.claude/skills/run-assert-eval/SKILL.md @@ -516,6 +516,15 @@ around X behavior", "add a stratify dimension for Y", or **govern the failure wi re-measure to prove the rate dropped** — see Step 8 and `workflows/govern-and-remeasure.md`). +**Non-ACS fixes are the user's call, outside this skill.** Troubleshooting a failure +may lead a coding agent to a non-ACS fix — upgrading or swapping the target model, +rewriting the agent's system prompt, or otherwise changing the agent itself. Those +are legitimate but are *agent changes*, not ACS governance: ACS can only restrict or +gate an output, never add a capability the model lacks. Keep the label honest — if a +governed run's improvement came from a non-ACS fix, do not record it under an +`acs-governed` run id; name the run for what actually produced the numbers (e.g. +`model-upgrade`) so the metrics aren't misattributed to ACS. + ## Authoritative references Team-maintained docs on `main`. Prefer linking these over restating their content here — diff --git a/.cursor/rules/assert.mdc b/.cursor/rules/assert.mdc index 37959c4d1..3e960ff82 100644 --- a/.cursor/rules/assert.mdc +++ b/.cursor/rules/assert.mdc @@ -274,6 +274,11 @@ read https://raw.githubusercontent.com/responsibleai/assert-ai-action/main/ONBOA - **Suggested next step**: one concrete action (tighten the system prompt around X, add a stratify dimension for Y, or govern the failure with ACS and re-measure to prove the rate dropped — see Step 8 and `../../.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md`). +- **Non-ACS fixes are the user's call, outside this skill**: troubleshooting may lead a coding agent to a + non-ACS fix — upgrading or swapping the target model, rewriting the system prompt, or otherwise changing the + agent. These are *agent changes*, not ACS governance (ACS can only restrict an output, never add a missing + capability). Keep the label honest — don't record such a run under `acs-governed`; name it for what produced + the numbers (e.g. `model-upgrade`) so metrics aren't misattributed to ACS. ### Authoritative references diff --git a/.github/prompts/run-assert-eval.prompt.md b/.github/prompts/run-assert-eval.prompt.md index 8c18d7bf5..473afe613 100644 --- a/.github/prompts/run-assert-eval.prompt.md +++ b/.github/prompts/run-assert-eval.prompt.md @@ -213,6 +213,8 @@ For each failure: **Suggested next step**: one concrete action (e.g. "tighten the system prompt around X behavior", "add a stratify dimension for Y", or **govern the failure with ACS and re-measure to prove the rate dropped** — see Step 8 and `../../.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md`). +**Non-ACS fixes are the user's call, outside this skill.** Troubleshooting a failure may lead a coding agent to a non-ACS fix — upgrading or swapping the target model, rewriting the agent's system prompt, or otherwise changing the agent itself. Those are legitimate but are *agent changes*, not ACS governance: ACS can only restrict or gate an output, never add a capability the model lacks. Keep the label honest — if a governed run's improvement came from a non-ACS fix, do not record it under an `acs-governed` run id; name the run for what actually produced the numbers (e.g. `model-upgrade`) so the metrics aren't misattributed to ACS. + ## Authoritative references Team-maintained docs under `docs/` on `main` — prefer them over restating product behavior here. `guides/create-evaluation.md` and `config/schema.md` (step 3), `targets/callable.md` and `targets/model-and-tools.md` (step 4), `guides/troubleshooting.md` (step 5), `guides/results.md` (step 6), `guides/use-local-viewer.md` (step 7), `guides/securing-agents-with-acs.md` (step 8). This skill owns the methodology — the Clarity → ASSERT → ACS → ASSERT loop; those docs own product behavior. From 41b5ca39e8e34d2e0523d00f043d4500ace6da43 Mon Sep 17 00:00:00 2001 From: Alex Ngo Date: Mon, 31 Aug 2026 12:11:44 -0700 Subject: [PATCH 2/2] fix(skill): forbid mixing agent changes into the ACS-governed A/B label. --- .claude/skills/run-assert-eval/SKILL.md | 13 ++++++++----- .cursor/rules/assert.mdc | 9 ++++++--- .github/prompts/run-assert-eval.prompt.md | 2 +- 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/.claude/skills/run-assert-eval/SKILL.md b/.claude/skills/run-assert-eval/SKILL.md index b179d1580..6b6f8c154 100644 --- a/.claude/skills/run-assert-eval/SKILL.md +++ b/.claude/skills/run-assert-eval/SKILL.md @@ -519,11 +519,14 @@ re-measure to prove the rate dropped** — see Step 8 and **Non-ACS fixes are the user's call, outside this skill.** Troubleshooting a failure may lead a coding agent to a non-ACS fix — upgrading or swapping the target model, rewriting the agent's system prompt, or otherwise changing the agent itself. Those -are legitimate but are *agent changes*, not ACS governance: ACS can only restrict or -gate an output, never add a capability the model lacks. Keep the label honest — if a -governed run's improvement came from a non-ACS fix, do not record it under an -`acs-governed` run id; name the run for what actually produced the numbers (e.g. -`model-upgrade`) so the metrics aren't misattributed to ACS. +are legitimate but are *agent changes*, not ACS governance: ACS can constrain inputs, +outputs, and tool calls, but it cannot add a capability the agent lacks. **Never fold +one into the ACS-governed run.** Step 8's A/B is readable only because the governed +run differs from the baseline in nothing but `run:` and `target.callable`; change the +model or the prompt inside it and two variables moved at once, so the delta is +attributable to neither — relabelling the run does not recover it. Measure an agent +change as its own arm instead: branch from the same baseline, reuse its exact test +set, and name that run for the change (e.g. `model-upgrade`). ## Authoritative references diff --git a/.cursor/rules/assert.mdc b/.cursor/rules/assert.mdc index 3e960ff82..b40572165 100644 --- a/.cursor/rules/assert.mdc +++ b/.cursor/rules/assert.mdc @@ -276,9 +276,12 @@ read https://raw.githubusercontent.com/responsibleai/assert-ai-action/main/ONBOA `../../.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md`). - **Non-ACS fixes are the user's call, outside this skill**: troubleshooting may lead a coding agent to a non-ACS fix — upgrading or swapping the target model, rewriting the system prompt, or otherwise changing the - agent. These are *agent changes*, not ACS governance (ACS can only restrict an output, never add a missing - capability). Keep the label honest — don't record such a run under `acs-governed`; name it for what produced - the numbers (e.g. `model-upgrade`) so metrics aren't misattributed to ACS. + agent. These are *agent changes*, not ACS governance (ACS can constrain inputs, outputs, and tool calls, but + cannot add a capability the agent lacks). **Never fold one into the ACS-governed run** — the A/B is readable + only because the governed run differs from the baseline in nothing but `run:` and `target.callable`, so a + model or prompt change inside it moves two variables at once and the delta is attributable to neither; + relabelling does not recover it. Measure an agent change as its own arm: branch from the same baseline, reuse + its exact test set, and name that run for the change (e.g. `model-upgrade`). ### Authoritative references diff --git a/.github/prompts/run-assert-eval.prompt.md b/.github/prompts/run-assert-eval.prompt.md index 473afe613..e08974091 100644 --- a/.github/prompts/run-assert-eval.prompt.md +++ b/.github/prompts/run-assert-eval.prompt.md @@ -213,7 +213,7 @@ For each failure: **Suggested next step**: one concrete action (e.g. "tighten the system prompt around X behavior", "add a stratify dimension for Y", or **govern the failure with ACS and re-measure to prove the rate dropped** — see Step 8 and `../../.claude/skills/run-assert-eval/workflows/govern-and-remeasure.md`). -**Non-ACS fixes are the user's call, outside this skill.** Troubleshooting a failure may lead a coding agent to a non-ACS fix — upgrading or swapping the target model, rewriting the agent's system prompt, or otherwise changing the agent itself. Those are legitimate but are *agent changes*, not ACS governance: ACS can only restrict or gate an output, never add a capability the model lacks. Keep the label honest — if a governed run's improvement came from a non-ACS fix, do not record it under an `acs-governed` run id; name the run for what actually produced the numbers (e.g. `model-upgrade`) so the metrics aren't misattributed to ACS. +**Non-ACS fixes are the user's call, outside this skill.** Troubleshooting a failure may lead a coding agent to a non-ACS fix — upgrading or swapping the target model, rewriting the agent's system prompt, or otherwise changing the agent itself. Those are legitimate but are *agent changes*, not ACS governance: ACS can constrain inputs, outputs, and tool calls, but it cannot add a capability the agent lacks. **Never fold one into the ACS-governed run.** Step 8's A/B is readable only because the governed run differs from the baseline in nothing but `run:` and `target.callable`; change the model or the prompt inside it and two variables moved at once, so the delta is attributable to neither — relabelling the run does not recover it. Measure an agent change as its own arm instead: branch from the same baseline, reuse its exact test set, and name that run for the change (e.g. `model-upgrade`). ## Authoritative references