From 291e8631351940f5560a25676e2f5cc0c38d12f8 Mon Sep 17 00:00:00 2001 From: Yeming Tang Date: Wed, 12 Aug 2026 23:29:12 +0000 Subject: [PATCH] docs+skill: onboarding UX (Node 22 prereq, version pin, explicit model selection) Three small onboarding-experience improvements grouped together because they all fix places where the current flow silently fails or silently picks a suboptimal default: README: - Option 1 (npx skills add ...) now states the Node.js >= 22 prerequisite up front and points users to Option 2 as the no-Node fallback. The underlying skills CLI imports styleText from node:util which is Node-22-only; running under Node 18/20 fails with a SyntaxError that looks unrelated to Node. wire-assert-ci skill (SKILL.md + prompt + mdc mirrors): - 'assert-ai version pin' section: at workflow-authoring time (not every CI run), resolve the latest stable assert-ai from PyPI and write that exact version into assert-ai-version. Pinning is required so the paired-binary McNemar test compares like against like across baseline and PR runs; a floating install would attribute version drift to code drift and produce false regressions. Users still own subsequent bumps. - 'Model selection' section: available models are user-specific (Azure deployment names, provider access), so the skill must ask the user before generating any behavior YAML. Ask for three roles in one prompt (judge, tester, default), use LiteLLM-style names, verify provider-env secrets match the chosen provider, and write every role explicitly. Never let pipeline.judge.model silently fall back to default_model (which is typically small/cheap and downgrades gate fidelity), and never write tester: null without explicit single-turn confirmation. Replaces the earlier draft that let the skill guess a default model. --- README.md | 2 + skills/wire-assert-ci/SKILL.md | 37 ++++++++++++++++++- skills/wire-assert-ci/assert-ci.mdc | 37 ++++++++++++++++++- .../wire-assert-ci/wire-assert-ci.prompt.md | 37 ++++++++++++++++++- 4 files changed, 110 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 345900c..7ef6a6f 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,8 @@ Two ways in. Both end with a working gate; pick based on whether you have Node. ### Option 1 — install the skills (recommended, 40+ agents) +Requires Node.js ≥ 22 (the `skills` CLI uses `node:util`'s `styleText`, added in Node 22). On Node 18/20 you'll see `SyntaxError: The requested module 'node:util' does not provide an export named 'styleText'` — use [Option 2](#option-2--paste-one-url-no-install-no-node) instead, or upgrade Node. + ```bash npx skills add responsibleai/ASSERT --skill run-assert-eval --yes npx skills add responsibleai/assert-ai-action --skill wire-assert-ci --yes diff --git a/skills/wire-assert-ci/SKILL.md b/skills/wire-assert-ci/SKILL.md index ac9d93c..0afb48b 100644 --- a/skills/wire-assert-ci/SKILL.md +++ b/skills/wire-assert-ci/SKILL.md @@ -71,6 +71,26 @@ Sample-size guidance is part of the gate design: Delegate config generation to `run-assert-eval` by invoking the installed `run-assert-eval` skill/prompt/rule and asking it to use `assert-ai init` for each behavior YAML. Do not hand-write pipeline internals unless you are only applying the route wrapper chosen above. +### Model selection + +Available models are user-specific: Azure OpenAI deployment names vary per resource, and OpenAI/Anthropic/other providers vary per account. Do not guess or silently default. Before generating any behavior YAML, ask the user for the deployed models they can use, then write those exact values into every file. + +Ask for three roles in one prompt — the user may answer with one, two, or three distinct models: + +1. **Judge model** — rates each conversation against the rubric. Judge fidelity dominates gate quality; steer the user toward their strongest available reasoning model (for example, a full GPT-5-class deployment). +2. **Tester (user-simulator) model** — drives the customer side of multi-turn conversations for Rung 1/2 behaviors. Latency and cost matter more than reasoning depth; a mini/small deployment is usually fine. If the behavior is single-turn, leave `pipeline.inference.tester` unset and tell the user. +3. **Default model** — used by `pipeline.systematize` (behavior taxonomy) and `pipeline.test_set` (prompt/scenario generation). Cheap/small is fine; correctness of intermediate reasoning is not the gate. + +Use LiteLLM-style names as they appear in the user's environment (e.g. `azure/`, `openai/gpt-5.4`, `anthropic/claude-4-sonnet`). When the user gives an Azure deployment name, verify the corresponding provider secrets (`AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`) are already listed in `provider-env`; if not, add them. Never inline API keys. + +Write the answers into every behavior YAML explicitly: + +- `default_model.name: ` +- `pipeline.inference.tester.model.name: ` (omit the block if single-turn) +- `pipeline.judge.model.name: ` + +Do not leave `pipeline.judge.model` unset; letting the judge fall back to `default_model` silently downgrades it to a small model and the gate becomes unreliable. Do not write `tester: null` unless the user confirmed the behavior is single-turn. + ## 4. Run the live baseline Delegate the live run to `run-assert-eval` only after the user confirms the spec and cost. Let it own pipeline execution and result reporting. Capture only the suite/run identifiers from `artifacts/results///` needed for ACS and CI setup. Do not commit generated artifacts. @@ -114,7 +134,7 @@ jobs: guard-dimensions: overrefusal alpha: '0.05' min-pairs: '30' - assert-ai-version: '0.1.0' + assert-ai-version: '' # resolve once at workflow creation, then pin; see "assert-ai version pin" below extras: regression,otel,langgraph target-install: python -m pip install -e . provider-env: | @@ -147,6 +167,21 @@ assert-ai acs validate --manifest artifacts/acs//manifest.yaml --suite /report.md`, the manifest, and generated policy. Propose the code/prompt/policy fixes and ask for confirmation before applying them. After confirmation, create a branch named `assert/acs-`, apply the fixes, and open a PR. When opening the PR, state: because this branch uses `gate-mode: improvement`, the gate passes only on a statistically significant `policy_violation` gain with no `overrefusal` regression; a change that merely trends better fails. +## assert-ai version pin + +At workflow-authoring time — not at every CI run — resolve the latest stable `assert-ai` release from PyPI and write that exact version into `assert-ai-version`. Pinning is required because the paired-binary McNemar test compares a baseline artifact against later PR runs, and any change in the test-set generator, tester, or judge behavior between versions would show up as a false regression. + +Use the first working command: + +```bash +curl -fsSL https://pypi.org/pypi/assert-ai/json | python -c "import sys,json;info=json.load(sys.stdin)['info'];print(info['version'])" +# fallback: pip index versions assert-ai | head -1 | awk '{print $2}' | tr -d '()' +``` + +Substitute the resolved version for `` in the workflow above. Tell the user what version was written and that they own subsequent bumps: re-baseline after any `assert-ai-version` change so the next PR run compares like against like. + +Do not write `assert-ai-version: latest` or omit the input — the action's default is a fixed older version, and floating installs would break paired comparisons. + ## Guardrails - Do not push without user approval. diff --git a/skills/wire-assert-ci/assert-ci.mdc b/skills/wire-assert-ci/assert-ci.mdc index 4840598..32d3ba4 100644 --- a/skills/wire-assert-ci/assert-ci.mdc +++ b/skills/wire-assert-ci/assert-ci.mdc @@ -69,6 +69,26 @@ Sample-size guidance is part of the gate design: Delegate config generation to `run-assert-eval` by invoking the installed `run-assert-eval` skill/prompt/rule and asking it to use `assert-ai init` for each behavior YAML. Do not hand-write pipeline internals unless you are only applying the route wrapper chosen above. +### Model selection + +Available models are user-specific: Azure OpenAI deployment names vary per resource, and OpenAI/Anthropic/other providers vary per account. Do not guess or silently default. Before generating any behavior YAML, ask the user for the deployed models they can use, then write those exact values into every file. + +Ask for three roles in one prompt — the user may answer with one, two, or three distinct models: + +1. **Judge model** — rates each conversation against the rubric. Judge fidelity dominates gate quality; steer the user toward their strongest available reasoning model (for example, a full GPT-5-class deployment). +2. **Tester (user-simulator) model** — drives the customer side of multi-turn conversations for Rung 1/2 behaviors. Latency and cost matter more than reasoning depth; a mini/small deployment is usually fine. If the behavior is single-turn, leave `pipeline.inference.tester` unset and tell the user. +3. **Default model** — used by `pipeline.systematize` (behavior taxonomy) and `pipeline.test_set` (prompt/scenario generation). Cheap/small is fine; correctness of intermediate reasoning is not the gate. + +Use LiteLLM-style names as they appear in the user's environment (e.g. `azure/`, `openai/gpt-5.4`, `anthropic/claude-4-sonnet`). When the user gives an Azure deployment name, verify the corresponding provider secrets (`AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`) are already listed in `provider-env`; if not, add them. Never inline API keys. + +Write the answers into every behavior YAML explicitly: + +- `default_model.name: ` +- `pipeline.inference.tester.model.name: ` (omit the block if single-turn) +- `pipeline.judge.model.name: ` + +Do not leave `pipeline.judge.model` unset; letting the judge fall back to `default_model` silently downgrades it to a small model and the gate becomes unreliable. Do not write `tester: null` unless the user confirmed the behavior is single-turn. + ## 4. Run the live baseline Delegate the live run to `run-assert-eval` only after the user confirms the spec and cost. Let it own pipeline execution and result reporting. Capture only the suite/run identifiers from `artifacts/results///` needed for ACS and CI setup. Do not commit generated artifacts. @@ -112,7 +132,7 @@ jobs: guard-dimensions: overrefusal alpha: '0.05' min-pairs: '30' - assert-ai-version: '0.1.0' + assert-ai-version: '' # resolve once at workflow creation, then pin; see "assert-ai version pin" below extras: regression,otel,langgraph target-install: python -m pip install -e . provider-env: | @@ -145,6 +165,21 @@ assert-ai acs validate --manifest artifacts/acs//manifest.yaml --suite /report.md`, the manifest, and generated policy. Propose the code/prompt/policy fixes and ask for confirmation before applying them. After confirmation, create a branch named `assert/acs-`, apply the fixes, and open a PR. When opening the PR, state: because this branch uses `gate-mode: improvement`, the gate passes only on a statistically significant `policy_violation` gain with no `overrefusal` regression; a change that merely trends better fails. +## assert-ai version pin + +At workflow-authoring time — not at every CI run — resolve the latest stable `assert-ai` release from PyPI and write that exact version into `assert-ai-version`. Pinning is required because the paired-binary McNemar test compares a baseline artifact against later PR runs, and any change in the test-set generator, tester, or judge behavior between versions would show up as a false regression. + +Use the first working command: + +```bash +curl -fsSL https://pypi.org/pypi/assert-ai/json | python -c "import sys,json;info=json.load(sys.stdin)['info'];print(info['version'])" +# fallback: pip index versions assert-ai | head -1 | awk '{print $2}' | tr -d '()' +``` + +Substitute the resolved version for `` in the workflow above. Tell the user what version was written and that they own subsequent bumps: re-baseline after any `assert-ai-version` change so the next PR run compares like against like. + +Do not write `assert-ai-version: latest` or omit the input — the action's default is a fixed older version, and floating installs would break paired comparisons. + ## Guardrails - Do not push without user approval. diff --git a/skills/wire-assert-ci/wire-assert-ci.prompt.md b/skills/wire-assert-ci/wire-assert-ci.prompt.md index 382dd84..f181adf 100644 --- a/skills/wire-assert-ci/wire-assert-ci.prompt.md +++ b/skills/wire-assert-ci/wire-assert-ci.prompt.md @@ -67,6 +67,26 @@ Sample-size guidance is part of the gate design: Delegate config generation to `run-assert-eval` by invoking the installed `run-assert-eval` skill/prompt/rule and asking it to use `assert-ai init` for each behavior YAML. Do not hand-write pipeline internals unless you are only applying the route wrapper chosen above. +### Model selection + +Available models are user-specific: Azure OpenAI deployment names vary per resource, and OpenAI/Anthropic/other providers vary per account. Do not guess or silently default. Before generating any behavior YAML, ask the user for the deployed models they can use, then write those exact values into every file. + +Ask for three roles in one prompt — the user may answer with one, two, or three distinct models: + +1. **Judge model** — rates each conversation against the rubric. Judge fidelity dominates gate quality; steer the user toward their strongest available reasoning model (for example, a full GPT-5-class deployment). +2. **Tester (user-simulator) model** — drives the customer side of multi-turn conversations for Rung 1/2 behaviors. Latency and cost matter more than reasoning depth; a mini/small deployment is usually fine. If the behavior is single-turn, leave `pipeline.inference.tester` unset and tell the user. +3. **Default model** — used by `pipeline.systematize` (behavior taxonomy) and `pipeline.test_set` (prompt/scenario generation). Cheap/small is fine; correctness of intermediate reasoning is not the gate. + +Use LiteLLM-style names as they appear in the user's environment (e.g. `azure/`, `openai/gpt-5.4`, `anthropic/claude-4-sonnet`). When the user gives an Azure deployment name, verify the corresponding provider secrets (`AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`) are already listed in `provider-env`; if not, add them. Never inline API keys. + +Write the answers into every behavior YAML explicitly: + +- `default_model.name: ` +- `pipeline.inference.tester.model.name: ` (omit the block if single-turn) +- `pipeline.judge.model.name: ` + +Do not leave `pipeline.judge.model` unset; letting the judge fall back to `default_model` silently downgrades it to a small model and the gate becomes unreliable. Do not write `tester: null` unless the user confirmed the behavior is single-turn. + ## 4. Run the live baseline Delegate the live run to `run-assert-eval` only after the user confirms the spec and cost. Let it own pipeline execution and result reporting. Capture only the suite/run identifiers from `artifacts/results///` needed for ACS and CI setup. Do not commit generated artifacts. @@ -110,7 +130,7 @@ jobs: guard-dimensions: overrefusal alpha: '0.05' min-pairs: '30' - assert-ai-version: '0.1.0' + assert-ai-version: '' # resolve once at workflow creation, then pin; see "assert-ai version pin" below extras: regression,otel,langgraph target-install: python -m pip install -e . provider-env: | @@ -143,6 +163,21 @@ assert-ai acs validate --manifest artifacts/acs//manifest.yaml --suite /report.md`, the manifest, and generated policy. Propose the code/prompt/policy fixes and ask for confirmation before applying them. After confirmation, create a branch named `assert/acs-`, apply the fixes, and open a PR. When opening the PR, state: because this branch uses `gate-mode: improvement`, the gate passes only on a statistically significant `policy_violation` gain with no `overrefusal` regression; a change that merely trends better fails. +## assert-ai version pin + +At workflow-authoring time — not at every CI run — resolve the latest stable `assert-ai` release from PyPI and write that exact version into `assert-ai-version`. Pinning is required because the paired-binary McNemar test compares a baseline artifact against later PR runs, and any change in the test-set generator, tester, or judge behavior between versions would show up as a false regression. + +Use the first working command: + +```bash +curl -fsSL https://pypi.org/pypi/assert-ai/json | python -c "import sys,json;info=json.load(sys.stdin)['info'];print(info['version'])" +# fallback: pip index versions assert-ai | head -1 | awk '{print $2}' | tr -d '()' +``` + +Substitute the resolved version for `` in the workflow above. Tell the user what version was written and that they own subsequent bumps: re-baseline after any `assert-ai-version` change so the next PR run compares like against like. + +Do not write `assert-ai-version: latest` or omit the input — the action's default is a fixed older version, and floating installs would break paired comparisons. + ## Guardrails - Do not push without user approval.