diff --git a/README.md b/README.md index 73252a4fd..622b0f7b3 100644 --- a/README.md +++ b/README.md @@ -341,7 +341,7 @@ Full catalog โ†’ [`CATALOG.md`](docs/CATALOG.md). | ๐Ÿณ **Recipes** | Bundled how-to sheets: [start a project](plugins/aidd-context/skills/12-cook/assets/recipes/start-a-project.md), [ship a feature](plugins/aidd-context/skills/12-cook/assets/recipes/ship-a-feature.md), [MCP installations](plugins/aidd-context/skills/12-cook/assets/recipes/mcp-installation.md), [token optimization](plugins/aidd-context/skills/12-cook/assets/recipes/token-optimization.md). Project recipes created by cook live in `aidd_docs/recipes/`. | | ๐Ÿ›๏ธ **[Architecture](docs/ARCHITECTURE.md)** | How the framework composes: plugins, skills, hooks, agents. | | ๐Ÿงฉ **[Create a plugin](docs/CREATE_PLUGIN.md)** | Build and publish your own. | -| ๐Ÿ›’ **[Marketplace](docs/MARKETPLACE.md)** | Install scopes, versioning, LLM tiers. | +| ๐Ÿ›’ **[Marketplace](docs/MARKETPLACE.md)** | Install scopes and versioning. | | โ“ **[FAQ & Troubleshooting](docs/FAQ.md)** ยท **[Glossary](docs/GLOSSARY.md)** | Common questions, fixes, and terms. | ## ๐Ÿ”’ Trust and safety diff --git a/docs/MARKETPLACE.md b/docs/MARKETPLACE.md index ffe728c67..9505c1c3f 100644 --- a/docs/MARKETPLACE.md +++ b/docs/MARKETPLACE.md @@ -1,6 +1,6 @@ # Marketplace, scopes & versioning -Reference for how the `aidd-framework` marketplace is registered, scoped, and versioned, plus the LLM tier mapping used by skills. +Reference for how the `aidd-framework` marketplace is registered, scoped, and versioned. ## ๐Ÿ›’ How marketplaces work @@ -38,13 +38,3 @@ Set scope at install time via the `/plugin` UI, or edit `enabledPlugins` directl - Each plugin and the root marketplace version independently via `release-please` (tags `-vX.Y.Z`, root `vX.Y.Z`). Tooling โ†’ [`deployment.md`](../aidd_docs/memory/deployment.md). - Pull updates inside Claude Code: `/plugin marketplace update aidd-framework`. - Full history โ†’ [`CHANGELOG.md`](../CHANGELOG.md). - -## ๐Ÿง  LLM tier reference - -Some skills target a model **tier** for a needed capability. The framework is authored against Claude; on another AI tool, map each tier to its nearest model. - -| Tier | Best for | Claude | Other tools (examples) | -| ---- | -------- | ------ | ---------------------- | -| **T1 Fast** | Mechanical, deterministic tasks, templates, git ops | Haiku 4.5 | GPT-5.5 mini, Gemini Flash, Grok fast | -| **T2 Balanced** | Implementation, validation, code generation | Sonnet 4.6 | GPT-5.5, Gemini Pro | -| **T3 Thinking** | Deep reasoning, synthesis, planning, onboarding | Opus 4.8 | GPT-5.5 (thinking), Gemini Pro thinking | diff --git a/plugins/aidd-context/skills/06-agent-generate/SKILL.md b/plugins/aidd-context/skills/06-agent-generate/SKILL.md index aac9ebe55..0ed301d81 100644 --- a/plugins/aidd-context/skills/06-agent-generate/SKILL.md +++ b/plugins/aidd-context/skills/06-agent-generate/SKILL.md @@ -12,7 +12,7 @@ Write one canonical agent from intent and render it per confirmed host tool, or | # | Action | Role | Input | | --- | --------------- | ---------------------------------------------- | ------------ | -| 01 | `capture-agent` | Gather the role, propose names, pick the model | user request | +| 01 | `capture-agent` | Gather the role and propose names | user request | | 02 | `write-agent` | Render the agent per tool and write | the role | | 03 | `validate` | Check each agent file | the files | diff --git a/plugins/aidd-context/skills/06-agent-generate/actions/01-capture-agent.md b/plugins/aidd-context/skills/06-agent-generate/actions/01-capture-agent.md index 08a33d17a..45010c424 100644 --- a/plugins/aidd-context/skills/06-agent-generate/actions/01-capture-agent.md +++ b/plugins/aidd-context/skills/06-agent-generate/actions/01-capture-agent.md @@ -8,12 +8,12 @@ A free-form description of the agent's purpose, tools, and instructions. ## Output -In-context: the role and its system prompt, the chosen name, the model, a quality score, and the write mode (host with confirmed tools, or plugin source). +In-context: the role and its system prompt, the chosen name, an optional user-specified model, a quality score, and the write mode (host with confirmed tools, or plugin source). ## Process 1. **Gate.** Run the asset-access precheck ([tool-paths.md](../references/tool-paths.md)). -2. **Clarify.** Ask about the purpose, tools, inputs, instructions, and preferred model. If anything stays vague, ask again before moving on. +2. **Clarify.** Ask about the purpose, tools, inputs, and instructions. If anything stays vague, ask again before moving on. 3. **Draft.** Write the canonical role ([agent-authoring.md](../references/agent-authoring.md)): a frontmatter intent + a system-prompt body. Include only the optional and orchestration sections it needs. 4. **Name.** Propose three short, catchy names that fit the purpose, and have the user pick one. 5. **Score.** Rate the agent 1-10 on clarity and completeness. If it scores under 8, revise and score again. @@ -23,5 +23,5 @@ In-context: the role and its system prompt, the chosen name, the model, a qualit ## Test -- The role, name, model, and score are stated and confirmed in writing. +- The role, name, and score are stated and confirmed in writing. A model is included only when the user specified one. - The score is at least 8 before writing. diff --git a/plugins/aidd-context/skills/06-agent-generate/actions/02-write-agent.md b/plugins/aidd-context/skills/06-agent-generate/actions/02-write-agent.md index c40746183..92c656e02 100644 --- a/plugins/aidd-context/skills/06-agent-generate/actions/02-write-agent.md +++ b/plugins/aidd-context/skills/06-agent-generate/actions/02-write-agent.md @@ -4,7 +4,7 @@ Render the canonical agent per confirmed tool and write it. ## Input -From 01: the role, the chosen name, the model, and the write mode. +From 01: the role, the chosen name, an optional user-specified model, and the write mode. ## Output diff --git a/plugins/aidd-context/skills/06-agent-generate/assets/agent-template.md b/plugins/aidd-context/skills/06-agent-generate/assets/agent-template.md index e3c19a169..6af2328ce 100644 --- a/plugins/aidd-context/skills/06-agent-generate/assets/agent-template.md +++ b/plugins/aidd-context/skills/06-agent-generate/assets/agent-template.md @@ -1,7 +1,6 @@ --- name: # required description: # required, third person -model: # optional, per tool-paths.md --- # Role diff --git a/plugins/aidd-context/skills/06-agent-generate/references/tool-paths.md b/plugins/aidd-context/skills/06-agent-generate/references/tool-paths.md index 5ca7a1250..90a1badde 100644 --- a/plugins/aidd-context/skills/06-agent-generate/references/tool-paths.md +++ b/plugins/aidd-context/skills/06-agent-generate/references/tool-paths.md @@ -16,14 +16,14 @@ Agents are supported on all five tools. ## Frontmatter per tool -The canonical agent carries `name`, `description`, `model`. Emit those a row accepts, drop the rest. Optional fields (also listed) only if the user asked. Never invent a value. +The canonical agent carries `name` and `description`. Emit those a row accepts, drop the rest. Optional fields, including `model`, only if the user asked. Never invent a value. | Tool | Accepts | | -------------- | ----------------------------------------------------------- | -| Claude Code | `name`, `description`, `model`, optional `color`, `tools` | -| Cursor | `name`, `description`, `model`, optional `readonly`, `is_background` | -| OpenCode | `name`, `description`, `model`, optional `temperature`, `permission` | -| GitHub Copilot | `name`, `description`, `model`, optional `tools` | +| Claude Code | `name`, `description`, optional `model`, `color`, `tools` | +| Cursor | `name`, `description`, optional `model`, `readonly`, `is_background` | +| OpenCode | `name`, `description`, optional `model`, `temperature`, `permission` | +| GitHub Copilot | `name`, `description`, optional `model`, `tools` | | Codex CLI | `name`, `description` (drops `model`) | ## Codex TOML conversion diff --git a/plugins/aidd-dev/agents/checker.md b/plugins/aidd-dev/agents/checker.md index 6f0752098..0122b6033 100644 --- a/plugins/aidd-dev/agents/checker.md +++ b/plugins/aidd-dev/agents/checker.md @@ -1,7 +1,6 @@ --- name: checker description: Judges finished work against its validator and the real need, leaving nothing unchecked. Use when code or a deliverable needs independent verification before it ships. Never edits the work, never implements the fix. -model: opus --- # Role diff --git a/plugins/aidd-dev/agents/executor.md b/plugins/aidd-dev/agents/executor.md index e2af667ce..10a4d6dfc 100644 --- a/plugins/aidd-dev/agents/executor.md +++ b/plugins/aidd-dev/agents/executor.md @@ -1,7 +1,6 @@ --- name: executor description: Turns a dispatched task into working, validated code that fits the project. Use when an approved scope must become code. Never plans, never judges its own work. -model: sonnet --- # Role diff --git a/plugins/aidd-dev/skills/03-assert/SKILL.md b/plugins/aidd-dev/skills/03-assert/SKILL.md index 14c689e86..2ccc5aa7d 100644 --- a/plugins/aidd-dev/skills/03-assert/SKILL.md +++ b/plugins/aidd-dev/skills/03-assert/SKILL.md @@ -2,7 +2,6 @@ name: 03-assert description: Assert the work behaves by iterating the project's coding assertions until they pass, plus optional architecture and frontend facets. Use to validate an implementation. Not for reviewing or writing tests. argument-hint: work | scope -model: sonnet --- # Skill: assert diff --git a/plugins/aidd-dev/skills/04-audit/SKILL.md b/plugins/aidd-dev/skills/04-audit/SKILL.md index 768ba5ec8..f0681a729 100644 --- a/plugins/aidd-dev/skills/04-audit/SKILL.md +++ b/plugins/aidd-dev/skills/04-audit/SKILL.md @@ -2,7 +2,6 @@ name: 04-audit description: Audit a codebase read-only across seven quality pillars into one ranked report. Use when the user wants to assess, health-check, or audit a codebase or one pillar. Not for fixing findings, reviewing a change, or checking a feature works. argument-hint: scope | pillar -model: opus --- # Skill: audit diff --git a/plugins/aidd-dev/skills/05-review/SKILL.md b/plugins/aidd-dev/skills/05-review/SKILL.md index 4da8ad889..d231e441b 100644 --- a/plugins/aidd-dev/skills/05-review/SKILL.md +++ b/plugins/aidd-dev/skills/05-review/SKILL.md @@ -2,7 +2,6 @@ name: 05-review description: Review a diff read-only on three axes, code, behavior versus the plan, and relevancy, into one verdict report. Use before shipping a change. Not for fixing findings or auditing a codebase. argument-hint: diff | plan -model: opus --- # Skill: review diff --git a/plugins/aidd-dev/skills/06-test/SKILL.md b/plugins/aidd-dev/skills/06-test/SKILL.md index 5645ef9f8..2f38dfc3b 100644 --- a/plugins/aidd-dev/skills/06-test/SKILL.md +++ b/plugins/aidd-dev/skills/06-test/SKILL.md @@ -2,7 +2,6 @@ name: 06-test description: Write and iterate developer-side tests until they pass, or validate a user journey end to end in the browser during implementation. Use when the user wants to add coverage, find what's untested, or walk a flow while building it. Do NOT use for independent acceptance QA or reviewer evidence, auditing test health, or debugging a failure. argument-hint: scope | journey -model: sonnet --- # Skill: test diff --git a/plugins/aidd-dev/skills/08-debug/SKILL.md b/plugins/aidd-dev/skills/08-debug/SKILL.md index 795597e68..4bd3e4dda 100644 --- a/plugins/aidd-dev/skills/08-debug/SKILL.md +++ b/plugins/aidd-dev/skills/08-debug/SKILL.md @@ -2,7 +2,6 @@ name: 08-debug description: Reproduce and fix a known bug, or find an unknown root cause by hypothesis validation. Use when the user wants to fix a bug, find why something breaks, or reopen a stuck investigation. Not for building a feature or reviewing a diff. argument-hint: bug | symptom -model: opus --- # Skill: debug diff --git a/scripts/__tests__/model-selection.test.js b/scripts/__tests__/model-selection.test.js new file mode 100644 index 000000000..7eaa010e6 --- /dev/null +++ b/scripts/__tests__/model-selection.test.js @@ -0,0 +1,63 @@ +const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); +const { spawnSync } = require("node:child_process"); +const test = require("node:test"); + +const ROOT = path.resolve(__dirname, "../.."); + +test("bundled skills and agents leave model selection to the host", () => { + const plugins = path.join(ROOT, "plugins"); + const files = fs.readdirSync(plugins, { recursive: true }).filter((file) => + /(?:\/SKILL\.md$|\/agents\/[^/]+\.md$|\/assets\/agent-template\.md$)/.test(file.replaceAll(path.sep, "/")) + ); + assert.ok(files.length > 0); + const overrides = files.filter((file) => { + const content = fs.readFileSync(path.join(plugins, file), "utf8"); + const frontmatter = content.match(/^---\r?\n([\s\S]*?)\r?\n---/); + return frontmatter && /^model\s*:/m.test(frontmatter[1]); + }); + assert.deepEqual(overrides, []); +}); + +for (const model of [undefined, "user-selected-model"]) { + test(`skill eval ${model ? "honors an explicit model" : "does not override the host model"}`, (t) => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "model-selection-")); + t.after(() => fs.rmSync(tmp, { recursive: true, force: true })); + const argsFile = path.join(tmp, "args.json"); + const stub = path.join(tmp, "claude.cjs"); + fs.writeFileSync(stub, [ + 'const fs = require("node:fs");', + 'fs.writeFileSync(process.env.MODEL_SELECTION_ARGS, JSON.stringify(process.argv.slice(2)));', + 'fs.writeFileSync("Makefile-shadow-report.md", "stub report");', + ].join("\n")); + const preload = path.join(tmp, "preload.cjs"); + fs.writeFileSync(preload, [ + 'const childProcess = require("node:child_process");', + 'const spawn = childProcess.spawn;', + `childProcess.spawn = (command, args, options) => command === "claude"`, + ` ? spawn(process.execPath, [${JSON.stringify(stub)}, ...args], options)`, + ' : spawn(command, args, options);', + 'require("node:module").syncBuiltinESMExports();', + ].join("\n")); + const result = spawnSync(process.execPath, [ + "--require", preload, + path.join(ROOT, "scripts/skill-eval.mjs"), + "--case=filename rule keeps a dotless name", + ...(model ? [`--model=${model}`] : []), + ], { + encoding: "utf8", + env: { ...process.env, MODEL_SELECTION_ARGS: argsFile }, + }); + assert.equal(result.status, 0, result.stderr + result.stdout); + const args = JSON.parse(fs.readFileSync(argsFile, "utf8")); + const at = args.indexOf("--model"); + if (model) { + assert.notEqual(at, -1); + assert.equal(args[at + 1], model); + } else { + assert.equal(at, -1); + } + }); +} diff --git a/scripts/skill-eval.mjs b/scripts/skill-eval.mjs index b25e5bc14..fbd545ec2 100644 --- a/scripts/skill-eval.mjs +++ b/scripts/skill-eval.mjs @@ -18,7 +18,7 @@ const CASES = JSON.parse(readFileSync(join(HERE, "skill-eval", "cases.json"), "u const args = process.argv.slice(2); const JUDGE = args.includes("--judge"); const KEEP = args.includes("--keep"); -const MODEL = (args.find((a) => a.startsWith("--model=")) || "--model=sonnet").split("=")[1]; +const MODEL = args.find((a) => a.startsWith("--model="))?.slice("--model=".length); const JOBS = Number((args.find((a) => a.startsWith("--jobs=")) || "--jobs=4").split("=")[1]); const filter = args.find((a) => !a.startsWith("--")); const REPEAT = Number((args.find((a) => a.startsWith("--repeat=")) || "--repeat=1").split("=")[1]); @@ -40,7 +40,7 @@ function runClaude(prompt, cwd) { // settings (hooks, output modes) so results are reproducible across machines. const child = spawn( "claude", - ["-p", prompt, "--model", MODEL, "--setting-sources", "project,local", "--add-dir", cwd, "--dangerously-skip-permissions"], + ["-p", prompt, ...(MODEL ? ["--model", MODEL] : []), "--setting-sources", "project,local", "--add-dir", cwd, "--dangerously-skip-permissions"], { cwd, timeout: 600000 }, ); let out = ""; @@ -176,7 +176,7 @@ async function runAll(list, jobs) { return out; } -console.log(`Running ${cases.length} case(s) on ${MODEL}, ${JOBS} at a time${JUDGE ? ", with --judge" : ""}.\n`); +console.log(`Running ${cases.length} case(s) on ${MODEL || "the host default model"}, ${JOBS} at a time${JUDGE ? ", with --judge" : ""}.\n`); const results = await runAll(cases, JOBS); const failed = results.filter(({ checks }) => checks.some((k) => !k.ok)).length; diff --git a/scripts/skill-eval/README.md b/scripts/skill-eval/README.md index a6a57d21e..5c7f7dac7 100644 --- a/scripts/skill-eval/README.md +++ b/scripts/skill-eval/README.md @@ -16,6 +16,8 @@ node scripts/skill-eval.mjs --keep # keep temp dirs to inspect Local and opt-in. Needs an authenticated `claude` CLI and spends tokens, so it is not a CI gate. +No model override is passed unless you supply `--model=`. + ## How it works Each case runs in a throwaway temp project. The skill under test is copied into