From ac1166e03111e6f51fc283e3bdc9a512abc8b427 Mon Sep 17 00:00:00 2001 From: Harsh Singh Date: Sun, 16 Aug 2026 01:04:05 +0530 Subject: [PATCH] fix: verify every claim the compiler makes about its targets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A pass over all eleven adapters, checking each claimed path and loading mode against that client's own source or documentation. All eleven loading modes are correct — the 14x-47x standing-cost benchmark stands. Three claims around them did not survive. Aider does not read CONVENTIONS.md on its own. Its docs are explicit that the file is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in .aider.conf.yml; there is no auto-discovery. Aider is one of only two eager targets, so reporting a standing cost beside a file that is never in context charged a cost nobody was paying. compile now distinguishes the three cases — unconfigured, configured but not reading, wired up — and the benchmark and README carry the same caveat. The numbers are unchanged; what they mean is not. The README's status line had drifted eight releases (still v0.15.0 at v0.23.0) and had omitted `import` since 0.16.0, because nothing checked either. The version is now stamped by site/build.mjs from package.json, so CI's --check gate catches it and the drift class is gone rather than patched. Three tests hold the paragraph to the real command surface; each was verified to fail on the exact drift it guards. The benchmark crashed after a docs edit whose backticks closed its template literal, and the failure hid behind `git diff --exit-code`, which reads "the script never ran" as "output unchanged" — the second time that pattern produced a false green. Fixed, and the gate checks exit codes explicitly. Three findings are now recorded in the adapters instead of assumed: .windsurf/rules/ is the legacy Devin path (.devin/rules/ supersedes it and wins where both exist) and is still written deliberately, since it is the form that works on current Devin and on unmigrated Windsurf; Claude Code merged custom commands into skills, so the .claude/commands/ shim is inert on current versions and kept only for older ones; and trigger: model_decision is confirmed lazy, which makes a rule with an empty description inert. A test pins the invariant that an emitted skill's frontmatter name equals its directory name — Cline drops a mismatch with a silent return null and Zed rejects it outright, so a wrong name is invisible skill loss. The invariant held; nothing enforced it. Adds 12 tests. 0.24.0. --- CHANGELOG.md | 13 ++++++ README.md | 4 +- docs/benchmarks/README.md | 2 + packages/cli/package.json | 2 +- packages/cli/scripts/benchmark.mjs | 2 + packages/cli/scripts/test.mjs | 72 ++++++++++++++++++++++++++++++ packages/cli/src/adapters.ts | 54 ++++++++++++++++++++-- site/build.mjs | 14 ++++++ site/changelog.html | 16 ++++++- site/index.html | 2 +- 10 files changed, 172 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d67cf3..f30e80c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ Format: [Keep a Changelog](https://keepachangelog.com). Versioning: semver — for skills *and* for this CLI, breaking prompt changes are breaking changes. +## [0.24.0] — 2026-08-12 + +A verification pass over every claim the compiler makes about the agents it targets, checked against each client's own source or documentation. **All eleven loading modes are correct** — the 14×–47× standing-cost benchmark stands — but three claims around them were overstated, and one of them was in the published numbers. + +### Fixed +- **Aider does not read `CONVENTIONS.md` on its own.** Its docs are explicit: the file is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Aider is one of only two eager targets, so reporting a standing cost beside a file that is never in context charged a cost nobody was paying. `compile` now says which of the two situations a repo is in — unconfigured, configured-but-not-reading, or wired up (silent) — and the benchmark and README carry the same caveat. The measured numbers are unchanged; what they mean is now stated correctly. +- **The README's status line had drifted eight releases** (it still said v0.15.0) and had omitted `import` from the working-command list since 0.16.0, because nothing checked either. Both corrected. The version is now stamped into the README by `site/build.mjs` from the same `package.json` everything else reads, so CI's existing `--check` gate catches it — the drift is eliminated rather than re-fixed. Three tests additionally hold the status paragraph to the real command surface, verified to fail on precisely the two drifts that occurred. +- The benchmark script crashed after a docs edit whose backticks closed its template literal, and the failure hid behind a `git diff --exit-code` check that reads "script never ran" as "output unchanged". Fixed, and the release gate now checks exit codes explicitly. + +### Changed +- Recorded, in the adapters themselves, three findings that were previously assumed: `.windsurf/rules/` is the **legacy** Devin path (`.devin/rules/` supersedes it and wins where both exist) and is still written deliberately, because it is the form that works on current Devin *and* on unmigrated Windsurf; Claude Code has **merged custom commands into skills**, so the `.claude/commands/` shim is inert on current versions and kept only for older ones; and `trigger: model_decision` is confirmed lazy, which makes a rule with an empty description inert. +- A test now pins the invariant that an emitted skill's frontmatter `name` equals its directory name. Cline drops a mismatch with a silent `return null` and Zed rejects it outright, so getting this wrong is invisible skill loss rather than an error. The invariant already held; nothing enforced it. + ## [0.23.0] — 2026-08-12 ### Added diff --git a/README.md b/README.md index e848e8c..d239a85 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ The identical instructions cost ~40 standing tokens on a target that lazy-loads Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. I have not found another tool that surfaces that number. -Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. `--strict` turns budget overruns and degradation warnings into build failures. +Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. (Aider does not read `CONVENTIONS.md` on its own — until you add `read: CONVENTIONS.md` to `.aider.conf.yml`, it costs nothing and does nothing, and `compile` says so.) `--strict` turns budget overruns and degradation warnings into build failures. ### Why not a sync script? @@ -61,7 +61,7 @@ Already have skills? A plain SKILL.md folder — the skills.sh / Claude Skills c Already carrying a hand-written `CLAUDE.md`, `.cursor/rules/`, `AGENTS.md`, and the rest of the copy-per-agent set? `kitbash import` reads them back into a single skill, measures what each one costs in standing context, and reports where the copies have drifted apart — so `kitbash compile` can regenerate them all from that one source. It touches nothing on disk until you remove the originals yourself. -**Status.** v0.15.0, on npm and Homebrew, zero runtime dependencies, Node 20+. The KSF core is frozen and additive-only within the major version ([RFC 0002](rfcs/0002-ksf-1.0-stabilization.md)). Everything around it is early and labeled as such: `init`, `install`, `remove`, `list`, `compile`, `doctor`, `update`, `diff`, `lint`, `preview`, `explain`, and `test` work today; `audit`, `gate`, `search`, `publish`, `lore`, and `run` exit `7` and are on the [roadmap](docs/roadmap.md). One first-party skill ships (`prereview`); six more are specified but not built. Adoption is single-digit stars — if the measurement above is what you want, you are early. +**Status.** v0.24.0, on npm and Homebrew, zero runtime dependencies, Node 20+. The KSF core is frozen and additive-only within the major version ([RFC 0002](rfcs/0002-ksf-1.0-stabilization.md)). Everything around it is early and labeled as such: `init`, `import`, `install`, `remove`, `list`, `compile`, `doctor`, `update`, `diff`, `lint`, `preview`, `explain`, and `test` work today; `audit`, `gate`, `search`, `publish`, `lore`, and `run` exit `7` and are on the [roadmap](docs/roadmap.md). One first-party skill ships (`prereview`); six more are specified but not built. Adoption is single-digit stars — if the measurement above is what you want, you are early.

npm version diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 5bd8dc9..089aad5 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -9,6 +9,8 @@ Two costs matter: - **Loaded** — tokens the agent reads when the skill is in play. - **Standing** — tokens sitting in the context window *every session, before the skill is even invoked*. Lazy targets keep only a stub; eager targets carry the whole body. +One caveat on the eager rows: `agentsmd` is loaded by the agent automatically, but **aider does not read `CONVENTIONS.md` on its own** — it is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and `compile` says which case a repo is in. + Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. ## `prereview` — manifested (budget 1500, lazy) diff --git a/packages/cli/package.json b/packages/cli/package.json index 46e7e4b..4e4f837 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,6 +1,6 @@ { "name": "kitbash", - "version": "0.23.0", + "version": "0.24.0", "description": "The package manager and compiler for AI agent skills — write once, run in every coding agent", "license": "Apache-2.0", "author": "Harsh Singh", diff --git a/packages/cli/scripts/benchmark.mjs b/packages/cli/scripts/benchmark.mjs index 291454d..3aed266 100644 --- a/packages/cli/scripts/benchmark.mjs +++ b/packages/cli/scripts/benchmark.mjs @@ -145,6 +145,8 @@ Two costs matter: - **Loaded** — tokens the agent reads when the skill is in play. - **Standing** — tokens sitting in the context window *every session, before the skill is even invoked*. Lazy targets keep only a stub; eager targets carry the whole body. +One caveat on the eager rows: \`agentsmd\` is loaded by the agent automatically, but **aider does not read \`CONVENTIONS.md\` on its own** — it is loaded with \`aider --read CONVENTIONS.md\` or a \`read:\` entry in \`.aider.conf.yml\`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and \`compile\` says which case a repo is in. + Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. `; diff --git a/packages/cli/scripts/test.mjs b/packages/cli/scripts/test.mjs index fd86c47..0083046 100644 --- a/packages/cli/scripts/test.mjs +++ b/packages/cli/scripts/test.mjs @@ -1849,6 +1849,78 @@ try { rmSync(curOnly, { recursive: true, force: true }); } +// ── the README's own claims stay true ──────────────────────────────────────── +// The status line drifted eight releases (it still said v0.15.0 at v0.23.0) and +// the command list omitted `import` for seven, because nothing checked either. +{ + const readme = readFileSync(join(repoRoot, "README.md"), "utf8"); + const pkgVersion = JSON.parse(readFileSync(join(here, "../package.json"), "utf8")).version; + const claimed = readme.match(/\*\*Status\.\*\* v(\d+\.\d+\.\d+)/)?.[1]; + check("readme: the status line names the current version", claimed === pkgVersion, `README says v${claimed}, package.json says v${pkgVersion}`); + + // Every command the CLI actually ships must be listed as working, and nothing + // that exits 7 may be — that is the promise the status paragraph makes. + const help = run(["--help"], repoRoot).out; + const planned = help.includes("Planned (not yet implemented):") ? help.split("Planned (not yet implemented):")[1] : ""; + const working = (help.split("Planned (not yet implemented):")[0] || "") + .split("\n") + .map((l) => l.match(/^ {2}(\S+) {2,}/)?.[1]) + .filter(Boolean); + const statusPara = readme.split("**Status.**")[1]?.split("\n\n")[0] ?? ""; + const missing = working.filter((c) => !statusPara.includes(`\`${c}\``)); + check("readme: every working command is listed in the status line", missing.length === 0, `missing: ${missing.join(", ")}`); + const plannedNames = planned.split("\n").map((l) => l.match(/^ {2}(\S+) {2,}/)?.[1]).filter(Boolean); + const misfiled = plannedNames.filter((c) => new RegExp(`\`${c}\`[^.]*work today`).test(statusPara)); + check("readme: no planned command is claimed as working", misfiled.length === 0, `misfiled: ${misfiled.join(", ")}`); +} + +// ── aider only pays the cost it is charged once it is wired up ─────────────── +// Aider does not auto-read CONVENTIONS.md; it needs `aider --read` or a `read:` +// entry in .aider.conf.yml. Reporting an eager standing cost without saying so +// would charge a file that is never in context. +const aidTmp = mkdtempSync(join(tmpdir(), "kitbash-aider-")); +try { + writeFileSync(join(aidTmp, "CONVENTIONS.md"), "# House rules\n"); + run(["init"], aidTmp); + run(["install", `file:${fixture}`, "--yes"], aidTmp); + + const bare = run(["compile"], aidTmp); + check("aider: unconfigured, compile says the file is not read", bare.out.includes("does not read it automatically"), bare.out); + + writeFileSync(join(aidTmp, ".aider.conf.yml"), "model: gpt-4\n"); + const unwired = run(["compile"], aidTmp); + check("aider: a conf without read: is still flagged", unwired.out.includes("does not read it"), unwired.out); + + writeFileSync(join(aidTmp, ".aider.conf.yml"), "model: gpt-4\nread: CONVENTIONS.md\n"); + const wired = run(["compile"], aidTmp); + check("aider: once wired up the note goes away", !wired.out.includes("does not read it"), wired.out); + check("aider: the standing cost is still reported when wired", wired.out.includes("aider is eager and cannot lazy-load"), wired.out); + check("aider: none of this fails --strict", run(["compile", "--strict"], aidTmp).status === 0); +} finally { + rmSync(aidTmp, { recursive: true, force: true }); +} + +// The emitted skill's frontmatter name must equal its directory name: Cline drops +// a mismatch with a silent `return null` and Zed rejects it outright, so a wrong +// name here is invisible skill loss rather than an error. +const nameTmp = mkdtempSync(join(tmpdir(), "kitbash-name-")); +try { + const odd = join(nameTmp, "odd"); + mkdirSync(odd, { recursive: true }); + // Frontmatter name that KSF must NOT propagate: wrong charset, differs from dir. + writeFileSync(join(odd, "SKILL.md"), "---\nname: Totally-Different_Name\ndescription: A bare skill whose frontmatter name differs from its folder\n---\n\nBody.\n"); + mkdirSync(join(nameTmp, ".agents"), { recursive: true }); + run(["init"], nameTmp); + run(["install", `file:${odd}`, "--yes"], nameTmp); + run(["compile"], nameTmp); + const emitted = join(nameTmp, ".agents/skills/odd/SKILL.md"); + check("name-invariant: the skill dir is named from the manifest", existsSync(emitted)); + const fm = readFileSync(emitted, "utf8"); + check("name-invariant: frontmatter name equals the directory name", /^---\nname: odd\n/.test(fm), fm.slice(0, 80)); +} finally { + rmSync(nameTmp, { recursive: true, force: true }); +} + if (failures) { console.error(`\n${failures} test(s) failed`); process.exit(1); diff --git a/packages/cli/src/adapters.ts b/packages/cli/src/adapters.ts index baa9395..5a84795 100644 --- a/packages/cli/src/adapters.ts +++ b/packages/cli/src/adapters.ts @@ -87,22 +87,48 @@ function eagerStandingNote(skill: LoadedSkill, adapter: Adapter, files: Compiled } /** Shared-file adapters (AGENTS.md, GEMINI.md): eager-loaded, marker-merged. */ -function mergedFileAdapter(id: string, file: string, detect: (root: string) => boolean): Adapter { +function mergedFileAdapter(id: string, file: string, detect: (root: string) => boolean, extraNote?: (root: string) => string | null): Adapter { return { id, capabilities: [], loading: "eager", detect, - emit(skill, body) { + emit(skill, body, root) { const { name } = skill.manifest.skill; const { begin, end } = markers(name); const section = `${begin}\n${header(skill)}\n\n## Skill: ${name}\n\n${body.trim()}\n${end}`; const files = [{ path: file, content: section, merge: true }]; - return { files, warnings: degradationWarnings(skill, this), notes: eagerStandingNote(skill, this, files) }; + const extra = extraNote?.(root); + return { + files, + warnings: degradationWarnings(skill, this), + notes: [...eagerStandingNote(skill, this, files), ...(extra ? [extra] : [])], + }; }, }; } +/** + * Aider does not read `CONVENTIONS.md` on its own. Its docs are explicit that the + * file is loaded with `aider --read CONVENTIONS.md`, or by a `read:` entry in + * `.aider.conf.yml` — there is no auto-discovery. That matters more here than it + * would elsewhere: this adapter is one of only two eager targets, and the standing + * cost reported beside it is real only once the file is actually wired in. + * Unconfigured, `CONVENTIONS.md` costs nothing and does nothing, so the compiler + * says which of the two situations it is looking at rather than quietly implying + * the cost is already being paid. + */ +function aiderWiringNote(root: string): string | null { + const conf = [".aider.conf.yml", ".aider.conf.yaml"].map((f) => join(root, f)).find((p) => existsSync(p)); + if (conf) { + const text = readFileIfExists(root, conf.slice(root.length + 1)); + // A `read:` entry naming the file (alone or in a list) is enough to load it. + if (/^\s*read:/m.test(text) && text.includes("CONVENTIONS.md")) return null; + return `aider: CONVENTIONS.md is written, but ${conf.slice(root.length + 1)} does not read it — add "read: CONVENTIONS.md" there, or the file is never in context and the standing cost above is not actually charged.`; + } + return `aider: CONVENTIONS.md is written, but aider does not read it automatically — run "aider --read CONVENTIONS.md", or add "read: CONVENTIONS.md" to .aider.conf.yml. Until then the file costs nothing and does nothing.`; +} + /** Simple per-skill file adapters that differ only in path, loading mode, and frontmatter. */ function fileAdapter( id: string, @@ -136,6 +162,13 @@ const claudeCode: Adapter = { // only alongside the emit code that produces its primitive. capabilities: [], loading: "lazy", + // Verified lazy: Claude Code keeps each skill's description in context and + // loads the body only on invoke. Note that custom commands have since been + // merged into skills — a skill at .claude/skills//SKILL.md already creates + // /, and when a .claude/commands/.md exists alongside it the skill wins + // the name. The command file is therefore inert on current Claude Code and is + // still written only because it remains the trigger on versions from before + // that merge; it costs nothing standing, since commands load on use. detect: (root) => existsSync(join(root, ".claude")), emit(skill, body) { const { name, description } = skill.manifest.skill; @@ -389,6 +422,20 @@ function clineConstraints(skill: LoadedSkill): string[] { * frontmatter — `model_decision` means the description is what sits in context * and the body loads only when the model judges it relevant, i.e. lazy. */ +/** + * Windsurf, now Devin Desktop. `.windsurf/rules/` is the LEGACY path: Devin's docs + * moved to `.devin/rules/`, and where both exist the `.devin/` copy takes + * precedence. The legacy path is still read by both Devin Desktop and the Devin + * CLI, which is why it is what gets written — a file at `.windsurf/rules/` works + * on current Devin *and* on older Windsurf, while `.devin/rules/` would silently + * do nothing for anyone who has not migrated. Revisit when the legacy fallback is + * announced for removal. + * + * `trigger: model_decision` is verified lazy: Devin's docs say only the + * `description` reaches the system prompt and the body is read when the model + * decides it is relevant. A rule with an empty description is therefore inert — + * nothing exists for the model to match on. + */ const windsurf = fileAdapter( "windsurf", [], @@ -417,6 +464,7 @@ const aider = mergedFileAdapter( "aider", "CONVENTIONS.md", (root) => existsSync(join(root, "CONVENTIONS.md")) || existsSync(join(root, ".aider.conf.yml")), + aiderWiringNote, ); /** The floor: everything that reads AGENTS.md (Codex and many others). */ diff --git a/site/build.mjs b/site/build.mjs index df19650..dbd2506 100644 --- a/site/build.mjs +++ b/site/build.mjs @@ -110,6 +110,20 @@ for (const e of readdirSync(site, { recursive: true, withFileTypes: false })) { else writeFileSync(p, out); } +// ---- stamp the README's status line ---- +// It drifted eight releases before anything checked it, which is what a +// hand-maintained version string in prose does. Stamped from the same source as +// everything else so it cannot disagree with what is published. +{ + const p = join(repoRoot, "README.md"); + const src = readFileSync(p, "utf8"); + const out = src.replace(/(\*\*Status\.\*\* )v\d+\.\d+\.\d+/, `$1v${version}`); + if (out !== src) { + if (checkOnly) stale.push(`README.md (status line is not v${version})`); + else writeFileSync(p, out); + } +} + if (checkOnly && stale.length) { console.error("site is stale — run `node site/build.mjs` and commit the result:"); for (const s of stale) console.error(` ${s}`); diff --git a/site/changelog.html b/site/changelog.html index dc504c5..ab8a636 100644 --- a/site/changelog.html +++ b/site/changelog.html @@ -91,7 +91,7 @@

Changelog

Releases follow Keep a Changelog and semver — for skills and for this CLI, breaking prompt changes are breaking changes. The CLI is published to npm as kitbash and to Homebrew via singhharsh1708/tap. Tagged builds are on the GitHub releases page.

-
v0.23.0Current CLI version
+
v0.24.0Current CLI version
8Compile targets
Apache-2.0License
@@ -105,10 +105,22 @@

Changelog

Confirm with kitbash --version, which reads the installed package.json. Install and uninstall routes are covered on the installation page.

+
+
+

v0.24.0

+ 2026-08-12latest +
+

A verification pass over every claim the compiler makes about the agents it targets, checked against each client's own source or documentation. All eleven loading modes are correct — the 14×–47× standing-cost benchmark stands — but three claims around them were overstated, and one of them was in the published numbers.

+

Fixed

+ +

Changed

+ +
+

v0.23.0

- 2026-08-12latest + 2026-08-12

Added

diff --git a/site/index.html b/site/index.html index c04680e..cd8c0d5 100644 --- a/site/index.html +++ b/site/index.html @@ -151,7 +151,7 @@ -

Open format for AI agent skills · v0.23.0 · stable spec (RFC 0002)

+

Open format for AI agent skills · v0.24.0 · stable spec (RFC 0002)

Write an agent skill once. Run it everywhere.

Get started