diff --git a/README.md b/README.md index a420ca9..506a236 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,9 @@ compiled 1 skill for 3 targets The identical instructions cost ~40 standing tokens on a target that lazy-loads and ~560 on one that can't — a **14× per-session tax**, charged before the skill is ever invoked, on every agent that has no lazy mode. A separate unmanifested fixture measures 19 against 885, a 47× gap. One real third-party skill from the [skills.sh](https://www.skills.sh) convention carries ~5,044 tokens of instructions, which compile to ~5,101 standing tokens on an eager target. -Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. I have not found another tool that surfaces that number. +Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. + +Counting a skill's tokens is no longer unusual — [`skills-check`](https://www.skillscheck.ai/commands/budget) does it with a real `cl100k_base` tokenizer, per section, and will fail a build over a ceiling. What is still specific to Kitbash is the *per-target* half: the same skill has a different standing cost on every agent, because each one loads it differently, and that number falls out of the compile step that already knows each target's loading mode. `~40 on a lazy target, ~560 on an eager one` is a fact about the pair, not about the file. (Kitbash's own estimate is `length / 4`, not a tokenizer — measured against `o200k_base` it runs about 2-11% high on this corpus, and the ratio it reports is within half a multiple of the real one.) Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. (Aider does not read `CONVENTIONS.md` on its own — until you add `read: CONVENTIONS.md` to `.aider.conf.yml`, it costs nothing and does nothing, and `compile` says so.) `--strict` turns budget overruns and degradation warnings into build failures. diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 089aad5..e22c020 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -11,7 +11,7 @@ Two costs matter: One caveat on the eager rows: `agentsmd` is loaded by the agent automatically, but **aider does not read `CONVENTIONS.md` on its own** — it is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and `compile` says which case a repo is in. -Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. +Token counts are estimates (`length / 4`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against `o200k_base` on this repo's corpus that estimator runs **about 2-11% high** — `prereview`'s body is 567 here and 515 in real tokens, its stub 40 against 35 — which is the safe direction for budget enforcement, since a skill fails its budget slightly early rather than late. The bias very largely cancels in the *ratio* the argument rests on: the 14x gap above is **14.7x** under a real tokenizer. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. ## `prereview` — manifested (budget 1500, lazy) diff --git a/packages/cli/scripts/benchmark.mjs b/packages/cli/scripts/benchmark.mjs index 3aed266..6a14d21 100644 --- a/packages/cli/scripts/benchmark.mjs +++ b/packages/cli/scripts/benchmark.mjs @@ -147,7 +147,7 @@ Two costs matter: One caveat on the eager rows: \`agentsmd\` is loaded by the agent automatically, but **aider does not read \`CONVENTIONS.md\` on its own** — it is loaded with \`aider --read CONVENTIONS.md\` or a \`read:\` entry in \`.aider.conf.yml\`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and \`compile\` says which case a repo is in. -Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. +Token counts are estimates (\`length / 4\`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against \`o200k_base\` on this repo's corpus that estimator runs **about 2-11% high** — \`prereview\`'s body is 567 here and 515 in real tokens, its stub 40 against 35 — which is the safe direction for budget enforcement, since a skill fails its budget slightly early rather than late. The bias very largely cancels in the *ratio* the argument rests on: the 14x gap above is **14.7x** under a real tokenizer. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. `; for (const r of rows) {