Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@ compiled 1 skill for 3 targets

The identical instructions cost ~40 standing tokens on a target that lazy-loads and ~560 on one that can't — a **14× per-session tax**, charged before the skill is ever invoked, on every agent that has no lazy mode. A separate unmanifested fixture measures 19 against 885, a 47× gap. One real third-party skill from the [skills.sh](https://www.skills.sh) convention carries ~5,044 tokens of instructions, which compile to ~5,101 standing tokens on an eager target.

Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. I have not found another tool that surfaces that number.
Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you.

Counting a skill's tokens is no longer unusual — [`skills-check`](https://www.skillscheck.ai/commands/budget) does it with a real `cl100k_base` tokenizer, per section, and will fail a build over a ceiling. What is still specific to Kitbash is the *per-target* half: the same skill has a different standing cost on every agent, because each one loads it differently, and that number falls out of the compile step that already knows each target's loading mode. `~40 on a lazy target, ~560 on an eager one` is a fact about the pair, not about the file. (Kitbash's own estimate is `length / 4`, not a tokenizer — measured against `o200k_base` it runs about 2-11% high on this corpus, and the ratio it reports is within half a multiple of the real one.)

Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. (Aider does not read `CONVENTIONS.md` on its own — until you add `read: CONVENTIONS.md` to `.aider.conf.yml`, it costs nothing and does nothing, and `compile` says so.) `--strict` turns budget overruns and degradation warnings into build failures.

Expand Down
2 changes: 1 addition & 1 deletion docs/benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Two costs matter:

One caveat on the eager rows: `agentsmd` is loaded by the agent automatically, but **aider does not read `CONVENTIONS.md` on its own** — it is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and `compile` says which case a repo is in.

Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`.
Token counts are estimates (`length / 4`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against `o200k_base` on this repo's corpus that estimator runs **about 2-11% high** — `prereview`'s body is 567 here and 515 in real tokens, its stub 40 against 35 — which is the safe direction for budget enforcement, since a skill fails its budget slightly early rather than late. The bias very largely cancels in the *ratio* the argument rests on: the 14x gap above is **14.7x** under a real tokenizer. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`.

## `prereview` — manifested (budget 1500, lazy)

Expand Down
2 changes: 1 addition & 1 deletion packages/cli/scripts/benchmark.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ Two costs matter:

One caveat on the eager rows: \`agentsmd\` is loaded by the agent automatically, but **aider does not read \`CONVENTIONS.md\` on its own** — it is loaded with \`aider --read CONVENTIONS.md\` or a \`read:\` entry in \`.aider.conf.yml\`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and \`compile\` says which case a repo is in.

Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`.
Token counts are estimates (\`length / 4\`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against \`o200k_base\` on this repo's corpus that estimator runs **about 2-11% high** — \`prereview\`'s body is 567 here and 515 in real tokens, its stub 40 against 35 — which is the safe direction for budget enforcement, since a skill fails its budget slightly early rather than late. The bias very largely cancels in the *ratio* the argument rests on: the 14x gap above is **14.7x** under a real tokenizer. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`.

`;
for (const r of rows) {
Expand Down
Loading