docs: name the prior art on token measurement, and state the estimator's real error - #119
Open
singhharsh1708 wants to merge 1 commit into
Open
docs: name the prior art on token measurement, and state the estimator's real error#119singhharsh1708 wants to merge 1 commit into
singhharsh1708 wants to merge 1 commit into
Conversation
…r's error The README claimed no other tool surfaces a skill's token cost. That is no longer true: skills-check (npx skills-check budget) counts tokens per skill with a real cl100k_base tokenizer, breaks them down per section, detects redundancy between skills, and fails a build over a ceiling. Leaving the claim up would be the same credibility problem the 0.11.0 pass existed to fix, on the one number this project is known for. Corrected to what is still specific here: the per-target half. The same skill has a different standing cost on every agent because each loads it differently, and that falls out of a compile step that already knows each target's loading mode — skills-check measures the file, not the pair. Also states the estimator's real error rather than "a few percent". Measured against o200k_base: length/4 runs 2-11% high on this corpus (prereview 567 vs 515, stub 40 vs 35), biased toward failing budgets early, and the 14x ratio is 14.7x under a real tokenizer.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two corrections to the claim this project is best known for. Docs only — no behaviour change, no version bump (the entry can ride the next release; merge after #118 to avoid a changelog collision).
1. "I have not found another tool that surfaces that number" is no longer true
skills-check(npx skills-check budget, open source) counts tokens per skill using a real cl100k_base tokenizer, gives a per-section breakdown, detects redundancy between skills via 4-gram Jaccard similarity, estimates cost across Haiku/Sonnet/Opus pricing, snapshots changes over time, and exits 1 over a configurable ceiling.Leaving that sentence up is exactly the credibility problem the 0.11.0 pass existed to fix, on the one number this repo is known for. Someone who has used
skills-checkreads it and stops trusting the rest of the page.What survives, and is now what the README says.
skills-checkmeasures the file. Kitbash measures the pair: the same skill has a different standing cost on every target because each one loads it differently, and that number falls out of a compile step that already encodes each target's loading mode. "~40 on a lazy target, ~560 on an eager one" is not a property of the skill — it is a property of the skill and the agent. That is the half nothing else does, and it is narrower and more defensible than the claim it replaces.Worth noting what
skills-checkdoes not do: it does not distinguish standing from activation cost, and it does no security or lint checking.2. The estimator's error is now stated, measured
The benchmark said absolute counts "will differ by a few percent against a model-specific tokenizer". I measured it rather than leaving it vague. Against
o200k_baseon this repo's corpus,length / 4runs 2-11% high:length / 4o200k_baseTwo things follow, both now in the doc: the bias is in the safe direction for budget enforcement (a skill fails its budget slightly early, never late), and it largely cancels in the ratio — the advertised 14x is 14.7x under a real tokenizer, so the headline understates rather than overstates.
That is also the answer to the obvious attack on the pitch ("your token numbers are made up"), which is worth having in writing given a competitor ships a real tokenizer.
Verification
Numbers produced by compiling
prereviewthrough the real pipeline and encoding the emitted output withjs-tiktoken'so200k_base. The corpus figure spansSKILL.md,README.md,spec/SPEC.md, anddocs/roadmap.md(+11.4%, -0.6%, +4.2%, +4.3%; +2.3% aggregate).Suite green,
site/build.mjs --checkcurrent, benchmark regenerated from its generator (the prose lives inscripts/benchmark.mjs, so the committed doc stays reproducible).