Validate ids and token counts, add required items, hygiene for 0.2.0

.github/workflows/ci.yml

@@ -0,0 +1,18 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          cache: npm
+      - run: npm ci
+      - run: npm test
+      - run: npm run build

.gitignore

@@ -1,6 +1,5 @@
 node_modules/
 dist/
 *.tsbuildinfo
-examples/
 .env
 .env.*

CHANGELOG.md

@@ -0,0 +1,44 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.0] - 2026-09-02
+
+### Added
+
+- `required?: boolean` on `Item`. When a required item (or the pair group containing it) cannot be included, `fit()` throws `[snug] Required item "id" does not fit ...` naming the item and the token shortfall. Required items are still placed by priority.
+- Validation that all items in a pair group agree on `required`.
+- `examples/demo.ts`, a short runnable demo, and a `demo` script that runs it with `tsx`.
+- `CHANGELOG.md` and a GitHub Actions CI workflow (test and build on Node 20).
+- `homepage`, `bugs`, `sideEffects: false` and `engines.node >= 18` in `package.json`.
+
+### Fixed
+
+- Duplicate item ids are now rejected with `[snug] Duplicate item id "x"`. Previously costs and inclusion were keyed by id with no uniqueness check, so two items sharing an id could both be excluded even when one of them fitted.
+- Token counts are validated. A `tokens` field or tokenizer result that is negative, `NaN`, `Infinity` or not a number now throws. Previously `tokens: -50` produced a negative `tokensUsed` and a `tokensRemaining` above the budget, and `tokens: NaN` silently never fitted.
+- `examples/` is no longer gitignored, so the `demo` script works for anyone cloning the repository.
+
+### Changed
+
+- README: the token counting section no longer claims tiktoken counts Anthropic tokens. Anthropic has its own tokenizer and a `count_tokens` endpoint; pass a count from the API via the `tokens` field or accept an approximation.
+- README documents `required` and every validation error.
+
+## [0.1.1] - 2026-04-05
+
+Initial release, published to npm as `@jeremysnr/snug`. The repository history begins at this version; 0.1.0 was an earlier publish of the same code with no separate record.
+
+### Added
+
+- `fit(items, options)`: greedy selection by descending priority within a token budget, preserving input order.
+- `pairId` for atomic pair groups (for example `tool_use` and `tool_result`), with validation that a group shares one priority.
+- `tokens` field for pre-counted costs, `reserve` option, and a character-based fallback tokenizer with a suppressible warning.
+- Validation of `budget`, `reserve` and `priority`.
+
+[Unreleased]: https://github.com/JeremySNR/snug/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/JeremySNR/snug/compare/v0.1.1...v0.2.0
+[0.1.1]: https://github.com/JeremySNR/snug/releases/tag/v0.1.1

README.md

@@ -7,7 +7,7 @@
 
 **Fit prioritised content into a token budget.**
 
-Every LLM application has the same problem: you have a context window of N tokens and need to fit a system prompt, conversation history, retrieved documents, and tool definitions into it — with space left for the model's reply. Every team writes their own solution from scratch.
+Every LLM application has the same problem: you have a context window of N tokens and need to fit a system prompt, conversation history, retrieved documents, and tool definitions into it, with space left for the model's reply. Every team writes their own solution from scratch.
 
 `snug` is a single function that solves this once.
 
@@ -16,15 +16,15 @@ import { fit } from '@jeremysnr/snug';
 
 const { included } = fit(
   [
-    { id: 'system',  content: systemPrompt,  priority: 100 },
+    { id: 'system',  content: systemPrompt,  priority: 100, required: true },
     { id: 'history', content: chatHistory,   priority:  60 },
     { id: 'rag',     content: retrievedDocs, priority:  40 },
   ],
   { budget: 8192, reserve: 1024, tokenizer: myTokenizer },
 );
 
-// included — items that fit, in original input order
-// excluded — items that didn't fit
+// included: items that fit, in original input order
+// excluded: items that didn't fit
 ```
 
 Items are selected greedily in descending priority order. The result preserves original input order. Zero dependencies. Works in Node, Deno, Bun, and edge runtimes.
@@ -47,17 +47,18 @@ fit(items: Item[], options: FitOptions): FitResult
 
 | Field | Type | Description |
 |-------|------|-------------|
-| `id` | `string` | Unique identifier |
-| `content` | `unknown` | Your content — not inspected by snug |
+| `id` | `string` | Unique identifier. Duplicates throw. |
+| `content` | `unknown` | Your content, not inspected by snug |
 | `priority` | `number` | Higher = included first |
-| `tokens` | `number` | Pre-counted cost (optional — see below) |
-| `pairId` | `string` | Atomic pair group (optional — see below) |
+| `tokens` | `number` | Pre-counted cost (optional, see below) |
+| `pairId` | `string` | Atomic pair group (optional, see below) |
+| `required` | `boolean` | Throw instead of excluding if it does not fit (optional, see below) |
 
 **FitOptions**
 
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `budget` | `number` | — | Token limit for included items |
+| `budget` | `number` | (required) | Token limit for included items |
 | `tokenizer` | `(text: string) => number` | built-in approx | Your token counter |
 | `reserve` | `number` | `0` | Tokens to hold back (e.g. for model response) |
 | `suppressApproximationWarning` | `boolean` | `false` | Silence the no-tokenizer warning |
@@ -75,7 +76,7 @@ fit(items: Item[], options: FitOptions): FitResult
 
 ## Pair constraints
 
-Anthropic's API requires strict 1:1 pairing between `tool_use` and `tool_result` messages — orphaning either half causes a 400 error. Mark paired items with a shared `pairId` and snug treats them as an atomic unit: both are included or neither is.
+Anthropic's API requires strict 1:1 pairing between `tool_use` and `tool_result` messages. Orphaning either half causes a 400 error. Mark paired items with a shared `pairId` and snug treats them as an atomic unit: both are included or neither is.
 
 ```ts
 fit(
@@ -87,33 +88,90 @@ fit(
 );
 ```
 
-All items in a pair group must share the same `priority`.
+All items in a pair group must share the same `priority` and the same `required` value.
+
+## Required items
+
+Some content must always be sent: a system prompt, the user's latest message, a tool result the model is waiting on. Mark it `required: true` and snug throws if it cannot be included, rather than silently dropping it and letting the request go out incomplete.
+
+```ts
+fit(
+  [
+    { id: 'system', content: systemPrompt, priority: 100, required: true },
+    { id: 'latest', content: latestTurn,   priority:  90, required: true },
+    { id: 'rag',    content: docs,         priority:  40 },
+  ],
+  { budget: 4096, reserve: 512, tokenizer },
+);
+// Error: [snug] Required item "latest" does not fit: it needs 700 tokens but only
+// 120 remain after higher-priority items (short by 580). Budget 4096, reserve 512.
+```
+
+Required items are still placed by priority. The flag does not promote an item above higher-priority optional items; it only changes what happens when the item does not fit. Give required items the highest priorities if they must be placed before optional content.
+
+If any item in a pair group is required, all items in that group must be required, and the error names the group.
+
+## Validation errors
+
+`fit()` throws an `Error` whose message starts with `[snug]` when the input cannot be trusted. Catch these during development; they indicate a bug in the calling code rather than a tight budget.
+
+| Condition | Message |
+|-----------|---------|
+| Two items share an `id` | `Duplicate item id "x". Item ids must be unique.` |
+| `tokens` is negative, `NaN`, `Infinity`, or not a number | `Item "x" has an invalid \`tokens\` value: -50. Token counts must be finite numbers >= 0.` |
+| The tokenizer returns a negative, `NaN`, `Infinity`, or non-number value | `Item "x" received an invalid token count from the tokenizer: NaN. ...` |
+| `priority` is not finite | `Item "x" has a non-finite priority: Infinity` |
+| Items in a pair group have different priorities | `All items in pair group "p" must have the same priority. Found 90 and 50.` |
+| Items in a pair group disagree on `required` | `All items in pair group "p" must agree on \`required\`. Found true and false.` |
+| A required item or pair group does not fit | `Required item "x" does not fit: it needs N tokens but only M remain after higher-priority items (short by S). Budget B, reserve R.` |
+| `content` is not a string and `tokens` is missing | `Item "x" has no \`tokens\` field and its \`content\` is not a string.` |
+| `budget` is not a positive finite number | `budget must be a positive finite number. Got: 0` |
+| `reserve` is negative or not less than `budget` | `reserve (100) must be less than budget (100).` |
 
 ## Token counting
 
-Pass any `(text: string) => number` function:
+Pass any `(text: string) => number` function.
+
+**OpenAI models** use tiktoken. Pass the model name so tiktoken picks the right encoding (`gpt-4o` and newer use `o200k_base`; `gpt-4` and `gpt-3.5-turbo` use `cl100k_base`):
 
 ```ts
-// tiktoken (OpenAI / Anthropic)
 import { encoding_for_model } from 'tiktoken';
 const enc = encoding_for_model('gpt-4o');
 const tokenizer = (text: string) => enc.encode(text).length;
 ```
 
-If you already have a token count (e.g. from an API usage response), pass it directly via the `tokens` field and skip counting entirely:
+**Anthropic models** do not use tiktoken. Claude has its own tokenizer, which Anthropic does not publish as a library, and a tiktoken count will be off by a variable margin. For an exact count call the [count_tokens endpoint](https://docs.anthropic.com/en/api/messages-count-tokens) (`client.messages.countTokens(...)` in the SDK) and pass the result through the `tokens` field so snug never needs to count:
+
+```ts
+const { input_tokens } = await client.messages.countTokens({ model, messages: [msg] });
+{ id: 'msg', content: msg, priority: 50, tokens: input_tokens }
+```
+
+If a network round trip per item is too expensive, use a tiktoken or character-based count as an approximation and keep a healthy `reserve` to absorb the error.
+
+If you already have a token count from any source (for example an API usage response), pass it via the `tokens` field and skip counting entirely:
 
 ```ts
 { id: 'msg', content: msg, priority: 50, tokens: 342 }
 ```
 
 When no tokenizer is supplied, snug falls back to `Math.ceil(text.length / 4)` and prints a warning. This is useful for prototyping but can be off by up to 37% in production.
 
+## Demo
+
+```
+npm install
+npm run demo
+```
+
+Runs [`examples/demo.ts`](./examples/demo.ts) with `tsx`.
+
 ## Ecosystem
 
 | Package | What it does |
 |---------|-------------|
-| `@jeremysnr/snug` | Zero-dependency core — bring your own tokenizer |
-| [`@jeremysnr/snug-tiktoken`](https://github.com/JeremySNR/snug-tiktoken) | Pre-wired with tiktoken, model-agnostic |
+| `@jeremysnr/snug` | Zero-dependency core, bring your own tokenizer |
+| [`@jeremysnr/snug-tiktoken`](https://github.com/JeremySNR/snug-tiktoken) | Pre-wired with tiktoken for OpenAI encodings |
 | [`@jeremysnr/snug-openai`](https://github.com/JeremySNR/snug-openai) | Accepts OpenAI SDK message arrays directly |
 | [`@jeremysnr/snug-anthropic`](https://github.com/JeremySNR/snug-anthropic) | Accepts Anthropic SDK message arrays, auto-pairs tool messages |
 

examples/demo.ts

@@ -0,0 +1,69 @@
+/**
+ * Runnable demo for snug. Run with `npm run demo` (uses tsx).
+ *
+ * Uses a deliberately crude tokenizer (one token per word) so the numbers are
+ * easy to follow. In real code, pass a tiktoken encoder or a count from your
+ * provider's API via the `tokens` field.
+ */
+import { fit } from '../src/index.js';
+import type { Item } from '../src/index.js';
+
+const wordTokenizer = (text: string): number => text.trim().split(/\s+/).filter(Boolean).length;
+
+const items: Item[] = [
+  {
+    id: 'system',
+    content: 'You are a concise travel assistant. Answer in British English.',
+    priority: 100,
+    required: true,
+  },
+  {
+    id: 'latest-user',
+    content: 'Which of the three hotels you mentioned has the best pool?',
+    priority: 90,
+    required: true,
+  },
+  {
+    id: 'tool-use',
+    content: 'search_hotels({ city: "Lisbon", stars: 4 })',
+    priority: 70,
+    pairId: 'call-1',
+  },
+  {
+    id: 'tool-result',
+    content: 'Found: Hotel Avenida (pool, rooftop), Casa do Bairro (no pool), Tejo Suites (indoor pool).',
+    priority: 70,
+    pairId: 'call-1',
+  },
+  {
+    id: 'old-history',
+    content:
+      'User: Hi, I am planning a long weekend in Lisbon in October with my partner. ' +
+      'Assistant: Lovely choice. October is warm and quieter than the summer months.',
+    priority: 40,
+  },
+  {
+    id: 'rag-doc',
+    content:
+      'Lisbon guide: The city has seven hills, a historic tram network, and excellent seafood. ' +
+      'Neighbourhoods include Alfama, Baixa, Chiado, Bairro Alto and Belem.',
+    priority: 30,
+  },
+];
+
+function show(label: string, budget: number, reserve: number): void {
+  console.log(`\n== ${label}: budget ${budget}, reserve ${reserve}`);
+  try {
+    const result = fit(items, { budget, reserve, tokenizer: wordTokenizer });
+    console.log(`  included : ${result.included.map(i => i.id).join(', ')}`);
+    console.log(`  excluded : ${result.excluded.map(i => i.id).join(', ') || '(none)'}`);
+    console.log(`  used ${result.tokensUsed}, remaining ${result.tokensRemaining}`);
+  } catch (err) {
+    console.log(`  threw: ${(err as Error).message}`);
+  }
+}
+
+show('Everything fits', 200, 20);
+show('Tight: low-priority items drop, tool pair stays together', 60, 10);
+show('Very tight: the tool pair is dropped as a unit', 40, 5);
+show('Too tight for a required item', 15, 5);