Ship repository setup doctor and bounded benchmark corpus (0.3.1)

RELEASE_EVIDENCE.md

@@ -4,6 +4,30 @@ Latest implementation and adoption results are in the
 [dogfood execution ledger](docs/DOGFOOD-EXECUTION.md). The dated snapshots below
 remain historical evidence; internal pilot acceptance is separate from G0–G4.
 
+## 0.3.1 shipment, 22 September 2026
+
+This patch adds an installed, read-only `doctor` command that verifies the exact
+Git checkout, all four repository tools and one source line's generation/hash
+through a fresh stdio connection. It reports client acceptance as `not-tested`;
+an SDK probe does not establish Claude/Codex task acceptance.
+
+The raw-evidence benchmark now uses versioned v2 synthetic collections, retaining
+every corpus chunk within the existing 128-record and 32-collection bounds.
+Each question queries every collection and counts every response. Per-call
+budgets and gate thresholds are unchanged; the total per-question budget scales
+with the collection count. See [the methodology](benchmarks/README.md).
+
+The version-bumped 0.3.1 local checks passed 386 tests, independent installed-package smoke,
+both benchmark gates, navigation stdio smoke, 22 scale tests and the 10k postings
+probe. Raw v2 measured 32.53x and navigation 270.71x, both with required-source
+recall 1.0. Commit-specific CI and artifact checksums accompany the GitHub release. No new client,
+consumer or whole-task savings acceptance follows from these checks.
+
+npm authentication returned HTTP 401 on 22 September. Registry publication
+remains blocked; matching release tarballs provide the independent installation
+route. The next adoption gate is an accepted Claude task with complete usage,
+repair and review accounting, alongside continuing Codex use.
+
 ## Current internal adoption review, 21 September 2026
 
 The [ForgeSworn dogfooding plan](docs/FORGESWORN-DOGFOOD-GOALS.md) now tracks

benchmarks/README.md

@@ -16,6 +16,41 @@ record IDs, authors and event IDs. Signature and grant verification are tested
 elsewhere; excluding fresh signatures prevents random signature bytes from
 changing tokenizer merges between benchmark runs.
 
+## Raw-evidence benchmark v2: growing corpora
+
+The original v1 runner rejected a corpus above 128 chunks because it used one
+synthetic signed-collection view. The repository grew to 136 chunks while adding
+the installation doctor. V2 retains the complete original file selection,
+3,600-character chunks, fixed questions and required sources. It partitions the
+ordered records into consecutive collections of at most 128 records, within the
+existing maximum of 32 collections. Nothing is dropped, and partitioning does
+not depend on the query or expected answer. Larger corpora still fail explicitly.
+Production signed-collection limits and authorisation are unchanged.
+
+For every question, v2 calls the existing `retrieveView` once for **every**
+collection, each with the original 8,192-byte, four-record response limits and
+related expansion disabled. Token and byte totals sum each complete response,
+including empty responses, metadata and duplicate evidence. Required-source
+recall uses the union of returned sources; accounting never deduplicates or
+discards responses. The report exposes each collection's cost, call count,
+collection sizes, indexed record count and a corpus digest.
+
+The per-call budgets are unchanged; the **total per-question budget scales with
+the number of collections**. This is an explicit benchmark runner loop, not a
+new production aggregate API, global ranking or cross-collection graph traversal.
+It measures response payloads only: request tokens, MCP framing, model work,
+indexing, signature verification and host/reviewer effort are outside this gate.
+The existing 10x regression floor and full required-source recall still apply.
+For a single collection the retrieval payload remains identical to v1. Multi-
+collection results have a new `forgesworn-context-token-reduction-v2` label and
+must not be presented as directly comparable to historical v1 results. The v1
+runner remains reproducible from Git history; locked D5 experiments are untouched.
+This synthetic naive-baseline comparison does not establish subscription or cash
+savings, or guarantee a sufficient answer merely because a source was returned.
+
+`npm run test:benchmark-corpus` checks partition boundaries, capacity rejection,
+evidence beyond record 128, empty-response accounting and v1 single-view parity.
+
 Run `npm run benchmark:tokens:parity` for the distinct source-navigation gate.
 It compares the full raw TypeScript corpus with exact compact JSON returned from
 the deterministic source graph, requiring every predeclared source to be found.

benchmarks/collection-corpus.mjs

@@ -0,0 +1,133 @@
+// Benchmarks/collection-corpus.mjs — synthetic retrieval fixture driver.
+
+import { createHash } from 'node:crypto';
+import { retrieveView } from '../packages/context/dist/retrieval.js';
+
+export const BENCHMARK_LIMITS = Object.freeze({
+  maxCollections: 32,
+  maxRecordsPerCollection: 128,
+  maxBytesPerCall: 8192,
+  maxRecordsPerCall: 4,
+});
+
+const HEX64 = /^[0-9a-f]{64}$/;
+const OWNER = '1d'.repeat(32);
+const V1_ID = '2e'.repeat(32);
+const V1_HEAD = '3f'.repeat(32);
+
+function sha64(label) {
+  return createHash('sha256').update(label).digest('hex');
+}
+
+function fail(msg) {
+  throw new Error(`[collection-corpus] ${msg}`);
+}
+
+export function partitionBenchmarkRecords(records, observedAt) {
+  if (!Array.isArray(records) || records.length === 0) fail('records must be non-empty array');
+  if (records.length > 4096) fail('records exceeds 4096');
+  if (!Number.isSafeInteger(observedAt) || observedAt < 0) fail('observedAt required');
+
+  const seen = new Set();
+  for (const rec of records) {
+    if (!rec || typeof rec.id !== 'string' || !HEX64.test(rec.id)) fail('invalid record id');
+    if (seen.has(rec.id)) fail(`duplicate record id ${rec.id}`);
+    seen.add(rec.id);
+  }
+
+  const slices = [];
+  for (let i = 0; i < records.length; i += BENCHMARK_LIMITS.maxRecordsPerCollection) {
+    slices.push(records.slice(i, i + BENCHMARK_LIMITS.maxRecordsPerCollection));
+  }
+
+  return slices.map((slice, index) => {
+    const first = index === 0;
+    const id = first ? V1_ID : sha64(`context-repo/benchmark/collection/${index}`);
+    const head = first ? V1_HEAD : sha64(`context-repo/benchmark/head/${index}`);
+    return Object.freeze({
+      id,
+      owner: OWNER,
+      title: 'Context repository benchmark',
+      scope: 'personal',
+      epoch: 1,
+      head,
+      revision: 1,
+      updatedAt: observedAt,
+      role: 'write',
+      uploaded: false,
+      records: slice,
+    });
+  });
+}
+
+function validateViews(views) {
+  if (!Array.isArray(views)) fail('views must be array');
+  if (views.length < 1 || views.length > BENCHMARK_LIMITS.maxCollections) {
+    fail(`views length ${views.length} not in 1..${BENCHMARK_LIMITS.maxCollections}`);
+  }
+  const ids = new Set();
+  for (const v of views) {
+    if (!v || typeof v.id !== 'string' || !HEX64.test(v.id)) fail('invalid view.id');
+    if (ids.has(v.id)) fail(`duplicate view.id ${v.id}`);
+    ids.add(v.id);
+    if (!Array.isArray(v.records) || v.records.length === 0) fail('view.records non-empty');
+    if (v.records.length > BENCHMARK_LIMITS.maxRecordsPerCollection) {
+      fail(`view.records exceeds ${BENCHMARK_LIMITS.maxRecordsPerCollection}`);
+    }
+  }
+}
+
+export function measureBenchmarkRetrieval(views, query, countTokens) {
+  validateViews(views);
+  if (typeof countTokens !== 'function') fail('countTokens must be function');
+  if (typeof query !== 'string') fail('query must be string');
+
+  let retrievedTokens = 0;
+  let bytesUsed = 0;
+  const sources = new Set();
+  const collections = [];
+
+  for (const view of views) {
+    const payload = retrieveView(view, {
+      query,
+      maxBytes: BENCHMARK_LIMITS.maxBytesPerCall,
+      maxRecords: BENCHMARK_LIMITS.maxRecordsPerCall,
+      includeRelated: false,
+    });
+    const serialized = JSON.stringify(payload);
+    const bytes = Buffer.byteLength(serialized, 'utf8');
+    if (bytes > BENCHMARK_LIMITS.maxBytesPerCall) {
+      fail(`payload bytes ${bytes} exceeds ${BENCHMARK_LIMITS.maxBytesPerCall}`);
+    }
+    if (!Array.isArray(payload.records) || payload.availableRecords !== view.records.length || payload.bytesUsed !== bytes) {
+      fail('retrieval returned inconsistent coverage or byte accounting');
+    }
+    const recs = payload.records;
+    if (recs.length > BENCHMARK_LIMITS.maxRecordsPerCall) {
+      fail(`payload.records ${recs.length} exceeds ${BENCHMARK_LIMITS.maxRecordsPerCall}`);
+    }
+    const tokens = countTokens(serialized);
+    if (!Number.isSafeInteger(tokens) || tokens < 0) fail('countTokens returned invalid');
+
+    retrievedTokens += tokens;
+    bytesUsed += bytes;
+    for (const rec of recs) {
+      if (rec && typeof rec.source === 'string') sources.add(rec.source);
+    }
+    collections.push(Object.freeze({
+      collection: view.id,
+      availableRecords: payload.availableRecords,
+      returnedRecords: recs.length,
+      retrievedTokens: tokens,
+      bytesUsed: bytes,
+    }));
+  }
+
+  return Object.freeze({
+    retrievalCalls: views.length,
+    retrievedTokens,
+    bytesUsed,
+    returnedSources: Object.freeze([...sources].sort()),
+    collections: Object.freeze(collections),
+  });
+}

benchmarks/collection-corpus.test.mjs

@@ -0,0 +1,78 @@
+import assert from 'node:assert/strict'
+import test from 'node:test'
+import { createHash } from 'node:crypto'
+import { encode } from 'gpt-tokenizer/encoding/o200k_base'
+import { retrieveView } from '../packages/context/dist/retrieval.js'
+import { partitionBenchmarkRecords, measureBenchmarkRetrieval } from './collection-corpus.mjs'
+
+const now = 1_800_000_000
+const tokens = text => encode(text).length
+const records = count => Array.from({ length: count }, (_, i) => ({
+  id: createHash('sha256').update(`record-${i}`).digest('hex'),
+  kind: 'evidence', text: i === count - 1 ? 'needle evidence 🌳' : 'unrelated background',
+  source: `fixture-${i}.ts`, observedAt: now, author: '1d'.repeat(32), event: '4e'.repeat(32),
+}))
+
+for (const count of [128, 129, 256, 4096]) {
+  test(`retains all ${count} records exactly once in deterministic bounded collections`, () => {
+    const input = records(count)
+    const before = structuredClone(input)
+    const views = partitionBenchmarkRecords(input, now)
+    assert.equal(views.length, Math.ceil(count / 128))
+    assert.ok(views.every(view => view.records.length > 0 && view.records.length <= 128))
+    assert.equal(new Set(views.map(view => view.id)).size, views.length)
+    assert.deepEqual(views.flatMap(view => view.records), before)
+    assert.deepEqual(input, before)
+    assert.deepEqual(partitionBenchmarkRecords(input, now), views)
+  })
+}
+
+test('rejects unsupported capacity and invalid identity without silently dropping records', () => {
+  assert.throws(() => partitionBenchmarkRecords([], now))
+  assert.throws(() => partitionBenchmarkRecords(records(4097), now))
+  const [record] = records(1)
+  assert.throws(() => partitionBenchmarkRecords([record, record], now))
+  assert.throws(() => partitionBenchmarkRecords([{ ...record, id: 'invalid' }], now))
+})
+
+test('charges empty responses and finds evidence beyond the first 128 records', () => {
+  const views = partitionBenchmarkRecords(records(129), now)
+  const seen = []
+  const result = measureBenchmarkRetrieval(views, 'needle', text => {
+    seen.push(text)
+    return tokens(text)
+  })
+  const expected = views.map(view => JSON.stringify(retrieveView(view, {
+    query: 'needle', maxBytes: 8192, maxRecords: 4, includeRelated: false,
+  })))
+  assert.deepEqual(seen, expected)
+  assert.equal(result.retrievalCalls, 2)
+  assert.deepEqual(result.returnedSources, ['fixture-128.ts'])
+  assert.equal(result.retrievedTokens, expected.reduce((sum, text) => sum + tokens(text), 0))
+  assert.equal(result.bytesUsed, expected.reduce((sum, text) => sum + Buffer.byteLength(text, 'utf8'), 0))
+  assert.equal(result.collections[0].returnedRecords, 0)
+  assert.ok(result.collections[0].retrievedTokens > 0)
+  assert.equal(result.collections[1].returnedRecords, 1)
+  assert.ok(result.collections.every(row => row.bytesUsed <= 8192 && row.returnedRecords <= 4))
+})
+
+test('one collection retains the original v1 retrieval payload and budget', () => {
+  const input = records(10)
+  const legacyView = { id: '2e'.repeat(32), owner: '1d'.repeat(32), title: 'Context repository benchmark',
+    scope: 'personal', epoch: 1, head: '3f'.repeat(32), revision: 1, updatedAt: now,
+    role: 'write', uploaded: false, records: input }
+  const payload = retrieveView(legacyView, { query: 'needle', maxBytes: 8192, maxRecords: 4, includeRelated: false })
+  const result = measureBenchmarkRetrieval(partitionBenchmarkRecords(input, now), 'needle', tokens)
+  assert.equal(result.retrievalCalls, 1)
+  assert.equal(result.retrievedTokens, tokens(JSON.stringify(payload)))
+  assert.equal(result.bytesUsed, payload.bytesUsed)
+})
+
+test('refuses invalid view sets before counting any response', () => {
+  const [view] = partitionBenchmarkRecords(records(1), now)
+  const counter = () => { assert.fail('invalid view sets must fail before retrieval') }
+  for (const views of [[], [view, view], Array.from({ length: 33 }, () => view),
+    [{ ...view, records: [] }], [{ ...view, records: records(129) }]]) {
+    assert.throws(() => measureBenchmarkRetrieval(views, 'needle', counter))
+  }
+})

benchmarks/token-reduction.mjs

@@ -3,7 +3,7 @@ import { readFile, readdir } from 'node:fs/promises'
 import { relative, resolve } from 'node:path'
 import { fileURLToPath } from 'node:url'
 import { encode } from 'gpt-tokenizer/encoding/o200k_base'
-import { retrieveView } from '../packages/context/dist/retrieval.js'
+import { BENCHMARK_LIMITS, partitionBenchmarkRecords, measureBenchmarkRetrieval } from './collection-corpus.mjs'
 
 const root = resolve(fileURLToPath(new URL('..', import.meta.url)))
 const parityTarget = 71.5
@@ -51,44 +51,44 @@ for (const path of [...sourceFiles, ...documentation].sort()) {
   const content = await readFile(path, 'utf8')
   for (const [part, text] of chunks(content).entries()) corpus.push({ source, part, text })
 }
-if (corpus.length > 128) throw new Error(`Corpus has ${corpus.length} chunks; ContextVault supports at most 128 records.`)
 
 const records = corpus.map(item => ({
   id: createHash('sha256').update(`${item.source}\0${item.part}\0${item.text}`).digest('hex'),
   kind: 'evidence', text: item.text, source: item.source, observedAt: now,
   author: '1d'.repeat(32),
   event: createHash('sha256').update(`event\0${item.source}\0${item.part}\0${item.text}`).digest('hex'),
 }))
-const view = {
-  id: '2e'.repeat(32), owner: '1d'.repeat(32), title: 'Context repository benchmark',
-  scope: 'personal', epoch: 1, head: '3f'.repeat(32), revision: 1,
-  updatedAt: now, role: 'write', uploaded: false, records,
-}
+const views = partitionBenchmarkRecords(records, now)
 
 // The naïve comparator is exactly what an agent would receive if every source
 // were inserted verbatim, including filenames, without signatures or graph data.
 const baselinePayload = corpus.map(({ source, part, text }) => ({ source, part, text }))
 const baselineTokens = tokens(baselinePayload)
 const results = questions.map(question => {
-  const payload = retrieveView(view, { query: question.query, maxBytes: 8192, maxRecords: 4, includeRelated: false })
-  const retrievedTokens = tokens(payload)
-  const returned = new Set(payload.records.map(record => record.source))
+  const measurement = measureBenchmarkRetrieval(views, question.query, tokens)
+  const { retrievedTokens } = measurement
+  const returned = new Set(measurement.returnedSources)
   const found = question.required.filter(source => returned.has(source))
   const recall = found.length / question.required.length
   return {
     id: question.id, query: question.query, requiredSources: question.required,
-    returnedSources: [...returned], baselineTokens, retrievedTokens,
+    ...measurement, baselineTokens,
     multiplier: baselineTokens / retrievedTokens,
     reductionPercent: (1 - retrievedTokens / baselineTokens) * 100,
-    evidenceRecall: recall, bytesUsed: payload.bytesUsed,
+    evidenceRecall: recall,
   }
 })
 const totalBaselineTokens = baselineTokens * results.length
 const totalRetrievedTokens = results.reduce((sum, result) => sum + result.retrievedTokens, 0)
 const aggregateMultiplier = totalBaselineTokens / totalRetrievedTokens
 const report = {
-  benchmark: 'forgesworn-context-token-reduction-v1', tokenizer: 'o200k_base',
-  corpus: { files: sourceFiles.length + documentation.length, chunks: corpus.length, baselineTokens },
+  benchmark: 'forgesworn-context-token-reduction-v2', tokenizer: 'o200k_base',
+  contract: 'synthetic-authorised-collections-payload-only',
+  limits: BENCHMARK_LIMITS,
+  corpus: { files: sourceFiles.length + documentation.length, chunks: corpus.length, baselineTokens,
+    sha256: createHash('sha256').update(JSON.stringify(corpus)).digest('hex'),
+    collections: views.length, collectionSizes: views.map(view => view.records.length),
+    indexedRecords: views.reduce((sum, view) => sum + view.records.length, 0) },
   queries: results.map(result => ({ ...result,
     multiplier: Number(result.multiplier.toFixed(2)), reductionPercent: Number(result.reductionPercent.toFixed(2)),
   })),