Clamp packet ranges and make handoff metadata optional

docs/WORKER-PACKETS.md

@@ -57,7 +57,12 @@ then call `repository_refresh` and retain its `generation`.
 
 Replace the example paths and anchors with located evidence. Use `mode: "build"`
 and `{path, startLine, endLine}` source entries for reviewed exact ranges,
-including languages without syntax planning. The server accepts an inline spec;
+including languages without syntax planning. Through MCP only `sources` is
+required: omitted handoff metadata gets a neutral task and acceptance check, and
+an `endLine` past the end of a file reads to its last line (the packet records
+the range actually read). A range starting past the end is still rejected with
+the file's line count. A plan anchor outside every supported block is rejected
+with a pointer to `mode: "build"`. The CLI keeps the strict spec. The server accepts an inline spec;
 it cannot accept a different root, spec-file path, output path or shell command.
 The configured root must be a Git repository with a committed HEAD. Internal
 `git rev-parse` calls read provenance; the tool does not execute acceptance checks.

docs/experiments/retrieval-recall-20260923/RESULTS.md

@@ -0,0 +1,85 @@
+# Retrieval measured from recorded transcripts
+
+No model calls. `analyse.mjs` reads the executor streams already recorded for
+the v1, v2, S5, v3 coverage, Flash smoke and aborted Pro runs (98 cells; the
+evidence stays private) and measures retrieval directly instead of through the
+reviewer:
+
+- **recall**: the share of each task's `requiredEvidence` tokens
+  (`../d5-20260921/acceptance/`) that appear in any tool result. The Context
+  tools print `local-source-unsigned` in their own metadata, so that token is
+  excluded; counting it would credit the Context arm for reading its own
+  status output.
+- **bytes to full recall**: cumulative tool-result bytes when the last required
+  token first appeared.
+- **tool errors**: tool results flagged as errors.
+
+## Retrieval is not what fails the structured tasks
+
+Structured tasks only (code-change tasks have no required evidence), all runs
+pooled:
+
+| Arm | Cells | Mean recall | Full recall | Accepted with full recall | Accepted without |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| plain | 23 | 0.90 | 16 | 4 of 16 | 1 of 7 |
+| graphify | 23 | 0.96 | 20 | 4 of 20 | 0 of 3 |
+| context | 29 | 0.92 | 23 | 6 of 23 | 1 of 6 |
+
+Every arm usually retrieves everything the rubric needs, and three quarters of
+full-recall answers are still rejected. On these tasks acceptance measures
+synthesis and rubric fit; a retrieval tool can only show up as cost. The most
+common miss in every arm is one test title in orientation-context
+(`successfully used cursor is consumed`). Per task, Graphify and Context reach
+full recall with fewer bytes than plain on the orientation tasks (median 63 to
+70 KB against 97 KB on orientation-context) but not consistently elsewhere;
+with three or four cells per task this is not a measured difference.
+
+## Tool errors: the Context arm's real friction
+
+| Run | plain | graphify | context |
+| --- | ---: | ---: | ---: |
+| v1 | 4 / 203 | 10 / 196 | 9 / 233 |
+| v2 | 3 / 116 | 3 / 111 | 7 / 128 |
+| S5 (DeepSeek Pro) | 3 / 206 | 4 / 152 | 37 / 179 |
+| v3 coverage | | | 10 / 228 |
+| Flash smoke | 0 / 60 | 0 / 36 | 8 / 68 |
+
+(errors / tool results). On DeepSeek the Context arm failed one call in five.
+Grouped by message, the Context errors were:
+
+- `line range exceeds source length`: 31. Models ask for a whole file with a
+  guessed `endLine`.
+- empty or placeholder packet metadata rejected by the schema
+  (`acceptanceChecks` empty, `allowedFiles` empty strings): 10, plus many
+  accepted calls padded with `"a"` or `"ok"`.
+- `repository packet already in progress`: 10 (fixed in #26, calls now queue).
+- `no supported syntax block contains <file>:<line>`: 7, mostly plan anchors on
+  imports.
+- one-identifier search terms rejecting literals: 5 (fixed in #26), and one
+  literal passed to `repository_explore`.
+- a coverage draft naming more than eight symbols: 1 (fixed in #26).
+- harness permission prompts in v1: 3 (not the tool).
+- stale navigation after the model's own edits: 3 (correct behaviour).
+
+## Changes made from this
+
+In `repository_packet` through MCP only (the CLI spec stays strict):
+
+- an `endLine` past the end of a file reads to its last line, and the packet's
+  `originalSpec` records the range read, so `verifyPacket` still rebuilds it
+  exactly; a range that starts past the end is still rejected;
+- only `sources` is required; omitted or blank handoff metadata gets a neutral
+  task and acceptance check;
+- a plan anchor outside every supported block now says to request an exact
+  range with `mode: "build"`.
+
+Together with #26 these remove 57 of the 71 recorded Context errors and
+redirect 7 more; the rest are harness prompts, stale navigation and one
+misused tool.
+The served tool listing fell to 5,550 bytes.
+
+## What this does not show
+
+Recall is token presence in tool output, not understanding; a token can appear
+in a search listing without the model reading its context. Runs differ in
+model, build and rubric, so cells are pooled only for this model-free measure.

docs/experiments/retrieval-recall-20260923/analyse.mjs

@@ -0,0 +1,81 @@
+// Model-free retrieval measures from recorded executor streams.
+// Usage: node analyse.mjs <label>=<evidence-dir> [...] > results.json
+import { readFileSync, existsSync, readdirSync, statSync } from 'node:fs'
+import { join, dirname } from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+const here = dirname(fileURLToPath(import.meta.url))
+const acceptanceDir = join(here, '../d5-20260921/acceptance')
+// Tokens a Context tool prints in its own metadata, so their presence proves nothing about retrieval.
+const toolMetadataTokens = new Set(['local-source-unsigned'])
+
+const textOf = (content) => typeof content === 'string'
+  ? content
+  : (content ?? []).map((part) => part.type === 'text' ? part.text : '').join('\n')
+
+function cells(dir) {
+  const found = []
+  const walk = (d, depth) => {
+    if (existsSync(join(d, 'receipt.json')) && existsSync(join(d, 'executor.stream.jsonl'))) found.push(d)
+    else if (depth < 4) for (const n of readdirSync(d)) {
+      const p = join(d, n)
+      if (n !== 'workspace' && statSync(p).isDirectory()) walk(p, depth + 1)
+    }
+  }
+  walk(dir, 0)
+  return found
+}
+
+function analyse(cellDir) {
+  const receipt = JSON.parse(readFileSync(join(cellDir, 'receipt.json'), 'utf8'))
+  const acceptance = JSON.parse(readFileSync(join(acceptanceDir, `${receipt.task}.json`), 'utf8'))
+  const required = acceptance.requiredEvidence ?? []
+  const first = new Map()
+  let turn = 0; let bytes = 0; let results = 0; let errors = 0; let requests = 0
+  const seenCalls = new Map()
+  let repeatedCalls = 0
+  for (const line of readFileSync(join(cellDir, 'executor.stream.jsonl'), 'utf8').split('\n')) {
+    if (!line.trim()) continue
+    const event = JSON.parse(line)
+    if (event.type === 'assistant') {
+      requests += 1
+      for (const part of event.message.content ?? []) {
+        if (part.type !== 'tool_use') continue
+        const key = `${part.name}:${JSON.stringify(part.input)}`
+        if (seenCalls.has(key)) repeatedCalls += 1
+        seenCalls.set(key, true)
+      }
+    }
+    if (event.type !== 'user' || !Array.isArray(event.message?.content)) continue
+    for (const part of event.message.content) {
+      if (part.type !== 'tool_result') continue
+      turn += 1; results += 1
+      if (part.is_error) errors += 1
+      const text = textOf(part.content)
+      bytes += Buffer.byteLength(text)
+      for (const [i, item] of required.entries()) {
+        if (!first.has(i) && text.includes(item.token)) first.set(i, { result: turn, bytes })
+      }
+    }
+  }
+  const scored = required.map((item, i) => ({ token: item.token, leaked: toolMetadataTokens.has(item.token), at: first.get(i) ?? null }))
+  const honest = scored.filter((s) => !s.leaked)
+  const run = receipt.executorRun
+  return {
+    task: receipt.task, arm: receipt.arm, accepted: receipt.accepted,
+    required: required.length,
+    recall: required.length ? scored.filter((s) => s.at).length / required.length : null,
+    recallExcludingToolMetadata: honest.length ? honest.filter((s) => s.at).length / honest.length : null,
+    bytesToFullRecall: honest.length && honest.every((s) => s.at) ? Math.max(...honest.map((s) => s.at.bytes)) : null,
+    missing: scored.filter((s) => !s.at).map((s) => s.token),
+    toolResults: results, toolErrors: errors, repeatedCalls, toolResultBytes: bytes,
+    turns: run.numTurns, inputTotal: run.inputTotal, inputUncached: run.inputUncached, output: run.output,
+  }
+}
+
+const out = {}
+for (const arg of process.argv.slice(2)) {
+  const [label, dir] = arg.split('=')
+  out[label] = cells(dir).map(analyse)
+}
+process.stdout.write(`${JSON.stringify(out, null, 1)}\n`)

packages/context-tools/src/repository-navigation-mcp.ts

@@ -17,13 +17,18 @@ const MAX_QUEUED_PACKETS = 8
 const formatSchema = z.enum(['text', 'json']).optional()
 const pathPrefixSchema = z.string().min(1).max(512).optional()
 
+// Handoff metadata is optional here: an interactive read needs only sources, and
+// recorded sessions filled required metadata with placeholders or were rejected.
+const DEFAULT_PACKET_TASK = 'read exact source'
+const DEFAULT_ACCEPTANCE_CHECK = 'cite exact source lines'
+const metadataStrings = z.array(z.string()).transform((items) => items.filter((item) => item.trim().length > 0))
 const packetMetadataSchema = z.object({
-  version: z.literal(1),
-  task: z.string().min(1),
-  acceptanceChecks: z.array(z.string().min(1)).min(1),
-  allowedFiles: z.array(z.string().min(1)).max(32),
-  exclusions: z.array(z.string().min(1)),
-  unresolvedQuestions: z.array(z.string().min(1)),
+  version: z.literal(1).default(1),
+  task: z.string().transform((task) => task.trim() || DEFAULT_PACKET_TASK).default(DEFAULT_PACKET_TASK),
+  acceptanceChecks: metadataStrings.transform((checks) => checks.length ? checks : [DEFAULT_ACCEPTANCE_CHECK]).default([DEFAULT_ACCEPTANCE_CHECK]),
+  allowedFiles: metadataStrings.pipe(z.array(z.string()).max(32)).default([]),
+  exclusions: metadataStrings.default([]),
+  unresolvedQuestions: metadataStrings.default([]),
 }).strict()
 const packetBuildSpecSchema = packetMetadataSchema.extend({
   sources: z.array(z.object({ path: z.string().min(1), startLine: z.number().int().min(1), endLine: z.number().int().min(1) }).strict()).max(32),
@@ -235,7 +240,8 @@ export function createRepositoryNavigationServer(root: string): RepositoryNaviga
     'repository_packet',
     {
       description:
-        'Verbatim source. mode build: exact ranges (sources path, startLine, endLine). ' +
+        'Verbatim source. mode build: exact ranges (sources path, startLine, endLine; ' +
+        'an endLine past the end reads to the end). ' +
         'mode plan: complete TypeScript/JavaScript blocks around anchors (sources path, ' +
         'line). Pass the current generation; stale navigation is rejected. Capped at ' +
         'maxBytes (64 KiB default).',
@@ -260,7 +266,7 @@ export function createRepositoryNavigationServer(root: string): RepositoryNaviga
           const before = await currentPacketNavigation(navigation, input.expectedGeneration, extra.signal)
           throwIfAborted(extra.signal)
           const result = input.mode === 'build'
-            ? { packet: await buildPacketInline({ root, spec: parsedSpec }) }
+            ? { packet: await buildPacketInline({ root, spec: parsedSpec, clampToEnd: true }) }
             : await planPacketInline({ root, spec: parsedSpec })
           throwIfAborted(extra.signal)
           const after = await currentPacketNavigation(navigation, input.expectedGeneration, extra.signal)

packages/context-tools/src/repository-packet-mcp.test.ts

@@ -220,14 +220,50 @@ describe('repository packet MCP adapter', () => {
     } finally { await connection.close() }
   })
 
-  it('reports the file length when a range runs past the end', async () => {
+  it('reads to the end of the file when a range runs past it, and verification rebuilds the clamped range', async () => {
     const value = await root(); const connection = await connect(value)
     try {
       const refreshed = await connection.client.callTool({ name: 'repository_refresh', arguments: {} }) as { content: unknown }
       const generation = (JSON.parse(text(refreshed)) as { generation: string }).generation
-      const past = await call(connection.client, { mode: 'build', spec: spec([{ path: 'alpha.ts', startLine: 1, endLine: 40 }]), expectedGeneration: generation })
-      expect(past.isError).toBe(true)
-      expect(text(past)).toMatch(/line range exceeds source length: alpha\.ts has 3 lines; request endLine 3 or less/)
+      const past = await call(connection.client, { mode: 'build', spec: spec([{ path: 'alpha.ts', startLine: 2, endLine: 40 }]), expectedGeneration: generation })
+      expect(past.isError).not.toBe(true)
+      const packet = JSON.parse(text(past)).packet
+      expect(packet.sources[0]).toMatchObject({ path: 'alpha.ts', startLine: 2, endLine: 3 })
+      expect(packet.sources[0].lines.map((line: { line: number }) => line.line)).toEqual([2, 3])
+      expect(packet.originalSpec.sources[0]).toEqual({ path: 'alpha.ts', startLine: 2, endLine: 3 })
+      const { verifyPacket, serializePacket } = await import('./source-packet.mjs') as unknown as { verifyPacket: (input: { root: string; packet: string }) => Promise<{ status: string }>; serializePacket: (value: unknown) => string }
+      const saved = join(value, 'packet.json')
+      await writeFile(saved, serializePacket(packet))
+      expect(await verifyPacket({ root: await realpath(value), packet: saved })).toEqual({ status: 'current' })
+      const beyond = await call(connection.client, { mode: 'build', spec: spec([{ path: 'alpha.ts', startLine: 9, endLine: 40 }]), expectedGeneration: generation })
+      expect(beyond.isError).toBe(true)
+      expect(text(beyond)).toMatch(/line range exceeds source length: alpha\.ts has 3 lines; request endLine 3 or less/)
+    } finally { await connection.close() }
+  })
+
+  it('accepts a spec with only sources and fills neutral handoff metadata', async () => {
+    const value = await root(); const connection = await connect(value)
+    try {
+      const refreshed = await connection.client.callTool({ name: 'repository_refresh', arguments: {} }) as { content: unknown }
+      const generation = (JSON.parse(text(refreshed)) as { generation: string }).generation
+      const bare = await call(connection.client, { mode: 'build', spec: { sources: [{ path: 'alpha.ts', startLine: 1, endLine: 1 }] }, expectedGeneration: generation })
+      expect(bare.isError).not.toBe(true)
+      expect(JSON.parse(text(bare)).packet.originalSpec).toMatchObject({ version: 1, task: 'read exact source', acceptanceChecks: ['cite exact source lines'], allowedFiles: [], exclusions: [], unresolvedQuestions: [] })
+      const empty = await call(connection.client, { mode: 'plan', spec: { version: 1, task: '', acceptanceChecks: [], allowedFiles: [], exclusions: [''], unresolvedQuestions: [], sources: [{ path: 'alpha.ts', line: 2 }] }, expectedGeneration: generation })
+      expect(empty.isError).not.toBe(true)
+    } finally { await connection.close() }
+  })
+
+  it('points a plan anchor outside any block at an exact build range', async () => {
+    const value = await root()
+    await writeFile(join(value, 'beta.ts'), "import { alpha } from './alpha'\nexport function beta() {\n  return alpha()\n}\n")
+    const connection = await connect(value)
+    try {
+      const refreshed = await connection.client.callTool({ name: 'repository_refresh', arguments: {} }) as { content: unknown }
+      const generation = (JSON.parse(text(refreshed)) as { generation: string }).generation
+      const outside = await call(connection.client, { mode: 'plan', spec: { sources: [{ path: 'beta.ts', line: 1 }] }, expectedGeneration: generation })
+      expect(outside.isError).toBe(true)
+      expect(text(outside)).toMatch(/no supported syntax block contains beta\.ts:1; request an exact range with mode build instead/)
     } finally { await connection.close() }
   })
 })

packages/context-tools/src/source-packet.mjs

@@ -166,7 +166,7 @@ async function scopeFor(root, policy, path, cache) {
   }
   return scope;
 }
-async function readSource(root, entry, policy, scopes, total) {
+async function readSource(root, entry, policy, scopes, total, clampToEnd = false) {
   const absolute = resolve(root, entry.path);
   await rejectSymlinkComponents(root, absolute, `source ${entry.path}`);
   const scope = await scopeFor(root, policy, entry.path, scopes);
@@ -177,6 +177,7 @@ async function readSource(root, entry, policy, scopes, total) {
   const text = strictText(bytes, `source ${entry.path}`);
   const lines = text.split('\n');
   const lineCount = text.endsWith('\n') ? lines.length - 1 : lines.length;
+  if (clampToEnd && entry.startLine <= lineCount && entry.endLine > lineCount) entry = { ...entry, endLine: lineCount };
   assert(entry.endLine <= lines.length, `line range exceeds source length: ${entry.path} has ${lineCount} lines; request endLine ${lineCount} or less`);
   const excerpt = [];
   for (let line = entry.startLine; line <= entry.endLine; line++) {
@@ -223,16 +224,18 @@ async function validateAllowedSnapshots(root, allowedFiles) {
   }
 }
 
-export async function buildPacketFromSpec({ root: rootInput, spec: specInput }) {
+export async function buildPacketFromSpec({ root: rootInput, spec: specInput, clampToEnd = false }) {
   const rootInfo = await canonicalRoot(rootInput);
   const root = rootInfo.root;
-  const spec = parseSpec(serialize(specInput));
+  let spec = parseSpec(serialize(specInput));
   const head = await gitHead(root);
   const policy = await NavigationPolicy.load(root);
   const scopes = new Map([['', policy.rootDirectoryScope()]]);
   const total = { value: 0, excerptBytes: 0 };
   const sources = [];
-  for (const entry of spec.sources) sources.push(await readSource(root, entry, policy, scopes, total));
+  for (const entry of spec.sources) sources.push(await readSource(root, entry, policy, scopes, total, clampToEnd));
+  // A clamped range is recorded as read, so verification rebuilds exactly what was returned.
+  if (clampToEnd) spec = { ...spec, sources: spec.sources.map((entry, index) => ({ ...entry, endLine: sources[index].endLine })) };
   const allowedFiles = [];
   for (const file of spec.allowedFiles) allowedFiles.push(await allowedState(root, file, policy, scopes, total));
   const initialManifest = await validatePolicy(root, spec, policy);
@@ -261,10 +264,10 @@ export async function buildPacket({ root: rootInput, spec: specInput }) {
   return buildPacketFromSpec({ root: rootInput, spec: parseSpec(strictText(specBytes, 'spec')) });
 }
 
-export async function buildPacketInline({ root: rootInput, spec: specInput }) {
+export async function buildPacketInline({ root: rootInput, spec: specInput, clampToEnd = false }) {
   const bytes = Buffer.from(serialize(specInput), 'utf8');
   assert(bytes.byteLength <= MAX_SPEC_BYTES, `spec exceeds ${MAX_SPEC_BYTES} bytes`);
-  return buildPacketFromSpec({ root: rootInput, spec: parseSpec(strictText(bytes, 'spec')) });
+  return buildPacketFromSpec({ root: rootInput, spec: parseSpec(strictText(bytes, 'spec')), clampToEnd });
 }
 
 function scriptKind(path) {
@@ -327,7 +330,7 @@ function resolveAnchor(source, path, line) {
   const lastLine = source.getLineAndCharacterOfPosition(source.end).line + 1;
   assert(line <= lastLine, `anchor line exceeds source length: ${path} has ${lastLine} lines`);
   const containing = source.__packetCandidates.filter((candidate) => line >= candidate.startLine && line <= candidate.endLine);
-  assert(containing.length > 0, `no supported syntax block contains ${path}:${line}`);
+  assert(containing.length > 0, `no supported syntax block contains ${path}:${line}; request an exact range with mode build instead`);
   containing.sort((a, b) => (a.end - a.start) - (b.end - b.start) || a.start - b.start || compare(a.kind, b.kind));
   const chosen = containing[0];
   const sameSpan = containing.filter((candidate) => candidate !== chosen && candidate.end - candidate.start === chosen.end - chosen.start);