Reviewer consistency evidence and model-free code-change acceptance

docs/experiments/REPEATED-RUNS.md

@@ -17,6 +17,9 @@ the changed tool was never called. Every later comparison uses this design.
 - **Executor.** One model per run, recorded with effort and client version.
   The next run uses `claude-opus-5-5`, with the reviewer unchanged
   (`claude-sonnet-5`, high) so verdicts stay comparable with earlier runs.
+- **Code-change acceptance.** Checker plus the deterministic scope check
+  (`"codeAcceptance": "checker-and-scope"`, see `code-acceptance-20260923/`);
+  the model reviewer judges structured answers only.
 - **Arms.** Plain tools, Graphify, and Context at its current build, which
   includes `repository_explore` and `repository_coverage` with the exact-quote
   check. A category checklist in the instructions, if tested, is a fourth arm

docs/experiments/code-acceptance-20260923/README.md

@@ -0,0 +1,46 @@
+# Code-change acceptance without a model reviewer
+
+On code-change tasks the pack's checker (`d5-20260921/accept.mjs`) already
+builds, runs the focused tests and probes each required behaviour. The recorded
+reviews added nothing correct on top: all 20 recorded code-change diffs passed
+the checker, and the reviewer's one rejection was of a diff whose code is
+identical to eight it accepted (`../reviewer-evidence-20260923/RESULTS.md`).
+
+`scope.mjs` replaces the reviewer's `scope` dimension with a deterministic
+check of the workspace against its base commit:
+
+- every changed path is inside the task's selection policy;
+- no package manifest, lockfile, TypeScript, Vite, Vitest or Jest config,
+  `.gitignore`, `.npmrc` or `.github/` file changed;
+- no test file deleted, no test declaration removed, no `skip`, `only`, `todo`,
+  `xit` or `xdescribe` added;
+- at least one file changed.
+
+Assertion counts are reported but not judged: a behaviour change legitimately
+rewrites assertions (S5 Graphify on code-change-kithmoot replaced two
+assertions of the old behaviour with one looped assertion over seven invalid
+inputs), and the checker runs its own behaviour probes regardless of the
+agent's tests.
+
+A task is accepted when the checker and the scope check both pass. The harness
+uses this rule only when a protocol sets `"codeAcceptance": "checker-and-scope"`;
+the locked v1, v2, S5 and v3 protocols do not, so their runs replay unchanged.
+
+## Recorded runs re-scored
+
+`node rescore.mjs v1=DIR v2=DIR s5=DIR v3=DIR` (no model calls):
+
+| Run | Arms | Recorded accepted | Checker and scope |
+|-----|------|------------------:|------------------:|
+| v1 | plain, Graphify, Context | 2, 1, 2 of 2 | 2, 2, 2 of 2 |
+| v2 | plain, Graphify, Context | 2, 2, 2 of 2 | 2, 2, 2 of 2 |
+| S5 | plain, Graphify, Context | 2, 2, 2 of 2 | 2, 2, 2 of 2 |
+| v3 | Context | 2 of 2 | 2 of 2 |
+
+The only change is v1 Graphify on code-change-context, rejected by the
+reviewer for the same code the other eight arms were accepted for. All 20
+cells pass the scope check. The rule would have avoided 20 reviews: 1,199
+seconds and $2.20 client-reported.
+
+These two tasks discriminate no arm under either rule; they measure cost, not
+acceptance.

docs/experiments/code-acceptance-20260923/rescore.mjs

@@ -0,0 +1,37 @@
+#!/usr/bin/env node
+// Re-score recorded code-change cells with checker plus scope check instead of the model reviewer.
+// No model calls. Usage: node rescore.mjs NAME=DIR [...]
+import { existsSync, readFileSync, readdirSync } from 'node:fs'
+import { dirname, join, resolve } from 'node:path'
+import { fileURLToPath } from 'node:url'
+import { checkScope } from './scope.mjs'
+
+const packDir = resolve(dirname(fileURLToPath(import.meta.url)), '../d5-20260921')
+const runs = process.argv.slice(2).map((arg) => arg.split('='))
+if (!runs.length) throw new Error('usage: rescore.mjs NAME=DIR [...]')
+const cells = []
+for (const [run, dir] of runs) {
+  for (const taskId of readdirSync(dir).sort()) {
+    const taskPath = join(packDir, 'tasks', `${taskId}.json`)
+    if (!existsSync(taskPath)) continue
+    const task = JSON.parse(readFileSync(taskPath, 'utf8'))
+    if (task.answerSchema) continue
+    for (const arm of ['plain', 'graphify', 'context']) {
+      const armDir = join(dir, taskId, arm)
+      if (!existsSync(join(armDir, 'receipt.json'))) continue
+      const receipt = JSON.parse(readFileSync(join(armDir, 'receipt.json'), 'utf8'))
+      const scope = checkScope({ workspace: join(armDir, 'workspace'), include: task.selectionPolicy.include })
+      const checker = Boolean(receipt.checker?.passed)
+      cells.push({ run, task: taskId, arm, checker, scope: scope.passed, scopeReasons: scope.reasons, recordedAccepted: Boolean(receipt.accepted), accepted: checker && scope.passed,
+        reviewerSeconds: receipt.reviewerRun?.seconds ?? null, reviewerCostUsd: receipt.reviewerRun?.totalCostUsd ?? null })
+    }
+  }
+}
+const changed = cells.filter((c) => c.accepted !== c.recordedAccepted)
+const byRun = {}
+for (const c of cells) {
+  const r = (byRun[`${c.run}|${c.arm}`] ??= { run: c.run, arm: c.arm, tasks: 0, recordedAccepted: 0, accepted: 0 })
+  r.tasks += 1; r.recordedAccepted += Number(c.recordedAccepted); r.accepted += Number(c.accepted)
+}
+const sum = (k) => Number(cells.reduce((s, c) => s + (c[k] ?? 0), 0).toFixed(2))
+process.stdout.write(`${JSON.stringify({ cells: cells.length, changed, byRunAndArm: Object.values(byRun), reviewerAvoided: { seconds: sum('reviewerSeconds'), costUsd: sum('reviewerCostUsd') }, all: cells }, null, 1)}\n`)

docs/experiments/code-acceptance-20260923/scope.mjs

@@ -0,0 +1,62 @@
+#!/usr/bin/env node
+// Deterministic scope check for code-change tasks: the agent's changes stay inside the task's
+// selection policy, leave build and test configuration alone, and do not delete or disable tests.
+// Assertion counts are reported but not judged: a behaviour change legitimately rewrites assertions
+// (one looped expect can replace several), and the pack checker runs its own behaviour probes.
+// With the pack's behaviour checker it replaces the model reviewer for code tasks.
+// Usage: node scope.mjs --workspace DIR --task ID    (prints the result; exit 1 when it fails)
+import { spawnSync } from 'node:child_process'
+import { readFileSync } from 'node:fs'
+import { dirname, join, resolve } from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+const ignored = [/^graphify-out\//, /^answer\.json$/]
+const protectedConfig = /(^|\/)(package(-lock)?\.json|npm-shrinkwrap\.json|tsconfig[^/]*\.json|(vitest|vite|jest)\.config\.[cm]?[jt]s|\.gitignore|\.npmrc)$|^\.github\//
+const testFile = /\.(test|spec)\.[cm]?[jt]sx?$|(^|\/)(test|tests|__tests__)\//
+const declarations = /\b(it|test)(\.each\([^)]*\))?\s*\(/g
+const assertions = /\bexpect\s*\(|\bassert(\.\w+)?\s*\(/g
+const disabled = /\b(it|test|describe)\.(skip|only|todo)\s*\(|\bx(it|describe)\s*\(/g
+
+function git(workspace, args, env) {
+  const r = spawnSync('git', args, { cwd: workspace, encoding: 'utf8', env, maxBuffer: 64 * 1024 * 1024 })
+  if (r.status !== 0) throw new Error(`git ${args.join(' ')} failed: ${r.stderr}`)
+  return r.stdout
+}
+const count = (text, pattern) => (text.match(pattern) ?? []).length
+
+export function checkScope({ workspace, include, env = process.env }) {
+  const reasons = []
+  const changes = git(workspace, ['status', '--porcelain=v1', '-z', '-uall'], env).split('\0').filter(Boolean)
+  const changed = []
+  for (let i = 0; i < changes.length; i += 1) {
+    const status = changes[i].slice(0, 2), path = changes[i].slice(3)
+    // With -z a rename or copy is followed by its source path.
+    const from = status.includes('R') || status.includes('C') ? changes[(i += 1)] : path
+    if (!ignored.some((re) => re.test(path))) changed.push({ status, path, from })
+  }
+  const tests = []
+  for (const { status, path, from } of changed) {
+    if (!include.some((prefix) => path === prefix || path.startsWith(`${prefix.replace(/\/$/, '')}/`))) reasons.push(`outside selection policy: ${path}`)
+    if (protectedConfig.test(path)) reasons.push(`build or test configuration changed: ${path}`)
+    if (!testFile.test(path)) continue
+    if (status.includes('D')) { reasons.push(`test file deleted: ${path}`); continue }
+    const before = status.includes('?') || status.includes('A') ? '' : git(workspace, ['show', `HEAD:${from}`], env)
+    const after = readFileSync(join(workspace, path), 'utf8')
+    const t = { path, tests: [count(before, declarations), count(after, declarations)], assertions: [count(before, assertions), count(after, assertions)], disabled: [count(before, disabled), count(after, disabled)] }
+    tests.push(t)
+    if (t.tests[1] < t.tests[0]) reasons.push(`fewer tests in ${path}: ${t.tests[0]} to ${t.tests[1]}`)
+    if (t.disabled[1] > t.disabled[0]) reasons.push(`skip, only or todo added in ${path}`)
+  }
+  if (!changed.length) reasons.push('no source change')
+  return { passed: reasons.length === 0, reasons, changed, tests }
+}
+
+if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
+  const arg = (key) => { const i = process.argv.indexOf(`--${key}`); return i >= 0 ? process.argv[i + 1] : undefined }
+  if (!arg('workspace') || !arg('task')) throw new Error('usage: scope.mjs --workspace DIR --task ID')
+  const packDir = resolve(dirname(fileURLToPath(import.meta.url)), '../d5-20260921')
+  const task = JSON.parse(readFileSync(join(packDir, 'tasks', `${arg('task')}.json`), 'utf8'))
+  const result = checkScope({ workspace: resolve(arg('workspace')), include: task.selectionPolicy.include })
+  process.stdout.write(`${JSON.stringify(result, null, 2)}\n`)
+  process.exit(result.passed ? 0 : 1)
+}

docs/experiments/graphify-20260922/run.mjs

@@ -2,12 +2,14 @@
 // Three-way retrieval comparison runner: plain tools, Graphify, Context.
 // Usage: node run.mjs --local /private/local.json [--protocol DIR] [--task ID | --all] [--arms plain,graphify,context] [--skip-review]
 // --protocol selects a directory holding protocol.json and context-instructions.txt (default: this directory).
+// protocol.json may set "codeAcceptance": "checker-and-scope" to accept code tasks without the model reviewer.
 // local.json (private, machine-specific): { evidence, roots: { context, kithmoot }, node, contextCli, graphifyBin, graphifyAlwaysOn }
 import { spawn, spawnSync } from 'node:child_process'
 import { createHash } from 'node:crypto'
 import { appendFileSync, existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'
 import { dirname, join, resolve } from 'node:path'
 import { fileURLToPath } from 'node:url'
+import { checkScope } from '../code-acceptance-20260923/scope.mjs'
 
 const here = dirname(fileURLToPath(import.meta.url))
 const packDir = join(here, '..', 'd5-20260921')
@@ -310,7 +312,14 @@ async function runArm({ taskId, arm, orderIndex, local, evidence, skipReview })
   log(evidence, `${taskId}/${arm}: checker ${checker.passed ? 'passed' : 'failed'}`)
   write()
 
-  if (!skipReview) {
+  // protocol.codeAcceptance 'checker-and-scope' (optional) accepts code tasks on the behaviour checker plus a
+  // deterministic scope check, with no model reviewer; protocols without it keep the reviewer.
+  const deterministicCode = acceptance.kind === 'code' && protocol.codeAcceptance === 'checker-and-scope'
+  if (deterministicCode) {
+    receipt.scope = checkScope({ workspace, include: task.selectionPolicy.include, env: { ...cleanEnv(), ...gitEnv } })
+    log(evidence, `${taskId}/${arm}: scope ${receipt.scope.passed ? 'passed' : `failed (${receipt.scope.reasons.join('; ')})`}`)
+  }
+  if (!skipReview && !deterministicCode) {
     const rprompt = reviewerPrompt({ task, acceptance, workspace, answer, checker, diff })
     const attempts = []
     for (let attempt = 1; attempt <= 2; attempt += 1) {
@@ -326,7 +335,8 @@ async function runArm({ taskId, arm, orderIndex, local, evidence, skipReview })
     const last = attempts[attempts.length - 1]
     receipt.reviewerRun = { ...last, attempts: attempts.length, seconds: attempts.reduce((a, r) => a + r.seconds, 0), inputTotal: attempts.reduce((a, r) => a + (r.inputTotal ?? 0), 0), output: attempts.reduce((a, r) => a + (r.output ?? 0), 0), allAttempts: attempts.map(r => ({ subtype: r.subtype, inputTotal: r.inputTotal, output: r.output, seconds: r.seconds, parsed: Boolean(r.verdict) })) }
   }
-  receipt.accepted = checker.passed && receipt.reviewerRun?.verdict?.accepted === true && (receipt.reviewerRun?.verdict?.materialIssues?.length ?? 1) === 0
+  receipt.acceptanceRule = deterministicCode ? 'checker-and-scope' : 'checker-and-reviewer'
+  receipt.accepted = deterministicCode ? checker.passed && receipt.scope.passed : checker.passed && receipt.reviewerRun?.verdict?.accepted === true && (receipt.reviewerRun?.verdict?.materialIssues?.length ?? 1) === 0
   receipt.armSeconds = (receipt.setup.graphify?.seconds ?? 0) + exec.seconds + checker.seconds + (receipt.reviewerRun?.seconds ?? 0)
   receipt.finishedAt = now()
   writeFileSync(receiptPath, JSON.stringify(receipt, null, 2))

docs/experiments/graphify-20260922/summarise.mjs

@@ -16,7 +16,7 @@ for (const task of readdirSync(evidence, { withFileTypes: true }).filter(d => d.
     const r = JSON.parse(readFileSync(path, 'utf8'))
     const e = r.executorRun ?? {}, v = r.reviewerRun ?? {}
     rows.push({
-      task, arm, order: r.orderIndex, accepted: r.accepted, checker: r.checker?.passed ?? null, reviewer: v.verdict?.accepted ?? null,
+      task, arm, order: r.orderIndex, accepted: r.accepted, checker: r.checker?.passed ?? null, reviewer: v.verdict?.accepted ?? null, scope: r.scope ? r.scope.passed : null,
       subtype: e.subtype, turns: e.numTurns, toolCalls: e.toolCallsTotal, toolCallsByName: e.toolCalls,
       inputTotal: e.inputTotal, inputUncached: e.inputUncached, cacheRead: e.usage?.cache_read_input_tokens ?? null, output: e.output,
       costUsd: e.totalCostUsd, executorSeconds: e.seconds, reviewerInput: v.inputTotal, reviewerOutput: v.output, reviewerSeconds: v.seconds,
@@ -26,7 +26,7 @@ for (const task of readdirSync(evidence, { withFileTypes: true }).filter(d => d.
 }
 const fmt = (n) => n === null || n === undefined ? '?' : typeof n === 'number' ? (Number.isInteger(n) ? n.toLocaleString('en-GB') : n.toFixed(1)) : String(n)
 const lines = ['| Task | Arm | Order | Accepted | Checker | Reviewer | Turns | Tool calls | Input total | Uncached input | Output | Cost est. USD | Executor s | Reviewer in/out | Arm s |', '| --- | --- | ---: | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |']
-for (const r of rows) lines.push(`| ${r.task} | ${r.arm} | ${r.order} | ${r.accepted ? 'yes' : 'no'} | ${r.checker ? 'pass' : 'fail'} | ${r.reviewer === null ? '?' : r.reviewer ? 'accept' : 'reject'} | ${fmt(r.turns)} | ${fmt(r.toolCalls)} | ${fmt(r.inputTotal)} | ${fmt(r.inputUncached)} | ${fmt(r.output)} | ${r.costUsd === null ? '?' : r.costUsd.toFixed(2)} | ${fmt(r.executorSeconds)} | ${fmt(r.reviewerInput)}/${fmt(r.reviewerOutput)} | ${fmt(r.armSeconds)} |`)
+for (const r of rows) lines.push(`| ${r.task} | ${r.arm} | ${r.order} | ${r.accepted ? 'yes' : 'no'} | ${r.checker ? 'pass' : 'fail'} | ${r.scope !== null ? (r.scope ? 'scope ok' : 'scope fail') : r.reviewer === null ? '?' : r.reviewer ? 'accept' : 'reject'} | ${fmt(r.turns)} | ${fmt(r.toolCalls)} | ${fmt(r.inputTotal)} | ${fmt(r.inputUncached)} | ${fmt(r.output)} | ${r.costUsd === null ? '?' : r.costUsd.toFixed(2)} | ${fmt(r.executorSeconds)} | ${fmt(r.reviewerInput)}/${fmt(r.reviewerOutput)} | ${fmt(r.armSeconds)} |`)
 const agg = {}
 for (const arm of arms) {
   const set = rows.filter(r => r.arm === arm)

docs/experiments/reviewer-agreement-20260923/README.md

@@ -0,0 +1,42 @@
+# Reviewer agreement probe (design, not yet run)
+
+The dimension rescore found that no arm separates and that one task's verdicts
+swing between runs, without being able to say whether the executor or the
+reviewer moved. This probe holds the answers fixed and re-reviews each of the
+80 recorded answers (v1, v2, S5, v3) three times under two conditions:
+
+- **replay**: the recorded reviewer prompt, byte for byte, with the recorded
+  schema;
+- **fixed**: the same prompt plus an explicit dimension list, enforced by the
+  schema, so every review reports the same dimension ids.
+
+No executor runs. Design, measures, decision rule and cost basis are in
+`protocol.json`, which is locked (`lockedAt`) before the first model call.
+
+```sh
+node docs/experiments/reviewer-agreement-20260923/agree.mjs \
+  --evidence v1=/private/v1 --evidence v2=/private/v2 \
+  --evidence s5=/private/s5 --evidence v3=/private/v3 \
+  --out /private/reviewer-agreement --estimate
+```
+
+Drop `--estimate` to run, add `--conditions replay` for the cheaper half, and
+use `--summarise` to rebuild the report from finished reviews. Finished
+reviews are never repeated, so an interrupted run resumes where it stopped. A
+provider or quota failure stops the probe without retry.
+
+## Cost estimate
+
+From the 80 recorded reviews (client-reported, Sonnet 5, high effort), with
+`--estimate` reproducing these figures:
+
+| Scope | Reviews | Client-reported cost | Input (uncached) | Output | Review time |
+|-------|--------:|---------------------:|-----------------:|-------:|------------:|
+| replay and fixed, 3 repeats | 480 | about $65 | 13.8M (7.6M) | 3.3M | 8.9 h serial, about 3 h at concurrency 3 |
+| replay only, 3 repeats | 240 | about $32 | 6.9M (3.8M) | 1.7M | 4.5 h serial, about 1.5 h |
+
+The client's cost is an API-price estimate; on a subscription the reviews draw
+on the Sonnet weekly quota instead. The pipeline was exercised end to end
+against a stub reviewer (no model calls) before this was committed.
+
+Results, when present, are in `RESULTS.md`. `../reviewer-evidence-20260923/RESULTS.md` already shows, without model calls, that at least three of the five v1 to v2 flips on unchanged arms came from the reviewer.