Calibration kit under tools/calibration/ (onboarding proposal Items 2-4.2): protocol template, classification, prompts with the forbidden prong, history-less checkout, manifest-driven grading, observations builder, workflow template, request printer, pack format by StepenkoAnatoli · Pull Request #106 · StepenkoAnatoli/SmartRouter

StepenkoAnatoli · 2026-10-18T21:58:26Z

Problem and resulting behavior

The calibration method behind runs 1-3 was reproducible in principle and not portable in practice: the workflow script, the graders' prompts, the observations builder, the grading steps and the checkout script lived in one host's session scratchpad and reached the evidence folder by hash or by copy. The onboarding proposal (docs/proposals/calibration-first-onboarding.md, #102) Items 2-4.2 ask for the kit under one path, host-independent in method, enforcing the run protocol and printing the results pull request for the operator to open.

This PR adds tools/calibration/, built from run 3's actual artefacts (the only complete lineage), each file with a lineage header:

README.md (how a host onboards a profile; the run protocol Items 3.1-3.5 as numbered rules), protocol.template.json (run 3's shape, values blank: cases with pass rules and tier, classification.prongs, judging_profile, relayed_operator_text, run_id null until the launch, amendments), classification.md (the three prongs, the S4 rule, G-1 and G-2 as the rules they produced).
grader.prompt.md, lens.prompt.md, cluster.prompt.md with their output schemas; the lens schema carries forbidden and a lens votes real only on all three prongs (G-2 closed).
historyless-checkout.sh (one root commit per cell; --link NAME=DIR for dependency trees, refusing a directory that reaches the pack or the source repository), grade.sh (mechanical pin and fix grading, every value from the pack manifest; each swapped source gets its own mtime so a cache keyed on mtime and size never runs the previous version), observations.py (the flat profile map the adapter's rate reads, scores.json with totals.tokens, adjudication_schema, category_mapping, would_fail_under_three_prong, the tables), run.template.js (the workflow for the Claude Code workflow harness, harness-specific parts marked; the candidate preamble that reconciles a relayed operator request; launch refusals for an unfrozen protocol, an unrecorded relayed text, a placeholder judging qualification or any placeholder left in the protocol, keys included, and a checkout that shares a path with the pack), request.py (prints the branch, the files and the pull-request body in this repository's template and asks the operator to open it; never runs git), subject-pack/README.md (the pack format: manifest with the base and accepted references, ground truth, required behaviour stated without the fix, pass rules per case and tier, source hashes, the isolation rule, the pack version) and the shared-pack stub naming the unit that builds it.
tools/test_calibration_kit.py (57 tests) registered in tools/run_regression.py after test_claude_code_host_artifacts; synthetic fixtures under tests/fixtures/calibration/; one CHANGELOG line and one README row.

The kit contains no provider calls, no keys and no host paths; its own test fails on an absolute path, a session id or a run id under tools/calibration/.

Built as the host unit sr-kit through the Claude Code adapter (cc-opus-high, decision seq 2, dispatch seq 3), refute-first reviewed: review 1 found F1 (S2, stale bytecode after a same-size source swap could grade a correct cell FAIL), F2 (S2, --link could reach the pack's accepted fix through a symlink) and F3 (S3, launch refusals passed on the template's own placeholders); fix round 1 closed them; re-review 1 found N1 (S3, the placeholder scan skipped object keys) and N2 (S4, / slipped past the path guard); fix round 2 closed them; re-review 2 found N3 (S4, a leading // or a . segment spelled a path past the guards), fixed by the lead at integration (229aa1f). Reports on the host branch: docs/orchestration/2026-10-13-github-sign-in/reports/sr-kit-*.md.

Related issue

Related to #102 (the proposal; Items 2-4.2), #101 (run 3, the lineage), #105 (the host adapter at revisions 5 and 6, whose rate reads the kit's observations). No issue number. The shared synthetic subject pack, the CI check (check_calibration_evidence) and the generated qualification table are the next units.

Validation

Run from the repo root on this branch at 229aa1f (Linux, Python 3.12 and 3.11, Node 24.21.0):

python3 tools/test_calibration_kit.py: 57 tests OK; every test shown red first (six red-first runs in the build, 4/2/4 and 1/2 and 3 failures before each fix) with 72 + 14 + 3 + 2 guard-removal proofs.
python3 tools/run_regression.py: "REGRESSION GATE: all checks PASS", standalone and through the pre-commit hook on every commit.
python3 tools/check_secrets.py: clean; the hygiene grep for /home/, /tmp/, session_, wf_, sk- and the MoonAliza file names under tools/calibration/ is empty.
node hosts/claude-code/route.mjs rate --trace <temp> --policy hosts/claude-code/host-policy.json on the observations built from the fixtures prints CALIBRATION lines (the reviewer's run).
The filled workflow template runs under Node with stub harness globals (every launch refusal, the cells, the prompts, the three-prong vote).

Remaining limits or risks

run.template.js has run only under the stub harness; its first real run is calibration run 4 (stage 4a), which is where its harness-specific parts are proven.
The F1 fix was tested with Python's bytecode cache only; another test runner's cache keyed the same way gets the same 2-second mtime step.
A --link NAME that already exists in the base tree (src, .git) puts the link inside that directory; only the host chooses NAME, as before.
The shared pack directory holds a README stub only; a host onboarding today runs its own host pack.
Draft for the reviewer.

Generated by Claude Code

…re journal and run file What changed: tools/calibration/observations.py builds a calibration run's results from the harness journal, the run file, a subject-pack manifest and the grades directory (optional --protocol for the profile order, the rates and the declared prongs; --void and --supplementary for the Item 3.2 re-run of a void cell). It writes observations.json (the flat profile map hosts/claude-code/route.mjs rate reads, each task object carrying tier), scores.json (totals.tokens, adjudication_schema, category_mapping, would_fail_under_three_prong), results-table.md, clusters.md, tokens-and-estimates.md, adjudication.json and void-cells.json. The harness journal format is documented at the top of the file. A cell that ran without a grade is written as ungraded (score null), never as not run. tools/test_calibration_kit.py starts with ten tests on the fixtures under tests/fixtures/calibration/ (manifest, protocol, journal, run file, two grades; three cells, synthetic). Why: onboarding proposal Item 2 (#102) asks for the builder every run so far rebuilt by hand; this generalises run 3's builder (its profile list, rates, ground-truth field names and paths come from the protocol and the manifest), and applies G-2: a lens record without the forbidden prong never carries a three-prong pass. What it touched: tools/calibration/observations.py, tools/test_calibration_kit.py, tests/fixtures/calibration/{manifest.json, protocol.json, journal.jsonl, run.json, grades/pin-alpha-low.json, grades/fix-alpha-low.json}. Not yet in the regression gate (the registration is this unit's last commit). What I verified: python3 tools/test_calibration_kit.py red before the builder existed (9 tests: failures=8, errors=1) - verified; green after (10 tests OK, Python 3.11.15, Node 22 for the adapter test) - verified; ten guard-removal proofs (each guard removed in a scratch edit: the review-grade key, the pin score, the token sum, the forbidden prong, the void refusal, the supplementary replacement, the PARTIAL rule, the missing-journal refusal, the rates, the void-cells.json write) each turn their tests red and green again on restore - verified; node hosts/claude-code/route.mjs rate on the built map counts pin-tests, fix and review per profile - verified (it is one of the tests); the regression gate through the pre-commit hook on this commit - verified by this commit existing. What I got wrong: nothing to report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

…a pack manifest, with smoke tests What changed: tools/calibration/historyless-checkout.sh (SOURCE_REPO COMMIT TARGET BRANCH, with --link NAME and --pack DIR) writes one root commit holding the base tree in a fresh repository, copying the tree and its blobs as git objects (no commit of the source is copied, and export-ignore cannot change the tree), links dependency trees only on request and hides them in .git/info/exclude, and refuses an existing target or one that shares a path with the pack. tools/calibration/grade.sh (MANIFEST pin|fix CHECKOUT) grades a pin or fix cell with every value run 3 hard-coded read from the manifest (revisions, source and test paths, the scope rule, the test command and its report mode exit-code or vitest-json, the lint command), prints the grade to stdout, restores the swapped source files and says whether it did, and refuses a checkout that shares a path with the pack (Item 2). Eleven smoke tests on temporary repositories and a temporary pack (a two-line module, a one-row test, the accepted fix). Why: proposal Item 2's history-less checkout and grader, host-independent: run 3's scripts carried host paths, a fixed source file and fixed commits. What it touched: tools/calibration/historyless-checkout.sh, tools/calibration/grade.sh, tools/test_calibration_kit.py (the Scripts class and its git helpers; tests run git with an isolated config so a machine's signing or hooks path does not change them). What I verified: the Scripts tests red before the scripts existed (10 tests, failures=10) - verified; green after (11, OK), and the whole file 21 OK standalone and with HOME at the repository root as the hook sets it - verified; bash -n on both scripts - verified; fifteen guard-removal proofs, each red then green on restore: a syntax error in each script, run 3's git archive copy (drops an export-ignore file, so the tree check fails), a fetch of the source commit into the checkout, the info/exclude line, the --pack refusal, the existing-target refusal, the fix scope rule, the pin source-clean rule, the pin red-at-head rule, the restore, the base swap, the accepted swap, the pack-path refusal in grade.sh, the counting-report parser - verified; the gate through the pre-commit hook - verified by this commit existing. The vitest-json mode is exercised with a stand-in command that writes vitest's counters, not with vitest itself - untested against a real vitest. What I got wrong: mistake: the pin-green-at-the-base test was added with an Edit that did not apply (the file had changed on disk), so its first guard proof ran against a missing test (ERROR both mutated and restored); where: tools/test_calibration_kit.py, Scripts; impact: none delivered, the proof log showed it; cause: the edit tool refused a stale read; fix: re-added the test by script and re-ran that guard; verified: the guard goes red (failures=1) and green on restore. The test was written after the rule existed, so its red is shown by guard removal only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

…lens and cluster prompts, with the template and hygiene tests What changed: protocol.template.json is run 3's protocol shape with the values blank: subject pack (with the manifest hash), profiles, cases pin/fix/review with category, tier and pass rule (the pin case carries the host category pin-tests, run 3 A-2), sources and the hash convention, metrics, accounting with per-model rates, timing, privacy, limits, classification.prongs (holds, reachable, forbidden), judging_profile with its qualification or the bootstrap limitation line, relayed_operator_text (null with the statement that the harness relays nothing, or the text verbatim), usage-window readings, run_id null until the launch, amendments []. classification.md holds the three-prong Definition (inserted verbatim into the grader and lens prompts), the S4 rule, and G-1 and G-2 as rules. grader.prompt.md, lens.prompt.md and cluster.prompt.md hold each prompt with its output schema in a fenced JSON block and a documented {{placeholder}} list; the lens schema carries forbidden and a lens votes real only on all three prongs; the cluster prompt carries the supplementary run's assignment variant. Tests: the template parses and carries every key, the fixture protocol is a filled template, the classification sections, the prompt schemas and placeholders, the lens vote, and the K3 hygiene scan (no host path, session or run id, MoonAliza name, environment read or provider call anywhere under tools/calibration/) with a self-test that the scan catches each kind. Why: proposal Item 2's protocol template, classification and prompts, with G-2 closed (the forbidden prong in the lens schema and the vote) and the bootstrap judging limitation and the A-1 relayed-text statement as template fields (Items 3.2-3.3). What it touched: tools/calibration/{protocol.template.json, classification.md, grader.prompt.md, lens.prompt.md, cluster.prompt.md}, tools/test_calibration_kit.py (TemplateAndPrompts, Hygiene), tests/fixtures/calibration/protocol.json (now a filled copy of the template: manifest_sha256, lens_schema, usage_window_readings added). What I verified: the new tests red before the files existed (7 tests: errors=6; the scanner self-test passed, it does not depend on the kit files) - verified; the filled-template test red before the fixture was brought in line - verified; green after, the whole file 29 OK - verified; seventeen guard-removal proofs, each red then green on restore (a template key, run_id non-null, the forbidden prong, the limitation line, the relayed statement, the pin-tests category, the S4 sentence, the G-2 rule, forbidden in the lens schema and in its required list, the three-prong vote sentence, a placeholder's documentation, the grader's truth field, the assignment variant, a host path and a run id written into a kit file, a hygiene pattern) - verified; the regression gate through the pre-commit hook - verified by this commit existing. What I got wrong: mistake: the first hygiene run read a git-ignored __pycache__ file under tools/calibration/ (UnicodeDecodeError); where: tools/test_calibration_kit.py, Hygiene.scan; impact: a red test, nothing delivered; cause: the observations test imports the builder, which writes bytecode beside it; fix: the scan skips __pycache__ paths; verified: Hygiene 2 OK. Mistake: the lens prompt documented three placeholders on one bullet; where: tools/calibration/lens.prompt.md, Placeholders; impact: the documentation test went red as designed; fix: one bullet per placeholder; verified: TemplateAndPrompts OK. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

…amble, and the request printer What changed: run.template.js is the calibration workflow for the Claude Code workflow harness, its harness-specific lines marked // HARNESS:. It is filled before the freeze with the run config (the pack manifest, the frozen protocol, the host's paths) and the prompt files, and it refuses at the launch, before any agent starts, when the protocol is not frozen, retry_limit is not 0, the relayed operator text is not recorded, the prongs are not the three, the judging profile lacks its qualification or the bootstrap line, or a candidate or lens checkout shares a path with the pack (Items 2, 3.1-3.4). It then runs pin and fix cells per profile and review cells per profile and review-only profile from the manifest, with the candidate preamble that reconciles a relayed operator request (A-1) and quotes a recorded one verbatim. Reviews are graded as they finish (pipelined), the other S1-S3 findings are clustered, and three lenses judge each cluster, a lens voting real only on all three prongs (G-2). Its supplementary mode re-runs void cells, uses the assignment prompt and keeps the main run's verdicts. request.py prints the branch, the file list, a title and the pull-request body in .github/pull_request_template.md's four sections, filled from protocol.json and scores.json (two-prong PASSes named as such, the judging qualification or limitation, the relayed-text statement), and ends by asking the operator to open the pull request; it runs no command. The fixture manifest gains run_one, scratch_test, commit_rule and the pin and fix instructions the candidate prompts read. Why: proposal Item 2 (the workflow script with the A-1 preamble) and Item 4.2 (the printed request the operator opens). What it touched: tools/calibration/run.template.js, tools/calibration/request.py, tools/test_calibration_kit.py (RunTemplate: the filled template executed under Node with stub agent, pipeline, parallel and log; Request), tests/fixtures/calibration/manifest.json. What I verified: the eleven new tests red before the files existed (failures=3, errors=8) - verified; green after, the whole file 40 OK - verified; py_compile on the kit's Python and the test - verified; seventeen guard-removal proofs, each red then green on restore (the RUN slot, the pack-path refusal, the relayed-text refusal, the frozen refusal, the prongs refusal, forbidden in the vote, the A-1 preamble, the quoted relayed text, the judging profile, the Definition in the grader prompt, the supplementary filter, the kept verdict, the operator sentence, a template section, the two-prong naming, the missing-scores refusal, a subprocess import) - verified; the regression gate through the pre-commit hook - verified by this commit existing. The template has been executed only under the stub harness, never in the real workflow harness - untested there; the first real run is its test. What I got wrong: mistake: the template's header quoted the fill slots literally, so a fill would replace the comment as well (the slot test counted two of each); where: tools/calibration/run.template.js header; impact: none delivered; fix: the header describes the slots without their literal text; verified: the slot test green. Mistake: request.py named only the first missing input; fix: it names every missing one; verified: the refusal test green. Mistake: the relayed-text guard proof first stayed green, because a second check refuses the same input; impact: the test did not pin the Item 3.2 refusal; fix: the test asserts that refusal's own message and also drops the whole record; verified: the guard goes red and green on restore. Mistake: two test expectations of mine were wrong (the fixture's required behaviour reads "add(a, b) answers a + b"; the template writes "Retry limit 0" with a capital); fix: the expectations corrected; verified: green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

…d-pack stub, and the CHANGELOG and README lines What changed: tools/calibration/README.md says how a host onboards a profile in nine steps (pack, protocol freeze, checkouts, fill, launch, grade, build, results, request) with the commands, the run protocol Items 3.1-3.5 as numbered rules, the run-id and amendment conventions, every key of observations.json the adapter reads and the keys of scores.json, and the kit's limits. subject-pack/README.md is the pack format: every manifest field (ground truth with severities and causes, the required behaviour stated without the fix, pass rules and tier per case, the paths and commands grade.sh reads, source hashes, the version) and the isolation rule of Item 2. subject-pack/shared/README.md is a stub naming the unit that builds the shared pack. run.template.js's header now says the protocol goes into the fill without its sources (they record the filled script's hash, which would otherwise depend on itself). CHANGELOG Unreleased gains one line; the README's documentation table gains one row. Why: proposal Item 2 (one path, a documented method a second host can follow) and K2's README and pack format; K5's two lines. What it touched: tools/calibration/{README.md, subject-pack/README.md, subject-pack/shared/README.md, run.template.js (header comment only)}, tools/test_calibration_kit.py (Docs), CHANGELOG.md, README.md. What I verified: the nine Docs tests red before the files existed (failures=2, errors=7) - verified; green after, the whole file 51 OK - verified; ten guard-removal proofs, each red then green on restore (a lineage header, a listed file, a documented observation key, a numbered protocol rule, an onboarding step, the README's fill command (its output must equal the test's fill byte for byte), a documented manifest field, the stub's sentence, the CHANGELOG line, the README row) - verified; the regression gate through the pre-commit hook, including test_docs_drift and validate_skills (untouched) - verified by this commit existing. What I got wrong: mistake: the README's first launch steps would have recorded the filled script's hash in a protocol the script embeds whole, a hash that depends on itself; where: tools/calibration/README.md step 4 and the run.template.js header; impact: none delivered (caught while writing the steps); cause: the fill took the protocol verbatim; fix: the fill takes the protocol without its sources, said in both places; verified: the fill-command test green. Mistake: two first versions of my own tests were wrong (the lineage check was case-sensitive though the JSON template carries the header as its _lineage key; the fill-command extraction matched from the earlier hash one-liner and kept the list indentation); fix: a case-insensitive check, and the fenced block taken and dedented; verified: both green, and both guard proofs red then green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

…he K3 scan also takes the literal /home/ and /tmp/ strings What changed: tools/run_regression.py gains one CHECKS entry, test_calibration_kit (python3 tools/test_calibration_kit.py), right after test_claude_code_host_artifacts; tools/test_calibration_kit.py gains the test that pins that place, and its K3 hygiene scan gains the literal rule K3 states (no /home/ or /tmp/ string anywhere under tools/calibration/ outside a placeholder, where the path rule only matched at a path's start), one hit per line, with a self-test sample only that rule catches. Why: K4: the kit's tests run on every commit through the pre-commit hook and in CI's gate job, so a change that breaks the kit stops the line. The scan change makes the K3 assertion say exactly what K3 says. What it touched: tools/run_regression.py (one line), tools/test_calibration_kit.py (Gate; Hygiene). What I verified: the registration test red before the line existed (test_calibration_kit not in CHECKS) and green after - verified; guard removal of the line: red, then green on restore - verified; with the forbidden prong removed from observations.py the full gate stops at test_calibration_kit (REGRESSION GATE: 1 failure(s): [('test_calibration_kit', 1)]) and passes again on restore - verified; python3 tools/run_regression.py standalone: REGRESSION GATE: all checks PASS in 66 s, the kit's 50 tests about 5 s of it - verified; the literal rule: a mid-word /tmp/ written into a kit file turns the kit scan red, and removing the rule turns the self-test red (after the self-test gained a sample only it catches; before that the removal stayed green) - verified; grep finds no /home/ or /tmp/ under tools/calibration/ - verified; the 50 tests on Python 3.12.3 as well as 3.11.15 - verified; python3 tools/check_secrets.py clean, py_compile on the kit's Python and bash -n on both scripts exit 0 - verified; the gate through the pre-commit hook on this commit - verified by this commit existing. Run here on Python 3.11.15 and Node 22; CI's Python 3.12 job - expected green, not run; where node is absent the seven Node-backed tests skip and say why - expected, not run here. What I got wrong: mistake: my first version of this message counted 52 kit tests (and the previous commit's message says 51); where: the commit reports of 3eb0e2c and this commit; impact: two wrong numbers in commit messages, no code; cause: I added test counts by hand instead of reading the runner's line; fix: this message amended to 50 (49 before the Gate test; 3eb0e2c's 51 stays as written and is corrected in the unit report); verified: python3 tools/test_calibration_kit.py prints Ran 50 tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

…f its own, so a cache keyed on mtime and size never runs the previous version What changed: grade.sh records each manifest source file's mtime before grading; after every put() and every intermediate restore() it sets the swapped files' mtime to a fresh value two seconds past the last one (os.utime with ns, from a clock that starts at max(now, the original mtimes)), and the final restore puts the original mtimes back; `restored` now also requires the original mtimes. The test module's git_env no longer sets PYTHONDONTWRITEBYTECODE for grade.sh (it pops it and PYTHONPYCACHEPREFIX, so the cache is on as in a host's shell), and two tests are added: a same-size swap (a - b against a + b) graded three times with a warmed __pycache__, for fix and pin; and a deterministic one whose test command logs the source's mtime at each run, asserting every run's mtime is at least 2 s from every other and the original mtime is back. Why: review finding F1 (S2). grade.sh swapped sources in place, keeping the size and moving the mtime by under a second, so Python's bytecode cache (mtime in whole seconds plus size) ran the previous version and graded a correct candidate FAIL depending on timing; the kit's test hid it by turning bytecode off. Set aside: invalidating caches per runner (deleting __pycache__, PYTHONDONTWRITEBYTECODE in the test command) - runner-specific, while the mtime bump works for any mtime-and-size cache. What it touched: tools/calibration/grade.sh (swap, restore, the restored check, the header), tools/test_calibration_kit.py (git_env, show_bytes, Scripts: two tests). What I verified: red first - with the cache on, the current grade.sh failed the two new tests and test_grade_fix_green_candidate_passes and test_grade_pin_red_then_green_with_the_accepted_fix_passes (failures=4; the reviewer's scratch reproduction gave 3, 3 and 2 failures over three runs here) - verified; green after the fix, Scripts 13 OK three runs in three - verified; guard removal, each restored byte-identical: the stamp in put() (5 red), the stamp in the intermediate restore() (the mtime test red), the final mtime restore (the mtime test red), a 1 us step instead of 2 s (5 red) - verified; python3 tools/test_calibration_kit.py: Ran 52 tests, OK - verified; bash -n grade.sh - verified; the gate through the pre-commit hook - verified by this commit existing. Python 3.11.15, Node 22, cwd /home/user/task5-handoff/wt/sr-kit. A future-dated mtime during a grade is harmless to Python's cache (an equality check) - expected, not run against other runners. What I got wrong: nothing to report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

… and refuses a dependency directory that reaches the pack What changed: --link now takes NAME=DIR and symlinks the real path of DIR into TARGET/NAME (still listed in .git/info/exclude). It refuses, before the target is made, a bare NAME (the old form, which linked SOURCE_REPO/NAME), a missing DIR, and a DIR whose physical path (cd + pwd -P, so a symlink is resolved) is inside SOURCE_REPO or the --pack directory or holds either; SOURCE_REPO is guarded with or without --pack. The pack and target checks share one within() helper. The kit README (step 3), the subject-pack README (Isolation and checkout.links) and the launch step of run.template.js now say that dependencies are installed outside the pack, that --link refuses a directory reaching it, and that symlinks inside a dependency tree, and checkouts made another way, are the host's to check (the harness script reads no filesystem, so its path check compares strings). Why: review finding F2 (S2). --link NAME linked TARGET/NAME to SOURCE_REPO/NAME, the pack repository holding the accepted fix, and neither --pack nor the launch check followed the link, so `git -C node_modules/.. show HEAD:<source>` printed the accepted fix from inside any checkout made with --link (reproduced here before the change). Set aside: copying the dependency tree into each checkout - safe but slow and large for a Node tree, and the copy still lands in the candidate's paths; scanning every symlink inside DIR - node_modules holds thousands, and the README names it the host's duty instead. What it touched: tools/calibration/historyless-checkout.sh, tools/calibration/README.md, tools/calibration/subject-pack/README.md, tools/calibration/run.template.js (comment only), tools/test_calibration_kit.py (Scripts: the link test uses a directory outside the pack and checks where the link resolves; a new test with five refusals). What I verified: red first - the new tests against the committed script: failures=2 (the link test, and the refusal test where the script linked instead of refusing) - verified; green after the change - verified; guard removal, each restored byte-identical: the NAME=DIR requirement (red after the test was tightened to assert each refusal's own reason), the whole inside/holding refusal, its holding-the-pack half, the SOURCE_REPO guard without --pack, the physical path (pwd -L instead of -P) - each red, then green - verified; the reviewer's reproduction re-run: --link node_modules refused (exit 2), --link node_modules=<pack>/repo/node_modules refused (exit 2), --link node_modules=<outside dir> ok, and `git -C node_modules/.. log` then finds no repository - verified; python3 tools/test_calibration_kit.py: Ran 53 tests, OK - verified; bash -n - verified; the gate through the pre-commit hook - verified by this commit existing. Python 3.11.15, Node 22, cwd /home/user/task5-handoff/wt/sr-kit. What I got wrong: mistake: the first refusal test asserted only that the message named --link, so removing the NAME=DIR guard stayed green (the bare form was then refused by the SOURCE_REPO check instead); where: test_historyless_checkout_refuses_a_link_that_reaches_the_pack; impact: one guard unpinned, nothing committed; cause: an assertion looser than the refusal it pins; fix: each case asserts its own reason; verified: that guard removal red, then green. Also, mistake: my first re-run of the reviewer's reproduction omitted the BRANCH argument and got the usage error; where: a scratch command; impact: none; cause: a dropped argument; fix: re-run with it; verified: the results above. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

…cation, a limitation that is not the bootstrap line, a missing relayed statement, and any placeholder left in the protocol What changed: the launch step of run.template.js now (1) refuses a relayed_operator_text without a filled statement (Item 3.2); (2) treats a qualification that is empty, not a string or a {{placeholder}} as no qualification, and then requires the limitation to begin with "judging profile not independently qualified" (Item 3.3), where it used to accept any non-placeholder text and took a placeholder qualification as set; (3) scans the whole protocol recursively and refuses any string still holding '{{', naming its path (protocol.profiles[0].key); the relayed text itself is the operator's words verbatim and is not scanned. The specific checks run before the scan so that each has its own message. The kit README's step 5 and the template header list the new refusals. Why: review finding F3 (S3). The README's "fill every placeholder" rule and Item 3.3's refusal were not enforced: under the stub harness, a template-placeholder qualification and limitation, a null qualification with the limitation "trust me", a relayed record without its statement, and the template's unfilled profiles each launched 10 agents (reproduced here before the change). Set aside: a separate launch.py that validates the protocol on disk - K2 lists no launch file, and the refusal belongs in the step that starts agents. What it touched: tools/calibration/run.template.js (launch step, header), tools/calibration/README.md (step 5), tools/test_calibration_kit.py (RunTemplate: a launch() helper and four tests). What I verified: red first - the four new tests against the committed template: failures=4 - verified; green after - verified; guard removal, each restored byte-identical: the statement check, the placeholder-qualification-as-unset rule (reverted to `== null`), the bootstrap-line prefix, the scan, and the relayed-text exemption (after its case was added) - each red, then green - verified; the reviewer's four cases re-run under the stub harness: each refused with its own message and 0 agents - verified; the existing launch tests (frozen, prongs, relayed text, pack paths, supplementary) still green - verified; python3 tools/test_calibration_kit.py: Ran 57 tests, OK - verified; the gate through the pre-commit hook - verified by this commit existing. Python 3.11.15, Node 22, cwd /home/user/task5-handoff/wt/sr-kit. The real workflow harness has not run the template - untested, as before. What I got wrong: mistake: the relayed-text exemption from the scan first had no test, so its removal stayed green; where: test_launch_refuses_any_placeholder_left_in_the_protocol; impact: an unpinned branch, nothing committed; cause: I wrote the exemption after the tests; fix: a case where a relayed text containing '{{' launches; verified: the exemption's removal red, then green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

…es a {{placeholder}} left as an object key What changed: placeholderAt in run.template.js (the launch step's recursive scan) now returns the path when an object key contains '{{', checked before it recurses into the value; before, it tested string values only. The relayed-text exemption is unchanged and still applies first. Why: review finding N1 (S3). protocol.template.json ships accounting.rates_per_mtok keyed by '{{model alias}}'. Under the stub harness, the fixture protocol with rates_per_mtok.alpha renamed to '{{model alias}}', and one with a top-level key '{{extra}}', each launched 10 agents with no error. observations.py then wrote null USD estimates for the cells whose model had no rate. That contradicts README step 5, the template header and the F3 fix, which all say the launch refuses any {{placeholder}} left in the protocol. Nothing was set aside: the finding's fix is the one obvious remedy. What it touched: tools/calibration/run.template.js (placeholderAt), tools/test_calibration_kit.py (RunTemplate.test_launch_refuses_any_placeholder_left_in_the_protocol: two new cases). README step 5 already says "any {{placeholder}}" and needs no change. What I verified: red first - the two new cases against the committed template: failures=1, '...rates_per_mtok.{{model alias}}' not found in '' - verified; green after - verified; guard removal (the key check deleted, file restored byte-identical by cmp): red on the rates-key case, and with that case removed, red on the top-level '{{extra}}' case alone, then green - verified; python3 tools/test_calibration_kit.py: Ran 57 tests, OK - verified; the gate through the pre-commit hook - verified by this commit existing. Python 3.11.15, Node 22.22.0, cwd /home/user/task5-handoff/wt/sr-kit. The real workflow harness has not run the template - untested, as before. What I got wrong: nothing to report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

…tory as holding every path, so neither --link NAME=/ nor --pack / slips past its guards What changed: within() in historyless-checkout.sh returns 0 when its second argument is '/'. Before, it matched "$1/" against the pattern "$2"/*, which for '/' is '//*' and matches no physical path. That made the root count as holding nothing. Why: review finding N2 (S4, plainly right). --link node_modules=/ was accepted, with --pack and without it, and the accepted fix's commit was then readable through the link: git -C <checkout>/node_modules<pack>/repo log --all listed it. Reproduced here before the change. The same blind spot let --pack / accept any target. pwd -P ends in '/' only for the root, so the root is the only special case; nothing else was set aside. What it touched: tools/calibration/historyless-checkout.sh (within(), with its comment), tools/test_calibration_kit.py (Scripts: two '/' cases in test_historyless_checkout_refuses_a_link_that_reaches_the_pack - with --pack, and against SOURCE_REPO with no --pack - and a '--pack /' case in test_historyless_checkout_refuses_an_existing_target_and_one_inside_the_pack). What I verified: red first - against the committed script: failures=2 (the link case 'node_modules=/' answered ok; '--pack /' answered ok) - verified; green after - verified; guard removal (the '/' clause deleted, the file restored byte-identical by cmp): both tests red, then green - verified; the reviewer's reproduction re-run: both '=/' forms and '--pack /' refused with exit 2, and no target created - verified; python3 tools/test_calibration_kit.py: Ran 57 tests, OK - verified; bash -n - verified; check_secrets clean - verified; the gate through the pre-commit hook - verified by this commit existing. Python 3.11.15, cwd /home/user/task5-handoff/wt/sr-kit. What I got wrong: nothing to report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

…: historyless-checkout.sh collapses a leading '//' from pwd -P, and the launch's norm() collapses '//' and drops '.' segments What changed: historyless-checkout.sh gains phys() (pwd -P with a leading '//' collapsed), used for the source, the pack, every --link directory and the target's parent; run.template.js's norm() splits on '/', drops empty and '.' segments and rejoins; tools/test_calibration_kit.py adds the '//' and '/./' cases to the --link, --pack and launch refusals (red first, 3 failures). Why: re-review 2's N3 (S4): bash keeps a leading '//' from pwd -P, so '--link node_modules=//', '--pack //' and a //-spelled pack slipped past the N2 guard, and shares() missed '<root>//pack/co' and '<root>/./pack/co'. Fixed by the lead at integration, the unit's two fix rounds being spent. What it touched: the Item 2 isolation guards only; no prompt, schema, manifest field or output changes. What you verified: the three new cases red before the change and green after (verified); guard proofs: restoring the bare pwd -P turns two tests red, restoring the old norm() turns one red, each green on restore (verified); the kit's 57 tests OK; the regression gate through the pre-commit hook on this commit (verified by the commit landing). What you got wrong: nothing to report. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

…are absolute and resolved (review residual) What changed: one sentence in tools/calibration/README.md step 4 (Fill): checkout_root, adjudication_checkout and pack_dir are absolute and already resolved, because the launch step of run.template.js compares strings and follows no symlink and no "..". Why: the deeper re-review of #106 noted the string-only path check and asked the README to say so if it did not; it did not. What you verified: the regression gate through the pre-commit hook (verified, on this commit); no code changed. What you got wrong: nothing to report. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

StepenkoAnatoli and others added 12 commits October 18, 2026 20:39

StepenkoAnatoli marked this pull request as ready for review October 19, 2026 03:15

Merge remote-tracking branch 'origin/main' into tools/calibration-kit

54e1c6c

StepenkoAnatoli merged commit d729890 into main Oct 19, 2026
5 checks passed

StepenkoAnatoli deleted the tools/calibration-kit branch October 19, 2026 03:23

StepenkoAnatoli mentioned this pull request Oct 19, 2026

Claude Code adapter: refresh with the sr-kit, sr-pack and sr-evidence traces, the cc-opus-high headroom ratified to 1492000, and the usage-windows test pinned at the ratification instant #109

Merged

Navigation Menu

Repository navigation

Calibration kit under tools/calibration/ (onboarding proposal Items 2-4.2): protocol template, classification, prompts with the forbidden prong, history-less checkout, manifest-driven grading, observations builder, workflow template, request printer, pack format - #106

StepenkoAnatoli commented Oct 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

StepenkoAnatoli commented Oct 18, 2026

Problem and resulting behavior

Related issue

Validation

Remaining limits or risks

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant