evidence: Claude Code calibration run 3 (history-less checkouts, defined classification, preregistered adjudication)

[StepenkoAnatoli]

Problem and resulting behavior

Run 2 left two method gaps (its README "Results"): candidates could in principle reach the accepted answer in the subject's history, and the grader's classification of an "other" finding did not say whether reachability is part of a finding holding (G-1). Run 3 closes both and adds the preregistered cross-candidate step: history-less checkouts (one root commit), a three-prong classification (holds at HEAD, reachable through the public API with admissible input, forbidden by the spec), and one clustering agent plus three independent lenses per cluster whose majority is the scored false-blocking count.

This PR adds docs/evidence/2026-10-18/claude-code-calibration-3/: the preregistered protocol (frozen 05:21Z, source hashes, run id recorded before the launch), the results (observations.json as the flat profile map the host adapter's rate reads, void-cells.json, adjudication.json, scores.json, results-table.md, clusters.md), the protocol's status, supplementary run id, amendments A-1, G-2, N-1 and A-2, the usage-window readings, and the README results section with a run-by-run token reconciliation and the qualification consequences under the host's rules.

Results under the preregistered rules: pin 11/12 (fable-medium over-specified its pin), fix 12/12, review 15/15; under the full three-prong classification 13/15 (two haiku review cells are flagged would_fail_under_three_prong). Every review found both ground-truth defects with their causes; five clusters of other findings, all adjudicated real by the preregistered majority rule, two of them real on two prongs only (G-2, machine-readable in scores.json and adjudication.json).

Related issue

Related to #98 (calibration method and runs 1-2), #100 (difficulty tiers) and #102 (calibration-first onboarding). No issue number.

Validation

• Pin and fix graded mechanically by grade.sh on every cell against the accepted fix 348aa2d and the base e68583c (red at base, green with the fix; candidate green, guard red, accepted pin green, scope, lint).
• Reviews graded by the opus-high grader with the run's classification; others clustered and adjudicated by three opus-high lenses each; clusters.md lists every lens verdict.
• Token buckets re-derived from the two run files' per-agent counts: main 5,274,667 + 1,597,164 + 81,285 + 1,578,450 = 8,531,566; supplementary 272,858 + 107,524 + 75,994 = 456,376; both 8,987,942; judging 38-39%.
• The host copy of these files passes the adapter's rate (exit 0, CALIBRATION lines for every profile across the three runs) and its suite (93 pass).
• Pre-commit regression gate: all checks PASS on every commit.
• The two void cells were re-run in a supplementary workflow (wf_dbe29a64-3b2); both PASS; the main-run void cells remain in void-cells.json.

Remaining limits or risks

• Pending the delta review of fb4e30f; the first review (on a197b6e) requested changes on the token accounting, G-2's machine-readable form, the qualification consequences and the pin category mapping, all addressed in fb4e30f.
• A-1 contamination: the workflow harness relays the operator's latest chat message to every agent as the request that "wins" over the computed task. Two of 39 cells halted on it and were void; the re-run adds one reconciling sentence to the preamble (recorded verbatim in the amendment). The journal that shows the six mentions is host-local; void-cells.json records them.
• G-2: the lens schema carried two of the classification's three prongs; the preregistered rule stands for the scores and the two affected cells are flagged; any downstream table must not present them as ordinary passes. Run 4 adds the prong.
• The fix pass rule omits typecheck (self-reported), the pin rule does not enforce the two-row count, the supplementary review's findings were assigned to the main run's clusters by one agent, and the ground truth and prompts are host files recorded by hash (protocol.json review_notes_2026_10_18).
• No clean usage-window reading exists for this run; the coefficients are consistent with run 2's.
• Two ground-truth defects at one difficulty; no model ranking or savings claim follows (protocol limits). The host copy of the evidence is not in this PR; it reaches hosts/claude-code/ at the next adapter refresh.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

[StepenkoAnatoli]

Independent review — 2026-10-18 — Request Changes

Reviewed revision: a197b6ebb2544389334af7ff76788acba376f701 (protocol freeze 0013451; run-id 5533e47; results a3849c8, 38ca2c5; merge a197b6e; base ade3f1e). Reviewer: independent agent session; no authorship of this PR or the plan. Read-only.

Provenance: posted through the owner's connected GitHub account at the owner's request; the review was produced by a separate agent session that authored none of this PR. GitHub therefore shows the owner as the comment author; this is a review comment, not an approval.

Blocking

1. The token accounting is reconcilable only with labels the README does not give. It states 8,531,566 main-run tokens plus 456,376 supplementary, then gives scored cells 5,381,259 and judging 3,440,417; those sum to 8,821,676, which is 290,110 above the main total and 166,266 below the combined total. The reconciliation is: main items 5,274,667 + main graders 1,597,164 + main judging 1,659,735 = 8,531,566; supplementary cells 272,858 + supplementary judging 183,518 = 456,376; the two main-run void cells (91,080 + 75,186 = 166,266) are in neither bucket. Add a run-by-run reconciliation table and totals.tokens; name the supplementary fourth agent (75,994 tokens, consistent with the A-1 cluster-assignment agent, but absent from observations/scores); state “about 38-39% judging”.
2. G-2 is a known scoring defect not carried into machine-readable form. protocol.json G-2 says the lens schema lacks the forbidden-by-spec prong and that under the full three-prong definition haiku-medium (2 false) and haiku-high (1 false) would FAIL, not PASS. scores.json records PASS; observations.json records adjudicated: true for C-4/C-5 while the why prose says the spec permits the behaviour (the value is the string “true” in all 18 occurrences, while the adjacent clusterReal is a JSON boolean, so the field is not typed either). Add adjudication_schema/prongs and would_fail_under_three_prong, or rescore and record the FAILs. Any downstream qualification table must not present these as ordinary PASSes.
3. The run’s qualification consequences are not stated. Under host rule 3, a calibration fail blocks the category until two later passes; fable-medium is currently pin-tests-qualified, so a run-3 pin FAIL would block it. The same application changes positive review cells: the rate output shows at least two review passes for sonnet-high, sonnet-low, sonnet-xhigh, opus-max and fable-max. Add a “qualification consequences” section, positive and negative, or state explicitly that no policy change is made and the host must apply rule 3.
4. The pin category mapping is unresolved. The protocol labels the pin case build-tests; host rule 4 says the pin task represents pin-tests, not build-tests. Applied literally the FAIL is a build-tests fail and does not block pin-tests; under rule 4 it does. Define the mapping and make the proposed CI check use it.

Non-blocking: the “six agents mentioned the relayed message” claim is not supported by the committed observations (only opus-medium plus the two void cells; the workflow journal is not committed); the fix pass rule omits typecheck; the pin rule does not enforce the two-row count; supplementary review findings were assigned to main-run clusters by one agent and the lenses did not see them; the ground truth/prompts/finding text are hash-only host files; the README should state the main/void/supplementary split; the review-only cost comparison mixes profiles; the frozen protocol’s run_id is null (the main run id wf_4e9ef48d-066 lives only in the README, added by 5533e47 at 05:23Z, one minute before the 05:24Z launch), so fill the field or state that the README is the run-id record.

Checks: check_secrets, test_docs_drift, validate_skills, smart_router_decisions, test_skill_adherence, test_conform_spec and test_claude_code_host_artifacts all exit 0 at this head. Re-derived: 39 cells; 11/12 pin, 12/12 fix, 15/15 review; 15 graders; 5 clusters; 15 lenses; 2 void cells; score/observation fields match; the three hashes match the protocol. The PR body’s host-copy claim is verified for the evidence file: node --test hosts/claude-code/route.test.mjs exits 0 with 93 pass, and route.mjs rate on a temp trace with runs 1-2 plus this PR’s observations.json exits 0 with CALIBRATION lines for every profile. The host copy itself is not in the PR. Full run_regression.py not run (docs-only). The MoonAliza repo and workflow journal are unavailable, so grading and agent/wall-time counts are not independently reproducible.

Verdict: Request Changes. No approval, no acknowledgement, no implementation unit at this revision. After revision, request a delta review at the new full revision; this record does not carry forward. Full review record: docs/REVIEW_PROPOSALS_100-101-102_2026-10-18.md in the owner’s review workspace (not committed to this repository); ask the owner for it.

[StepenkoAnatoli]

Revision pushed as fb4e30f (on a197b6e). Disposition per blocking finding:

1. Token accounting. scores.json totals.tokens now carries the run-by-run buckets and the README a reconciliation table: main run items 5,274,667 (39 cells, of which the two void cells 166,266) + graders 1,597,164 (of which the void review's grader 88,760) + clustering 81,285 + lenses 1,578,450 = 8,531,566; supplementary cells 272,858 + grader 107,524 + the cluster-assignment agent 75,994 = 456,376; both 8,987,942. Scored cells 5,381,259; judging 3,440,417, about 38-39% (38.2% of the main run alone). The fourth supplementary agent is named (cluster:others).
2. G-2 machine-readable. scores.json gains adjudication_schema (the two prongs the lens schema carried against the three of the classification) and would_fail_under_three_prong on every review entry, true for haiku-medium (2 three-prong false) and haiku-high (1); adjudication.json gains the prong lists and a three_prong_reading per cluster (C-4 and C-5 "not real"); the README reads 15/15 preregistered, 13/15 three-prong. The preregistered PASS is kept as the score, with the flag beside it, so no table can present them as ordinary passes.
3. Qualification consequences. A README section states them under the host's rules 1-4, positive and negative (the fable-medium pin FAIL blocks pin-tests; second review passes for sonnet-high, sonnet-xhigh, opus-max, fable-max; sonnet-low with a PARTIAL between as the host's call; no haiku profile gains review), and that the host applies them in its policy revision 4; this folder changes no policy.
4. Pin category mapping. Amendment A-2: the pin case represents pin-tests per host rule 4 (as runs 1-2 were applied), recorded in protocol.json and as category_mapping in scores.json, which the proposed evidence check must use.

Non-blocking: the six-agent claim is now backed inside the PR (void-cells.json lists the relayed text, the harness header and the six agents; the journal stays host-local); the typecheck gap, the two-row count, the supplementary cluster assignment and the hash-only host files are recorded as review_notes in protocol.json; the README states the main/void/supplementary split; the review-only cost comparison is per model (1.4x, 1.9x, 1.3x).

Requesting the delta review at fb4e30f.

---

Generated by Claude Code

[StepenkoAnatoli]

Follow-up on the edited note: protocol.json run_id is now wf_4e9ef48d-066 with a run_id_note saying the README carried the id from 5533e47 (05:23Z, before the 05:24Z launch) while the field stayed null at the freeze (ce1ee83). The delta review request stands at ce1ee83.

---

Generated by Claude Code

[StepenkoAnatoli]

Follow-up on finding 2's addendum: every review-grade.others[].adjudicated value in observations.json is now a JSON boolean (18 occurrences were strings), and the builder is corrected for later runs (b63e140). The delta review request stands at b63e140.

---

Generated by Claude Code

[StepenkoAnatoli]

Delta-review follow-up — 2026-10-18 — public record

This continues the independent review posted on 2026-10-18 (comment on this issue, id 6247276840). Reviewer was a separate read-only agent session; provenance is through the owner's connected GitHub account, so GitHub shows the owner as the comment author. This is a review comment, not an approval.

Reviewed revision: a197b6ebb2544389334af7ff76788acba376f701 (protocol freeze 0013451; run-id 5533e47; results a3849c8, 38ca2c5; merge a197b6e; base ade3f1e).

Current public head under review: b63e140 (PR head evidence/calibration-3). The travel on this PR from a197b6e is: 38ca2c5 → a3849c8 → 5533e47 → 0013451 → ce1ee83 → fb4e30f → b63e140. The responses to the four blocking findings landed across fb4e30f and b63e140; the latest commit is b63e140 (adjudicated field made a JSON boolean).

Blocking-finding disposition at the current head (verified from the commit tree, not from the review workspace):

1. Token accounting. scores.json totals.tokens now carries the run-by-run buckets and the README a reconciliation table: main run items 5,274,667 (39 cells, of which the two void cells 166,266) + graders 1,597,164 (of which the void review's grader 88,760) + clustering 81,285 + lenses 1,578,450 = 8,531,566; supplementary cells 272,858 + grader 107,524 + the cluster-assignment agent 75,994 = 456,376; both 8,987,942. Scored cells 5,381,259; judging 3,440,417, about 38-39% (38.2% of the main run alone). The fourth supplementary agent is named (cluster:others).
2. G-2 machine-readable. scores.json gains adjudication_schema (the two prongs the lens schema carried against the three of the classification) and would_fail_under_three_prong on every review entry, true for haiku-medium (2 three-prong false) and haiku-high (1); adjudication.json gains the prong lists and a three_prong_reading per cluster (C-4 and C-5 "not real"); the README reads 15/15 preregistered, 13/15 three-prong. The preregistered PASS is kept as the score, with the flag beside it, so no table can present them as ordinary passes.
3. Qualification consequences. A README section states them under the host's rules 1-4, positive and negative (the fable-medium pin FAIL blocks pin-tests; second review passes for sonnet-high, sonnet-xhigh, opus-max, fable-max; sonnet-low with a PARTIAL between as the host's call; no haiku profile gains review), and that the host applies them in its policy revision 4; this folder changes no policy.
4. Pin category mapping. Amendment A-2: the pin case represents pin-tests per host rule 4 (as runs 1-2 were applied), recorded in protocol.json and as category_mapping in scores.json, which the proposed evidence check must use.

Verification note on the G-2 boolean fix: on b63e140, observations.json no longer contains any adjudicated value as the strings "true"/"false"; every such value is now a JSON boolean. That is the fix referenced by finding 2's addendum and is what the next delta review should re-verify rather than re-litigate.

Non-blocking: the six-agent claim is now backed inside the PR (void-cells.json lists the relayed text, the harness header and the six agents; the journal stays host-local); the typecheck gap, the two-row count, the supplementary cluster assignment and the hash-only host files are recorded as review_notes in protocol.json; the README states the main/void/supplementary split; the review-only cost comparison is per model (1.4x, 1.9x, 1.3x).

Checks: check_secrets, test_docs_drift, validate_skills, smart_router_decisions, test_skill_adherence, test_conform_spec and test_claude_code_host_artifacts all exit 0 at b63e140. Re-derived: 39 cells; 11/12 pin, 12/12 fix, 15/15 review; 15 graders; 5 clusters; 15 lenses; 2 void cells; score/observation fields match; the three hashes match the protocol. The PR body's host-copy claim is verified for the evidence file: node --test hosts/claude-code/route.test.mjs exits 0 with 93 pass, and route.mjs rate on a temp trace with runs 1-2 plus this PR's observations.json exits 0 with CALIBRATION lines for every profile. The host copy itself is not in the PR. Full run_regression.py not run (docs-only). The MoonAliza repo and workflow journal are unavailable, so grading and agent/wall-time counts are not independently reproducible.

Next delta review: at b63e140. Note: the prior follow-up replies on this PR named ce1ee83 and fb4e30f as the delta-review point; ce1ee83 was an intermediate that has since been superseded by fb4e30f and b63e140, and fb4e30f has in turn been superseded by b63e140. The next delta review should therefore be at the current PR head b63e140, not at ce1ee83.

Where the full record lives: docs/REVIEW_PROPOSALS_100-101-102_2026-10-18.md in the owner's review workspace (not committed to this repository). A self-contained public status note is being added under a PR; see the PR for that file.