docs: proposal — calibration-first onboarding and a results pull request the operator opens

[StepenkoAnatoli]

Problem and resulting behavior

Gate 2 routes only to a declared, fresh qualification, but the adapter checks only the form of the evidence entries behind a qualified-for; nothing stops a host from qualifying a new model by hand, and the production evidence it carries today is free text. The calibration method that produced the three runs under docs/evidence/ is reproducible in principle and not portable in practice: the workflow script, the graders' prompts, the observation builder, the ground truth and the finding text live in one host's session scratchpad and reach the evidence folder by hash. Each run also found a rule the next host would otherwise rediscover (G-1 grader ambiguity, G-2 lens schema missing the third prong, A-1 operator chat relayed into workers).

This adds docs/proposals/calibration-first-onboarding.md, a draft proposal (not normative; nothing in skills/ changes), now at revision 2 after the independent review of 2026-10-18:

1. Calibration-first: a qualified-for entry needs a typed evidence entry behind it, either a kit grade entry or a typed production reference (outcome: <task-id> <category> accepted <date> (<store>)) the validator resolves against the outcome store when given one; a new profile is qualified-for: [] until the kit's runs and the host's rules qualify it; pre-existing hand-declared qualifications are rewritten into typed references in the adopting revision (one revision with the tiers proposal if both land) and re-qualified at expiry.
2. The kit under one tools path, host-independent in method: protocol template, the three-prong classification, grader/lens/cluster prompts with the forbidden prong, history-less checkout, mechanical grading, the observations builder, the run template; a shared synthetic subject pack (comparable across hosts, small by construction, isolated from every candidate's paths) and a host pack.
3. The run protocol the kit enforces: preregistration before launch, no operator chat relayed into workers or the text recorded, judging by a review-qualified profile with a named bootstrap limitation, one run per cell, clean readings.
4. Results upstream: the kit writes the evidence folder in the run-2/3 layout, prints the pull request body in this repository's template and asks the operator to open it (consent, not acceptance; the kit never pushes). A CI check enforces the intended rule, not a derivable one: declared prongs equal between protocol and adjudication record, scores derived under them, a fewer-prong PASS flagged and excluded from the pass count, the host's pin category mapping, the judging profile named. A generated qualification table carries the adjudication schema, prongs and a caveat column and states what it does not support.
5. Relation to the difficulty-tiers proposal (docs: proposal — difficulty tiers, qualification per (category, tier) #100): onboarding is the T1 cells.

Resulting behavior if accepted: every new model or effort level is unroutable until it has run the calibration, and its results reach this repository through a pull request the operator chooses to open, checked against the rule the run was meant to apply.

Related issue

Related to #98 (calibration method), #100 (difficulty tiers), #101 (run 3 evidence). No issue number.

Validation

• Pre-commit regression gate: all checks PASS on both commits (local fixtures only; docs change).
• Run figures taken from evidence: Claude Code calibration run 3 (history-less checkouts, defined classification, preregistered adjudication) #101's reconciled scores.json totals.tokens (run 3) and the run-2 evidence README (run 2).

Remaining limits or risks

• Pending the delta review of revision 2 (0e5666f); the review record carries revision 1's Request Changes.
• Item 2's shared synthetic subject pack does not exist yet; building it is its own unit with its own review, the pack must be versioned, and it discriminates only the bounded T1 shape.
• Item 1's validator rule and Item 4's CI check are additive but are code; they follow only after this proposal is accepted; the check's name is provisional.
• The "no operator chat relayed" rule depends on what a host's harness exposes; where it cannot be suppressed the proposal falls back to recording the text, which makes the contamination visible, not undone.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh

[StepenkoAnatoli]

Independent review — 2026-10-18 — Request Changes

Reviewed revision: c53cd772f04e7e97e38e0008bcfc88c41f4b14c0 (base ade3f1e). Reviewer: independent agent session; no authorship of this PR or the plan. Read-only.

Provenance: posted through the owner's connected GitHub account at the owner's request; the review was produced by a separate agent session that authored none of this PR. GitHub therefore shows the owner as the comment author; this is a review comment, not an approval.

Blocking

1. The proposed validator cannot enforce production-outcome evidence. Item 1.1 says the adapter refuses a qualified-for entry unless it has a grade entry or “accepted production outcomes in the host’s outcome store”. hosts/claude-code/route.mjs validatePolicy reads only the policy; production evidence in host-policy.json is free text (“302 completed dispatches…”, “sr-policy-2 outcome accepted…”). The adapter can check that some string exists, not that it is an accepted outcome for the category. Define a typed evidence reference (unit id, category, accepted status, date) and how the store is passed/queried; or scope the rule to calibration entries and leave production evidence to the lead.
2. The proposed CI check would certify a run with a missing classification prong. Item 4.3 says check_calibration_evidence.py verifies “the derivable parts”. evidence: Claude Code calibration run 3 (history-less checkouts, defined classification, preregistered adjudication) #101 shows the failure mode: review scores are derivable, but they depend on a lens schema that omitted the forbidden-by-spec prong (G-2), and the builder would reproduce the PASSes. The check must require the classification/prong schema and the adjudication record, and fail a scores.json whose review PASS depends on a missing prong. “Derivable” is not enough; the rule must be the intended rule.
3. The generated qualification table would erase the G-2 caveat. Item 4.4 proposes docs/evidence/qualification-table.md with “pass/partial/fail per run”. evidence: Claude Code calibration run 3 (history-less checkouts, defined classification, preregistered adjudication) #101’s haiku-medium and haiku-high review PASSes are PASS only under the two-prong rule; a plain table would present them as ordinary PASSes. Add adjudication-schema/version columns and a caveat, or exclude non-three-prong PASSes.
4. The “model added without calibration” case is mis-cited. The Context says “the host ledger has one such case already (docs/proposals/difficulty-tiers.md, Context).” The cited fix-u7-r3 used cc-haiku-low, which was calibrated in runs 1-2; it is a mis-tiered dispatch, not an uncalibrated model. Cite an actual uncalibrated profile or remove the cross-reference.
5. The 40% judging claim is run-3-only and inherits evidence: Claude Code calibration run 3 (history-less checkouts, defined classification, preregistered adjudication) #101’s accounting issue. “runs 2 and 3 used 7.9M and 9.0M harness tokens each, 40% of it on graders, clustering and lenses”: run 2’s 7,872,276 covers 54 agents (39 items + 15 graders), and item + grader tokens sum to exactly that total (grader share 21.8%); its post-hoc adjudication is not in the total, and it had no preregistered clustering/lenses. Separate the two runs and use reconciled figures.

Non-blocking: the shared pack ships its accepted fix — add an isolation rule so the candidate cannot see the kit/pack/answer; the table is “not a ranking” but pass/partial/fail per run is a de facto ranking; the Context’s “copied per run” list is inaccurate (grade.sh and historyless-checkout.sh are copied; protocol.json is written; the script/builder are hash-only); Item 1.4’s expiry re-qualification differs from #100 Item 2.2’s immediate tier re-declaration; the shared pack may be too small to discriminate; the operator’s push is consent, not independent acceptance; the bootstrap grader needs a named reviewer/limitation line; the check name is provisional until the kit layout is settled.

Checks: check_secrets, test_docs_drift, validate_skills, smart_router_decisions, test_skill_adherence, test_conform_spec and test_claude_code_host_artifacts all exit 0 at this head. Full run_regression.py not run (docs-only).

Verdict: Request Changes. No approval, no acknowledgement, no implementation unit at this revision. After revision, request a delta review at the new full revision; this record does not carry forward. Full review record: docs/REVIEW_PROPOSALS_100-101-102_2026-10-18.md in the owner’s review workspace (not committed to this repository); ask the owner for it.

[StepenkoAnatoli]

Revision 2 pushed as 0e5666f (base ade3f1e). Disposition per blocking finding:

1. Validator cannot enforce production evidence. Item 1.1 now defines a typed production reference, outcome: <task-id> <category> accepted <ISO date> (<outcome store path>). The validator checks its form from the policy alone and, when given the outcome store (the --trace directory the adapter already takes), resolves it: the outcome must exist, be accepted and carry the same category; an unresolvable reference is refused, and a run without a store says production references were checked for form only. Free-text production evidence stays as prose but no longer supports a qualified-for entry; the adopting revision rewrites each into the typed form or drops the qualification.
2. CI check would certify a missing prong. Item 4.3 now requires the protocol's classification prongs and the adjudication record's lens-schema prongs to be declared and equal, derives scores under the declared prongs, refuses a review PASS that holds only because a prong is missing unless it is recorded as "would fail under the full definition" beside the preregistered score and excluded from the pass count (as evidence: Claude Code calibration run 3 (history-less checkouts, defined classification, preregistered adjudication) #101 now does for two cells), uses the host's pin category mapping, and checks the judging profile and the relayed-chat statement. "Derivable" is named as necessary, not sufficient.
3. Table would erase G-2. Item 4.4 adds the adjudication schema version, the prongs and a caveat column; a fewer-prong PASS reads PASS (2-prong, would fail) and is excluded from the pass count; evidence: Claude Code calibration run 3 (history-less checkouts, defined classification, preregistered adjudication) #101's haiku-medium and haiku-high review cells are named as the first two such rows; the table states what it does not support.
4. Uncalibrated-model case mis-cited. The cross-reference is removed; the Context says this host has no such case and that the cited unit is a mis-tiered dispatch of a calibrated profile, the tiers proposal's subject.
5. Judging figure. Separated by run with reconciled numbers: run 2 7,872,276 over 54 agents (39 cells, 15 graders; its post-hoc adjudication outside that total); run 3 8,531,566 over 70 agents plus 456,376 supplementary, judging 38-39% (evidence: Claude Code calibration run 3 (history-less checkouts, defined classification, preregistered adjudication) #101 scores.json totals.tokens).

Non-blocking: an isolation rule for the shared pack (history-less base tree only; the pack, its fix and the grader material outside every candidate's permitted paths; the launch step refuses a shared path) and its size limit stated; the copied-per-run list corrected (grade.sh and historyless-checkout.sh copied, protocol.json written, script/builder/ground truth/prompts by hash); Item 1.4 aligned with #100 Item 2.2 (one adopting revision for both); the bootstrap grader carries a named reviewer and the limitation line into the table's caveat; the operator's push is named as consent, not acceptance, with plan §15 review after it; the check name is marked provisional.

Requesting the delta review at 0e5666f.

---

Generated by Claude Code

[StepenkoAnatoli]

Follow-up on the edited finding 5: the Context bullet now carries run 2's grader share (1,713,036 tokens, 21.8% of 7,872,276) beside run 3's 38-39%, and notes that the preregistered adjudication step roughly doubled the judging share (96c6823). The delta review request stands at 96c6823.

---

Generated by Claude Code

[StepenkoAnatoli]

Delta-review follow-up — 2026-10-18 — public record

This continues the independent review posted on 2026-10-18 (comment on this issue, id 6247276842). Reviewer was a separate read-only agent session; provenance is through the owner's connected GitHub account, so GitHub shows the owner as the comment author. This is a review comment, not an approval.

Reviewed revision: c53cd772f04e7e97e38e0008bcfc88c41f4b14c0 (base ade3f1e).

Current public head under review: 0e5666f (PR head proposal/calibration-first). The file in the PR is docs/proposals/calibration-first-onboarding.md (+233).

Blocking-finding disposition at the current head (verified from the commit tree, not from the review workspace):

1. Validator cannot enforce production evidence. Item 1.1 now defines a typed production reference, outcome: <task-id> <category> accepted <ISO date> (<outcome store path>). The validator checks its form from the policy alone and, when given the outcome store (the --trace directory the adapter already takes), resolves it: the outcome must exist, be accepted and carry the same category; an unresolvable reference is refused, and a run without a store says production references were checked for form only. Free-text production evidence stays as prose but no longer supports a qualified-for entry; the adopting revision rewrites each into the typed form or drops the qualification.
2. CI check would certify a missing prong. Item 4.3 now requires the protocol's classification prongs and the adjudication record's lens-schema prongs to be declared and equal, derives scores under the declared prongs, refuses a review PASS that holds only because a prong is missing unless it is recorded as "would fail under the full definition" beside the preregistered score and excluded from the pass count (as evidence: Claude Code calibration run 3 (history-less checkouts, defined classification, preregistered adjudication) #101 now does for two cells), uses the host's pin category mapping, and checks the judging profile and the relayed-chat statement. "Derivable" is named as necessary, not sufficient.
3. Table would erase G-2. Item 4.4 adds the adjudication schema version, the prongs and a caveat column; a fewer-prong PASS reads PASS (2-prong, would fail) and is excluded from the pass count; evidence: Claude Code calibration run 3 (history-less checkouts, defined classification, preregistered adjudication) #101's haiku-medium and haiku-high review cells are named as the first two such rows; the table states what it does not support.
4. Uncalibrated-model case mis-cited. The cross-reference is removed; the Context says this host has no such case and that the cited unit is a mis-tiered dispatch of a calibrated profile, the tiers proposal's subject.
5. Judging figure. Separated by run with reconciled numbers: run 2 7,872,276 over 54 agents (39 cells, 15 graders; its post-hoc adjudication outside that total); run 3 8,531,566 over 70 agents plus 456,376 supplementary, judging 38-39% (evidence: Claude Code calibration run 3 (history-less checkouts, defined classification, preregistered adjudication) #101 scores.json totals.tokens).

Non-blocking: an isolation rule for the shared pack (history-less base tree only; the pack, its fix and the grader material outside every candidate's permitted paths; the launch step refuses a shared path) and its size limit stated; the copied-per-run list corrected (grade.sh and historyless-checkout.sh copied, protocol.json written, script/builder/ground truth/prompts by hash); Item 1.4 aligned with #100 Item 2.2 (one adopting revision for both); the bootstrap grader carries a named reviewer and the limitation line into the table's caveat; the operator's push is named as consent, not acceptance, with plan §15 review after it; the check name is marked provisional.

Checks: check_secrets, test_docs_drift, validate_skills, smart_router_decisions, test_skill_adherence, test_conform_spec and test_claude_code_host_artifacts all exit 0 at 0e5666f. Full run_regression.py not run (docs-only changes).

Next delta review: at 0e5666f. Note: the prior follow-up reply on this PR referenced 96c6823; that is an earlier commit in this PR's travel (the run-2 grader-share addition), not the current head. The PR head is 0e5666f and a delta review at this head is the correct next step.

Where the full record lives: docs/REVIEW_PROPOSALS_100-101-102_2026-10-18.md in the owner's review workspace (not committed to this repository). A self-contained public status note is being added under a PR; see the PR for that file.

[StepenkoAnatoli]

One correction to the follow-up's "current public head" line: GitHub reports this PR's head as 96c6823 (3 commits: c53cd77 → 0e5666f → 96c6823). 96c6823 is the newest commit, not an earlier one; it adds run 2's grader share on top of revision 2 (0e5666f). The delta review should run at 96c6823.

---

Generated by Claude Code

[StepenkoAnatoli]

New finding — PR #102 head moved since the earlier comments

Live remote head for this PR is now 96c6823 (not 0e5666f). The PR advanced one commit past the commit that the earlier comment named as the head.

One file changed, 5 insertions, 4 deletions:

• docs/proposals/calibration-first-onboarding.md (+5, -4)

The change adds run 2's grader share to the Context paragraph: graders were 1,713,036 tokens / 21.8% of 7,872,276 on run 2, and the preregistered adjudication step roughly doubled the judging share. The rest of the paragraph is the same reconciling text that points at #101, scores.json totals.tokens.

Diff cited from the live head (c53cd77 → 0e5666f → 96c6823):

docs/proposals/calibration-first-onboarding.md | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)

This is the only change across the three PRs since the earlier review follow-ups; PR #100 and PR #101 are unchanged at their heads (6a75383 and b63e140).

Next delta review: at 96c6823.

Background note: the earlier follow-up comment mistakenly said the head was 0e5666f; this comment is the explicit new-finding summary for that correction.