From c53cd772f04e7e97e38e0008bcfc88c41f4b14c0 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli <318295322+StepenkoAnatoli@users.noreply.github.com> Date: Sun, 18 Oct 2026 06:59:30 +0000 Subject: [PATCH 1/3] =?UTF-8?q?docs:=20proposal=20=E2=80=94=20calibration-?= =?UTF-8?q?first=20onboarding=20and=20a=20results=20pull=20request=20the?= =?UTF-8?q?=20operator=20opens?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit What changed: docs/proposals/calibration-first-onboarding.md, a draft proposal (not normative) in five items: no hand-declared qualification (the validator refuses a qualified-for entry with no evidence entry behind it; a new profile is qualified-for [] until the kit's runs qualify it); the kit under one tools path, host-independent in method, with a shared synthetic subject pack for cross-host comparability and a host pack for the host's own categories; the run protocol the kit enforces (preregistration, no operator chat relayed or the text recorded, judging by a review-qualified profile, one run per cell, clean readings); the results folder, the pull request body the kit prints and asks the operator to open, the CI check on derivable parts and a generated qualification table; the relation to the difficulty-tiers proposal. Why: the operator asked that every new model run the calibration first and that results be pushed upstream. Three runs exist and each closed a method gap (G-1, G-2, A-1) that a kit can carry; today the method lives in one host's scratchpad and a lead's memory. The alternative set aside - the kit opening the pull request itself - is excluded: the operator's push is their statement that the evidence may be shared. What you verified: the regression gate ran in the pre-commit hook (verified, this commit). Cited facts re-read this session: spec §4 Gate 2; the host README's qualification rules; the three evidence folders' layouts; runs 2 and 3 token totals and amendments. What you got wrong: nothing to report. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh --- .../proposals/calibration-first-onboarding.md | 177 ++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 docs/proposals/calibration-first-onboarding.md diff --git a/docs/proposals/calibration-first-onboarding.md b/docs/proposals/calibration-first-onboarding.md new file mode 100644 index 0000000..e2975c3 --- /dev/null +++ b/docs/proposals/calibration-first-onboarding.md @@ -0,0 +1,177 @@ +# Proposal: calibration-first onboarding — a new model or effort enters a host policy only through the kit, and its results go upstream as a pull request + +Status: **draft proposal, not normative.** Nothing in `skills/` is amended by +this document. If accepted after independent review (plan §15), the items +below would become one additional host-policy validation rule, one new tools +path holding the calibration kit, one evidence-folder contract with a CI +check, and an additive host-policy field; until then hosts MUST NOT emit or +require them. + +## Context + +- Gate 2 passes a profile only on a declared, fresh qualification; labels, + price tier and self-confidence are not capability proof + (`skills/smart-router/references/spec-v2.md` §4). The Claude Code host + declares qualifications from graded calibration runs and production outcomes + under its own rules: two independent passes for a high-risk category, one + for a normal-risk one, a fail blocking the category until two later passes + (`hosts/claude-code/README.md`, "Qualification rules"). +- The adapter checks only the **form** of a qualification's evidence entries; + it does not count passes, and nothing stops a host from writing + `qualified-for: ["fix"]` on a profile with no evidence at all. Today that + restraint is the lead's, not the policy's. +- Three calibration runs exist (`docs/evidence/2026-10-17/claude-code-calibration/`, + `.../claude-code-calibration-2/`, `docs/evidence/2026-10-18/claude-code-calibration-3/`), + each an improvement on the last: a preregistered protocol with source + hashes, history-less checkouts, a reachability-defined classification, a + preregistered three-lens adjudication. The method is reproducible in + principle and **not portable in practice**: the workflow script, the graders, + the observation builder and the subject material live in one host's session + scratchpad and are copied per run into the evidence folder (`grade.sh`, + `historyless-checkout.sh`, `protocol.json`). A second host, or a new model + on this host, starts from a copy of the last run and a lead's memory. +- The runs also found what a kit must guard against: the grader instruction + ambiguity of run 2 (G-1), the lens schema missing the third prong of its own + classification in run 3 (G-2), and the host harness relaying the operator's + chat into every worker as the request that "wins" (A-1, two cells void). + Each was caught after the run, by reading outputs, and each is a rule a kit + can carry so that the next host does not rediscover it. +- Cost: runs 2 and 3 used 7.9M and 9.0M harness tokens each, 40% of it on + graders, clustering and lenses. A model added without calibration costs + nothing to add and an unknown amount later; the host ledger has one such + case already (`docs/proposals/difficulty-tiers.md`, Context). + +The request behind this proposal, from the operator of the Claude Code host: +every new model or effort level runs the calibration first, and the operator +is asked to push the results upstream. + +## Item 1 — calibration-first: no hand-declared qualification + +1. A profile may enter a host policy with any `qualified-for`, but every + category it names MUST be supported by evidence entries the adapter can + read: grade entries in the kit's form (`calibration-: + PASS|PARTIAL|FAIL (n=)`, with the run id entry beside them) or + accepted production outcomes in the host's outcome store with that + category. The adapter's policy validator refuses a `qualified-for` entry + with no supporting evidence entry; it still does not count passes (the + host's rules do), but it refuses the entry with nothing behind it. +2. A new profile with no evidence is valid only as `qualified-for: []`, which + Gate 2 already refuses to route to. That is the onboarding state: present, + priced, unroutable. +3. The onboarding run is the kit's T1 cells of every category the host uses + (pin-tests, fix, review today), under the host's pass rule; a profile + becomes routable for a category exactly when the host's qualification + rules say so, and the policy edit that grants it names the runs. +4. The exception that exists today stays visible: a qualification carried from + before this rule (the Claude Code host's production-derived `cc-opus-high`, + `cc-fable-high`, `cc-fable-xhigh`) keeps the reviewer line `host-declared + (lead), not independent` and is re-qualified by the kit at its next + expiry, never silently extended. + +## Item 2 — the kit, under one path, host-independent in method + +A new path, `tools/calibration/` (one directory; the exact name is the +reviewer's call), holding what every run so far rebuilt by hand: + +| File | What it is | Lineage | +|------|------------|---------| +| `protocol.template.json` | the preregistration form: subject, sources with hash convention, cases with pass rules, metrics, accounting, timing, privacy, limits, amendments; frozen before the launch with the run id recorded | run 3 `protocol.json` | +| `classification.md` | the three-prong definition of a true other finding (holds at HEAD; reachable through the public API with admissible input; forbidden by the spec or the module's promise) and the rule that a contract hole no admissible input reaches is at most S4 | run 3 (G-1 closed) | +| `grader.prompt.md`, `lens.prompt.md`, `cluster.prompt.md` | the grader, lens and clustering prompts with their output schemas; the lens schema carries `forbidden`, and a lens votes real only on all three prongs | run 3 (G-2 closed) | +| `historyless-checkout.sh` | one root commit per cell, so the accepted answer is unreachable | run 3 | +| `grade.sh` | mechanical grading of pin and fix cells against the subject pack's base and accepted trees | runs 1-3 | +| `observations.py` | builds `observations.json` (the flat profile map the adapter's `rate` reads), `scores.json`, `results-table.md`, `clusters.md`, `tokens-and-estimates.md` from the harness journal and run file | run 3 | +| `run.template.js` | the workflow script: items per profile and task, pipelined grading, clustering and adjudication, with the preamble that reconciles a relayed operator request | run 3 (A-1) | +| `subject-pack/README.md` | the format of a subject pack: base tree, accepted fix, ground-truth findings with severities, the required behaviour stated without the fix, pass rules per case, source hashes | runs 1-3 | + +Two kinds of subject pack: + +- **Shared pack** (shipped in the kit): a small self-contained module with two + seeded defects of the same shape the runs used (a rejection that breaks a + "nothing rejects" promise; a bound skipped on an unexpected input), with its + spec, its tests and its accepted fix. Every host runs it, so its cells are + **comparable across hosts**. It is synthetic on purpose: no host's private + code travels, and the pack can be rewritten when it leaks into training + data (the kit records the pack's version in the protocol). +- **Host pack** (the host's own): a real accepted defect pair from the host's + repository, as runs 1-3 used. Its cells qualify the profile for the host's + own categories and are not comparable across hosts. A host may run both; + onboarding requires the shared pack at least. + +The kit contains no provider calls: the host's harness runs the agents, the +kit supplies the prompts, the checkouts, the graders and the builders. + +## Item 3 — the run protocol the kit enforces + +1. **Preregistration before launch**: the frozen protocol with source hashes + and the run id recorded before any agent starts; a cell whose prompt or + checkout differs from the frozen hash is void (run 3 A-1 shows the form). +2. **No operator chat in the workers**: the kit's launch step checks what the + harness will prepend to each worker; a harness that relays operator chat + must either suppress it for the run or record the relayed text verbatim in + the protocol before the launch. A cell that halts on relayed text is void + and re-run once, never graded as capability. +3. **Judging by a qualified reviewer**: the grader, the clustering agent and + the lenses run on a profile the host policy qualifies for `review`; a host + onboarding its first model has no such profile and says so in the protocol + (the bootstrap case: the first grader is host-declared and the run's README + carries that line). +4. **One run per profile and task**, retry limit 0, harness retries recorded + as such (run 2 H-1); the two-pass rule needs two runs, so onboarding is at + least two runs or one run plus accepted production outcomes. +5. **Readings with nothing in flight** before and after, so the run also + measures the host's usage-window coefficients (Gate 4 on subscription + hosts); run 3 could not, and says so. + +## Item 4 — results go upstream as a pull request the operator opens + +1. The kit writes `docs/evidence//-calibration-/` in the layout + runs 2 and 3 use: `protocol.json` (frozen, then amended), `observations.json` + (flat map), `void-cells.json`, `adjudication.json`, `scores.json`, + `results-table.md`, `clusters.md`, `grade.sh` and `historyless-checkout.sh` + as run, and `README.md` with the frozen sections above the results. +2. The kit then prints the request: the branch name, the files, and a pull + request body in this repository's template (Problem and resulting + behavior; Related issue; Validation; Remaining limits or risks), filled + from the protocol and scores, and **asks the operator to open the pull + request**. The kit never pushes and never opens it: the operator's push is + the operator's statement that the evidence may be shared, which no agent + makes for them. +3. What a results pull request must carry to be merged: the frozen protocol + with the run id recorded before launch; `scores.json` derivable from + `observations.json` by the kit's builder; every grade entry in the + adapter's form; the judging profile named with its own qualification; the + relayed-chat statement from Item 3.2; no secrets, prompts bodies or model + output beyond the structured returns (`tools/check_secrets.py` already runs + in the gate). A CI check, `tools/check_calibration_evidence.py`, verifies + the derivable parts and refuses the rest with a named reason, in the + regression gate. +4. A merged results pull request updates one generated table, + `docs/evidence/qualification-table.md`: host, profile (model, effort, + config), category, pass/partial/fail per run, run ids. The table is a + record, not a ranking; it is what another host reads before choosing which + models to onboard first. + +## Item 5 — relation to the difficulty-tiers proposal + +If `docs/proposals/difficulty-tiers.md` is accepted, the onboarding run is the +T1 cells; T3 cells are run only for profiles qualified at T1 and on a host +pack, since a shared pack's T3 task would need a realistic multi-module +subject that a synthetic pack does not carry well. Nothing in this proposal +depends on that one. + +## Non-goals + +- No model ranking and no cross-host claim beyond the shared pack's cells. +- The kit does not call providers, hold keys, or run outside the host's own + harness and permissions. +- No automatic qualification: the kit grades; the host's rules qualify; the + policy edit is the host's, named and dated as today. +- No retroactive void of existing qualifications; Item 1.4 says how they age + out. + +## Review record (Items 1-5 pending) + +| Item | Decision | Reviewer | Date | +|------|----------|----------|------| +| 1-5 | pending | — | — | From 0e5666f4d9be47083ba37c9df8eac118e43cbb7c Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli <318295322+StepenkoAnatoli@users.noreply.github.com> Date: Sun, 18 Oct 2026 07:54:14 +0000 Subject: [PATCH 2/3] docs: calibration-first onboarding proposal revision 2 - the five blocking review findings and the notes What changed: docs/proposals/calibration-first-onboarding.md. (1) Item 1.1 defines a typed production reference ('outcome: accepted ()') the validator checks for form from the policy and against the outcome store when given --trace; free-text production evidence no longer supports a qualification on its own. (2) Item 4.3: the CI check requires the protocol's classification prongs and the adjudication record's lens-schema prongs to be equal, derives scores under the declared prongs, excludes a PASS that holds only because a prong is missing, uses the host's pin category mapping, and checks the judging profile and the relayed-chat statement; 'derivable' is named as necessary, not sufficient. (3) Item 4.4: the table carries the adjudication schema version, prongs and a caveat column; a fewer-prong PASS is shown as 'PASS (2-prong, would fail)' and excluded from the pass count; the table states what it does not support. (4) the uncalibrated-model cross-reference removed; this host has no such case and the cited unit is a mis-tiered calibrated profile. (5) the judging figures separated by run with reconciled numbers (run 2: 7,872,276 over 54 agents, post-hoc adjudication not included; run 3: 38-39%). Notes: an isolation rule for the shared pack and its size limit; the copied-per-run list corrected; Item 1.4 aligned with the tiers proposal's 2.2 (one adopting revision); the bootstrap grader's limitation line; the operator's push named as consent, not acceptance; the check name marked provisional. Why: the independent review of c53cd77 (Request Changes). What you verified: the run figures from #101's reconciled totals; the regression gate in the pre-commit hook (verified, this commit). What you got wrong: the first revision cited a mis-tiered dispatch as an uncalibrated-model case and gave a run-3-only judging share for both runs; corrected here. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh --- .../proposals/calibration-first-onboarding.md | 117 +++++++++++++----- 1 file changed, 86 insertions(+), 31 deletions(-) diff --git a/docs/proposals/calibration-first-onboarding.md b/docs/proposals/calibration-first-onboarding.md index e2975c3..94934f7 100644 --- a/docs/proposals/calibration-first-onboarding.md +++ b/docs/proposals/calibration-first-onboarding.md @@ -1,7 +1,8 @@ # Proposal: calibration-first onboarding — a new model or effort enters a host policy only through the kit, and its results go upstream as a pull request -Status: **draft proposal, not normative.** Nothing in `skills/` is amended by -this document. If accepted after independent review (plan §15), the items +Status: **draft proposal, not normative; revision 2 after the independent +review of 2026-10-18 (Request Changes on c53cd77).** Nothing in `skills/` is +amended by this document. If accepted after independent review (plan §15), the items below would become one additional host-policy validation rule, one new tools path holding the calibration kit, one evidence-folder contract with a CI check, and an additive host-policy field; until then hosts MUST NOT emit or @@ -25,21 +26,29 @@ require them. each an improvement on the last: a preregistered protocol with source hashes, history-less checkouts, a reachability-defined classification, a preregistered three-lens adjudication. The method is reproducible in - principle and **not portable in practice**: the workflow script, the graders, - the observation builder and the subject material live in one host's session - scratchpad and are copied per run into the evidence folder (`grade.sh`, - `historyless-checkout.sh`, `protocol.json`). A second host, or a new model - on this host, starts from a copy of the last run and a lead's memory. + principle and **not portable in practice**: the workflow script, the graders' + prompts, the observation builder, the ground truth and the finding text live + in one host's session scratchpad; per run, `grade.sh` and + `historyless-checkout.sh` are copied into the evidence folder, `protocol.json` + is written there, and the script, the builder, the ground truth and the + prompts are recorded by hash only. A second host, or a new model on this + host, starts from a copy of the last run and a lead's memory. - The runs also found what a kit must guard against: the grader instruction ambiguity of run 2 (G-1), the lens schema missing the third prong of its own classification in run 3 (G-2), and the host harness relaying the operator's chat into every worker as the request that "wins" (A-1, two cells void). Each was caught after the run, by reading outputs, and each is a rule a kit can carry so that the next host does not rediscover it. -- Cost: runs 2 and 3 used 7.9M and 9.0M harness tokens each, 40% of it on - graders, clustering and lenses. A model added without calibration costs - nothing to add and an unknown amount later; the host ledger has one such - case already (`docs/proposals/difficulty-tiers.md`, Context). +- Cost: run 2 used 7,872,276 harness tokens over 54 agents (39 cells and 15 + graders; its cross-candidate adjudication was post hoc and is not in that + total); run 3 used 8,531,566 over 70 agents plus 456,376 in a supplementary + run, of which graders, clustering and lenses were 38-39% (`#101`, + `scores.json` `totals.tokens`). A model added without calibration costs + nothing to add and an unknown amount later. This host has no such case: + every profile in its policy was calibrated or carries a production-derived + qualification marked as not independent; the ledger case cited by the + difficulty-tiers proposal is a mis-tiered dispatch of a calibrated profile, + which is that proposal's subject, not this one's. The request behind this proposal, from the operator of the Claude Code host: every new model or effort level runs the calibration first, and the operator @@ -48,13 +57,24 @@ is asked to push the results upstream. ## Item 1 — calibration-first: no hand-declared qualification 1. A profile may enter a host policy with any `qualified-for`, but every - category it names MUST be supported by evidence entries the adapter can - read: grade entries in the kit's form (`calibration-: - PASS|PARTIAL|FAIL (n=)`, with the run id entry beside them) or - accepted production outcomes in the host's outcome store with that - category. The adapter's policy validator refuses a `qualified-for` entry - with no supporting evidence entry; it still does not count passes (the - host's rules do), but it refuses the entry with nothing behind it. + category it names MUST be supported by an evidence entry of one of two + typed forms. (a) A grade entry in the kit's form, `calibration-: + PASS|PARTIAL|FAIL (n=)`, with the run id entry beside it; + the adapter's policy validator reads it from the policy alone, as today. + (b) A **typed production reference**, `outcome: + accepted ()`, which the validator checks + for form from the policy alone and, when it is given the outcome store + (the `--trace` directory the adapter already takes), checks against the + store: the referenced outcome must exist, be `accepted`, and carry the + same category as the entry; a reference that cannot be resolved against a + store the validator was given is refused, and a validation run without a + store says that production references were checked for form only. The + free-text production evidence the host policy carries today ("302 + completed dispatches ...", "sr-policy-2 outcome accepted ...") stays as + prose but no longer supports a `qualified-for` entry on its own; the + adopting revision rewrites each into form (b) or drops the qualification. + The validator still does not count passes (the host's rules do); it refuses + an entry with nothing typed behind it. 2. A new profile with no evidence is valid only as `qualified-for: []`, which Gate 2 already refuses to route to. That is the onboarding state: present, priced, unroutable. @@ -65,8 +85,11 @@ is asked to push the results upstream. 4. The exception that exists today stays visible: a qualification carried from before this rule (the Claude Code host's production-derived `cc-opus-high`, `cc-fable-high`, `cc-fable-xhigh`) keeps the reviewer line `host-declared - (lead), not independent` and is re-qualified by the kit at its next - expiry, never silently extended. + (lead), not independent`, is rewritten into typed production references in + the adopting revision (and, if the difficulty-tiers proposal is adopted, + re-declared at its tier in that same revision, as its Item 2.2 says: the + two revisions are one when both proposals land), and is re-qualified by + the kit at its next expiry, never silently extended. ## Item 2 — the kit, under one path, host-independent in method @@ -98,6 +121,15 @@ Two kinds of subject pack: own categories and are not comparable across hosts. A host may run both; onboarding requires the shared pack at least. +**Isolation of the answer.** A candidate's checkout is a history-less tree of +the pack's base state only (`historyless-checkout.sh`); the pack's accepted +fix, ground truth and grader material live outside every candidate's +permitted paths, the candidate prompt names only its checkout, and the kit's +launch step refuses a run whose candidate checkouts share a filesystem path +with the pack directory. A shared pack is also small by construction, and the +kit says so: it discriminates the bounded T1 shape only, and a host that needs +more runs its own pack; pack versions can grow. + The kit contains no provider calls: the host's harness runs the agents, the kit supplies the prompts, the checkouts, the graders and the builders. @@ -114,8 +146,10 @@ kit supplies the prompts, the checkouts, the graders and the builders. 3. **Judging by a qualified reviewer**: the grader, the clustering agent and the lenses run on a profile the host policy qualifies for `review`; a host onboarding its first model has no such profile and says so in the protocol - (the bootstrap case: the first grader is host-declared and the run's README - carries that line). + (the bootstrap case: the first grader is host-declared; the run's README + names the reviewer of that declaration and carries the limitation line + "judging profile not independently qualified", and the results carry it + into the qualification table's caveat column). 4. **One run per profile and task**, retry limit 0, harness retries recorded as such (run 2 H-1); the two-pass rule needs two runs, so onboarding is at least two runs or one run plus accepted production outcomes. @@ -135,22 +169,42 @@ kit supplies the prompts, the checkouts, the graders and the builders. behavior; Related issue; Validation; Remaining limits or risks), filled from the protocol and scores, and **asks the operator to open the pull request**. The kit never pushes and never opens it: the operator's push is - the operator's statement that the evidence may be shared, which no agent - makes for them. + the operator's consent that the evidence may be shared, which no agent + gives for them. It is consent, not acceptance: the pull request is then + reviewed under plan §15 like any other, by a reviewer who did not run it. 3. What a results pull request must carry to be merged: the frozen protocol with the run id recorded before launch; `scores.json` derivable from `observations.json` by the kit's builder; every grade entry in the adapter's form; the judging profile named with its own qualification; the relayed-chat statement from Item 3.2; no secrets, prompts bodies or model output beyond the structured returns (`tools/check_secrets.py` already runs - in the gate). A CI check, `tools/check_calibration_evidence.py`, verifies - the derivable parts and refuses the rest with a named reason, in the - regression gate. + in the gate). A CI check (provisionally `tools/check_calibration_evidence.py`; + the name follows the kit layout once settled) runs in the regression gate + and refuses with a named reason when any of these fails: the frozen + protocol declares the classification's prongs and the adjudication record + declares the prongs its lens schema carried, and the two sets are equal; + `scores.json` derives from `observations.json` by the kit's builder **under + the declared prongs**, so a review PASS that holds only because a prong is + missing is not a PASS the check accepts (it must be recorded as + `would_fail_under_` with the preregistered score kept + beside it, as #101 now does, and such a row is excluded from the table's + pass count); the pin case's category mapping is the host's (`pin-tests`, + not the protocol's label); every grade entry is in the adapter's form; the + judging profile is named with its qualification or the bootstrap + limitation; the relayed-chat statement from Item 3.2 is present. "Derivable" + is necessary and not sufficient: the rule the check enforces is the intended + rule, not the one the run happened to run. 4. A merged results pull request updates one generated table, `docs/evidence/qualification-table.md`: host, profile (model, effort, - config), category, pass/partial/fail per run, run ids. The table is a - record, not a ranking; it is what another host reads before choosing which - models to onboard first. + config), category, pass/partial/fail per run, run ids, **the adjudication + schema version and prongs each run's review grade rests on, and a caveat + column**. A PASS obtained under fewer prongs than the classification + defines is shown as `PASS (2-prong, would fail)` and excluded from the pass + count, never as an ordinary PASS; #101's `haiku-medium` and `haiku-high` + review cells are the first two such rows. The table will be read as a + ranking whatever it says, so it states what it does not support: no + ordering between models, no cross-host comparison outside the shared pack's + cells, and no claim about tiers it did not run. ## Item 5 — relation to the difficulty-tiers proposal @@ -174,4 +228,5 @@ depends on that one. | Item | Decision | Reviewer | Date | |------|----------|----------|------| -| 1-5 | pending | — | — | +| 1-5 (revision 1, c53cd77) | Request Changes: five blocking findings (the validator cannot enforce free-text production evidence; the CI check must enforce the intended rule, not a derivable one; the table would erase the G-2 caveat; the uncalibrated-model case mis-cited; the judging figure run-3-only) and eight non-blocking notes | independent agent session, read-only | 2026-10-18 | +| 1-5 (revision 2) | pending delta review | — | — | From 96c68230ae1667607071ed072d917cde243c3fd0 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli <318295322+StepenkoAnatoli@users.noreply.github.com> Date: Sun, 18 Oct 2026 07:58:30 +0000 Subject: [PATCH 3/3] docs: onboarding proposal - run 2's grader share (21.8%) beside run 3's judging share What changed: one Context bullet: run 2's graders 1,713,036 tokens (21.8% of 7,872,276) and the observation that the preregistered adjudication step roughly doubled the judging share in run 3. Why: the edited review of c53cd77 gives the run-2 grader share; the proposal's cost bullet should carry both runs' figures. What you verified: run 2's grader total from its evidence folder's tokens table (1,713,036); the regression gate in the pre-commit hook (verified, this commit). What you got wrong: nothing to report. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01VdJ51CTnHT4Zet6eScsBBh --- docs/proposals/calibration-first-onboarding.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/proposals/calibration-first-onboarding.md b/docs/proposals/calibration-first-onboarding.md index 94934f7..8b65196 100644 --- a/docs/proposals/calibration-first-onboarding.md +++ b/docs/proposals/calibration-first-onboarding.md @@ -40,10 +40,11 @@ require them. Each was caught after the run, by reading outputs, and each is a rule a kit can carry so that the next host does not rediscover it. - Cost: run 2 used 7,872,276 harness tokens over 54 agents (39 cells and 15 - graders; its cross-candidate adjudication was post hoc and is not in that - total); run 3 used 8,531,566 over 70 agents plus 456,376 in a supplementary - run, of which graders, clustering and lenses were 38-39% (`#101`, - `scores.json` `totals.tokens`). A model added without calibration costs + graders, the graders 1,713,036 or 21.8%; its cross-candidate adjudication + was post hoc and is not in that total); run 3 used 8,531,566 over 70 agents + plus 456,376 in a supplementary run, of which graders, clustering and lenses + were 38-39% (`#101`, `scores.json` `totals.tokens`). The preregistered + adjudication step roughly doubled the judging share. A model added without calibration costs nothing to add and an unknown amount later. This host has no such case: every profile in its policy was calibrated or carries a production-derived qualification marked as not independent; the ledger case cited by the