Skip to content

Register C3: the forward publication rule and its prospective test - #52

Merged
MaxGhenis merged 4 commits into
mainfrom
c3-registration
Nov 1, 2026
Merged

MaxGhenis merged 4 commits into
mainfrom
c3-registration

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

This registers C3, the rule Max asked for in d2202: no new average now, and a fixed rule for future releases.

What changes

  • docs/METHOD.md gains a frozen C3 section after C2. C2's section keeps its bytes and its hash.
  • README.md says what later releases will do.
  • scripts/c3_operating_characteristics.py and docs/uncertainty/c3-operating-characteristics.json show how often the test passes on simulated outcomes.
  • tests/test_c3_registration.py pins the section's hash and every rate it quotes.

No source file, input or result changes. Nothing here computes the rule's delta or target, and no one has computed them.

What the rule fixes

  • Form: the original primary. No score promotes another.
  • Calibration: one log-odds parameter, fitted on every complete spring origin and applied to today's floor probabilities, then an entropy tilt that holds the OBR means and shifts each year's floor share, within a 10% limit on how far it may reshape the distribution.
  • Test: C2's thresholds, on determination years from 2027 only.
  • Policy: every future release publishes both averages. The test decides only the word "estimate", and a failed test never withdraws a figure.

Decision worth reading first: the first verdict waits for October 2034

The obvious design was a verdict in October 2030, when the first complete four-year window of new outcomes arrives. The simulation rules that out: on four years the test passes a correct model 36% of the time and a model stating double the true floor rate 35%. From six years on the gap between them holds at 21 to 28 points, so eight sets no statistical minimum. We chose it because a correct model then passes about two times in three, and because it gives two non-overlapping windows. The forward policy brings the averages back at the next release either way, so the wait costs only the word "estimate".

The test stays weak at eight years, passes a model that is too wide more often than a correct one, and cannot check an average that runs to 2039-40. The registration says so.

Invariants

  • The frozen section's bytes match the pinned SHA-256, and C2's rule hash still verifies.
  • The section names no fiscal figure.
  • Every rate it quotes matches the committed simulation.
  • On one four-year window, the simulation's statistics equal ts_backtest.candidate_score (differential Hypothesis test).
  • Passing every check never exceeds passing any one check (property test).
  • Each simulated truth is what its label says on the span it covers, and main() scores each truth on its own span.

Synthetic checks of the frozen estimator

These use made-up data and a grid of delta values, never the real delta, targets or draws.

  • With equal weights, the net first-phase term equals C1's to rounding.
  • On the first model, coverage of the 95% margin held at 93.7% to 95.4% with no, mild and strong tilts, with bias within 0.04 SE.
  • On a panel of ten correlated years with a tilt shaped like step 4, coverage held at 93.5% to 95.7% across delta from 0 to -2. The 100-effective-run gate began to bind only at -2, where coverage still held, so on this panel it withholds before coverage fails. The panel omits the calendar-mean constraints and the other gates, and it reports unconditional coverage, so it shows the shape and sets no limit.
  • The gate fires on concentrated weights that pass the Kish limit of 500. The gates read fiscal outcomes only from sampled runs, so they cannot see a heavily tilted draw that no slot sampled, as the frozen text says. Each release prints the largest tilted weight so a reader can check.

Plan

  1. Independent review, with reviewers told not to compute delta or any target. Rounds 1 to 3 asked for changes, and the later commits answer them. Round 4 approved 999635a.
  2. Full CI: passed at 999635a.
  3. Max rules on the merge. A public commitment needs his go.
  4. After the merge: record the commit in the unfrozen note and tag c3-preregistration, as C2 did.

🤖 Generated with Claude Code

Max's d2202 ruling: no new average now, and a registered rule for future
releases. They publish the model's average as a labelled model-conditional
sensitivity, and a test on outcomes nobody has seen decides only whether it
may be called an estimate.

The rule fixes the form (the original primary), the calibration target
(one log-odds parameter from every complete spring origin) and the test
(C2's thresholds on determination years from 2027). The first verdict
waits for eight such years, October 2034: simulated outcomes show that four
years cannot separate a correct model from one stating double the true
floor rate.

Docs, a simulation of model draws and tests only. No code path, input or
result changes, and nobody computed the rule's delta or target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
uk-triple-lock Ready Ready Preview Oct 31, 2026 2:11pm UTC

Request Review

Sol's review asked for changes at 6575c23. Each change here answers one of
its ten findings:

- Write out the net estimator's variance, because the routine the rule
  cited assumes a different estimator. Synthetic checks give bias of about
  0.01 SE and 94-96% coverage.
- Score tilted draws by resampling them into candidate_score, so one scorer
  serves both figures.
- Drop the duty to publish four other forms' averages, which Max did not
  rule on.
- Fix the least-squares fits, the bootstrap interval and the four
  diagnostics, leaving no choice open.
- Tilt each year to its own odds-shifted target, with no new parameter, and
  limit how far the tilt may reshape the distribution.
- Set each simulated truth on the span it is scored on, and cover every
  span from four to twelve years.
- Say that eight years is a choice, not a statistical minimum, and that no
  pass covers an average running to 2039-40.
- Test truths() and main(), which the old tests never reached.

Nobody computed delta or a target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the net first-phase term with the repo's reweighted form, which
equals C1's with equal weights. Fix the variance convention behind the 10%
limit. Add C1's 100-effective-run limit as a third gate, name the tilt's
scales, print the largest tilted weight, and state what the gates cannot see.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agreement gate reads surrogate averages over every draw, so the gates
do not see only the sampled slots. They read fiscal outcomes only from
sampled runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 31, 2026 14:10
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Audit at head 999635adbd3c9609406eb06adb7321e528093402.

Reviews. Four independent GPT-6.1 Sol reviews:

Run Head Verdict
20261031-082656-tl-c3-r1 6575c23 Requested changes: ten findings, answered in d6d5064
20261031-084527-tl-c3-r2 d6d5064 Requested changes: two findings, answered in 07ca0a3
20261031-093841-tl-c3-r3 07ca0a3 Requested changes: one finding, answered in 999635a
20261031-101051-tl-c3-r4 999635a Approved

Checks. gh pr checks exits 0 on the latest run: all four pass, including the pipeline tests and committed-output checks. GitHub reports the PR as mergeable, and it is not a draft. There is no changes-requested review.

Restrictions. No review computed the rule's delta, any calibrated target or any calibrated quantity on real data, and none compared the March 2023 to March 2026 forecasts with outcomes.

Merge. This stays open for Max's ruling. It carries out d2202, but it contains choices he has not yet ruled on, and the text becomes a permanent public commitment.

@MaxGhenis
MaxGhenis merged commit bd6fdbb into main Nov 1, 2026
4 of 6 checks passed
@MaxGhenis
MaxGhenis deleted the c3-registration branch November 1, 2026 16:47

This branch was successfully deployed

1 active deployment
Preview — 999635ad Deployed Oct 31, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant