Repository navigation
Register C3: the forward publication rule and its prospective test - #52
Merged
Merged
Conversation
Max's d2202 ruling: no new average now, and a registered rule for future releases. They publish the model's average as a labelled model-conditional sensitivity, and a test on outcomes nobody has seen decides only whether it may be called an estimate. The rule fixes the form (the original primary), the calibration target (one log-odds parameter from every complete spring origin) and the test (C2's thresholds on determination years from 2027). The first verdict waits for eight such years, October 2034: simulated outcomes show that four years cannot separate a correct model from one stating double the true floor rate. Docs, a simulation of model draws and tests only. No code path, input or result changes, and nobody computed the rule's delta or target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Sol's review asked for changes at 6575c23. Each change here answers one of its ten findings: - Write out the net estimator's variance, because the routine the rule cited assumes a different estimator. Synthetic checks give bias of about 0.01 SE and 94-96% coverage. - Score tilted draws by resampling them into candidate_score, so one scorer serves both figures. - Drop the duty to publish four other forms' averages, which Max did not rule on. - Fix the least-squares fits, the bootstrap interval and the four diagnostics, leaving no choice open. - Tilt each year to its own odds-shifted target, with no new parameter, and limit how far the tilt may reshape the distribution. - Set each simulated truth on the span it is scored on, and cover every span from four to twelve years. - Say that eight years is a choice, not a statistical minimum, and that no pass covers an average running to 2039-40. - Test truths() and main(), which the old tests never reached. Nobody computed delta or a target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the net first-phase term with the repo's reweighted form, which equals C1's with equal weights. Fix the variance convention behind the 10% limit. Add C1's 100-effective-run limit as a third gate, name the tilt's scales, print the largest tilted weight, and state what the gates cannot see. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agreement gate reads surrogate averages over every draw, so the gates do not see only the sampled slots. They read fiscal outcomes only from sampled runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
marked this pull request as ready for review
October 31, 2026 14:10
Contributor
Author
|
Audit at head Reviews. Four independent GPT-6.1 Sol reviews:
Checks. Restrictions. No review computed the rule's delta, any calibrated target or any calibrated quantity on real data, and none compared the March 2023 to March 2026 forecasts with outcomes. Merge. This stays open for Max's ruling. It carries out d2202, but it contains choices he has not yet ruled on, and the text becomes a permanent public commitment. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This registers C3, the rule Max asked for in d2202: no new average now, and a fixed rule for future releases.
What changes
docs/METHOD.mdgains a frozen C3 section after C2. C2's section keeps its bytes and its hash.README.mdsays what later releases will do.scripts/c3_operating_characteristics.pyanddocs/uncertainty/c3-operating-characteristics.jsonshow how often the test passes on simulated outcomes.tests/test_c3_registration.pypins the section's hash and every rate it quotes.No source file, input or result changes. Nothing here computes the rule's delta or target, and no one has computed them.
What the rule fixes
Decision worth reading first: the first verdict waits for October 2034
The obvious design was a verdict in October 2030, when the first complete four-year window of new outcomes arrives. The simulation rules that out: on four years the test passes a correct model 36% of the time and a model stating double the true floor rate 35%. From six years on the gap between them holds at 21 to 28 points, so eight sets no statistical minimum. We chose it because a correct model then passes about two times in three, and because it gives two non-overlapping windows. The forward policy brings the averages back at the next release either way, so the wait costs only the word "estimate".
The test stays weak at eight years, passes a model that is too wide more often than a correct one, and cannot check an average that runs to 2039-40. The registration says so.
Invariants
ts_backtest.candidate_score(differential Hypothesis test).main()scores each truth on its own span.Synthetic checks of the frozen estimator
These use made-up data and a grid of delta values, never the real delta, targets or draws.
Plan
999635a.999635a.c3-preregistration, as C2 did.🤖 Generated with Claude Code