Measure paired capacity-controlled drift and recurrence - #9
Open
abrahamabel wants to merge 3 commits into
Open
abrahamabel wants to merge 3 commits into
abrahamabel wants to merge 3 commits into
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first drift diagnostic compared growing graphs with unequal starting capacities and independently seeded conditions. This successor tests prior geometric history across 20 paired seeds, with equal total updates, identical intervening B and return-A samples, and a common 100-node cap. Realized live-node capacity matched at every preupdate and postupdate step; all seeds are retained.
This PR is stacked on #8. Merge the evidence/reproduction foundation first, then retarget this PR to
mainand obtain code-owner review.Closes #7.
Results and interpretation
The primary paired mean QE difference is -0.062270330687 with descriptive 95% seed-bootstrap interval [-0.064298691355, -0.060502391565]. All 20 seeds favor prior A on return QE, and all 20 have higher return topographic error. The preceding B phase costs more for prior A. An explicitly post hoc mean over both equal-length B and A phases is 0.039019255843 versus 0.037049038227, about 5.32% worse for prior A. This is a return-specific history benefit with costs, not general adaptation superiority or explicit/semantic memory.
Protocol and implementation were committed at
e7b5e91before the full panel. The canonical engine and both completed predecessor experiments remain unchanged. Recovery separates retrospective onset, actual confirmation and right-censoring; thresholds use independent pre-change evaluation. All outcome windows, seed-level comparisons and limits are retained.The numeric artifact, source-backed HTML/PNG report and reproduction registry are included. Plotting uses optional matplotlib; the experiment, tests and numerical replay use the Python standard library.
Validation
python3 scripts/validate_repository.py— passed in the current worktree.python3 -m unittest discover -s tests -p 'test_*.py'— 57 tests passed locally./environment.9a24419— passed: Linux validation and all 57 tests, followed by macOS 26 arm64 / CPython 3.14.7 validation, all 57 tests and exact full replay of all three experiments.docs/portability.mdrecords the runtime boundary and issue Characterize exact-replay portability across numerical runtimes #11 tracks the unresolved cross-platform question.git diff --check— passed; historical experiment and engine bytes are unchanged.Repository-owned MIT implementation and generated synthetic data only; no third-party implementation code, external datasets or model artifacts introduced. The numerical boundary bugs in the preserved reference are tracked separately in #10 and are not repaired by this experiment.
Note
Medium Risk
Changes CI merge gates and registers a large frozen evidence artifact with zero-tolerance JSON replay; scientific integrity depends on the new macOS reference job, though the canonical engine bytes are unchanged.
Overview
Adds completed experiment
exp-20260907-8313df3e-capacity-controlled-recurrence: a preregistered 20-seed panel comparing prior-A vs B-only history under a shared 100-node cap, with frozen JSON evidence, smoke fixture, optionalrender_report.py/ matplotlib HTML, and registry coverage soreproduce_experiments.pychecks all three experiments.CI now runs Linux validation + unit tests first, then
macos-26/ CPython 3.14.7 arm64 for full exact replay; the macOS job fails if Linux contract checks fail.Docs introduce
docs/portability.mdand update reproduction/roadmap/README copy to state that full-panel exact replay is scoped to the reference runtime, while preserving (not waiving) two ~1e-12 Linux window-loss mismatches observed on an earlier Linux run.Reviewed by Cursor Bugbot for commit 9a24419. Bugbot is set up for automated code reviews on this repo. Configure here.