Skip to content

Measure paired capacity-controlled drift and recurrence - #9

Open
abrahamabel wants to merge 3 commits into
abe/feature/reproducible-drift-foundationfrom
abe/feature/capacity-controlled-recurrence
Open

abrahamabel wants to merge 3 commits into
abe/feature/reproducible-drift-foundationfrom
abe/feature/capacity-controlled-recurrence

Conversation

@abrahamabel

@abrahamabel abrahamabel commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

The first drift diagnostic compared growing graphs with unequal starting capacities and independently seeded conditions. This successor tests prior geometric history across 20 paired seeds, with equal total updates, identical intervening B and return-A samples, and a common 100-node cap. Realized live-node capacity matched at every preupdate and postupdate step; all seeds are retained.

This PR is stacked on #8. Merge the evidence/reproduction foundation first, then retarget this PR to main and obtain code-owner review.

Closes #7.

Results and interpretation

Return-to-A measurement Prior A, then B B-only history
Mean squared quantization error 0.006351514506 0.068621845192
Mean topographic error 0.018481445313 0.010400390625
QE recovery confirmed within 4,096 samples 20 / 20 0 / 20

The primary paired mean QE difference is -0.062270330687 with descriptive 95% seed-bootstrap interval [-0.064298691355, -0.060502391565]. All 20 seeds favor prior A on return QE, and all 20 have higher return topographic error. The preceding B phase costs more for prior A. An explicitly post hoc mean over both equal-length B and A phases is 0.039019255843 versus 0.037049038227, about 5.32% worse for prior A. This is a return-specific history benefit with costs, not general adaptation superiority or explicit/semantic memory.

Protocol and implementation were committed at e7b5e91 before the full panel. The canonical engine and both completed predecessor experiments remain unchanged. Recovery separates retrospective onset, actual confirmation and right-censoring; thresholds use independent pre-change evaluation. All outcome windows, seed-level comparisons and limits are retained.

The numeric artifact, source-backed HTML/PNG report and reproduction registry are included. Plotting uses optional matplotlib; the experiment, tests and numerical replay use the Python standard library.

Validation

  • python3 scripts/validate_repository.py — passed in the current worktree.
  • python3 -m unittest discover -s tests -p 'test_*.py' — 57 tests passed locally.
  • Independent full 20-seed rerun — zero deterministic differences, ignoring only /environment.
  • Independent accounting reconstructed the bootstrap interval, all capacity gates and every recovery decision.
  • Final CI at 9a24419 — passed: Linux validation and all 57 tests, followed by macOS 26 arm64 / CPython 3.14.7 validation, all 57 tests and exact full replay of all three experiments.
  • Full local replay of all three registered experiments at the final revision — exact deterministic match; frozen files unchanged.
  • Earlier full Linux replay — successor failed in exactly two rounded window-loss fields by about 1e-12; every other compared field matched. The comparator and frozen evidence are unchanged. docs/portability.md records the runtime boundary and issue Characterize exact-replay portability across numerical runtimes #11 tracks the unresolved cross-platform question.
  • git diff --check — passed; historical experiment and engine bytes are unchanged.
  • Visual report checked in the browser and plotted values inspected against the numeric evidence.

Repository-owned MIT implementation and generated synthetic data only; no third-party implementation code, external datasets or model artifacts introduced. The numerical boundary bugs in the preserved reference are tracked separately in #10 and are not repaired by this experiment.


Note

Medium Risk
Changes CI merge gates and registers a large frozen evidence artifact with zero-tolerance JSON replay; scientific integrity depends on the new macOS reference job, though the canonical engine bytes are unchanged.

Overview
Adds completed experiment exp-20260907-8313df3e-capacity-controlled-recurrence: a preregistered 20-seed panel comparing prior-A vs B-only history under a shared 100-node cap, with frozen JSON evidence, smoke fixture, optional render_report.py / matplotlib HTML, and registry coverage so reproduce_experiments.py checks all three experiments.

CI now runs Linux validation + unit tests first, then macos-26 / CPython 3.14.7 arm64 for full exact replay; the macOS job fails if Linux contract checks fail.

Docs introduce docs/portability.md and update reproduction/roadmap/README copy to state that full-panel exact replay is scoped to the reference runtime, while preserving (not waiving) two ~1e-12 Linux window-loss mismatches observed on an earlier Linux run.

Reviewed by Cursor Bugbot for commit 9a24419. Bugbot is set up for automated code reviews on this repo. Configure here.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant