P3: enforce covariance support and recover the N580 matrix without new sampling

.github/workflows/ci.yml

@@ -1,16 +1,8 @@
 name: ci
 
-# What CI is for here: catching the breakage you would not notice yourself.
-# It is not a gate, not a claim, and not evidence. See GOVERNANCE.md sections 0 and 2.
-#
-# The tests are deterministic standard-library mathematics, so running the whole
-# suite on three interpreters bought almost nothing and cost three times the budget.
-# It now runs once, on 3.11; 3.9 and 3.13 are compile-checked, which is what actually
-# catches a version incompatibility in this codebase.
-#
-# If branch protection is ever turned on for main, the checks to require are the ones
-# this file defines: "Compile on 3.9", "Compile on 3.13", "Tests", and
-# "C++17 build and self-tests". That is a note, not a machine-checked contract.
+# CI catches computational breakage, not scientific truth.
+# Run the complete suite once on 3.11; compile-check 3.9 and 3.13.
+# The full suite includes the P598 NumPy/SciPy balanced-realization tests.
 
 on:
   push:
@@ -60,18 +52,15 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: "3.11"
-
       - name: Install dependencies
         shell: bash
         run: |
           python -m pip install --upgrade pip
-          [[ -f requirements.txt ]] && python -m pip install -r requirements.txt
-
+          python -m pip install -r requirements-test.txt
       - name: Run tests
         shell: bash
         run: |
           [[ -d tests ]] && python -m unittest discover -s tests -p 'test_*.py'
-
       - name: Smoke check
         shell: bash
         run: |
@@ -83,7 +72,6 @@ jobs:
     timeout-minutes: 20
     steps:
       - uses: actions/checkout@v4
-
       - name: Compile standalone C++ sources
         shell: bash
         run: |
@@ -93,15 +81,13 @@ jobs:
             echo "No C++ sources on this branch."
             exit 0
           fi
-
           mkdir -p build/ci
           for source in "${sources[@]}"; do
             name="$(basename "${source%.cpp}")"
             binary="build/ci/$name"
             echo "Compiling $source"
             g++ -std=c++17 -O2 -Wall -Wextra -Wpedantic -Werror=return-type \
               -fopenmp "$source" -o "$binary"
-
             if grep -q -- '--self-test' "$source"; then
               echo "Running $name --self-test"
               timeout 120 "$binary" --self-test

.github/workflows/n580-recovery.yml

@@ -0,0 +1,28 @@
+name: N580 existing-block recovery
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - 'scripts/recover_n580_covariance.py'
+      - 'scripts/projective_inference.py'
+      - 'tests/test_projective*.py'
+      - 'tests/test_n580_covariance_recovery.py'
+      - '.github/workflows/n580-recovery.yml'
+permissions:
+  contents: read
+jobs:
+  recover:
+    runs-on: ubuntu-latest
+    timeout-minutes: 3
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - run: python -m pip install mpmath
+      - name: Targeted mathematical tests
+        run: |
+          python -m unittest discover -s tests -p 'test_projective*.py'
+          python -m unittest discover -s tests -p 'test_n580_covariance_recovery.py'
+      - name: Recover and report without rerunning the experiment
+        run: python scripts/recover_n580_covariance.py

notes/n580-complete-covariance-20260912.md

@@ -0,0 +1,90 @@
+# N580: the missing covariance recovered, without another experiment
+
+Date: 2026-09-12. Analysis implemented in PR #703.
+
+## Executed, not proposed
+
+GitHub Actions run 34679359201, job 103514945963, completed successfully.
+Command: `python scripts/recover_n580_covariance.py`.
+Head: `eb78265a7016c99c75f5ecc4a162973c448bf4ee`;
+actual tested merge checkout: `dda2a8e2dcb9e5a4a9215cb1f1877a4aa016b189`.
+Runner: Python 3.11.16, mpmath 1.4.1. The 14 inference checks and 2 recovery
+checks passed in the same job. Full-repository CI is separate from this job.
+
+The reader consumed the EXISTING 100 aligned delete-one rows in
+`results/aspect-ladder-n580/shards/rung_r{1,2,4}.json`. It checked replay
+seed/offset/sample/batch metadata and recovered the previously stored standard
+errors and cov(r1,r2), cov(r1,r4). No Monte Carlo or histogram reconstruction
+was run. All historical results and freezes are unchanged.
+
+The result summary is in
+`results/research-control-20260912/n580-complete-covariance-summary.json`.
+Fields there were extracted from the successful job stdout, then all eight
+pure-ray statistics were independently recomputed locally from the recovered
+vector/matrix; the largest D difference was zero. That local calculation was
+NOT a second shard replay or independent experiment.
+
+## Recovered matrix and decision
+
+The missing cov(r2,r4) is 5.5621485097801135e-9, correlation
+0.11001108422637242. The full matrix has rank 3 and condition number 1.96789.
+It is not singular; the exact-support bug fixed in #703 does not invalidate
+this well-conditioned calculation.
+
+| Pure model, retrospective three-rung test | D, df=2 | nominal Gaussian-reference p | equivalent sigma | declared 3-sigma decision |
+|---|---:|---:|---:|---|
+| bare aspect ratio | 10.8644599 | 0.00437333 | 2.84990 | not rejected |
+| no modulus dependence | 89.8585102 | 3.07236e-20 | 9.21640 | rejected |
+| area scaling | 57.5124933 | 3.24579e-13 | 7.28379 | rejected |
+| Q4 weight-4 shape | 55.5877105 | 8.49732e-13 | 7.15288 | rejected |
+| weight-12 E12 | 132.267324 | 1.89896e-29 | 11.26749 | rejected |
+| weight-12 E4 cubed | 132.226117 | 1.93849e-29 | 11.26568 | rejected |
+| weight-12 delta | 513.659218 | 2.88614e-112 | not resolved by inverse-erf conversion | rejected |
+| weight-8 E8 | 115.804378 | 7.13508e-26 | 10.51803 | rejected |
+
+The previous missing-covariance ambiguity for bare_aspect_ratio is resolved:
+it is **not rejected at the declared nominal 3-sigma cutoff**. This does not
+confirm that law, make it unique physically, or license a post-hoc 2-sigma
+cutoff. The original frozen two-rung comparison remains a different test with
+its recorded underpowered verdict; this is the retrospective three-rung test
+on the same block, conditional on its pure-amplitude/readout assumptions.
+
+The raw divided difference is
+
+    (m4 - 3*m2 + 2*m1)/6 = -0.00046630613986373587
+    SE                    =  0.0001499079748299409
+    nominal z             = -3.110615965513005.
+
+This 1-df contrast and the 2-df line test need not cross the same cutoff.
+The sign alone cannot reject an arbitrary signed multiple of a convex shape;
+any class-level sign claim must additionally declare the amplitude sign.
+No continuum operator or percolation threshold is identified.
+
+## What the nuisance sensitivity says, and does not say
+
+Allowing a second column lambda*(-v1,+v2,-v4), lambda=1148/21025, assumes
+that spin-8 has the SAME modulus shape and a common ratio to spin-4. Under
+that enlarged, unbounded plane the nominal residuals are bare 0.25187/1,
+area 6.54806/1, Q4 4.32890/1. The small-ratio bounds matter: at |rho|<=1,
+minimum D is 5.72524, 50.81440, 48.88288 respectively. These bounded-cone
+minima are not assigned an ordinary chi-square calibration.
+
+This is a sensitivity to a changed model, not a rescue of a rejected pure
+ray and not a measurement identifying H8. A bound on |rho| across rungs does
+not imply a COMMON rho; the latter is a separate stronger assumption. In
+particular the older `required_spin8_ratio` wording that called common rho
+an implication of a common bound is not logically valid.
+
+## Decision for the queue and manuscript
+
+Cancel any proposed N580 replay whose sole purpose is this covariance entry.
+Use the recovered matrix and clearly separated pure-line/nuisance readings in
+P3. Replace 'undetermined because covariance missing' with the narrow decision
+above. Keep prospective and retrospective evidence separate. The manuscript
+still needs the support/df, amplitude-sign, shared-ratio and prior-art language
+corrected; a numerical recovery is not publication readiness.
+
+The scientific next step is not an unbudgeted larger amplitude ladder. First
+resolve the angular/Smith-class identifiability in #589 and the within-model
+shape question in #622. All p-values here are Gaussian-reference diagnostics
+using estimated jackknife covariance, not exact finite-sample coverage.

notes/projective-support-repair-20260912.md

@@ -0,0 +1,63 @@
+# P3: covariance support repair and N580 recovery without production
+
+Date: 2026-09-12. Companion to the exact-foundations correction (#702).
+
+## Mathematical correction
+
+For nonsingular S and a fixed full-column-rank signed-real design V,
+D=min_a (y-Va)^T S^{-1}(y-Va) has the ordinary Gaussian-reference residual
+interpretation. Two coordinates/one line give exactly Fieller's contrast squared.
+Neither this algebra nor correct Fieller inference creates new information.
+The three-versus-two-rung comparison changes the data used, not the two-rung test.
+
+For singular S whose nullspace is KNOWN as an exact deterministic constraint,
+feasibility requires U0^T(y-Va)=0, where U0 spans ker S. Find a particular
+solution a0 and a null basis Z for U0^T V. On the stochastic support whiten
+with W=Lambda_+^{-1/2} U_+^T and fit W V Z against W(y-Va0). The correct df is
+rank(S)-rank(W V Z), not blindly rank(S)-number_of_original_amplitudes.
+If the support constraint is infeasible, there is no compatible model mean.
+
+Concrete regressions: S=diag(1,1,0), y=(0,0,1), v=(1,0,0) was D=0,p=1 in
+blob `1b6270238db586c5c48849bd607376bf3c7d14fa`; it now raises an explicit
+incompatibility in exact_support mode. For v=(1,0,1), the deterministic
+coordinate forces a=1, giving D=1,df=2. A matrix with eigenvalue -0.1 is
+rejected rather than clipped into an apparently valid covariance.
+
+Default `covariance_mode='strict'` refuses rank-deficient/truncated inputs.
+`exact_support` is an explicit scientific assertion, NOT something inferred
+from too few jackknife batches. Small positive variances cut off numerically
+are not silently made deterministic. Exact-support computation is numerical
+at the recorded precision/tolerance; it is not an exact rational certificate.
+For empirical low rank, explicitly choose/justify a reduced observation space
+or acquire adequate covariance information before using a reference p-value.
+
+A signed line {a*v:a in R} is not a positive ray. Cone/bounded-amplitude tests
+need their own reference law. Estimated covariance likewise does not supply
+exact finite-sample chi-square coverage. These distinctions also correct the
+broader claims in the P3 manuscript; this patch does not promote its readiness.
+
+## N580: recover before replaying
+
+The main-branch files `results/aspect-ladder-n580/shards/rung_r{1,2,4}.json`
+retain 100 aligned `_deleted` rows, plus seed, replica offset and sample count.
+The missing cov(r2,r4) therefore need not be set to zero or commissioned as a
+new run. `scripts/recover_n580_covariance.py` reads these arrays, checks the
+#577 alignment metadata and previously saved diagonal/cross entries, then
+reports the complete covariance, raw curvature and the existing eight ray tests.
+It writes only stdout: historical results and freezes remain untouched.
+
+An additional labelled sensitivity fits a common-modulus H8 nuisance plane and
+bounded common ratios |rho|<=0.2 and <=1. The bounded results are MINIMUM
+RESIDUALS ONLY, not chi-square-calibrated tests. This does not identify H8,
+validate a common ratio across moduli, or turn the reanalysis into new evidence.
+
+## Validation and limits
+
+Locally: 14 projective checks and 2 covariance-reader controls passed. The
+full-rank implementation exactly reproduced all returned fields in 80
+fixed-seed comparisons against the saved original module. No production data
+were changed. The real-shard end-to-end command is also put in a bounded,
+read-only, path-triggered GitHub Actions job (three-minute cap), so it can run
+where a complete repository checkout is available. Its actual result must be
+read before any numerical N580 verdict is claimed. Full repository CI is a
+separate check; it was not run in this local connector-only checkout.

requirements-test.txt

@@ -0,0 +1,5 @@
+-r requirements.txt
+# Existing P598 balanced-realization tests import the scientific Python stack.
+# Keep these test dependencies separate from the lightweight core runtime.
+numpy>=1.24,<3
+scipy>=1.10,<2

results/research-control-20260912/n580-complete-covariance-summary.json

@@ -0,0 +1,48 @@
+{
+  "schema": "matching-one.n580-complete-covariance-summary.20260912",
+  "provenance": {
+    "origin": "Selected fields extracted from successful GitHub Actions stdout; eight D scores independently recomputed locally from the recovered vector and covariance, not from a local shard checkout.",
+    "run_id": 34679359201,
+    "job_id": 103514945963,
+    "head_sha": "eb78265a7016c99c75f5ecc4a162973c448bf4ee",
+    "checkout_sha": "dda2a8e2dcb9e5a4a9215cb1f1877a4aa016b189",
+    "command": "python scripts/recover_n580_covariance.py",
+    "runner_python": "3.11.16",
+    "runner_mpmath": "1.4.1",
+    "local_max_statistic_difference": 0.0
+  },
+  "standing": "Retrospective three-rung reanalysis of the same existing block. Original frozen two-rung verdict unchanged.",
+  "batches": 100,
+  "rungs": [1, 2, 4],
+  "response_vector": [0.0009016433036753112, 0.0029109770266212776, 0.004131807633330795],
+  "covariance": [
+    [6.207020770273794e-8, 1.3480872826970252e-9, -8.011196220988015e-9],
+    [1.3480872826970252e-9, 6.713835319545046e-8, 5.5621485097801135e-9],
+    [-8.011196220988015e-9, 5.5621485097801135e-9, 3.807514679910218e-8]
+  ],
+  "covariance_rank": 3,
+  "covariance_condition_number": 1.9678931965099733,
+  "correlation_r2_r4": 0.11001108422637242,
+  "curvature": {
+    "value": -0.00046630613986373587,
+    "standard_error": 0.0001499079748299409,
+    "z_nominal": -3.110615965513005
+  },
+  "competitors": {
+    "bare_aspect_ratio": {"ray": [1,2,4], "statistic": 10.86445991488253, "degrees_of_freedom": 2, "p_value_nominal": 0.004373332575404287, "equivalent_sigma_diagnostic": 2.849897460536986, "excluded_at_nominal_3_sigma": false},
+    "no_modulus_dependence": {"ray": [1,1,1], "statistic": 89.85851023621422, "degrees_of_freedom": 2, "p_value_nominal": 3.072362291533769e-20, "equivalent_sigma_diagnostic": 9.216402139538845, "excluded_at_nominal_3_sigma": true},
+    "plain_area_scaling": {"ray": [1,4,16], "statistic": 57.512493327374884, "degrees_of_freedom": 2, "p_value_nominal": 3.245792509791233e-13, "equivalent_sigma_diagnostic": 7.2837862071923, "excluded_at_nominal_3_sigma": true},
+    "q4_jordan_weight4": {"ray": [1,2.75,10.9908008589], "statistic": 55.58771053243131, "degrees_of_freedom": 2, "p_value_nominal": 8.497321045543934e-13, "equivalent_sigma_diagnostic": 7.1528843204156916, "excluded_at_nominal_3_sigma": true},
+    "weight12_E12": {"ray": [1,32.515625,2080.30719731], "statistic": 132.26732359383956, "degrees_of_freedom": 2, "p_value_nominal": 1.898959111319179e-29, "equivalent_sigma_diagnostic": 11.267494938720372, "excluded_at_nominal_3_sigma": true},
+    "weight12_E4_cubed": {"ray": [1,20.796875,1327.6635036], "statistic": 132.22611687853123, "degrees_of_freedom": 2, "p_value_nominal": 1.938489879957022e-29, "equivalent_sigma_diagnostic": 11.265680304319943, "excluded_at_nominal_3_sigma": true},
+    "weight12_delta": {"ray": [1,0.125,2.79010739704e-5], "statistic": 513.6592181764852, "degrees_of_freedom": 2, "p_value_nominal": 2.8861439189966654e-112, "equivalent_sigma_diagnostic": null, "excluded_at_nominal_3_sigma": true},
+    "weight8_E8": {"ray": [1,7.5625,120.79770352], "statistic": 115.80437790454678, "degrees_of_freedom": 2, "p_value_nominal": 7.135080016926461e-26, "equivalent_sigma_diagnostic": 10.518028644032594, "excluded_at_nominal_3_sigma": true}
+  },
+  "boundaries": [
+    "Not rejected at the declared nominal 3-sigma threshold does not establish a law.",
+    "Pure-line decisions are conditional on the declared amplitude/readout contract.",
+    "A common H8/H4 ratio is stronger than a uniform bound on rung-specific ratios.",
+    "Estimated jackknife covariance; Gaussian-reference p-values, not exact finite-sample coverage.",
+    "Null extreme-tail sigma is an unresolved numerical conversion, not infinite evidence."
+  ]
+}

results/research-control-20260912/projective-support.json

@@ -0,0 +1,19 @@
+{
+  "local_tests_passed": 14,
+  "full_rank_exact_output_matches": 80,
+  "historical_source_blob": "1b6270238db586c5c48849bd607376bf3c7d14fa",
+  "exact_support_constraint_example": {
+    "statistic": 1.0,
+    "degrees_of_freedom": 2,
+    "amplitudes": [1.0],
+    "covariance_rank": 2,
+    "covariance_condition_number": 1.0,
+    "p_value": 0.6065306597126334,
+    "equivalent_sigma": 0.515031998812217,
+    "covariance_mode": "exact_support",
+    "deterministic_constraint_rank": 1,
+    "fitted_stochastic_rank": 0,
+    "support_numerical_tolerance": "1.0e-25"
+  },
+  "full_repository_suite": "not run locally"
+}