Limited-MC-statistics de-biasing for binned template fits (continuous-M + cross-fit)

[bendavid]

Limited-MC-statistics de-biasing for binned template fits

Adds opt-in de-biasing of the bias and under-coverage that finite MC template
statistics induce in chi-squared / profile-likelihood template fits (the
errors-in-variables / Neyman–Scott "incidental parameters" effect). Two
complementary methods, both injected at the scalar-NLL level so they flow through
rabbit's existing TF-autodiff grad/Hessian/HVP machinery and SciPy minimizers.

Draft — opening for design review and discussion of the interfaces before polish
(the test scripts are standalone validation drivers, not yet pytest/CI-wired).

Methods

• continuous-M — a frozen noise-floor matrix M (supplied to the TensorWriter)
  enters the objective as -½ θᵀ M θ, giving the de-biased point (H−M)⁻¹g and
  curvature H−M. Lowest variance; requires the de-biased parameters to enter the
  prediction linearly (e.g. --allowNegativeParam POIs); a warning fires otherwise.
• two-half / k-fold cross-fit — the user fills histograms with an extra MC-stat
  fold_axis; the writer slices it (full derived by projecting it out). The fitter
  forms the jackknife 2L_full − ½L_A − ½L_B; the --covMode fisher curvature uses
  the complete pairwise U-statistic over all k folds (k-fold averaging, O(k), no
  partition enumeration, → continuous-M as k→N). General / nonlinearity-robust.

Interfaces

• TensorWriter.add_mc_stat_moment(M, param_names); add_process(..., fold_axis=),
  add_systematic(..., fold_axis=) (split-logk, symmetric/asymmetric, dense/sparse).
• --mcStatDebias {none,continuousM,twoHalf,kfold},
  --mcStatDebiasCov {curvature,sandwich,dataPropagated},
  --covMode {observed,fisher} (applies to the standard cov too).
• Robust (sandwich) covariance; BB-lite compatible; the de-biased covariance flows
  through to impacts (a new mcStatDebias impact group) and output histograms.

Validation

• Per-toy exact match to an independent numpy reference (de-biased point,
  curvature, sandwich) for continuous-M and two-half; k-fold U-statistic == numpy and
  RMS↓ with k; sparse == dense and split-logk sparse == dense to 1e-6.
• A native template-fluctuating coverage harness (tests/mcstat_coverage.py):
  the de-bias removes the central-value attenuation bias and inflates σ→σ∞; the
  proper sandwich (BB-lite σ_MC in the meat) covers ~0.68 with no calibration,
  while the bare-Hessian meat under-covers and the data-propagated meat over-covers.

Known limitations (documented)

• continuous-M needs linear parameters; for nonlinear use two-half.
• BB-lite on pathologically degenerate problems can give a rank-deficient covariance.
• dataPropagated/sparse-split-logk are continuous-M/dense and log_normal-symmetric
  respectively; large-nparams --noHessian (matrix-free) de-biased covariance is
  not yet implemented (the HVPs are automatic; only a sandwich-aware Hessian-free row
  extractor is missing).

🤖 Generated with Claude Code