scientific(selection): add leakage-safe rolling-origin K validation

Finding

#679 corrects the name of the current fitted-selection diagnostic: TEPP maximizes an in-sample Schwarz score ℓ - (p ln N)/2; it does not currently measure held-out predictive likelihood. The existing selected_k_root_mean_square_error(...) fixture shows recovery only on a small fixed corpus under alternative seeds. That is useful regression coverage, but it is not the leakage-safe scientific acceptance evidence required for temporal relational psychometrics.

This matters especially for TRSL-TM because document availability, event-valid time, relation-connected groups, time-varying Membership, and topic prevalence all create leakage channels that an ordinary random document split can violate. corpus_split already owns strictly ordered rolling_origin_windows(...); model selection is not consuming it.

Current STM practice also keeps held-out likelihood distinct from in-sample/bound diagnostics. stm 1.3.8 searchK computes held-out likelihood by document completion in addition to semantic coherence, exclusivity, residual dispersion and bounds; see Roberts, Stewart, & Tingley (2019), Journal of Statistical Software, 91(2), https://doi.org/10.18637/jss.v091.i02.

Required scientific RED / acceptance evidence

Use realistic, non-toy recovery data and owner temporal boundaries to demonstrate that candidate-K selection is evaluated out of temporal sample:

build strictly increasing KnowledgeCutoff rolling-origin windows through corpus_split rather than random splitting;
train evidence must satisfy the train cutoff; evaluation evidence may be available only by the next cutoff and must never be visible during fitting;
preserve relation-connected grouping and multilevel/cross-classified/multiple-membership structure rather than atomizing documents;
fit candidate K only on the train state, then evaluate a separately named predictive/held-out diagnostic on admissible evaluation evidence without re-estimating against that evidence;
report selected-K RMSE/bias across replications plus failure denominator and Monte Carlo uncertainty; retain the in-sample Schwarz score as a separate diagnostic rather than relabeling it;
prove a deliberately leaked future-evidence fixture fails or produces a distinct rejected audit receipt;
preserve owner-authenticated future EventTime that was already available before cutoff; do not conflate EventTime with KnowledgeCutoff (scientific(time): bind admitted document event times to cutoff-qualified source evidence #671/scientific(cutoff): bind corpus snapshots to one knowledge horizon #672).

Synthetic micro-data may remain unit coverage, but scientific acceptance must use true-parameter trajectory/state recovery under realistic temporal composition as required by the TEPP test strategy.

Design boundary

Do not implement a second corpus splitter in model_selection. Consume the canonical corpus_split rolling-origin/connected-group contracts. Do not copy Evidence receipts, Membership projections, or relation promotion logic. #527/#658/#671/#675 and #605 remain the authorities for those identities once landed/released.

No LLM output may choose K or certify recovery.

Literature / traceability

Roberts, M. E., Stewart, B. M., & Tingley, D. (2019). stm: An R Package for Structural Topic Models. Journal of Statistical Software, 91(2), 1–40. https://doi.org/10.18637/jss.v091.i02.
Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464. The existing TEPP score is a Schwarz-style in-sample criterion and must remain distinct from held-out evidence.

Refs #638 #639 #671 #672 #676 #677 #679.

test(selection): require recovery denominator and Monte Carlo error (#…

feat(selection): type insufficient recovery replication failures (#680)

feat(selection): preserve recovery failures and Monte Carlo error (#680)

feat(selection): export recovery acceptance summary (#680)

test(selection): harden recovery summary edge cases (#680)

docs(selection): record recovery denominator and Monte Carlo contract (…

Ordinary-forward progress on the recovery-evidence part of #680:

API RED a9be59b66df98d2031a772d02d5622595e0cbe02 adds a public contract that requires recovery evidence to retain attempted/successful/failed replication counts together with bias, RMSE, and Monte Carlo standard errors. The RED head's hosted workflows were cancelled by subsequent forward commits, so this is source-level RED evidence only.
a7bfaba115c5e6c97d53ac167c1230c21d57d418 adds a typed InsufficientRecoveryReplications failure.
db20fc249c5af0f69083a9eb85b87f06417784e6 adds SelectedKRecoverySummary / selected_k_recovery_summary. Bias and RMSE are explicitly conditional on successful replications; attempted and failed counts remain in the same result so failed fits cannot disappear from the denominator. Bias MCSE uses the sample variance of selected-K residuals; RMSE MCSE uses the delta method on MSE. Fewer than two successful replications fail closed.
fef0c07afe3e0f469e1ebe3b1193dcae8795afa8 exports the owner contract.
eee4cb7a794ab5b8cb217741d26b64597c89d101 hardens empty/all-failed/single-success/exact-recovery cases.
f7e1b10780cf59b91b4ce161055590dd9e4fdcf8 records the contract boundary in CHANGELOG.

This does not close #680. It closes only the recovery-summary denominator/Monte-Carlo-uncertainty prerequisite. Canonical rolling-origin train/evaluation admission and a separately measured held-out/predictive diagnostic remain outstanding. The existing in-sample Schwarz score remains separate and is not being relabeled as predictive evidence.

Review finding for the next slice: ReferenceTopicInput currently standardizes EventTime independently inside each constructed input. A future rolling-origin evaluator cannot safely reuse fitted prevalence coefficients on a separately constructed evaluation input unless the training-time prevalence transform/basis is retained and reapplied; independently standardizing the evaluation horizon would make the evaluation design depend on test-horizon statistics. I am leaving that as an owner-design prerequisite rather than fabricating a held-out score that silently uses a different coordinate system.

refactor(selection): remove unreachable recovery finite branch (#680)

Follow-up on the recovery-summary slice: c7eef41e354c65c9c470be7bff62dde56bc29de0 removes the post-computation is_finite() failure branch from selected_k_recovery_summary. With u32 topic counts and a bounded usize replication vector, the residual/squared-error aggregate cannot approach binary64 overflow, so that branch was unreachable owner production code and would create a false branch-coverage obligation. The typed invalid-K and insufficient-replication failures remain. No acceptance threshold or denominator was weakened.

#682 adds one missing scientific-recovery evidence component without changing #680's acceptance boundary. On #639, RED d9a00a0... → repair 813b9c9... → CHANGELOG c277f85... now gives the empirical failure proportion its own Monte Carlo SE, sqrt(p_hat(1-p_hat)/R), over all attempted replications. Conditional bias/RMSE semantics are unchanged.

This is still prerequisite-only: canonical corpus_split rolling-origin windows, #681 frozen training prevalence projection, fixed-training-state predictive evaluation, realistic temporal/multilevel recovery, and deliberate future-leak rejection remain required here.

#681 prerequisite now has an ordinary-forward source repair on #639 through ADR/current head 09ed71ae95850148321d85e0307e91e1e806869c: ReferenceTopicTrainingInput atomically binds the admitted training input to a frozen PrevalenceDesignBasis, and evaluation rows reuse the training EventTime origin/location/scale plus exact ordered prevalence-feature basis. A/B coverage fixes the same focal evaluation row across different future batch companions; unseen covariate/Membership coordinates fail closed.

This removes the preprocessing-coordinate leakage prerequisite but does not satisfy #680 itself. The remaining #680 work is still canonical corpus_split::rolling_origin_windows, fixed-training-state held-out/document-completion scoring without global-parameter re-estimation, connected/multilevel/multiple-membership preservation, deliberately leaked-future rejection/audit evidence, and realistic recovery with the #680/#682 denominator + Monte Carlo summaries. Do not relabel the existing Schwarz score as held-out evidence.

Next implementation guard after #681: the predictive scorer should consume one owner-issued training state (fit plus its frozen prevalence basis), not accept a detachable ReferenceTopicFit and PrevalenceDesignBasis as two independent public arguments. Otherwise an A-fit/B-basis substitution would recreate the same dimension-compatible authority gap already repaired in #665/#670. This can be done inside #680 when the fixed-global-parameter scorer is added; no second basis/digest authority is needed.

Follow-on prerequisite from #681: #683 now binds the frozen training prevalence basis to the exact fitted training state instead of leaving ReferenceTopicFit and PrevalenceDesignBasis independently pairable. Source lineage on #639 is RED 1c1b85aa... → owner aggregate d650c1dc... → export eaf205e7... → CHANGELOG/current head ec13e76c....

This removes one substitution path before held-out scoring, but #680 remains open: canonical rolling-origin admission, fixed-global-parameter predictive/document-completion evaluation, relation/group-preserving realistic recovery, and deliberate future-leak rejection/audit evidence are still required.

#684 now supplies the next numerical prerequisite on #639: RED 5491f6cc... → fixed-training prevalence-mean scorer 3d6cd4e... → CHANGELOG e921ddf... → owner/coverage cleanup 0b5dfa2... → edge hardening/current head 87e5378f....

The scorer consumes the #683 owner-issued ReferenceTopicTrainingFit, reuses #681's frozen training EventTime/feature coordinates, and scores evaluation counts with retained training prevalence coefficients + topic-term probabilities without updating global parameters from evaluation counts. A/B coverage keeps the same focal evaluation score unchanged when the surrounding future batch differs.

This still does not satisfy #680. It is explicitly not STM document-completion likelihood and carries no cutoff/source-admission authority. Canonical corpus_split::rolling_origin_windows, cutoff-qualified evaluation admission, relation/group/multilevel/multiple-membership-preserving realistic recovery, deliberate future-evidence rejection/audit evidence, and recovery acceptance using the existing denominator/MCSE summaries remain outstanding.

Coverage-calibration execution provenance advanced on #639 through #739. Exact source head 015d9a5a73f367ef04f3131b01658e0b5691a28f makes digest-verified shard rehydration the only public persistence/resume path; the strict JSON-only parser is private and schema/source/design/scenario/range/outcome drift tests use the originally persisted owner digest. This is reconstructability/integrity work only. The prospectively fixed tepp.coverage.nominal95.v1 × tepp.simulation.rolling_origin_coverage.v1 10,000-DGP study has still not been executed or persisted as schema-v4 evidence, and no post-hoc numerical-failure acceptance threshold is introduced.

#740 prospective-design repair is now ordinary-forward on #639 through aeee7cdd2a9ddd6e1f6b24e43f9a580afaf1bb82: CoverageCalibrationDesign::tepp_nominal_95_v1() owns the exact two-sided normal critical value 1.959963984540054, fingerprint v2 binds its IEEE-754 bits, and the realistic executor consumes that owner value rather than a local constant. The complete design fingerprint is now 4f937a115d273f5e1171de03eeeea92e30681772849a590365f4933668e8e557.

This closes prospective interval-construction identity only. It does not satisfy #680: the 10,000 independent-DGP run has not been executed/persisted, no numerical-failure-rate acceptance threshold is added post hoc, and no scientific/release claim is promoted.

Prospective coverage-design integrity advanced through #741 on #639 current head 9c757db9e7e0f38ba7a1ebd60c55c7c9694393b1.

Review found that #740 completed the numerical criterion/interval-rule identity, but the design fingerprint still did not state which population and repeated-observation aggregation the criterion applied to. The current executor admits leakage-safe rolling-origin partitions, fits on the admitted training snapshot, constructs fit-bound ALR marginal intervals for those fitted training documents, pools document×coordinate intervals into one coverage proportion per window, then gives the declared windows equal weight inside each independent DGP before between-DGP Monte Carlo inference. That is now explicitly owned as tepp.coverage.estimand.training_fit_alr_marginal_equal_window.v1 and bound into fingerprint domain v3. The resulting design fingerprint is a3a0b8d65388627d2360b05731c5e410feac343a6068da4f9192a353791bc0c0.

This is important for #680 claim separation: the coverage-calibration estimand is training-fit latent-state interval calibration and is not held-out/predictive calibration. Existing held-out predictive/state recovery remains a separate validation claim. Nominal 0.95, z 1.959963984540054, [0.91,0.98], 10,000 attempts, MCSE <=0.005, and the no-post-hoc-failure-threshold boundary are unchanged.

#680 remains open. The actual 10,000 independent-DGP run still has not been executed/persisted, and exact-head hosted gates/current independent review remain outstanding.

#741 follow-up: final coverage evidence is now schema v5 on #639 head bdb641f091a2456d2c0cf042ecff677b38324a36, adding explicit validation_estimand_id beside the design ID/fingerprint. This makes the training-fit ALR marginal equal-window calibration claim self-describing in persisted evidence rather than requiring fingerprint lookup. The scientific boundary is unchanged: this is not held-out/predictive calibration, and the 10,000-DGP run remains unexecuted.

#742 hardens the persisted coverage-evidence claim boundary on #639 current head 08958f62e725fb1e5ed47ac701ac3cfb101ee96e. Evidence schema v6 no longer serializes/exposes one aggregate supports_calibration_claim boolean while numerical-failure acceptability remains prospectively undefined; it retains separate practical-band and MCSE decisions plus attempted/success/failure counts, failure rate and failure-rate MCSE.

This is not #680 acceptance evidence and does not resolve held-out predictive/state validation. The tepp.coverage.nominal95.v1 × tepp.simulation.rolling_origin_coverage.v1 10,000-DGP run is still unexecuted, and no post-hoc numerical-failure threshold has been introduced.

#743 currentizes the coverage-calibration claim surface on PR #639: CoverageCalibrationAssessment::supports_calibration_claim() is removed at exact head 30e916b9101e3f4e03c4e07fca4828319bb2403f, leaving the practical-band decision, MCSE decision, and unconditional failure denominator as separate evidence. This does not satisfy #680 and does not add a post-hoc numerical-failure threshold. The exact-source/scenario/design/estimand 10,000-DGP run and the separate held-out predictive/state acceptance remain outstanding.

Coverage-calibration prerequisite currentization on #639: #745 now makes empirical reporting percentiles 0.025/0.975 part of the prospective validation owner and coverage-design fingerprint (domain v4; canonical fingerprint 0d4aa337cbc22b91f0ca39db3a44dcdac9f56464946e9f6585f6678fccf3908e). Schema-v6 evidence no longer accepts caller-selected percentile probabilities, and final shard assembly consumes the owner design directly.

This is reporting-configuration provenance, not #680 acceptance evidence. The exact-head 10,000-DGP run has not been executed, numerical-failure acceptability remains undefined, and the training-fit interval calibration estimand remains distinct from #680 held-out predictive/state validation.

#746 currentizes the prospective reporting-method provenance before the 10,000-DGP run. The coverage design now fingerprints tepp.coverage.percentile.inclusive_nearest_rank.v1 together with the existing 0.025/0.975 probabilities; schema-v7 evidence persists that method identity. Canonical design fingerprint is b4733bdaea49439e1d45749227d5e6131ebd14a0fb56df3dc0878e4fc3bf9954 at #639 head 9c5544fc87c8eb7ae88884c7efbcdea092da7018.

This is provenance/currentization, not acceptance evidence. The exact-source 10,000-DGP study and #680's separate held-out predictive/state validation remain unexecuted; no post-hoc numerical-failure-rate threshold was introduced.

#747 currentizes the prospective coverage precision provenance on #639: tepp.coverage.nominal95.v1 now binds tepp.coverage.mcse.sample_sd_n_minus_1_over_sqrt_n.v1, the existing Welford/sample-variance (n-1) / sqrt(n) coverage-mean MCSE estimator. Fingerprint domain is v6, canonical design fingerprint is a12ca4a5a6bba7f3bb7af926b937c2efe80c60b48f7f6a5193d3237e889e5a81, and durable calibration evidence is schema v8 with the MCSE method ID persisted. This changes no numerical arithmetic or acceptance threshold and is not acceptance evidence. The 10,000-DGP training-fit coverage run remains unexecuted, and this issue's leakage-safe held-out K/state/predictive validation remains a separate unmet scientific claim.

#748 adds one prospective reporting-provenance binding on #639: the unconditional numerical-failure-rate MCSE already computed as sqrt(p_hat * (1 - p_hat) / n) now has validation-owner identity tepp.coverage.failure_rate_mcse.bernoulli_plugin_sqrt_p_one_minus_p_over_n.v1, is included in coverage-design fingerprint domain v7, and is persisted in schema-v9 evidence. Current #639 head is 6ea0dea9ed2bfec55d836ced9fbee1d4dd68f277; current design fingerprint is 1342c76126e06d3d56c9fbd49bb8d7d5a2b59fa95835a7bb39959e4f4ebd562b.

This does not supply #680 acceptance evidence, does not turn training-fit interval calibration into held-out/predictive validation, and does not define an acceptable numerical-failure rate. The exact 10,000-DGP run and separate held-out/state requirements remain open.

#749 adds a final-evidence recovery prerequisite on #639: persisted schema-v9 CoverageCalibrationEvidenceRecord JSON is now reconstructed from its canonical indexed outcome ledger through the validation owner and accepted only when every persisted field matches owner recomputation. This is evidence recovery/integrity, not 10,000-DGP acceptance evidence and not external storage/build authentication. The tepp.coverage.nominal95.v1 × tepp.simulation.rolling_origin_coverage.v1 10,000-DGP run remains unexecuted; held-out predictive/state validation here also remains separate. No numerical-failure-rate acceptance threshold was added.

#750 repairs prospective coverage-estimand completeness before any acceptance run: tepp.coverage.nominal95.v1 now owns exactly five rolling-origin windows per successful DGP, binds that count in design fingerprint v8 (2871ae32842a65273d15d1106b977244981cbf6bd69773ec44d427a35ba4d130), and rejects shortened/expanded successful window vectors before Monte Carlo aggregation. Evidence schema remains v9; numerical failures remain explicit None outcomes in the unconditional attempted denominator. This does not execute the 10,000-DGP study, does not satisfy held-out predictive/state validation, and does not define a numerical-failure-rate acceptance threshold.

Coverage-calibration owner-path handoff: #751 repaired a duplicated cardinality authority left after #750. Current #639 exact head c7db4b224084397a39d4877c64739f00c03e4705 makes analysis_engine consume the validation-owned declared rolling-origin window count instead of a local 5, and currentizes the study rustdoc to schema-v9. This is contract/DDD repair only. The tepp.coverage.nominal95.v1 × tepp.simulation.rolling_origin_coverage.v1 10,000 independent-DGP study is still unexecuted, and this training-fit interval calibration remains distinct from #680's held-out predictive/state acceptance. No numerical-failure-rate acceptance threshold is introduced.

#753 is now implemented on #639 exact head 5eaa422e318903407c1ed8a2ee0a7e82c07d2b0f as a prospective cutoff-geometry provenance/owner repair, not acceptance evidence. tepp.coverage.nominal95.v1 now binds geometry ID tepp.coverage.rolling_origin_geometry.latest_event_availability_adjacent_expanding.v1, first training event index 3, and declared window count 5 under fingerprint domain v9; canonical fingerprint is f6fd3855ec7ef413309cbb0b48b3572853d4fa87d4c46e47f265e06074aeffa0. Execution consumes the owner start index while actual cutoff/snapshot/rolling-origin construction remains in analysis/corpus owners. The exact-head 10,000-DGP schema-v9 evidence and the separate held-out predictive/state validation required by #680 are still unexecuted; no numerical-failure acceptability threshold was introduced.

#754 adds a missing prospective numerical-fit owner before the large coverage run. The current executor had local [7,11,19] / 2000 / 0.001 fit controls plus generic reference hyperparameter defaults; these can change convergence, the numerical-failure denominator, covariance, and coverage without changing the validation/scenario identities.

Current #639 exact head 4eb6eb655ec8386f8667273a33e02de9862586d0 now has CoverageCalibrationFitDesign::truth_k_v1() in model_selection, with explicit fit controls/hyperparameters and a domain-separated concrete truth-K fingerprint. This is prerequisite provenance only: the executor still has to consume that owner and shard/final evidence still has to bind its identity/fingerprint. No 10,000-DGP outcome has been executed or promoted, no failure-rate acceptance policy is introduced, and held-out predictive/state acceptance under #680 remains separate.

Navigation Menu

Repository navigation

scientific(selection): add leakage-safe rolling-origin K validation #680

Finding

Required scientific RED / acceptance evidence

Design boundary

Literature / traceability

61 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

scientific(selection): add leakage-safe rolling-origin K validation #680

Description

Finding

Required scientific RED / acceptance evidence

Design boundary

Literature / traceability

Activity

seonghobae commented on Sep 22, 2026

seonghobae commented on Sep 22, 2026

seonghobae commented on Sep 22, 2026

seonghobae commented on Sep 22, 2026

seonghobae commented on Sep 22, 2026

seonghobae commented on Sep 22, 2026

seonghobae commented on Sep 22, 2026

61 remaining items

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

seonghobae commented on Sep 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions