You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR #409 (pareto_candidate_k_v1) currently has deterministic unit/integration examples and selected-K RMSE arithmetic, but no profile-level scientific acceptance evidence showing recovery behavior across realistic data-generating conditions. Hand-built exact/two-point fixtures are useful contracts and are not sufficient scientific acceptance.
Owner boundary
model_selection continues to own Pareto selection and selected-K RMSE arithmetic. This issue owns the scientific validation design/evidence for the Analysis Run profile; it must not introduce an LLM numerical judge, a second Pareto implementation, or mutable external model authority.
Required evidence
Use a realistic Monte Carlo design with true K varied across an explicitly stated design, nontrivial topic separation/prevalence/document-count conditions, deterministic replay seeds, and an explicit attempted/recovered/failed denominator. Report at minimum true-K recovery/selection frequency, selected-K RMSE and bias with Monte Carlo uncertainty, failure/convergence rate, and stability across the stated design. Where temporal availability affects the buyer path, evaluation must be leakage-safe and rolling-origin/cutoff-aware rather than fitting future evidence into earlier runs.
Synthetic data is acceptable for estimator/unit verification only; any product scientific-acceptance claim must be tied to the repository's declared validation-evidence policy and should include a realistic corpus or an explicitly justified generative benchmark. Do not relax thresholds after observing results.
Acceptance boundary
Do not call #409 scientifically accepted or release-ready until this evidence is checked in, TRACEABILITY points to the exact test/study/artifact, and the surviving Analysis Run head has current required checks, coverage, review, and release evidence. This issue does not block fixing the temporal provenance/resource/API defects already identified in #409; it blocks scientific-acceptance claims.
Activity
dk5488 commentedon Sep 19, 2026
I'm interested in working on the Monte Carlo validation for Pareto candidate‑K recovery. Could you please assign this issue to me?