Skip to content

scientific(selection): require identical training state across recovery candidates #700

Open

Description

@seonghobae

Finding

#693 requires every rolling-origin recovery window to cover the full predeclared candidate-K grid, and #694 binds each surviving fit to the exact candidate-specific numerical configuration. The recovery path still permits one scientific comparison defect: two candidate K fits can use the same training document identities but different training numerical state (term counts, EventTime rows, prevalence design, Membership weights, or admitted relation pairs).

rolling_origin_prevalence_mean_predictive_log_likelihood(...) currently binds a fit to a RollingOriginPartition by training identity set. RollingOriginRecoveryEvaluation::new(...) retains each fit's topic dimension/configuration but does not require all candidates in a window to have been fit from one exact ReferenceTopicTrainingInput. A K=2 fit trained on corpus A and a K=3 fit trained on dimension-compatible corpus B can therefore enter the same recovery comparison when A/B use the same document UUIDs.

For scientific K recovery this is invalid. Candidate K is the experimental factor; the training observations and frozen prevalence coordinates must be held constant within a replication/window. Otherwise score differences are confounded with different training data and can make true-K recovery non-reproducible.

Required RED / repair

  • Add a realistic contract using owner-issued ReferenceTopicTrainingFit values with the same training document IDs and declared scientific(selection): bind recovery candidate fits to declared numerical configuration #694 configuration, but alter at least one training count row for one candidate.
  • The recovery composition must fail closed before predictive scoring rather than accepting the candidates as comparable.
  • Keep the comparison inside topic_measurement owner semantics: expose only a numerical training-state equivalence contract sufficient to compare exact retained training inputs; do not mint Evidence/source provenance authority or copy source receipts.
  • Preserve scientific(selection): aggregate candidate K across canonical rolling-origin windows #688 generic operational survivor semantics. This stricter invariant belongs to the scientific-recovery path.
  • Do not compare only document IDs or only PrevalenceDesignBasis: term/count, event-time/design, Membership/relation-derived design and transition coordinates must be covered by the retained owner input.

Boundary

This is numerical experiment integrity only. #527/#658/#671 remain source/Evidence/event-time authentication authorities, #605 remains Membership authority, and #675 remains observed-relation activation authority. A matching training state does not prove external provenance.

Refs #639 #680 #681 #683 #693 #694.

Activity

  1. seonghobae commented on Sep 23, 2026

    @seonghobae
    ContributorAuthor

    Ordinary-forward source repair is now on Draft #639.

    Lineage: realistic source/API RED 22b8e56aaa2996c600dfb24dc90e443d28326bc4 → topic-measurement retained numerical state 75598fdeb52b1f7ed87557f971a680c0bc37db0c → typed PredictiveCandidateTrainingStateMismatch 4979b1761b86b36de239f02a9bc6be85f82a8600 → recovery composition gate 016b6fa3dab72c18ab8dcee36cdb2400f2352531 → CHANGELOG/current exact 4d5c4278c4a303e64abe305fa01b06c4d67c233f.

    ReferenceTopicTrainingInput::shares_numerical_training_state(...) compares the retained numerical estimator state: ordered document identities, canonical term rows/vocabulary, exact EventTime rows, frozen prevalence design/features, and admitted observed transition endpoint pairs. RollingOriginRecoveryEvaluation::new(...) now refuses mixed-state candidate fits before predictive scoring. Generic #688 operational selection is unchanged.

    This remains numerical experiment integrity only. It does not authenticate Evidence/source provenance, Membership ownership, relation activation, or availability receipts. Exact-head Actions are currently queued with Rust Foundation jobs not yet assigned to a runner, so no hosted GREEN or RED is claimed. Keep #700 open until exact-head gate evidence is terminal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions