Skip to content

scientific(simulation): own true topic trajectories for realistic rolling-origin K recovery #689

Open

Description

@seonghobae

Finding

#680 now has the numerical pieces needed to evaluate candidate K without temporal preprocessing leakage: #684 fixed-training prevalence-mean prediction, #685 owner-admitted newly available partitions, #686 exact partition↔numerical row binding, #687 one-window predictive K selection, and #688 contiguous multi-window score aggregation.

The remaining acceptance cannot honestly be built from the current tepp_simulation::TruthManifest. The simulation owner currently records latent events, documents, availability, memberships, true/observed relations, method variants and digests, but it does not record a true topic count, topic-term probabilities, document logistic-normal coordinates, prevalence coefficients/covariance, topic trajectories, or generated term counts. SimulationConfig likewise has no topic-measurement DGP parameters. Consequently an apparent rolling-origin K-recovery test would have to hand-author counts outside the truth owner or infer "truth" from the estimator input, defeating the TEST_STRATEGY requirement for true-parameter recovery.

Required owner work

Extend tepp_simulation rather than adding a model-selection-local fixture. The owner contract must be deterministic/reproducible and expose enough truth to build realistic rolling-origin recovery without source rewriting:

  • explicit true K and vocabulary size with bounded nondegenerate topic-term probability matrices;
  • declared prevalence trajectory parameters over EventTime, including stable/interpretable temporal drift and covariance rather than independent per-window topic identities;
  • document latent logistic-normal coordinates and resulting topic mixtures generated from those declared parameters;
  • weighted multilevel/cross-classified/multiple-membership contribution truth aligned with the generated document memberships;
  • relation structure that can affect generated topic coordinates while retaining the existing event/revision/translation/copy truth and connected leakage groups;
  • document term counts generated from the declared topic/content state with nonzero realistic document lengths and overlap/sparsity/imbalance controls;
  • availability delays independent from EventTime semantics so deliberately future-available evidence can be constructed and rejected by corpus_split;
  • immutable truth-manifest digest coverage for all topic DGP parameters, latent document states and term-count observations;
  • deterministic CPU f64 generation with explicit seed domains and no LLM-generated numerical truth.

The public recovery fixture must not be a four-row micro-corpus. It should exercise multiple canonical rolling-origin windows, changing prevalence, overlapping topics, relation-connected observations and weighted memberships; candidate selection must consume #688 and each replication outcome must feed #680's denominator-preserving SelectedKRecoverySummary.

Required RED / acceptance

Before implementation, add a truth-owner RED demonstrating that a generated scenario exposes declared true_k, true topic/content/prevalence state and term counts whose identities align one-to-one with the manifest documents. The manifest digest must change when any scientific truth parameter or generated count changes.

Then add the #680 recovery acceptance separately: recover known K and declared trajectories/states over repeated realistic scenarios, report bias/RMSE/failure denominator/MCSE, and include a deliberately future-available document whose attempted admission is rejected/audited. Synthetic micro-data remains unit coverage; do not promote the owner generator itself as release acceptance without the full recovery experiment.

Boundary

tepp_simulation owns the known-truth DGP. topic_measurement owns the estimator. corpus_split owns rolling-origin/leakage admission. model_selection owns predictive K comparison/recovery summaries. Evidence, Membership and relation promotion remain #527/#658/#671, #605 and #675 respectively. Do not copy those owner contracts or treat simulation truth as production evidence authority.

Refs #639 #680 #688.

Activity

  1. seonghobae commented on Sep 22, 2026

    @seonghobae
    ContributorAuthor

    Ordinary-forward source lineage for the known-topic simulation owner is now on #639's canonical branch.

    • API/manifest RED: 55ec09a462fe95e4c1141c4618d3ef7a5604148b requires a digest-bound true-K/topic/content/prevalence/document-state contract.
    • Bounded DGP config: e53486c0b406ba900d5a173372629875db16f381 adds validated true K, vocabulary/document length, topic separation, temporal drift, logistic-normal residual scale, membership strength, and relation strength.
    • Deterministic Rust DGP: fbe756d9df42d5e408da5504e0470042288eb473 adds a separate topic seed domain, positive normalized topic-term truth, prevalence intercept/slope/covariance, per-document ALR coordinates/mixtures, membership/relation/method contributions, and fixed-length term counts.
    • Corpus binding: c788bf5536ad7c1c9ec5b5111af8735a16ad64f6 generates topic truth from the already-created temporal/event/document/relation corpus and includes every DGP setting in the configuration fingerprint.
    • Manifest integrity: 87d347d64b221d5488d10585db3eecd322cd766a binds the entire topic truth into TruthManifest verification and content hashing using exact f64 bit patterns plus term counts/document identities.
    • Realistic membership RED: ea989ec4c6eeff2f7f5e302ba623998894510bfc requires exact 10,000-bps membership weights plus a recurring classification across events rather than atomized document-only groups.
    • Membership repair: 3d0d23be03bbd92a7384cf0d2b05a992d23c6d11 makes role-specific group identities recur across independently crossed event classifications, preserves derivative-document grouping, and makes weights sum exactly to 10,000 bps.
    • CHANGELOG: 4b48bb5432af3d4c062284aa14ae9104d3bfac7d.
    • ADR 0012 currentization: 03af74c6295674cb344b3b5649accf16179f1cb4 records this as simulation/recovery truth only, not production Evidence/Membership/relation/source authority or scientific(selection): add leakage-safe rolling-origin K validation #680 acceptance.

    The first RED was an API-absence/source RED and the second a realistic behavioral RED; neither is being promoted as hosted RED evidence. Exact-head hosted checks for the current lineage are still non-terminal, so #689 remains open and no GREEN/merge/release claim is made.

  2. seonghobae commented on Sep 22, 2026

    @seonghobae
    ContributorAuthor

    Follow-up: cross-classification edge coverage is now current-head 3f060b20824e1bf92d55e00ed69ca1cbfce79bfd, extending the integration fixture to exercise all four configured classification lanes. #639 body has also been currentized to this exact head and fresh base compare (ahead 245 / behind 0). This is test hardening only; the #689 claim boundary and no-hosted-GREEN status are unchanged.

  3. seonghobae commented on Sep 22, 2026

    @seonghobae
    ContributorAuthor

    Follow-up scientific hardening from #690 is now on the same #639 owner branch: RED 73f4c79a85071d5fe9804b47be746f1931385b95 → config repair 7d4bf62e8dac8afa5edd07392e880d055c59497e → CHANGELOG/current head 9b4aef768ed77db8f1d6786e9e2731f2b24bd968.

    The finding was that SimulationConfig accepted more membership targets than the generator's six unique role labels, while TruthManifest requires per-document role uniqueness. The owner now fails closed above six instead of generating a manifest that violates its own invariant. This is #689 DGP integrity hardening only; it does not change the known-topic arithmetic, production Membership authority, or the still-outstanding #680 replicated rolling-origin recovery acceptance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions