Skip to content

scientific(coverage): predeclare recovery-fit numerical design before the 10k run #754

Open

Description

@seonghobae

Finding

The prospective coverage design and simulation scenario are now versioned/fingerprinted, but analysis_engine::coverage_calibration_execution still creates the truth-K estimator configuration from analysis-local constants: seeds [7, 11, 19], maximum iterations 2_000, tolerance 0.001, plus the generic FittedCandidateKConfig ADR defaults. Those choices materially affect convergence, the numerical-failure denominator, fitted covariance, and interval coverage.

#694/#706 ensure that scientific recovery fits match the FittedCandidateKConfig supplied for a run, but they do not prospectively identify which FittedCandidateKConfig the 10,000-DGP coverage experiment must use. A future executor can therefore change initialization seeds, convergence budget/tolerance, or recovery hyperparameters while retaining the same tepp.coverage.nominal95.v1 validation fingerprint and tepp.simulation.rolling_origin_coverage.v1 scenario fingerprint. Exact source_head would reveal that code changed, but the experimental numerical design would not be a separately versioned owner contract fixed before outcomes are observed.

First owner repair

Introduce a model_selection-owned coverage-calibration fit design that:

  • has a stable versioned identity distinct from validation and simulation identities;
  • declares the current candidate strategy as one truth-K candidate per rolling-origin fit;
  • owns the exact estimator seeds [7, 11, 19], 2,000-iteration budget, tolerance 0.001, prior variance 1.0, relation strength 0.25, ridge 0.01, topic smoothing 0.05, and step size 0.2 explicitly rather than inheriting mutable generic defaults;
  • constructs the exact FittedCandidateKConfig for a supplied truth K and rejects invalid K through the existing model-selection owner;
  • exposes a deterministic domain-separated SHA-256 fingerprint for the concrete truth-K configuration using canonical integer/string lengths and exact binary64 bits;
  • pins the v1 fingerprint in a public contract and proves the produced FittedCandidateKConfig retains the declared seeds/iteration/tolerance.

This is the foundation prerequisite. The next consumer repair must make coverage_calibration_execution consume this owner instead of its local constants and bind the fit-design identity/fingerprint into shard/final evidence before the 10,000-DGP run. Do not copy this scientific truth into validation or simulation owners.

Boundary

This does not execute the 10,000-DGP study, change estimator arithmetic, define numerical-failure acceptability, or promote #680. model_selection owns the numerical fit design; analysis_engine remains the composition owner; validation_core remains the coverage criterion/evidence arithmetic owner; tepp_simulation remains the DGP owner.

Refs #639 #680 #694 #706 #733 #753.

Activity

  1. seonghobae commented on Sep 24, 2026

    @seonghobae
    ContributorAuthor

    Owner-foundation ordinary-forward status on #639:

    • f05313de79e11db4c224884e69e3e544c4982a54: source compile RED requiring public CoverageCalibrationFitDesign and a pinned concrete truth-K fingerprint. This exact RED was not observed failing on a hosted runner, so it remains source-level RED only.
    • c022d88e3ef17b306db0586af9ae2396427b157c: added the model-selection owner with explicit seeds [7, 11, 19], 2,000 iterations, tolerance 0.001, and explicit 1.0 / 0.25 / 0.01 / 0.05 / 0.2 reference hyperparameters plus domain-separated SHA-256.
    • 50b781dbf931797cb634cd0b8583b6bc4cdaf6ab: added the existing workspace sha2 dependency to model_selection.
    • e0992bf2700072dce5564bf4f4593336e3f0f46c: exported the owner from the crate root.
    • 2a205557f561de34c52290d65c9f4f3607e7e238: repaired the seed manifest to a static owner lifetime before hosted validation.
    • 4eb6eb655ec8386f8667273a33e02de9862586d0: recorded the CHANGELOG and is the current exact source head.

    The pinned v1 fingerprint for concrete truth K=3 is f7a353e89aa97a33a51b776406138a53fe5e776d1987c31fade270868bf55983.

    This is deliberately only the foundation prerequisite. At this exact head analysis_engine::coverage_calibration_execution still owns the old local fit constants, and shard/final schema-v9 evidence does not yet bind the model-selection fit-design identity/fingerprint. Keep #754 open through that consumer/evidence repair; do not treat this slice as the 10,000-DGP run or #680 acceptance.

  2. seonghobae commented on Sep 24, 2026

    @seonghobae
    ContributorAuthor

    Second ordinary-forward slice landed on the same owner lane:

    • 8ee5cec07aa480dca221a89c5871bd4180701879: analysis_engine::coverage_calibration_execution now consumes CoverageCalibrationFitDesign::truth_k_v1().config_for_truth_k(truth_k). The diff against the prior exact head is only this file, 7 additions / 12 deletions, removing the local [7,11,19], 2_000, 0.001 constants and direct FittedCandidateKConfig::new(...) construction.
    • 1cdce36099ff04cc0e49a53c6a0302fae699108c: CHANGELOG currentization; current exact fix(release): bind fail-closed reconstructable topic-lineage results (#638) #639 head.

    The remaining #754 gap is now narrower: shard and final schema-v9 evidence still omit the model-selection fit-design identity/fingerprint. Keep this issue open until those persisted/recovery contracts bind the owner-issued numerical design. The 10,000-DGP run remains unexecuted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions