feat: re-derive as-published history by default

CHANGELOG.md

@@ -9,6 +9,10 @@ their own keyspaces. Newest first.
 
 ## 2026-09-08
 
+`./reproduce` now re-derives the as-published history end to end under the version
+live at each stamp, using each version's own lookback history. `--receipts` offers
+the fast per-observation check, and `--version` selects a single version.
+
 Published history now serves the version that was live at each observation. A
 methodology version's keyspace holds that version's full re-derivation from its
 first observation, including rows before its effective time, for verification;

README.md

@@ -64,54 +64,64 @@ pip install -e .
 install local; on systems that allow bare `pip install`, the venv lines are
 optional.)
 
-That verifies the current UTC day's published receipts and artifact digests.
-By default it reads the as-published keyspace advertised by `latest.json` as
-`data.versions[].history_path`. Until that field is available, it prints a
-notice and falls back to `current_version`. Each result names the version
-and `methodology_id` verified.
-
-Exit 0: every verifiable value matched; withheld receipts degrade to digest-only
-verification with a notice. Exit 1 means a mismatch, invalid artifact, or digest
-failure; exit 2 means verification could not run or the arguments were invalid.
-To check another accelerator or time, run
+That re-derives the current UTC day's as-published index end to end, including
+attendance events and factors, liveness scores, weights, votes, IQM, value, and
+stability band. It reads the history advertised by `latest.json` as
+`data.versions[].history_path`, selects the version live at each stamp by its
+succession `effective_from`, and uses that version's own `v<n>/` lookback history.
+Before public launch, it selects the launch version. Each result reports the
+version and `methodology_id`; observations before that version's effective time
+are labeled `back-calculated`.
+
+The raw inputs are disclosed prices and dispersions, recorded currency and FX,
+upstream status, carry basis, filter verdicts, timing, top-level flags, and
+`calc_params`. Published attendance factors, liveness scores, and weights are
+comparison outputs, never derivation inputs. Every artifact read is digest
+verified. Missing required history or withheld raw inputs cause a full-reproduction
+refusal. If `history_path` is absent, the command prints a notice and uses
+`current_version` (or the legacy flat record when no version pointer exists).
+
+Exit 0: every verifiable value matched. Exit 1 means a mismatch, invalid artifact,
+or digest failure; exit 2 means verification could not run or the arguments were
+invalid. To check another accelerator or time, run
 `./reproduce <h100|h200|b300|b200> YYYY-MM-DD`; add `THH` to check one UTC
-hour. `--receipts` explicitly selects this default receipts check.
+hour. A full day takes a few minutes.
 
-To verify a particular version's re-derivation, pass `--version <n>`:
+`--full` explicitly selects the default. `--receipts` is the fast opt-in: it
+recomputes each value and band from that observation's own published receipts,
+without re-deriving attendance or weights. In receipts mode only, withheld
+contributing receipts degrade to digest-only verification with a notice.
+
+To verify one version's re-derivation throughout, pass `--version <n>`:
 
 ```
+./reproduce --version 5 h100 2026-09-01
+./reproduce --version 6 h100 2026-09-01
 ./reproduce --receipts --version 5 h100 2026-09-01
-./reproduce --receipts --version 6 h100 2026-09-01
-./reproduce --full --version 5 h100 2026-09-01
 ```
 
-Explicit-version observations before that version's `effective_from` are labeled
-`back-calculated`. `--full` derives attendance events and factors, liveness
-scores, weights, votes, and values from raw disclosed history without consuming
-published derived intermediates. It stays within one version and takes a few
-minutes; pass `--version <n>`. During the cutover, `--full` without a version
-retains the current-version fallback only when `history_path` is absent.
+Rows before the selected version's `effective_from` are labeled
+`back-calculated` in both modes.
 
-Recorded 2026-09-01 under version 5; reproduce it with
-`./reproduce --full --version 5 h100 2026-09-01`. Taken live against
-https://data.getcomputable.com at 07:29 UTC and left exactly as recorded; the
-repeated middle lines and each observation's derived 16-source weight vector
-are elided.
+Recorded 2026-09-08 against https://data.getcomputable.com from a checkout with
+no `./data`. This day spans two versions: 76 observations under version 5 and
+20 under version 6. Repeated lines and derived source weight vectors are elided.
 
 ```
-$ ./reproduce h100 "$(date -u +%F)"
+$ ./reproduce h100 2026-09-03
 published record: full history via public HTTPS front https://data.getcomputable.com
 raw-only full reproduction: prices, dispersions, upstream status, carry basis, filter verdicts, timing, top-level flags, and calc_params are inputs; published derived intermediates are not
-H100 2026-09-01T00 derived 3.456577 (band 0.556277) published 3.456577 (band 0.556277) MATCH public digests OK
-H100 2026-09-01T00:15 derived 3.457619 (band 0.557319) published 3.457619 (band 0.557319) MATCH public digests OK
-[... 26 more MATCH lines, each followed by its derived weight vector ...]
-H100 2026-09-01T07 derived 3.465561 (band 0.565261) published 3.465561 (band 0.565261) MATCH public digests OK
-H100 2026-09-01T07:15 derived 3.465393 (band 0.565093) published 3.465393 (band 0.565093) MATCH public digests OK
-summary: 30 observation(s): 30 MATCH, 0 MISMATCH
+H100 2026-09-03T00 derived 3.519725 (band 0.520902) published 3.519725 (band 0.520902) MATCH public digests OK version 5 methodology_id h100_sxm_v1_calc_v8
+H100 2026-09-03T00:15 derived 3.519749 (band 0.520978) published 3.519749 (band 0.520978) MATCH public digests OK version 5 methodology_id h100_sxm_v1_calc_v8
+[... 73 more MATCH lines under version 5 ...]
+H100 2026-09-03T18:45 derived 3.505789 (band 0.605489) published 3.505789 (band 0.605489) MATCH public digests OK version 5 methodology_id h100_sxm_v1_calc_v8
+H100 2026-09-03T19 derived 3.508224 (band 0.518224) published 3.508224 (band 0.518224) MATCH public digests OK version 6 methodology_id h100_sxm_v1_calc_v10
+[... 18 more MATCH lines under version 6 ...]
+H100 2026-09-03T23:45 derived 3.547149 (band 0.646849) published 3.547149 (band 0.646849) MATCH public digests OK version 6 methodology_id h100_sxm_v1_calc_v10
+summary: 96 observation(s): 96 MATCH, 0 MISMATCH, 0 degraded
 ```
 
-Run `./reproduce --receipts` for the receipts-only value check; `--producer`
-and `--lane` replay a local collection record rather than the published one.
+`--producer` and `--lane` replay a local collection record.
 
 To compare the latest print with prices visible at its sources now, run
 `./reproduce --collect <h100|h200|b300|b200>`. It reports `SAME`, `MOVED`,

reproduce

@@ -15,18 +15,19 @@
 # files under observations/YYYY/MM/DD.json) -- a recompute-and-match run
 # (scripts/verify_published_record.py): every file's envelope digest is
 # verified, and each observation's index value and stability band are
-# recomputed from the published receipts and matched exactly. The record
+# re-derived from raw history and matched exactly. The record
 # is read from a local downloaded copy under GPU_INDEX_DATA_DIR (default
 # ./data) when it holds the requested day; otherwise from the public
 # HTTPS front -- GPU_INDEX_PUBLIC_BASE_URL when set, defaulting to the
 # official record host https://data.getcomputable.com. Exit 0 all-match /
 # 1 any mismatch or digest FAIL / 2 could not verify (front unreachable,
 # or no published artifacts for the date) -- the default NEVER exits 0
-# without verifying something; withheld sources degrade an observation
-# to digest-only with a distinct message.
+# without verifying something. --receipts is the fast opt-in; withheld
+# contributing sources degrade that mode to digest-only with a message.
 #
-# --full --version <n> replays one version's full public
-# calculation from raw disclosed inputs:
+# --full (default) replays each stamp under its effective version, using
+# that version's own history. --version <n> selects one version throughout.
+# The calculation uses raw disclosed inputs:
 # per-source prices and dispersions, status/timing facts, filter verdicts,
 # carried basis, and published calculation parameters. It derives the
 # attendance events and factors, liveness scores, weight vector, votes,
@@ -57,7 +58,7 @@ usage: reproduce <h100|h200|b300|b200> <YYYY-MM-DD[THH]>
        reproduce --frozen <b300|b200> <YYYY-MM-DD>
 
   <sku>             public SKU. Verifies the as-published record from
-                    its receipts when latest.json advertises history_path;
+                    raw history when latest.json advertises history_path;
                     otherwise prints a notice and uses current_version.
                     Reads a local copy under GPU_INDEX_DATA_DIR when it
                     holds the requested day, else the public front
@@ -66,9 +67,9 @@ usage: reproduce <h100|h200|b300|b200> <YYYY-MM-DD[THH]>
                     verifiable value matched; 1 mismatch/digest FAIL;
                     2 could not verify
   --version <n>     verify this version's full re-derivation keyspace
-  --full            derive weights and values from raw history; pass
-                    --version <n> (a day takes a few minutes)
-  --receipts        the default fast check: recompute each value from its own
+  --full            default: derive weights and values from raw history
+                    under the version live at each stamp (takes a few minutes)
+  --receipts        fast opt-in: recompute each value from its own
                     published receipts only (no weight re-derivation)
   --collect         collect current source prices for the latest print
                     and compare them with its receipts. Past source
@@ -85,9 +86,8 @@ EOF
 
 MODE=sku
 PRODUCER=0
-# Receipt verification can follow a mixed-methodology as-published day.
-# Raw full re-derivation remains an explicit per-version operation.
-FULL=0
+# Re-derive the as-published series with each stamp's effective version.
+FULL=1
 FULL_EXPLICIT=0
 COLLECT=0
 VERSION=""

scripts/verify_published_record.py

@@ -58,6 +58,7 @@
     FullReproductionRefusal,
     read_full_history,
     reproduce_full_history,
+    reproduce_published_history,
 )
 from gpu_index.published.reader import PublishedRecordReader  # noqa: E402
 from gpu_index.published.verify import (  # noqa: E402
@@ -148,8 +149,11 @@ def _run_full(
         "calc_params are inputs; published derived intermediates are not"
     )
     try:
-        history = read_full_history(reader, sku=sku, target_date=date, version=version)
-        run = reproduce_full_history(history, target_date=date)
+        if version is None and pointer and "history_path" in pointer:
+            run = reproduce_published_history(reader, sku=sku, target_date=date)
+        else:
+            history = read_full_history(reader, sku=sku, target_date=date, version=version)
+            run = reproduce_full_history(history, target_date=date)
     except FullReproductionRefusal as exc:
         print(f"FULL REFUSAL [{exc.code}]: {exc}", file=sys.stderr)
         return 2
@@ -180,7 +184,6 @@ def _run_full(
         )
         return 2
 
-    identities = {row["observed_at"]: row for row in history if row["sku"] == sku}
     matched = mismatched = 0
     for check in checks:
         if check.verdict == VERDICT_MATCH:
@@ -189,8 +192,8 @@ def _run_full(
             mismatched += 1
         verdict = "MATCH" if check.verdict == VERDICT_MATCH else "MISMATCH"
         identity = _identity_label(
-            identities[check.observed_at], pointer,
-            explicit_version=version if explicit else None,
+            {"observed_at": check.observed_at, "methodology_id": check.methodology_id},
+            pointer, explicit_version=check.version or (version if explicit else None),
         )
         print(
             f"{check.sku} {_stamp_label(check.observed_at)} "
@@ -250,7 +253,7 @@ def main(argv=None) -> int:
     )
     parser.add_argument(
         "--version", type=_positive_version,
-        help="verify this integer version's re-derivation (required for --full)",
+        help="verify this integer version's re-derivation",
     )
     args = parser.parse_args(argv)
 
@@ -276,9 +279,6 @@ def main(argv=None) -> int:
     try:
         pointer = reader.version_pointer(sku)
         version = args.version
-        if args.full and version is None and pointer and "history_path" in pointer:
-            print("--full re-derives one version; pass --version <n>", file=sys.stderr)
-            return 2
         if version is None and (pointer is None or "history_path" not in pointer):
             version = pointer["current_version"] if pointer else None
             target = f"current_version {version}" if pointer else "legacy flat keyspace"