Skip to content

v0.2: add structural-null audit and retire predictive headline - #2

Merged
gigichengnc merged 1 commit into
mainfrom
v0.2-null-audit
Aug 21, 2026
Merged

gigichengnc merged 1 commit into
mainfrom
v0.2-null-audit

Conversation

@gigichengnc

Copy link
Copy Markdown
Owner

Why this PR exists

A post-publication review identified a material target-construction problem in the v0.1 headline result. The legacy future RRG-style coordinate targets share substantial information with the lagged coordinate features because the rolling windows overlap the decision-time history.

This is not conventional look-ahead leakage. The inputs remain causal. The problem is that the target contains a mechanically predictable component, so beating current-state persistence is not sufficient evidence of market predictability.

Material correction

This PR independently reproduces the issue with a no-signal synthetic geometric-random-walk benchmark. Across 12 deterministic trials, the legacy coordinate pipeline achieves roughly:

Horizon Null Linear Null persistence Null edge
1w 81.0% 72.8% +8.2 pp
2w 73.8% 63.2% +10.6 pp
4w 63.2% 50.5% +12.8 pp
8w 47.4% 31.9% +15.5 pp

The previously reported real-data one-week result was 80.6% vs 74.0%. v0.2 therefore retires the interpretation that the old holdout demonstrated forecastable market dynamics.

What changes

  • add null_benchmark.py and a reproducible synthetic-null runner;
  • add tests for the structural-null benchmark;
  • add a fully future relative-return target:
    log(asset[t+h]/asset[t]) - log(benchmark[t+h]/benchmark[t]);
  • add an L2-regularized logistic baseline for the redesigned research question;
  • rewrite the README around the post-publication correction rather than the old 80.6% headline;
  • retain the old holdout table as a historical record, adjacent to the null correction;
  • correct the old development-vs-final interpretation: better holdout performance is not evidence of stability;
  • clarify that 572 scored rows are not 572 independent observations;
  • reframe the first strategy result as not supported by this backtest, not proof of no alpha;
  • clarify preprocessing: model scaling is fold-local, coordinate normalization is causal but pre-split;
  • document min_periods=1, unregularized OLS, adjusted-close restatement and dependence/inference limitations;
  • bump the package metadata to 0.2.0;
  • add CHANGELOG.md and CITATION.cff.

Historical provenance repair

The first public PUBLIC-SUBSET.md listed 15 historical evidence files, but only 12 were actually committed. The three absent paths were:

  • crates/data/src/data_fetcher.rs
  • crates/data/src/external_data.rs
  • crates/data/src/etf_data.rs

This PR does not publish more private competition-era source merely to repair the manifest. Instead it removes the missing paths from the public subset inventory and marks the corresponding implementation claims as not publicly verifiable from this repository.

The 12 committed evidence files also use LF in Git while the recovered archive hashes were calculated on CRLF bytes. This PR adds historical/verify_public_subset.py, documents that transformation, and runs the verifier in CI. .gitattributes prevents future text conversion of the evidence directory.

Interpretation boundary after v0.2

The old statement Linear outscored persistence on the frozen coordinate holdout remains a factual description of the recorded experiment.

It is not sufficient evidence of market predictability.

Any future market-signal claim must:

  1. use a target whose future window begins after the decision timestamp;
  2. beat a relevant structural/base-rate baseline;
  3. use dependence-aware uncertainty for shared-week, cross-sector and overlapping-horizon structure;
  4. reserve a genuinely prospective evaluation window.

The old 2025-08-22 to 2026-08-14 holdout is retired for new confirmatory claims.

Scope

This PR does not change ReasonedOps, repository visibility, licensing, tags/releases, or PyPI publication.

@gigichengnc
gigichengnc marked this pull request as ready for review August 21, 2026 15:50
@gigichengnc
gigichengnc merged commit 66a6e86 into main Aug 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant