Skip to content

The 2026-09-30 n=100 pooled review is underpowered: n_eff is about 39, and it can only detect a 20-point edge #427

Description

@eaitbrahim

#359 schedules the pooled forward-trades review for 2026-09-30 at a floor of n=100. That floor treats 100 pooled trades as 100 observations. They are not, because the signals fire in herds and the outcomes are correlated.

Raised after reading the Keel Quant Lab modelling note against #368 and #359 — the note supplies the power arithmetic, #359 supplies the schedule, and neither states the consequence of putting them together.

Method

Reconstructed the live rule on ONE_HOUR bars with its own parameters — entry_lookback: 40, adx_period: 14, adx_threshold: 25, atr_period: 20, atr_stop_mult: 2, target_rr: 6 — resolving each entry to win/loss by first touch, across 25 products over 5 years of cached candles.

It validates against keel's own published measurement, which is what makes the rest usable:

reconstruction #359
signals per asset-year 46.7 49.4
median n per rule-product 241 268

(Omitting the ADX gate gives 70.6/asset-year — the gate accounts for the difference, and its absence would have inflated everything downstream.)

Result

episodes (UTC days with >=1 signal):   1,355
size-weighted mean episode size k:      8.43
largest episode:                          24 assets firing the same day
ICC of outcomes within an episode:     0.212
design effect DEFF = 1 + (k-1)*rho:     2.58

n_eff at n = 100 pooled:                  39
detectable edge at 80% power:          20.0%    (12.4% if the trades were independent)

When a breakout triggers it triggers on about eight assets the same day, and those trades then win or lose together. A pooled sample of 100 carries roughly 39 independent observations.

What that means on 30 September

Detecting an edge at 80% power would need p ~ 21.4% + 20.0% = 41.4%. The reconstructed win rate is 14.9%.

The review is therefore near-certain to return "no edge", and that is the correct answer — but it must not be written up as "we measured no edge at n=100". The honest statement is "at n=100 pooled this test can only see an edge of 20 points or more." Those read the same to most audiences and mean very different things, and #304 / #348 have already committed keel to stating its non-results precisely.

What the floor would have to be

to detect n_eff pooled trades
12.4 pts 101 259
7.5 pts 275 708
5 pts 618 1,593
100 258 — what "n=100" is assumed to deliver

Raising the pooled floor from 100 to ~258 makes n_eff equal what everyone already believes n=100 means. That is the one-line change; min_trades already exists as a setting (keel/commands/rules.py).

Caveats, stated

  1. DEFF is a lower bound. Clusters are UTC days. These are trend trades held for many hours, so correlated outcomes plausibly persist across day boundaries — the true design effect is larger and n_eff smaller.
  2. ICC is from backtest outcomes, the only 5-year sample available. Forward paper outcomes may differ.
  3. k scales with the pool. DEFF 2.58 reflects ~25 products. A narrower live pool clusters less and loses less; the correction is not a constant.

Prerequisite

None of this is reachable yet: keel-paperhourly.db has 0 recorded trades, and every detected setup is being vetoed (#426). The power question only becomes live once accrual starts — but the floor should be corrected before the review, not after it reports.

Metadata

Metadata

Assignees

No one assigned

    Labels

    phase-3Execution/rails/agentresearchMeasurement, validation or KB work (Research & validation)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions