Skip to content

backtest(): a pending setup never expires, silently freezing a strategy for the rest of the series #254

Description

@eaitbrahim

Found while investigating a 34× non-monotonic trade count in #253. This is a correctness defect
in the instrument every experiment in docs/experiments/ was produced with.

The defect

keel/strategy/backtest.py::backtest:

if position is None and pending is None:
    pending = rule.detect(candles_by_tf)
    continue

if position is None and pending is not None:
    entry_touched = _touches(candle, pending.entry)
    stop_touched = _touches(candle, pending.stop)
    if not entry_touched:
        continue          # <-- pending persists, unconditionally

A pending setup has no expiry. The only path that clears it without a fill is the stop being
touched first and _resolve_order confirming the stop preceded entry. If price drifts away so that
neither entry nor stop is ever touched again, pending stays set forever, the
position is None and pending is None branch never runs again, and rule.detect() is never
called for the remainder of the series.

The strategy freezes silently. Nothing errors, no warning is emitted, and the result is
indistinguishable from "the rule found no further setups" — which is how this survived unnoticed.
A frozen backtest looks exactly like a selective one.

Reproduction

rsi_meanrev on UNI-USD, hourly, 43,640 bars, everything at defaults except oversold:

oversold=30: closed=309  open=0/1  last exit 2026-08-04   (trades throughout the series)
oversold=35: closed=9    open=0    last exit 2021-11-15   (dead for ~40,000 bars / 4.7 years)

Loosening the entry threshold — which can only make detection easier — reduced the trade count
34×, because the looser setting produced an early setup whose entry was never revisited.

Also visible as non-monotonic n on BTC-USD (181→80 between oversold 30 and 35) and AAVE-USD.

Why it matters beyond the anomaly

What is already known about the blast radius

#252's headline results were checked and are clean — the three turtle assets below the trade
floor each trade to within days of their series end, so their low counts are genuine short history:

PAXG-USD  n=61  last exit 2026-08-11, series ends 08-12   dead tail 1d
WLD-USD   n=45  last exit 2026-07-21, ends 07-23                    2d
TON-USD   n=29  last exit 2026-07-22, ends 07-23                    1d

What is not established is the effect on the lower-n cells of every prior sweep.

Suggested fix, and the reason it needs a decision rather than a patch

The obvious repair is a bar-count expiry on pending (re-detect if the entry has not filled within
N bars). But N is a modelling choice with real consequences — it changes every stored number in
docs/experiments/ — and it should match what the live path actually does, which is re-detect every
cycle. A faithful fix may be "re-detect each bar and replace the pending setup", not "expire after
N".

Whatever is chosen, it should ship with:

  1. a regression test built from the UNI-USD case above;
  2. a diagnostic that reports a dead tail (last exit far from series end) so a freeze is visible
    rather than silent — the same principle as fix(strategy): price fills at the taker rate, and wire the dormant PBO gate into promotion #247 printing the fee rate;
  3. a re-run of the affected sweeps, since until then every low trade count in docs/experiments/
    carries an asterisk.

Deliberately not fixed inside #253's research PR: a documentation change should not carry an
engine change that alters every prior measurement.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions