Skip to content

The simulator fills at a price live never waits for: entries are market orders in production, resting orders in backtest #257

Description

@eaitbrahim

Found while validating the #254 fix. Distinct from #254 and larger: that was a freeze; this is
a systematic optimistic bias in every backtest number the project has produced.

The divergence

Production entries are market orders. keel/execution/executor.py::_order_row:

order_type="market",
limit_price=None,
expected_fill=intent.entry,

The Setup's entry is an expected fill reference. Live never places a resting order at it —
when engine.evaluate emits a signal and the gates pass, the executor buys at market, now.

The simulator waits for that price instead. keel/strategy/backtest.py:

entry_touched = _touches(candle, pending.entry)
...
position = _OpenPosition(entry_fill=pending.entry * (1 + slippage_pct), ...)

It holds the setup, waits for a later bar's range to touch entry, and fills at that level.

Why this biases results optimistically

The backtest only ever enters at its chosen price. Production enters at whatever the market is
doing when the signal fires. Two effects, both favourable to the simulation:

  1. Free optionality on the entry price. A setup whose entry is below market fills only if
    price comes down to it — the simulator gets the good fill or no trade at all. Live pays the
    current price immediately, favourable or not.
  2. Unbounded patience. With backtest(): a pending setup never expires, silently freezing a strategy for the rest of the series #254 fixed, a setup is re-derived each bar, but the fill rule is
    unchanged: any bar that touches the level fills there. Live has no concept of waiting.

This is the opposite direction from #254, which suppressed trades. Fixing #254 did not address it,
and my commit message on #256 overstated the case by saying the fix "makes the simulator match the
live path" — that is true of detection and false of fills. Recorded here rather than left
standing.

It also complicates #247's fee reasoning

backtest.py's TAKER_FEE_PCT comment justifies the taker rate as: "a pending Setup fills the
moment a later bar's range touches its entry, at that level plus slippage. That is a marketable
order crossing the spread -- taker behaviour."

That holds for a breakout entry (entry above market — touching it means price rose into a stop
order, which is marketable and genuinely taker). It does not hold for a mean-reversion
entry (entry below market — touching it means price fell to a resting limit, which is a maker
fill). So turtle_breakout is priced correctly and rsi_meanrev is arguably overcharged.

Since production places market orders in both cases, taker remains the right conservative rate;
but the stated justification is only valid for one of the two rule shapes, and the comment should
say so.

Options

  1. Fill at the next bar's open, ignoring entry as a trigger. Closest to what live does. The
    largest change to historical numbers.
  2. Fill at entry only if the signal bar's own next bar touches it, else drop the setup — caps
    the optionality at one bar without abandoning the level entirely.
  3. Keep current semantics and document them as a deliberate limit-entry model that production
    does not implement — cheapest, but leaves every number optimistic by an unmeasured amount.

Option 1 is the faithful one and should probably win, but it moves every stored figure in
docs/experiments/ again, so it wants a deliberate decision rather than a drive-by patch — the
same argument that kept #254's fix out of a research PR.

Suggested first step

Quantify before choosing: run the same 90-combination matrix under option 1 and report the delta.
If the gap is small the decision is cheap; if it is large, that magnitude is itself the most
important number in the project, because it bounds how much of every prior conclusion was an
artifact of an entry price production never achieves.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions