You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Found while validating the #254 fix. Distinct from #254 and larger: that was a freeze; this is
a systematic optimistic bias in every backtest number the project has produced.
The divergence
Production entries are market orders.keel/execution/executor.py::_order_row:
The Setup's entry is an expected fill reference. Live never places a resting order at it —
when engine.evaluate emits a signal and the gates pass, the executor buys at market, now.
The simulator waits for that price instead.keel/strategy/backtest.py:
It holds the setup, waits for a later bar's range to touch entry, and fills at that level.
Why this biases results optimistically
The backtest only ever enters at its chosen price. Production enters at whatever the market is
doing when the signal fires. Two effects, both favourable to the simulation:
Free optionality on the entry price. A setup whose entry is below market fills only if
price comes down to it — the simulator gets the good fill or no trade at all. Live pays the
current price immediately, favourable or not.
This is the opposite direction from #254, which suppressed trades. Fixing #254 did not address it,
and my commit message on #256 overstated the case by saying the fix "makes the simulator match the
live path" — that is true of detection and false of fills. Recorded here rather than left
standing.
backtest.py's TAKER_FEE_PCT comment justifies the taker rate as: "a pending Setup fills the
moment a later bar's range touches its entry, at that level plus slippage. That is a marketable
order crossing the spread -- taker behaviour."
That holds for a breakout entry (entry above market — touching it means price rose into a stop
order, which is marketable and genuinely taker). It does not hold for a mean-reversion
entry (entry below market — touching it means price fell to a resting limit, which is a maker
fill). So turtle_breakout is priced correctly and rsi_meanrev is arguably overcharged.
Since production places market orders in both cases, taker remains the right conservative rate;
but the stated justification is only valid for one of the two rule shapes, and the comment should
say so.
Options
Fill at the next bar's open, ignoring entry as a trigger. Closest to what live does. The
largest change to historical numbers.
Fill at entry only if the signal bar's own next bar touches it, else drop the setup — caps
the optionality at one bar without abandoning the level entirely.
Keep current semantics and document them as a deliberate limit-entry model that production
does not implement — cheapest, but leaves every number optimistic by an unmeasured amount.
Option 1 is the faithful one and should probably win, but it moves every stored figure in docs/experiments/ again, so it wants a deliberate decision rather than a drive-by patch — the
same argument that kept #254's fix out of a research PR.
Suggested first step
Quantify before choosing: run the same 90-combination matrix under option 1 and report the delta.
If the gap is small the decision is cheap; if it is large, that magnitude is itself the most
important number in the project, because it bounds how much of every prior conclusion was an
artifact of an entry price production never achieves.
Found while validating the #254 fix. Distinct from #254 and larger: that was a freeze; this is
a systematic optimistic bias in every backtest number the project has produced.
The divergence
Production entries are market orders.
keel/execution/executor.py::_order_row:The
Setup'sentryis an expected fill reference. Live never places a resting order at it —when
engine.evaluateemits a signal and the gates pass, the executor buys at market, now.The simulator waits for that price instead.
keel/strategy/backtest.py:It holds the setup, waits for a later bar's range to touch
entry, and fills at that level.Why this biases results optimistically
The backtest only ever enters at its chosen price. Production enters at whatever the market is
doing when the signal fires. Two effects, both favourable to the simulation:
price comes down to it — the simulator gets the good fill or no trade at all. Live pays the
current price immediately, favourable or not.
unchanged: any bar that touches the level fills there. Live has no concept of waiting.
This is the opposite direction from #254, which suppressed trades. Fixing #254 did not address it,
and my commit message on #256 overstated the case by saying the fix "makes the simulator match the
live path" — that is true of detection and false of fills. Recorded here rather than left
standing.
It also complicates #247's fee reasoning
backtest.py'sTAKER_FEE_PCTcomment justifies the taker rate as: "a pendingSetupfills themoment a later bar's range touches its entry, at that level plus slippage. That is a marketable
order crossing the spread -- taker behaviour."
That holds for a breakout entry (entry above market — touching it means price rose into a stop
order, which is marketable and genuinely taker). It does not hold for a mean-reversion
entry (entry below market — touching it means price fell to a resting limit, which is a maker
fill). So
turtle_breakoutis priced correctly andrsi_meanrevis arguably overcharged.Since production places market orders in both cases, taker remains the right conservative rate;
but the stated justification is only valid for one of the two rule shapes, and the comment should
say so.
Options
entryas a trigger. Closest to what live does. Thelargest change to historical numbers.
entryonly if the signal bar's own next bar touches it, else drop the setup — capsthe optionality at one bar without abandoning the level entirely.
does not implement — cheapest, but leaves every number optimistic by an unmeasured amount.
Option 1 is the faithful one and should probably win, but it moves every stored figure in
docs/experiments/again, so it wants a deliberate decision rather than a drive-by patch — thesame argument that kept #254's fix out of a research PR.
Suggested first step
Quantify before choosing: run the same 90-combination matrix under option 1 and report the delta.
If the gap is small the decision is cheap; if it is large, that magnitude is itself the most
important number in the project, because it bounds how much of every prior conclusion was an
artifact of an entry price production never achieves.