From 8ddcb167e39589ccd018775527bf4840f092f895 Mon Sep 17 00:00:00 2001 From: Elmehdi Aitbrahim Date: Mon, 17 Aug 2026 18:38:50 -0400 Subject: [PATCH] docs(experiments): the honest-cost restatement and the DCA dip-bonus ablation (#339) --- ...onest-cost-restatement-and-dca-ablation.md | 100 ++++++++++++++++++ docs/experiments/trials-ledger.jsonl | 2 + 2 files changed, 102 insertions(+) create mode 100644 docs/experiments/2026-08-17-honest-cost-restatement-and-dca-ablation.md diff --git a/docs/experiments/2026-08-17-honest-cost-restatement-and-dca-ablation.md b/docs/experiments/2026-08-17-honest-cost-restatement-and-dca-ablation.md new file mode 100644 index 00000000..92e23aab --- /dev/null +++ b/docs/experiments/2026-08-17-honest-cost-restatement-and-dca-ablation.md @@ -0,0 +1,100 @@ +# 2026-08-17: The honest-cost simulate re-run, and the DCA dip-bonus ablation + +Two measurements, one date, both enabled by Phase 9: the first `keel simulate` run under +per-product slippage (#334, v0.9.0), and the first-ever measurement of the DCA family in a +fee-explicit harness. Recorded together because they answer the two questions Phase 10's +re-measurement step (#339) posed: do the thin-asset outliers survive honest pricing, and +does the one unswept rule family behave as its citation predicts. + +## What was measured, on what + +- Engine: `keel 0.9.0+eebf12b88292 [release]`, deployed 2026-08-17 (four distributions at + 0.9.0, clean build identity). +- Data: the deployment's `keel.db` candles, post-repair (2026-08-17: every series 0 bars + behind, all remaining gaps proven absent at the venue). +- Instrument A (simulate): `keel simulate` — the paper book's rules over the 8-asset + paper universe, edge pass priced at 1.2% taker per leg **plus per-product slippage** + (`floor × sqrt(anchor / median_daily_quote_volume)`, floor 5bp, cap 50bp, anchor + $500M/day — an assumption, not a measurement, stated beside every table the run prints). +- Instrument B (ablation): `sim/portfolio_sim.run` directly, `fee_pct=0.012`, + `slippage_pct=0.0005`, BTC-USD hourly (44,537 bars, 2021-08 → 2026-08), three cells + differing only in `dip_bonus_pct` (0 / 1 / 2) at constructor defaults + (`cadence_days=7, budget_usd=50, lookback_days=90`). Metric: terminal DCA-sleeve + mark-to-market vs deployed (qty × average cost basis) — never a profit factor, because + the sleeve never closes and DCA is exempt from the promotion gate (§12.6). + +## Result A — the honest-cost simulate: verdict unchanged, direction conservative + +Verdict: **TRAIN MORE**, on the same gates as the morning's flat-5bp run (the `[default]` +trio is the disabled-DCA artifact, not a measurement; `[trend_follow] n_trades 80 < 100` is +the live gate). The per-asset edge table moved exactly as #334 predicted — **thin assets +got worse, never better**: + +| turtle_breakout | PF, flat 5bp (08-17 AM) | PF, per-product (this run) | assumed slippage | +| --- | --- | --- | --- | +| BTC | 1.5216 | 1.5216 | 5.0bp (floor/anchor) | +| ETH | 1.2320 | 1.2284 | 6.2bp | +| SOL | 0.2309 | 0.2285 | 10.7bp | +| ADA | 1.7638 | 1.7087 | 24.1bp | +| XLM | 4.7567 | 4.4435 | 35.7bp | +| PAXG | 1.3152 | 1.3152 | 50.0bp (capped) | + +**No flattered outlier survived.** The TON-class result #259 worried about (gross PF 3.751 +on n=9 under flat 5bp) cannot print at that price anymore — products without a liquidity +statistic fall back to the floor and are flagged, and the thin end of the measured universe +(XLM at 35.7bp, ADA at 24.1bp, PAXG capped) now pays what its books plausibly demand. The +pooled reading (N=80, PF 1.4703) is below the promotion floor on sample size alone, as it +was. **Comparison caveat: pre-#334 numbers in older experiment documents were produced +under the flat model and are not comparable cell-for-cell with this table; they stand as +annotated records of what the engine of their day printed.** + +## Result B — the DCA ablation: the pre-registered expectation was NOT confirmed + +Terminal outcomes over the same ~5-year BTC window, `budget_usd=50` weekly, entry fees +1.2% + 5bp per leg, no exit leg ever: + +| `dip_bonus_pct` | qty accumulated | deployed (qty × avg basis) | terminal MTM | MTM/deployed | +| --- | --- | --- | --- | --- | +| 0 (plain) | 0.054628 | $2,495.79 | $3,512.63 | **1.40742** | +| 1 | 0.053646 | $2,456.50 | $3,449.50 | **1.40424** | +| 2 | 0.054829 | $2,494.65 | $3,525.55 | **1.41325** | + +The citation behind the proposal (bestinterest.blog, BTC 2013–2021, costs not modeled, +not independently replicated) found plain cadence beating every dip-conditioning variant, +monotonically worse at deeper thresholds. **That ordinal prediction does not transfer:** +the three cells sit within 0.65% relative of each other with no monotone direction +(dip-1 slightly under plain, dip-2 slightly over). Two honest readings: + +1. **Mechanism mismatch, stated plainly.** The citation's scheme holds cash until a + threshold dip — it changes WHEN buys happen and bears round-trip-adjacent timing risk. + `dip_bonus_pct` always buys and only scales size up on dip weeks. The nearest + expressible analogue was measured, not the cited scheme, and on this window the two + behave alike. +2. **Power.** One asset, one window, no replication. A ≤0.65% spread in terminal ratio is + inside path noise for anything this unreplicated; the result is "no detectable effect + at keel's cost structure", not "an effect of zero". + +**Consequence for the friction-sizing thesis (#342): none either way.** DCA's cost +structure (one leg, never exits, ad-valorem fees) is precisely the case the round-trip +argument does not govern, and the ablation was never a test of it. **Consequence for the +operator: no basis to prefer any cell — the live rule stays at `dip_bonus_pct=0`, which is +also its constructor default.** First-ever coverage of the family is now on record; the +proposal's value was the coverage, not the tuning. + +## Deployment actions recorded with this run + +- The disabled paper DCA twins (rules 19, 20) were re-enabled through the supported CLI + path — `keel rules enable` (→ candidate, the lifecycle floor) then `keel rules promote + --force` (the documented bypass for a paper-forward whose backtest can never reach the + floor; both warnings printed and `rules.promote_forced` events logged). The paper + pipeline now has an always-firing family alongside the turtles, at $55/week combined + against $550 synthetic cash — either the operator tops up or the cash caps veto, and + both are admissible evidence. + +## Caveats + +- In-sample, single-asset (BTC), single-window; the ablation's cells share one price path. +- The simulate run's dollar account pass and benchmarks still price at the flat fallback; + the report says so beside both sections (#334's stated asymmetry). +- Per-product slippage is an assumption with recoverable parameters (floor/cap/anchor + printed with every table); it is not a measured spread. diff --git a/docs/experiments/trials-ledger.jsonl b/docs/experiments/trials-ledger.jsonl index abcc97fe..592880d0 100644 --- a/docs/experiments/trials-ledger.jsonl +++ b/docs/experiments/trials-ledger.jsonl @@ -83,3 +83,5 @@ {"decision":"rejected","kind":"ablation","params":{"anchor":"oversold=20 reused from the #252 intersection run (identical in every other parameter and cost treatment)","assets":24,"caveat_declared_statistic":"the slope was UNDERPOWERED for its own question -- the relationship is a threshold effect at the floor, not linear, and two barely-trading assets supply most of its variance. The partition on the pre-declared n>=100 floor is the decisive reading and is reported alongside it.","conditional_arm":"declared in advance: widen support_proximity_pct if FEWER THAN 8 of 24 assets reach n>=100 at oversold=40. DID NOT FIRE (21/24 reached it).","design":"MONOTONICITY TEST, not a search -- declared statistic is the SLOPE of gross PF against n, per asset then averaged. The best cell is never reported as a result.","document":"docs/experiments/2026-08-12-rsi-meanrev-scale-vs-selectivity.md","fees":["0","0.006 maker","0.012 taker"],"granularity":"ONE_HOUR","held_fixed":{"overbought":80.0,"support_proximity_pct":"0.005","why_overbought_held":"it governs the EXIT side, so moving it changes trade outcomes and not merely trade counts, which would make the slope uninterpretable"},"instrument_defect_found":"backtest() never expires a pending setup, silently freezing a strategy for the rest of a series -- filed as #254. #252's headline results were checked and are clean.","predeclaration_location":"the script docstring, written before the run -- the correction to the defect #252 recorded against itself","slippage_pct":"0.0005 pinned explicitly","trials":"4 new levels x 24 assets = 96 combinations x 3 fees = 288 backtests","variable_axis":{"oversold":[20.0,25.0,30.0,35.0,40.0]}},"per_bar_pnl":[],"per_trade_pnl":[],"prev_hash":"19ecedc7148cb12c8a33d59f5e9cbbba5145beb1ddabd7cbffa1e63dd755ba6c","provenance":"a_priori","row_hash":"a16af414291b6034ccaa58ffa1577e56aa42eab1c72edb9364ac77988acd6b83","rule":"rsi_meanrev","series_missing":true,"session":"rsi-meanrev-scale-vs-selectivity-2026-08-12","summary":{"assets_negative_slope":"15","assets_reaching_floor_at_oversold_40":"21","cells_gross_positive_over_floor":"11","cells_net_positive_over_floor_any_fee":"0","cells_over_floor":"76","mean_slope_per_100_trades":"-0.0386","median_gross_pf_at_defaults":"1.1631","median_gross_pf_at_n_over_100":"0.8938","median_slope_per_100_trades":"-0.0197","trade_count":96},"timestamp":1786010000,"trial_id":"rsi-meanrev-scale-vs-selectivity-2026-08-12"} {"decision":"diagnostic_only","kind":"ablation","params":{"bias_directions":"OPPOSITE -- #256 suppressed opportunity, #258 flattered execution. Correcting both moved everything one way: n rose in 87/90, gross PF fell in 69/90.","conclusion_changed":"ONE -- ZEC-turtle no longer clears the maker line (1.034 -> 0.968), so the three-probe narrative of the original describes a survivor the faithful engine does not produce","document":"docs/experiments/2026-08-13-restated-under-a-production-faithful-engine.md","engine_changes":"#256 pending setup never expired (detector froze); #258 entries filled at a price production never waits for (live places market orders)","identical_design":"same 90 combinations, same 3 fee levels, same explicit 0.0005 slippage pin; only the engine differs","restates":"docs/experiments/2026-08-12-shipped-defaults-intersection.md"},"per_bar_pnl":[],"per_trade_pnl":[],"prev_hash":"a16af414291b6034ccaa58ffa1577e56aa42eab1c72edb9364ac77988acd6b83","provenance":"a_priori","row_hash":"605ecff048e50fa9cb0d1b649e1eab9e254f1463dd17e91b622d81fd7ef1f31e","rule":"turtle_breakout+rsi_meanrev+pullback_continuation","series_missing":true,"session":"restated-production-faithful-engine-2026-08-13","summary":{"arm_b_in_sample_net_pf":"0.5770","arm_b_oos_net_pf":"0.5427","arm_b_transfer_gap":"0.0343","cells_gross_positive_over_floor":"7","cells_net_positive_at_maker":"0","cells_net_positive_at_taker":"0","pullback_median_gross":"0.7736","rsi_median_gross":"1.1251","trade_count":90,"turtle_median_gross":"0.9262"},"timestamp":1786100000,"trial_id":"intersection-restated-faithful-engine-2026-08-13"} {"decision":"rejected","kind":"ablation","params":{"conditional_arm":"did not fire -- 22/24 assets reached n>=100 at oversold=40","design_unchanged":"oversold {20,25,30,35,40} x 24 assets, overbought and support_proximity_pct held; anchor re-derived on the same engine","document":"docs/experiments/2026-08-13-restated-under-a-production-faithful-engine.md","engine_changes":"#256 and #258, as above","monotonicity_check":"the anomaly that exposed both defects is structurally gone: 3 non-monotonic assets (UNI 34.3x, BTC 2.3x, AAVE 1.1x) -> 0","restates":"docs/experiments/2026-08-12-rsi-meanrev-scale-vs-selectivity.md"},"per_bar_pnl":[],"per_trade_pnl":[],"prev_hash":"605ecff048e50fa9cb0d1b649e1eab9e254f1463dd17e91b622d81fd7ef1f31e","provenance":"a_priori","row_hash":"a782b48b637e5e9f83b8c0a25af352edc38dc9d21115f2dbdd4974745751b8b5","rule":"rsi_meanrev","series_missing":true,"session":"restated-production-faithful-engine-2026-08-13","summary":{"assets_reaching_floor_at_oversold_40":"22","cells_gross_positive_over_floor":"6","cells_net_positive_at_maker":"0","cells_over_floor":"82","mean_slope_per_100_trades":"-0.0328","median_gross_at_anchor":"1.1251","median_gross_over_floor":"0.8396","non_monotonic_assets":"0","trade_count":96},"timestamp":1786100001,"trial_id":"rsi-scale-restated-faithful-engine-2026-08-13"} +{"decision":"diagnostic_only","kind":"ablation","params":{"cost_model":"per-product slippage #334: clamp(5bp x sqrt($500M/day / median_daily_quote_volume), 5bp, 50bp); fee 1.2%/leg taker","design_unchanged":"shipped defaults, paper universe of 8 assets, same candles as the flat-5bp run that morning; only the cost model differs","document":"docs/experiments/2026-08-17-honest-cost-restatement-and-dca-ablation.md","verdict":"TRAIN MORE, unchanged; thin assets worsened (XLM 4.7567->4.4435, ADA 1.7638->1.7087), floor-anchored assets unchanged (BTC identical); no flattered outlier survived"},"per_bar_pnl":[],"per_trade_pnl":[],"prev_hash":"a782b48b637e5e9f83b8c0a25af352edc38dc9d21115f2dbdd4974745751b8b5","provenance":"a_priori","row_hash":"3206bad9711d5ba1cd46958b127fbe2dc455afe95454e652d5fc0397ed387018","rule":"turtle_breakout","series_missing":true,"session":"honest-cost-restatement-2026-08-17","summary":{"flattered_outliers_surviving":0,"pooled_n":80,"pooled_pf":"1.4703"},"timestamp":1787006281,"trial_id":"honest-cost-simulate-2026-08-17"} +{"decision":"diagnostic_only","kind":"ablation","params":{"cells":"dip_bonus_pct {0,1,2} at cadence_days=7, budget_usd=50, lookback_days=90","document":"docs/experiments/2026-08-17-honest-cost-restatement-and-dca-ablation.md","instrument":"sim/portfolio_sim.run, fee_pct=0.012, slippage_pct=0.0005, BTC-USD hourly 44537 bars 2021-08..2026-08","metric":"terminal sleeve MTM / deployed; never a profit factor (sleeve never closes)","outcome":"NOT CONFIRMED -- cells within 0.65% relative, no monotone direction; mechanism mismatch stated (citation holds cash until threshold dips, param always buys and scales size); no cell preferred, live rule stays at dip_bonus_pct=0","pre_registered_expectation":"plain >= dip-scaled, monotonically worse deeper (bestinterest.blog, BTC 2013-2021, costs not modeled, not replicated)"},"per_bar_pnl":[],"per_trade_pnl":[],"prev_hash":"3206bad9711d5ba1cd46958b127fbe2dc455afe95454e652d5fc0397ed387018","provenance":"a_priori","row_hash":"8433009c6d8b0ee9fbda29a6fd5d696016b034267d43bf6edc539b107e779518","rule":"dca","series_missing":true,"session":"honest-cost-restatement-2026-08-17","summary":{"cells":3,"deployed_dip0":"2495.79","deployed_dip1":"2456.50","deployed_dip2":"2494.65","ratio_dip0":"1.40742","ratio_dip1":"1.40424","ratio_dip2":"1.41325"},"timestamp":1787006281,"trial_id":"dca-dip-bonus-ablation-2026-08-17"}