diff --git a/config.live-sandbox.yaml b/config.live-sandbox.yaml index b843ad31..5b69871b 100644 --- a/config.live-sandbox.yaml +++ b/config.live-sandbox.yaml @@ -5,12 +5,52 @@ # PREVIEWED and requires a typed `y`. Caps are deliberately tiny so a bad fill can lose only a # few dollars. # -# Authored in the dev repo (gitignored). DEPLOYED to ~/keel at arm-time alongside its -# OWN database keel-live.db. There it runs as: +# Authored in the dev repo and TRACKED there since 183dcdb. DEPLOYED to ~/keel at arm-time +# alongside its OWN database keel-live.db. There it runs as: # .venv/bin/keel --config config.live-sandbox.yaml --db keel-live.db agent (interactive) +# The repo copy is the backup of record: edit both, or they drift. -# The 5-trend Turtle set (same as the paper-forward + the walk-forward/PBO analysis). -# SOL/LTC/LINK are deliberately excluded -- they're the unvalidated skip set. +# The 5 assets this sandbox trades. Keeping SOL/LTC/LINK off this list is the RIGHT call -- but +# an earlier version of this comment gave reasons that were not true, so the reasons are restated +# here. Full working: docs/experiments/2026-08-07-unvalidated-skip-set-reassessment.md. +# +# What that comment got RIGHT: these 5 are exactly the `paper`-status rules in keel.db +# (10/11/12 BTC/ETH/PAXG + 22 XLM + 24 ADA), so "same as the paper-forward" was accurate. +# +# What it got WRONG: it also credited "the walk-forward/PBO analysis". No walk-forward or PBO run +# has ever covered ADA or XLM -- every such run in docs/experiments/ stopped at BTC/ETH/PAXG, and +# the walk-forward that exists tuned a PARAMETER (turtle_breakout.py:113, entry_lookback 20 -> 40), +# it never validated a set of assets. And these 5 did not clear the promotion gate: keel-live.db +# rules 1-5 carry promoted_at IS NULL, seeded straight to `live` through the bypass `rules seed` +# warns about. (Rule 6, the BTC dca, IS promoted -- and has a filled live order and an open +# position, so this deployment is NOT inert.) +# +# The reason that DOES hold up -- established 2026-08-07 by running the backtest nobody had run, +# so it is a post-hoc vindication of the 2026-07-24 exclusion, not the reason it was made. +# `keel rules backtest` on a COPY of keel.db, in-sample, one window (SOL/LTC/LINK candles 6 days +# stale; the allowlisted rows are current): +# 21 SOL n=13 win 23.1% expectancy -12.41 pf 0.26 <- excluded +# 23 LTC n= 5 win 0.0% expectancy -13.09 pf 0.00 <- excluded +# 25 LINK n=15 win 20.0% expectancy -0.66 pf 0.63 <- excluded +# 22 XLM n= 8 win 37.5% expectancy +0.06 pf 11.44 +# 24 ADA n= 6 win 50.0% expectancy +0.12 pf 5.52 +# 10 BTC n=13 win 46.2% expectancy +1353.99 pf 1.61 +# 11 ETH n=13 win 30.8% expectancy +45.72 pf 1.21 +# 12 PAXG n= 4 win 50.0% expectancy +212.15 pf 3.25 +# COMPARE ON pf, NOT expectancy: expectancy is quote-currency per trade, so it tracks the asset's +# price (BTC ~64k vs XLM ~0.30) and is meaningless across rows. On the scale-free metric the split +# is clean -- every excluded asset is pf < 1 (losing), every included one is pf > 1. +# +# That is a real adverse result for SOL/LTC/LINK, and NOT a compliance one: all three were attested +# 2026-07-23 alongside ADA/XLM under the same governing source (KB source-86; LINK's is more +# carefully qualified, as a utility token), all three ADMIT `keel assets screen`, and all three +# carry MORE daily history (1827 bars) than PAXG (456 native-USD bars, on this list only via a +# documented screen_exceptions waiver). +# +# Read the numbers with the sample sizes in view: n=4..15 against min_trades=100. NOT ONE of the +# eight rules above can clear the promotion gate on this evidence -- the live five included. The +# split is directionally supported, not validated. Do not treat a positive pf here as a licence to +# add an asset, and do not add SOL/LTC/LINK on the strength of their screen/attestation status. allowlist: - BTC - ETH diff --git a/docs/experiments/2026-08-07-unvalidated-skip-set-reassessment.md b/docs/experiments/2026-08-07-unvalidated-skip-set-reassessment.md new file mode 100644 index 00000000..a6f11975 --- /dev/null +++ b/docs/experiments/2026-08-07-unvalidated-skip-set-reassessment.md @@ -0,0 +1,218 @@ +# Re-assessing "SOL/LTC/LINK are the unvalidated skip set" + +**Date:** 2026-08-07 +**Status:** documentation correction + one measurement. **No allowlist changed, no asset +attested, no rule promoted, no production database written** (backtests ran against a copy). +**Trigger:** a standing re-assessment of whether that claim — carried in a live-money config — +is always valid. + +## The claim under test + +`config.live-sandbox.yaml`, directly above the `allowlist:` that governs the real-money sandbox: + +```yaml +# The 5-trend Turtle set (same as the paper-forward + the walk-forward/PBO analysis). +# SOL/LTC/LINK are deliberately excluded -- they're the unvalidated skip set. +allowlist: [BTC, ETH, PAXG, ADA, XLM] +``` + +**Verdict: the decision is right; the stated reasons were mostly wrong; and "always valid" is +the wrong frame for it.** One half of the parenthetical is accurate, the other half describes +analysis that does not exist, the "unvalidated" label misdescribes which assets were validated, +and the whole thing named a gate that does not do the excluding. Then — on a check nobody had +run until this re-assessment — the *conclusion* turned out to be supported by evidence the +comment never cited. + +## Provenance: no decision record exists + +The phrase "unvalidated skip set" appears **nowhere else in the repository**. The file entered +git fully formed on 2026-08-03 in `183dcdb` ("chore: track the deployment's operator configs"), +a *backup* commit; `36c931a` had gitignored it on 2026-07-24. No commit, PR, or design doc ever +introduced or justified the exclusion. + +That said, a coherent origin story does exist and the comment half-names it — see below. + +## Four gates, and the comment names the wrong one + +keel enforces four independent gates. Three are asset-level, one is rule-level: + +| # | Gate | Object | Enforced by | +|---|---|---|---| +| 1 | Compliance / attestation | asset | `screen.py::screen_asset` — `attestation=None` fails closed | +| 2 | Mechanical screen | asset | same fn — history ≥1460 bars, liquidity, settlement, spot-only | +| 3 | Allowlist | asset | `guards.py:369` rail 1 — **consults neither gate 1 nor 2** | +| 4 | Statistical evidence | **rule** | `agent.py` filters on `rules.status`; `promotion.py::can_promote` | + +The comment attaches a **gate-4 (rule-level)** reason to a **gate-3 (asset-level)** control. +Rail 1 is a flat membership test against whatever list the loaded config contains. What actually +makes SOL/LTC/LINK inert is the absence of a `live`-status `rules` row. + +## Gates 1 and 2: SOL/LTC/LINK passed, and have since 2026-07-23 + +SOL, XLM, LTC and ADA were attested at the **same second** (`attested_at` 1784823312 = +2026-07-23 16:15:12 UTC / 12:15:12 EDT), same attestor, `pays_yield=0`, "bare unstaked spot +only". LINK followed 160s later under the same governing source (KB source-86) — though its +attestation text is materially +more qualified, classifying it as a **utility token** (ERC-20, `native` chosen only because the +backing enum lacks a `utility` slot). That nuance is worth preserving; "same source" flattens it. + +Screening a copy of `keel.db`, all three ADMIT — 1827 daily bars each, median notionals +$108,273,790 / $15,129,648 / $21,156,873 against a $1M floor. + +| asset | daily bars | attested | screen verdict | in live allowlist | +|---|---:|:---:|:---:|:---:| +| ADA / XLM | 1833 | 2026-07-23 | ADMIT | yes | +| SOL / LTC / LINK | 1827 | 2026-07-23 | ADMIT | **no** | +| PAXG | **456** | 2026-07-23 | admit **only via waiver** | yes | + +The excluded assets carry *more* history than an included one. PAXG is on the list solely via +the single `screen_exceptions` row in the table. + +**There is no compliance asymmetry.** Whatever separates these two groups, it is not gates 1–2. + +## The parenthetical: half true, half false + +**True half — "same as the paper-forward."** The `paper`-status rules in `keel.db` are exactly +10 (BTC), 11 (ETH), 12 (PAXG), 22 (XLM), 24 (ADA) — precisely the live 5. `paper.py:15` loads +`repo.get_rules("paper")`. So at the *rule* level this is an accurate description, and it is a +perfectly coherent origin story for the exclusion: the sandbox mirrored the paper-forward's +promoted rule set. (Note `config.paperforward.yaml`'s *allowlist* is all eight assets — the +match is between rule sets, not config files.) + +**False half — "the walk-forward/PBO analysis."** Every walk-forward, PBO/CSCV and ablation run +in `docs/experiments/` stops at **BTC/ETH/PAXG**. The `trials-ledger.jsonl` symbol census across +69 rows and 4 sessions: BTC 32, ETH 33, PAXG 32, and **0** for each of ADA, XLM, SOL, LTC, LINK. +ADA and XLM have been through that machinery exactly as much as SOL/LTC/LINK have: not at all. +The walk-forward that does exist tuned a *parameter* — `turtle_breakout.py:113`: + +```python +entry_lookback: int = 40, # Donchian-high entry (days); walk-forward OOS default (was 20) +``` + +## Gate 4: the live set didn't pass it either + +`keel-live.db` rules 1–5 (BTC/ETH/PAXG/ADA/XLM) were created in the same second +(`created_at` 1784914944 = 2026-07-24 17:42:24 UTC / 13:42:24 EDT) at `status = live` with +**`promoted_at IS NULL`** — seeded through the bypass +`rules seed` warns about in its own output: + +> ⚠️ seeded at LIVE status, bypassing the promotion gate. This is for the supervised live-order +> test only … **Do not leave live-seeded rules in place afterwards.** + +They have been in place 14 days. `rules seed` performs no backtest: `--status` is an +operator-supplied string written straight through by `insert_rule` (`commands/rules.py:351`). + +In `keel.db`, XLM (22) and ADA (24) *do* carry a `promoted_at`, while SOL (21), LTC (23) and +LINK (25) remain `candidate`. That promotion cannot have been earned: `rules promote` gates on a +candle-driven backtest (`commands/rules.py:180`), and XLM/ADA return **n_trades = 8 and 6** +against `min_trades = 100`. It is inference, not record, that `--force` was used — the `journal` +table is empty and `log_event` writes to logs, not the DB — but the gate demonstrably could not +have passed. + +> **Two corrections this investigation made to itself.** +> **(a)** The empty `backtests` table proves nothing. It is declared at `keel/data/db.py:140` +> and indexed, but there is no `INSERT INTO backtests` anywhere — `rules backtest` / `rules +> promote` compute in memory and never persist. An empty table is expected even where backtests +> *have* run. Consequently "SOL/LTC/LINK have never been backtested" is **unprovable** and is no +> longer claimed. +> **(b)** "Nothing has traded" was **false**. `trade_outcomes` is empty in both DBs, but +> `keel-live.db` holds `signals=1`, a **filled live order** (BTC-USD BUY 0.000778 @ 64267.30, +> `rule_id=6`) and an **open position**, both 2026-08-07 01:12:50, from the `dca` rule 6 — +> which *is* properly promoted. This deployment is not inert and does carry exposure. + +## The measurement nobody had run + +The original comment guessed. Running `keel rules backtest` — against a **copy** of `keel.db`, +never the original — settles the direction. The three excluded rules first, then the five the +sandbox actually trades, for contrast: + +| rule | asset | n_trades | win rate | expectancy | profit factor | max DD | | +|---|---|---:|---:|---:|---:|---:|---| +| 21 | SOL | 13 | 23.08% | −12.41 | **0.259** | 161.3 | excluded | +| 23 | LTC | 5 | 0.00% | −13.09 | **0.000** | 65.5 | excluded | +| 25 | LINK | 15 | 20.00% | −0.665 | **0.630** | 19.4 | excluded | +| 22 | XLM | 8 | 37.50% | +0.056 | 11.44 | 0.04 | | +| 24 | ADA | 6 | 50.00% | +0.121 | 5.52 | 0.10 | | +| 10 | BTC | 13 | 46.15% | +1353.99 | 1.61 | 12177.6 | | +| 11 | ETH | 13 | 30.77% | +45.72 | 1.21 | 1285.0 | | +| 12 | PAXG | 4 | 50.00% | +212.15 | 3.25 | 377.1 | | + +**Compare on profit factor, not expectancy.** Expectancy is quote-currency per trade, so it +tracks the asset's price — BTC's +1354 and XLM's +0.056 are not commensurable, and reading the +column as a ranking would put BTC four orders of magnitude "ahead" of ADA purely because a +bitcoin costs more. On the scale-free metric the split is clean: **every excluded asset is +pf < 1 (losing), every included asset is pf > 1.** LTC won none of five trades. + +So the earlier draft of this document was wrong to say "there is no negative result anywhere for +SOL, LTC or LINK" — there is one, and it was ten seconds away in data already sitting in +`keel.db`. + +**Read it with the sample sizes in view.** n = 4–15 against `min_trades = 100`, in-sample, one +window, no walk-forward, and on candles six days stale for the excluded three. By this project's +own standards (`2026-07-20-first-pbo-run.md` on why high PBO with a flat slope is the *good* +shape; §79.13 on 47 of 55 assets failing `t = 1.65`) this is a cheap directional check, not +validation. It cannot promote anything and should not. **Not one of the eight rules above clears +the gate** — the live five included, and ETH's 30.77% win rate also sits under the canonical +`min_win_rate` of 0.55. + +Note the ordering of events: this measurement is dated 2026-08-07, roughly two weeks after the +2026-07-24 exclusion. It is a **post-hoc vindication** of that decision, not the reason it was +made. The reason it was made remains unrecorded. + +What it does do is remove the symmetry argument: the two groups are not indistinguishable on +evidence, and the difference runs the way the original comment assumed. + +## What survives + +- The exclusion of SOL/LTC/LINK is **correct**, and now for a stated, reproducible reason. +- The label "unvalidated skip set" is still **wrong**, because it implies the other five were + validated. None of the five was. All five sit far below the promotion floor. +- "Same as the paper-forward" was **right**; "the walk-forward/PBO analysis" was **invented**. +- The exclusion is not, and never was, a compliance judgment. + +## When does the claim stop holding? + +The compliance half expired on 2026-07-23 when all eight assets were attested. The "unvalidated" +half was never accurate as a *distinguishing* claim. What now stands in for it — adverse +backtest expectancy — is itself provisional: it rests on 5–15 trades and would want re-running +after a `keel fetch`, and properly a walk-forward before anyone treats it as settled. + +## Action taken + +The comment in `config.live-sandbox.yaml` was rewritten to state the accurate reasons and to +carry the measured numbers. **The allowlist itself is unchanged** — the measurement supports the +status quo, and admission is the operator's call through the deterministic gate regardless. + +## Open items for the operator + +1. **Deployment drift.** Only the repo copy was edited; `~/keel/config.live-sandbox.yaml` is + still byte-identical to the old version. Sync it. +2. **`keel-live.db` carries almost no compliance state.** Screening the live allowlist against a + copy, **0 of 5 admit** — BTC/ETH/ADA/XLM reject on missing attestation, PAXG on attestation + *and* history, because the attestations and the PAXG waiver exist only in `keel.db`. + Operationally harmless — rail 1 reads `config.allowlist`, not the DB — but anyone running + `keel assets screen` against the live DB gets the opposite of the comment's picture. +3. **The one attestation `keel-live.db` does hold deserves a look, and it is not SOL's + `keel.db` one.** The row is: + + ``` + asset=SOL sector='layer-1 smart-contract blockchain infrastructure' backing=native + pays_yield=0 source='https://www.solana.com/staking' + attested_by='Elmehdi Aitbrahim' attested_at=1786148983 + (2026-08-08 00:29:43 UTC / 2026-08-07 20:29:43 EDT) + ``` + + Two things worth the operator's attention. **(a)** It is same-evening, written ~44 minutes + before this investigation's first commit — not historical state. It was not written by this + work (which touched only scratchpad copies; `~/keel/keel.db`'s mtime is unchanged and every + query here ran `mode=ro`), and `keel-live.db` is the live agent's own DB, so it is written + continuously. But its provenance should be confirmed rather than assumed. **(b)** Its + `source` is a **staking page**, which sits awkwardly against `pays_yield=0` and against every + `keel.db` attestation's "Attests bare unstaked spot only" framing — including SOL's own. That + is a compliance question independent of this document, and it is the operator's to settle. +4. **Live-seeded rules left in place 14 days**, against `rules seed`'s own instruction. +5. **Stale candles.** `keel fetch` follows the allowlist, so SOL/LTC/LINK daily bars in `keel.db` + end 2026-07-31 vs 2026-08-06. Re-run the backtests above after a fetch before relying on them. +6. **Two open attestation questions**, neither a defect: LINK's staking floor is funded from + emissions rather than user fees, and LTC has carried **MWEB** — opt-in confidential + transactions — since May 2022, a narrower form of the ZEC privacy question.