docs(experiments): rsi_meanrev's edge is selectivity, not alpha — the last live lead closes - #255
Merged
Merged
Conversation
… last live lead closes #252 left one open question: does rsi_meanrev's gross edge survive being made to fire more often? It does not. The rule reaches the promotion floor trivially — 21 of 24 assets at oversold=40 — and has no edge when it gets there. gross PF at oversold=20 (median n=38) : median 1.1631 gross>1 14/24 gross PF at every cell with n>=100 : median 0.8938 gross>1 11/76 net > 1.0 at n>=100, at ANY fee : 0 of 76, including maker The edge evaporates exactly when the rule becomes measurable. The 1.1631 that made this look like the best of the three rules IS what n=38 looks like. Pre-registration lives in the script docstring this time, which is the correction to the defect #252 recorded against itself. The declared statistic — a linear slope — is reported as declared (-0.0386/100 trades, 15/24 negative) AND flagged as underpowered: the phenomenon is a threshold effect at the floor, and two barely-trading assets supply most of its variance. Investigating a 34x non-monotonic trade count on UNI-USD found a simulator defect: backtest() never expires a pending setup, so a strategy whose entry is never revisited freezes silently for the rest of the series. Filed as #254, deliberately not fixed here. #252's headline results were checked against it and are clean. Closes #253. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eaitbrahim
added a commit
that referenced
this pull request
Aug 13, 2026
…ine, and record why the defects were invisible (#261) Both experiments re-run under #256 (pending setups no longer freeze the detector) and #258 (entries fill at the next bar's open, as production's market orders do). Same designs, same combinations, same fee grid and slippage pin -- only the engine differs. EVERY CONCLUSION SURVIVES EXCEPT ONE. ZEC-turtle no longer clears the maker line (1.034 -> 0.968), so #252 section 6's three-probe narrative describes a survivor the faithful engine never produces. The replacement is simpler and worse for the library: the viable quadrant is empty at every reachable fee -- 0 of 90 in #252, 0 of 82 in #255 -- with nothing needing three gates to die. #255 strengthens: the level shift across the trade floor widens from 1.1631 -> 0.8938 to 1.1251 -> 0.8396, gross-positive cells at the floor nearly halve (11/76 -> 6/82), and the 34x UNI-USD monotonicity anomaly that exposed both defects is structurally gone (3 assets -> 0). The two defects pushed in OPPOSITE directions -- #256 suppressed opportunity, #258 flattered execution -- so correcting both moved everything one way rather than adding noise: trade counts rose in 87 of 90, gross profit factors fell in 69 of 90. Arm B's transfer is restated on a single engine: 0.5770 in-sample vs 0.5427 out-of-sample, a gap of 0.034. #252's 0.6335 vs 0.6346 compared figures from one engine and was partly luck. The conclusion is unchanged -- the sweep winner is not overfit, it is stably unprofitable. Records the operational takeaway as section 5: NEITHER DEFECT WAS FOUND BY LOOKING FOR DEFECTS, and neither was findable by the means we had. 2,712 tests passed throughout. A frozen backtest and a highly selective strategy produce identical-looking output, so no summary ledger distinguishes them. The fix is invariants the engine reports about itself -- a dead-tail warning, intent-vs-fill divergence logging (#260), and cost anchored to output (#247, shipped, and the model for the rest) -- not more unit tests, which only assert behaviour someone already imagined. Annotates rather than rewrites, per the convention #247 set: the original numbers were real outputs of the code as it stood. Both documents keep their figures and carry a banner pointing here. Two ledger rows; chain verifies clean at 85. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #253. Answers the single open question left by #252. Documentation only — the simulator
defect found along the way is filed as #254 and deliberately not fixed here.
Result
rsi_meanrevreaches the promotion floor trivially — 21 of 24 assets atoversold=40, so thepre-declared conditional arm never fired — and has no edge when it gets there.
The edge evaporates exactly when the rule becomes measurable — as a level shift across the
floor, not a gradual decline. The 1.1631 that made this look like the best of the three rules in
#252 is what n=38 looks like. All 11 gross-positive cells at
n≥100are listed in the document(enumeration, not selection); the best is 1.215 gross and 0.678 at maker.
Hypothesis (b) from the pre-registration, confirmed: the rule is unpromotable by construction.
Pre-registration in the file this time
§7 of #252 recorded against itself that its declaration lived in a dispatch brief rather than the
script. This one's is in the docstring, written before the run — including the conditional arm's
trigger, threshold and reporting rule, so the widening could not become a post-hoc rescue.
Two method notes against ourselves
1. The declared statistic was underpowered for its own question. The slope is reported as
declared — mean −0.0386 per +100 trades, median −0.0197, 15/24 negative — and then flagged:
the relationship is a threshold effect at the floor, not linear, and two barely-trading assets
(PAXG-USD n 12→145, WLD n 1→71) supply most of its 0.35 stdev. The partition on the pre-declared
n≥100floor is the decisive reading. Reporting the stronger analysis while omitting that itwasn't the declared one is the exact failure pre-registration exists to prevent, so both are in.
2. The monotonicity assumption is violated on 3 of 24 assets — and the cause is not the rule.
The simulator defect (#254)
UNI-USD's trade count collapses 309 → 9 between
oversold30 and 35 — loosening the entrythreshold reduced trades 34×. Diagnosis: 9 closed trades, 0 open, last exit 2021-11-15,
i.e. the rule stopped detecting for ~40,000 bars.
A pending setup never expires. If price drifts away so neither entry nor stop is touched again,
rule.detect()is never called for the rest of the series. The strategy freezes silently, and afrozen backtest is indistinguishable from a selective one — which is how it survived unnoticed.
It biases toward low trade counts, i.e. toward exactly the "unmeasurable" verdicts these documents
have been issuing, and it diverges from production semantics (the live path re-detects each cycle).
#252's headline results were checked and are clean — the three sub-floor
turtleassets eachtrade to within 1–2 days of their series end, so their low counts are genuine short history. What
is not established is the effect on lower-n cells of earlier sweeps; that needs a re-run, which
is why #254 exists rather than a guess here.
What this closes
Every signal rule the codebase ships has now been measured at defaults across 24 assets and
along its own frequency axis. There is no asset-rule-parameter combination that is simultaneously
measurable and profitable at any reachable fee.
Priority order for what remains is in §6: fix #254 first (it's a correctness issue in the
instrument every one of these conclusions was produced with), then feed the PBO/CSCV gate that
#247 deployed and nothing supplies. The maker-execution and queue-simulation work has no target —
#252 removed its only candidate and this removes the rule that might have supplied another.
Ledger
One row,
decision: rejectedrather thandiagnostic_only— this trial made a decision and shouldcount toward the multiple-testing N.
verify_chainclean at 83 rows.🤖 Generated with Claude Code