From 4c69f0de8e91838de1aa35126c4a69e55685e3de Mon Sep 17 00:00:00 2001 From: abdulsaheel Date: Sun, 2 Aug 2026 15:34:46 +0530 Subject: [PATCH] repin analytics + protocol to main, kAlgoVersion 53 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The sleep-stager rewrite (analytics#34) and the protocol RR-bound fix (protocol#21) were both on their respective mains but not reaching users, because edge was still pinned behind. Analytics was two hops back — sitting on analytics#32's PR-BRANCH head even though #32 merged two days ago. analytics cbbe06a -> f0d1153 (main @ #34) protocol 5bb8606 -> 7edcb3e (main @ #21) analytics#34 replaces the deep/REM boolean conjunctions with weighted robust-z scores. The old deep rule AND-ed one informative axis with one null one (rmssd, d -0.13) and one INVERTED one (mean HR, d +0.31 — deep sleep runs slightly FASTER than light on the wrist), so all three only co-fired by luck. That's what produced 30-second deep specks the 3-min bout rule then deleted. Against 99 PSG-labelled wrist nights: kappa 0.036 -> 0.128 (0.132 held out), deep 10.3/5.7 -> 53.0/12.9 sens/PPV, REM 30.6/12.1 -> 52.6/20.7. protocol#21 bounds realtimeRr to the same 200-2500ms range parseRealtimeHr and parseR24 already use, so a misaligned 0x28 frame can't hand a 5ms "beat" to live HRV. The other half of that commit (historical-family activity/steps_inc null instead of 0) isn't read by edge. kAlgoVersion 52 -> 53: every day's hypnogram, stage minutes and sleep-derived scalars change, so every day re-derives. Verified both SHAs actually contain what this claims, per the v43 lesson: git show f0d1153:lib/src/onehz/sleep/cardio_stager.dart | grep classifyCardioEpochs git show 7edcb3e:lib/src/live.dart | grep kMinRrMs Reproduced CI locally rather than trusting the path overrides: moved pubspec_overrides.yaml aside, pub get, confirmed lock resolved-ref matches both pins, ran check_sibling_pins.sh (both agree), then analyze + full suite against the REAL pinned packages. 1088 tests, analyze clean. Lock diff is the four ref lines only — no path: sources. --- lib/compute/derivation_engine.dart | 17 +++++++++++- pubspec.lock | 8 +++--- pubspec.yaml | 43 +++++++++++++++++------------- 3 files changed, 45 insertions(+), 23 deletions(-) diff --git a/lib/compute/derivation_engine.dart b/lib/compute/derivation_engine.dart index ed07e56..4a27646 100644 --- a/lib/compute/derivation_engine.dart +++ b/lib/compute/derivation_engine.dart @@ -376,7 +376,22 @@ import 'substrate.dart'; // ~17.5% of subjects get WORSE from personalization), and (3) discarding // pre-tracking profiles, which cannot be repaired, so they rebuild honestly. // Bump so every day re-stages without the corrupt blend. -const int kAlgoVersion = 52; +// v53: repin analytics to main @ #34 — the sleep-stager decision layer is +// rewritten. Deep and REM were boolean conjunctions AND-ing one informative +// axis with one null one (rmssd, Cohen's d -0.13 deep / -0.02 REM) and one +// INVERTED one (mean HR, d +0.31 for deep, i.e. deep sleep runs slightly +// FASTER than light on the wrist), so all three could only co-fire by +// coincidence — which is why deep sleep came out as isolated 30-second specks +// that the 3-min minimum-bout rule then deleted. Scored against 99 PSG-labelled +// wrist nights those rules managed kappa 0.036, with deep PPV 5.7% against a +// 4.5% base rate and REM 12.1% against 14.0% — at or below chance for both. +// Now weighted robust-z scores (weights = the measured effect sizes) over +// Rk / hrSd / sdnn / lfhf, with rmssd and mean HR dropped: kappa 0.128, 0.132 +// on held-out subjects, deep 53.0/12.9 and REM 52.6/20.7 sens/PPV. Every day's +// hypnogram, stage minutes and sleep-derived scalars change, so every day must +// re-derive. Also picks up the protocol realtimeRr bound (live HRV no longer +// sees implausible sub-100ms "beats" from a misaligned 0x28 frame). +const int kAlgoVersion = 53; // Fold idempotency, the minimum-nights warm-up, and legacy-payload handling // all live in SleepProfilePolicy (pure, unit-tested) — see diff --git a/pubspec.lock b/pubspec.lock index bd6ebe1..550fd1b 100644 --- a/pubspec.lock +++ b/pubspec.lock @@ -964,8 +964,8 @@ packages: dependency: "direct main" description: path: "." - ref: cbbe06addec1cb78b4ea2c75f64e8a281ac09294 - resolved-ref: cbbe06addec1cb78b4ea2c75f64e8a281ac09294 + ref: f0d115308aa5e9e3c82ee113c3d52e59756121d4 + resolved-ref: f0d115308aa5e9e3c82ee113c3d52e59756121d4 url: "https://github.com/OpenStrap/analytics.git" source: git version: "1.0.0" @@ -973,8 +973,8 @@ packages: dependency: "direct main" description: path: "." - ref: "5bb8606964f2d94d9a73dc42c57416cb74282d52" - resolved-ref: "5bb8606964f2d94d9a73dc42c57416cb74282d52" + ref: "7edcb3e377329968118c62cb03a81d95e2f6db8e" + resolved-ref: "7edcb3e377329968118c62cb03a81d95e2f6db8e" url: "https://github.com/OpenStrap/protocol.git" source: git version: "1.0.0" diff --git a/pubspec.yaml b/pubspec.yaml index 2b5c900..422986a 100644 --- a/pubspec.yaml +++ b/pubspec.yaml @@ -41,27 +41,34 @@ dependencies: # hexToBytes rejects odd-length hex instead of silently flooring it. This # PR sat unpinned for a full day after merging — see fix/issues #22 for # why (edge's own release pipeline shipped the pre-fix decoder). - # NOTE: this ref predates protocol fix/issues#21 (realtimeRr RR-bound, - # historical-family activity/steps_inc null-vs-0) — repin again once #21 - # merges to pick those up too. - # Verified present: `git show :lib/src/framing.dart | grep 'crc8(Uint8List'`. - ref: 5bb8606964f2d94d9a73dc42c57416cb74282d52 + # protocol main @ #21 merge. Picks up the realtimeRr RR-bound (live.dart + # accepted ANY positive int16 as an interval, unlike parseRealtimeHr and + # parseR24 which both gate 200-2500ms, so a misaligned 0x28 frame could + # hand a 5ms "beat" to live HRV/coherence) plus the historical-family + # activity/steps_inc null-instead-of-0 fix. Edge does not read the latter + # two fields, so only the RR bound is behaviour-visible here. + # Verified present: `git show :lib/src/live.dart | grep -c kMinRrMs`. + ref: 7edcb3e377329968118c62cb03a81d95e2f6db8e openstrap_analytics: git: url: https://github.com/OpenStrap/analytics.git - # PR-BRANCH HEAD, not main — OpenStrap/analytics#32 (fix/issue-170-...) - # is open, not yet merged. Repin to the merge commit on main once it - # lands (same pattern v49 used briefly for protocol/analytics PR heads). - # autoDetectWorkouts now bypasses the motion-confirmation gate on a - # genuine HR-onset (fast rise vs. immediately-preceding baseline), - # fixing low-limb-swing cardio (cycling/rowing) going undetected while - # still rejecting slow-drifting elevations (fever/heat/anxiety) that - # have no discernible onset. - # Verified present at THIS sha: `git show :lib/src/onehz/workout/auto_detect.dart | grep onsetRiseBpm`. - # Moved to the branch's new head (cbbe06a) after CodeRabbit caught a real - # off-by-one in the onset window on analytics#32 (earlyMean was 181s vs - # preMean's 180s) — fixed there, not worth a separate edge changelog line. - ref: cbbe06addec1cb78b4ea2c75f64e8a281ac09294 + # analytics main @ #34 merge. Two hops in one: #32 (the HR-onset bypass + # for low-limb-swing cardio) had already merged and this pin was still + # sitting on its PR-branch head, and #34 lands the sleep-stager rewrite. + # + # #34 replaces the deep/REM boolean conjunctions with weighted robust-z + # scores. Measured against 99 PSG-labelled wrist nights (DREAMT), the old + # rules scored kappa 0.036 — deep PPV 5.7% against a 4.5% base rate, REM + # 12.1% against 14.0%, i.e. at or below chance — because the deep rule + # AND-ed one good axis, one null axis (rmssd, d -0.13) and one INVERTED + # axis (mean HR, d +0.31), so they could only co-fire by luck. That is + # what produced 30-second deep specks the 3-min bout rule then deleted. + # Now kappa 0.128 (0.132 held out), deep 53.0/12.9 sens/PPV, REM + # 52.6/20.7. Cutoffs are calibrated on OUR captures, not DREAMT — the + # DREAMT-optimal values under-called REM badly on real WHOOP data. + # Verified present: `git show :lib/src/onehz/sleep/cardio_stager.dart + # | grep -E 'classifyCardioEpochs|_remScoreCut = 0.5'`. + ref: f0d115308aa5e9e3c82ee113c3d52e59756121d4 # BLE — flutter_blue_plus is the maintained cross-platform GATT client. flutter_blue_plus: ^1.36.8