fix(oura): gate RSA respiration to no-data when the R-R stream is banked - #1127
Merged
Merged
Conversation
An Oura night reports a respiratory rate that looks entirely normal while a WHOOP strap worn on the SAME nights measures ~16.4. Respiration is derived by RSA from the R-R stream, and the ring's banked overnight IBI cannot support RSA: it stores R-R VALUES against coarse per-record timestamps, so the beat-to-beat variation RSA reads is not in the data. Fix: respRateFromRR (both platforms) refuses a stream that is not beat-accurate, reusing HRVAnalyzer.beatAccurateFraction / beatValuesAreTrustworthy (ryanbr#1108) rather than carrying a second copy of the same judgement and its two constants. That is the collapse ryanbr#1108's PR body promised: one boundary, one definition, so a threshold change moves respiration and SDNN together. WHOOP R-R and the synthetic RSA fixtures measure ~100% and pass unchanged. MEASURED 2026-08-07 on two real nights (31,460 and 30,754 in-bed beats; the real SleepStager.respRateFromRR, plus a validated port to switch the gate off): beatAccurateFraction 0.0246 / 0.0235 (threshold 0.50) shipped path NaN / NaN both protections disabled 13.33 bpm / 13.33 bpm Three things this corrects in the original ryanbr#883 writeup, all of which strengthen it: 1. The number is 13.33, not the "~7-10" first reported. It sits inside respPlausibleRangeBpm (8-25) and inside the real adult sleeping range, so the range clamp is no protection whatsoever. Only 1 of 113 and 0 of 114 windows fall below 10 bpm. 2. The estimate carries zero information. SHUFFLING or REVERSING the night's R-R values returns the same 13.3333 to four decimals, on both nights. The tachogram's breath band is a flat 1/f shelf with no peak; 13.33 is the peak-picker's own floor on the 4 Hz grid. This is the ryanbr#194 bar failed outright, which is why the gate is on BANKED-ness and not on whether the output looks sane. 3. The mechanism in the old comment was wrong. The time AXIS is not corrupted: sum(R-R) over wall span is 1.030 / 1.008, so beat-time reconstructs the night to 1-3%. What is unusable is the interval VALUES - the ~6.6 s record decomposed into ~6 intervals whose sum is right to ~1% while the individuals are not beat-to-beat measurements, the same decomposition documented on beatValuesAreTrustworthy. Also documents, and pins with a test, that this gate and ryanbr#977's splice skip catch OPPOSITE banking geometries and neither subsumes the other: ryanbr#977 catches banking that TILES time (the real ring - it independently discards 113/113 and 114/114 windows on these nights), this catches banking that COMPRESSES it (the batched fixture, where no gap ever exceeds rsaGapToleranceS and ryanbr#977 is blind). And one known limitation, pinned deliberately: the gate detects banking by its symptom, and that symptom is repairable. Re-timing each record's beats by cumsum from the record's own timestamp moves the fraction 0.0246/0.0235 -> 0.875/0.863, defeating both this gate and ryanbr#977 - while the estimate stays 13.3333 and stays unchanged under shuffling. A well-meant decoder change would silently switch respiration back on for a stream carrying no breathing information. testRetiming...knownLimitation fails the day that happens, and says what to do: re-base the gate on provenance, not timestamp shape. Tests: Swift StrandAnalytics 1270/0 (RespRateRsaTests 7/7 incl. 3 new); Android compileFullDebugKotlin clean, 3554 tests / 3 failed - all 3 (AiCoachContextTest, 2x StandardHrSensorFormatTest locale) reproduce on clean upstream/main, pre-existing. No hardware behaviour changes: this is analytics-only, no BLE path touched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XXdjctbkxqo359NuuJascK
Owner
|
Cross-ref: this gate has an unintended interaction with WHOOP 4.0 — see #1331. WHOOP 4.0's R-R is over-counted (#1008 stamps several R-R in one coarse second), which is timing-indistinguishable from an Oura banked stream, so the beat-accuracy gate here catches 4.0 too and blanks respiratory rate on every 4.0 night (regression since V10). The gate itself is correct — 4.0's over-counted beats genuinely can't support RSA (beat-time is rebuilt from the R-R values, so duplication reads the rate low). The real fix is the #1118 two-channel R-R de-dup, after which 4.0 passes this gate legitimately. Noting here so the interaction is on record. |
This was referenced Aug 15, 2026
ryanbr
added a commit
that referenced
this pull request
Aug 15, 2026
… fix (#1331/#1008/#1118) (#1352) * diag(hrv): shadow-log a deduped R-R stream to validate the over-count fix (#1331/#1008/#1118) The WHOOP 4.0 R-R over-count (#1008 — each beat emitted 2-3x, an exact dup plus a two-optical-channel twin ~34ms off) corrupts HRV (#1008/#1118) AND, via #1127's RSA beat-accuracy gate, blanks respiratory rate on 4.0 since V10 (#1331). The fix is a de-dup, but it changes a physiological signal, so it must be validated against ground truth before it becomes the read path. Stage 1 (this PR) — instrument, do not change behaviour: - HRVAnalyzer/HrvAnalyzer.collapseOverCount: pure, byte-identical de-dup that collapses same-second beats within ~40ms of one already kept (the exact dups + the ~34ms channel twins). Returns the deduped (tsSec, rrMs). - On an over-count night the always-on hrv diag appends a `hrv dedup` line with the DEDUPED stream's rmssd/sdnn/meanNN/coverage/beatAccurate beside the raw. Shipped HRV/resp is UNCHANGED — this only logs a candidate. So a strap log now shows, per 4.0 over-count night, whether de-dup drives coverage→~1.0 and beatAccurate high (which would pass #1127's RSA gate → resp returns, the #1331 fix) and rmssd/sdnn toward physiological — validated against WHOOP's own numbers and @artemc's Polar H10 (#1118) before Stage 2 flips it live. Tests: HrvCollapseOverCount{Test,Tests} pin a synthetic 3x over-count collapsing 180→60 beats, coverage ~3x→~1.0, clean streams untouched. compileFullDebugKotlin + the Kotlin test green; StrandAnalytics via swift-packages; IntelligenceEngine shadow is app-target → app-build gate. Refs #1331, #1008, #1118, #1127. * diag(hrv): log an exact-dup reference beside the ~40ms de-dup candidate (#1331) Re-review caught that the ~40ms same-second collapse can over-merge two REAL neighbouring beats whose R-R values sit within 40ms (normal at sleeping HR), so its coverage undershoots 1.0 even when de-dup is working — confounding the validation signal. Add the EXACT-duplicate collapse (rrTolMs=0, same ts AND value, provably no real-beat loss) as a safe reference line, so the shadow logs the honest spectrum: covExact (safe floor, the unambiguous exact-dup magnitude) alongside cov40/beatAcc40/rmssd40 (aggressive upper bound). The real de-dup sits between them. Test pins the exact-dup floor (180->120, coverage ~2x, twins kept). * fixup: reattach collapsedCoverage doc (collapseOverCount was inserted between doc + fn) doc_comment_lint flagged HrvAnalyzer.kt:503 — collapseOverCount landed between collapsedCoverage's #550 KDoc and its declaration, detaching the doc. Move collapseOverCount above the #550 doc so it reattaches. No code change.
simoncad7
pushed a commit
to simoncad7/noop
that referenced
this pull request
Aug 17, 2026
…ked (ryanbr#1127) An Oura night reports a respiratory rate that looks entirely normal while a WHOOP strap worn on the SAME nights measures ~16.4. Respiration is derived by RSA from the R-R stream, and the ring's banked overnight IBI cannot support RSA: it stores R-R VALUES against coarse per-record timestamps, so the beat-to-beat variation RSA reads is not in the data. Fix: respRateFromRR (both platforms) refuses a stream that is not beat-accurate, reusing HRVAnalyzer.beatAccurateFraction / beatValuesAreTrustworthy (ryanbr#1108) rather than carrying a second copy of the same judgement and its two constants. That is the collapse ryanbr#1108's PR body promised: one boundary, one definition, so a threshold change moves respiration and SDNN together. WHOOP R-R and the synthetic RSA fixtures measure ~100% and pass unchanged. MEASURED 2026-08-07 on two real nights (31,460 and 30,754 in-bed beats; the real SleepStager.respRateFromRR, plus a validated port to switch the gate off): beatAccurateFraction 0.0246 / 0.0235 (threshold 0.50) shipped path NaN / NaN both protections disabled 13.33 bpm / 13.33 bpm Three things this corrects in the original ryanbr#883 writeup, all of which strengthen it: 1. The number is 13.33, not the "~7-10" first reported. It sits inside respPlausibleRangeBpm (8-25) and inside the real adult sleeping range, so the range clamp is no protection whatsoever. Only 1 of 113 and 0 of 114 windows fall below 10 bpm. 2. The estimate carries zero information. SHUFFLING or REVERSING the night's R-R values returns the same 13.3333 to four decimals, on both nights. The tachogram's breath band is a flat 1/f shelf with no peak; 13.33 is the peak-picker's own floor on the 4 Hz grid. This is the ryanbr#194 bar failed outright, which is why the gate is on BANKED-ness and not on whether the output looks sane. 3. The mechanism in the old comment was wrong. The time AXIS is not corrupted: sum(R-R) over wall span is 1.030 / 1.008, so beat-time reconstructs the night to 1-3%. What is unusable is the interval VALUES - the ~6.6 s record decomposed into ~6 intervals whose sum is right to ~1% while the individuals are not beat-to-beat measurements, the same decomposition documented on beatValuesAreTrustworthy. Also documents, and pins with a test, that this gate and ryanbr#977's splice skip catch OPPOSITE banking geometries and neither subsumes the other: ryanbr#977 catches banking that TILES time (the real ring - it independently discards 113/113 and 114/114 windows on these nights), this catches banking that COMPRESSES it (the batched fixture, where no gap ever exceeds rsaGapToleranceS and ryanbr#977 is blind). And one known limitation, pinned deliberately: the gate detects banking by its symptom, and that symptom is repairable. Re-timing each record's beats by cumsum from the record's own timestamp moves the fraction 0.0246/0.0235 -> 0.875/0.863, defeating both this gate and ryanbr#977 - while the estimate stays 13.3333 and stays unchanged under shuffling. A well-meant decoder change would silently switch respiration back on for a stream carrying no breathing information. testRetiming...knownLimitation fails the day that happens, and says what to do: re-base the gate on provenance, not timestamp shape. Tests: Swift StrandAnalytics 1270/0 (RespRateRsaTests 7/7 incl. 3 new); Android compileFullDebugKotlin clean, 3554 tests / 3 failed - all 3 (AiCoachContextTest, 2x StandardHrSensorFormatTest locale) reproduce on clean upstream/main, pre-existing. No hardware behaviour changes: this is analytics-only, no BLE path touched. Claude-Session: https://claude.ai/code/session_01XXdjctbkxqo359NuuJascK Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
simoncad7
pushed a commit
to simoncad7/noop
that referenced
this pull request
Aug 17, 2026
… fix (ryanbr#1331/ryanbr#1008/ryanbr#1118) (ryanbr#1352) * diag(hrv): shadow-log a deduped R-R stream to validate the over-count fix (ryanbr#1331/ryanbr#1008/ryanbr#1118) The WHOOP 4.0 R-R over-count (ryanbr#1008 — each beat emitted 2-3x, an exact dup plus a two-optical-channel twin ~34ms off) corrupts HRV (ryanbr#1008/ryanbr#1118) AND, via ryanbr#1127's RSA beat-accuracy gate, blanks respiratory rate on 4.0 since V10 (ryanbr#1331). The fix is a de-dup, but it changes a physiological signal, so it must be validated against ground truth before it becomes the read path. Stage 1 (this PR) — instrument, do not change behaviour: - HRVAnalyzer/HrvAnalyzer.collapseOverCount: pure, byte-identical de-dup that collapses same-second beats within ~40ms of one already kept (the exact dups + the ~34ms channel twins). Returns the deduped (tsSec, rrMs). - On an over-count night the always-on hrv diag appends a `hrv dedup` line with the DEDUPED stream's rmssd/sdnn/meanNN/coverage/beatAccurate beside the raw. Shipped HRV/resp is UNCHANGED — this only logs a candidate. So a strap log now shows, per 4.0 over-count night, whether de-dup drives coverage→~1.0 and beatAccurate high (which would pass ryanbr#1127's RSA gate → resp returns, the ryanbr#1331 fix) and rmssd/sdnn toward physiological — validated against WHOOP's own numbers and @artemc's Polar H10 (ryanbr#1118) before Stage 2 flips it live. Tests: HrvCollapseOverCount{Test,Tests} pin a synthetic 3x over-count collapsing 180→60 beats, coverage ~3x→~1.0, clean streams untouched. compileFullDebugKotlin + the Kotlin test green; StrandAnalytics via swift-packages; IntelligenceEngine shadow is app-target → app-build gate. Refs ryanbr#1331, ryanbr#1008, ryanbr#1118, ryanbr#1127. * diag(hrv): log an exact-dup reference beside the ~40ms de-dup candidate (ryanbr#1331) Re-review caught that the ~40ms same-second collapse can over-merge two REAL neighbouring beats whose R-R values sit within 40ms (normal at sleeping HR), so its coverage undershoots 1.0 even when de-dup is working — confounding the validation signal. Add the EXACT-duplicate collapse (rrTolMs=0, same ts AND value, provably no real-beat loss) as a safe reference line, so the shadow logs the honest spectrum: covExact (safe floor, the unambiguous exact-dup magnitude) alongside cov40/beatAcc40/rmssd40 (aggressive upper bound). The real de-dup sits between them. Test pins the exact-dup floor (180->120, coverage ~2x, twins kept). * fixup: reattach collapsedCoverage doc (collapseOverCount was inserted between doc + fn) doc_comment_lint flagged HrvAnalyzer.kt:503 — collapseOverCount landed between collapsedCoverage's ryanbr#550 KDoc and its declaration, detaching the doc. Move collapseOverCount above the ryanbr#550 doc so it reattaches. No code change.
ryanbr
added a commit
that referenced
this pull request
Aug 17, 2026
…w candidate (#1331/#1118) The #1331 4.0 logs read `crossSecondOverCount` every night: the same-second shadow collapses (#1352's exact + 40 ms candidates) can't reach the duplicate twins because they straddle the second boundary, so `cov40` stays ~1.7-2.0 on the heavy nights and RSA-respiration stays blanked by #1127's beat-accuracy gate. Add a THIRD shadow candidate, `xsec` — the 40 ms collapse widened to a 1-second window — logged beside the others so we can SIZE how much of a night is cross-second (does coverage fall to ~1.0, does beat-accuracy clear the 0.5 gate?). `collapseOverCount` gains a `windowSec` param (default 0 = same-second, byte- identical for every existing caller). Instrumentation ONLY — the shipped HRV/resp path is unchanged. The window is a strict UPPER BOUND, deliberately NOT a shippable de-dup: a steady real HR has near-identical intervals one second apart, so a cross-second collapse over-merges real beats (the test pins this: a 10-beat steady stream drops to 5 at window=1). The real fix is density/timeline-based and must be validated against ground truth (@artemc's H10) before it becomes the read path — this just quantifies the target. Both platforms (byte-identical `hrv dedup` line + twin tests).
ryanbr
added a commit
that referenced
this pull request
Aug 17, 2026
…w candidate (#1331/#1118) (#1399) The #1331 4.0 logs read `crossSecondOverCount` every night: the same-second shadow collapses (#1352's exact + 40 ms candidates) can't reach the duplicate twins because they straddle the second boundary, so `cov40` stays ~1.7-2.0 on the heavy nights and RSA-respiration stays blanked by #1127's beat-accuracy gate. Add a THIRD shadow candidate, `xsec` — the 40 ms collapse widened to a 1-second window — logged beside the others so we can SIZE how much of a night is cross-second (does coverage fall to ~1.0, does beat-accuracy clear the 0.5 gate?). `collapseOverCount` gains a `windowSec` param (default 0 = same-second, byte- identical for every existing caller). Instrumentation ONLY — the shipped HRV/resp path is unchanged. The window is a strict UPPER BOUND, deliberately NOT a shippable de-dup: a steady real HR has near-identical intervals one second apart, so a cross-second collapse over-merges real beats (the test pins this: a 10-beat steady stream drops to 5 at window=1). The real fix is density/timeline-based and must be validated against ground truth (@artemc's H10) before it becomes the read path — this just quantifies the target. Both platforms (byte-identical `hrv dedup` line + twin tests).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-land of #883, which was closed in the 2026-07-31 batch rather than on merit. Reworked onto
HRVAnalyzer.beatAccurateFraction(#1108 landed the canonical definition after #883 was written), andre-validated on two fresh nights — which corrected the original writeup in three places.
The problem
An Oura night reports a respiratory rate that looks entirely normal, while a WHOOP strap worn on the
same nights measures ~16.4. Respiration is derived by RSA from the R-R stream, and the ring's banked
overnight IBI cannot support RSA: it stores R-R values against coarse per-record timestamps, so the
beat-to-beat variation RSA reads is not in the data.
The fix
respRateFromRR(both platforms) refuses a stream that is not beat-accurate, callingHRVAnalyzer.beatAccurateFraction/beatValuesAreTrustworthyinstead of carrying a second copy of thesame judgement and its two constants. That is the collapse #1108's body promised — one boundary, one
definition, so a threshold change moves respiration and SDNN together instead of letting them drift.
WHOOP R-R and the synthetic RSA fixtures measure ~100% and pass unchanged.
Measured, 2026-08-07, two real nights
31,460 and 30,754 in-bed beats, through the real
SleepStager.respRateFromRR(temporary probeagainst
Packages/StrandAnalytics), plus a port validated against it — both synthetic fixturesreproduce (15.0000 / 10.9091) and both nights reproduce NaN — used to switch the gate off, since the
Swift constants are
static let.beatAccurateFractionThree corrections to the original #883 writeup — all of which strengthen it
1. The number is 13.33, not "~7–10". It sits inside
respPlausibleRangeBpm(8–25) and inside thereal adult sleeping range, so the range clamp is no protection whatsoever. Only 1 of 113 and 0 of
114 windows fall below 10 bpm. This also answers @vishk23's reading in
#883 (comment) — the exposed band is
not 8–10 with the sub-8 part already clamped; it is dead centre. Everything else in that comment holds,
and the consequence chain it traced (Charge inflation via the lower-is-better recovery term →
contaminated
"resp"baseline → second-order illness FP, no same-night false positive) is the rightone. It is just worse than it looked: a fabricated 13.3 is indistinguishable by eye from a real reading.
2. The estimate carries zero information. Shuffling or reversing the night's R-R values returns the
same 13.3333 to four decimals, on both nights. The tachogram's breath band is a flat 1/f shelf with
no peak (neighbouring bins at 0.86–0.96 of the maximum); 13.33 is the peak-picker's own floor on the
4 Hz grid, given its 2.5 s minimum distance. That is the #194 bar failed outright — hence a gate on
banked-ness, not on whether the output looks sane.
3. The mechanism in the old comment was wrong. The time axis is not corrupted: sum(R-R) over wall
span is 1.030 / 1.008, so beat-time reconstructs the night to 1–3%. What is unusable is the interval
values — the ~6.6 s record decomposed into ~6 intervals whose sum is right to ~1% while the
individuals are not beat-to-beat measurements, the same decomposition already documented on
beatValuesAreTrustworthy. Proof: the batched fixture, run ungated, returns the correct 15.0, becausecumsum recovers relative beat timing. Banked timestamps alone do not break RSA.
Why both this and #977's splice skip are kept
They catch opposite banking geometries, and neither subsumes the other — now pinned by a test:
interval reads as a splice; on these two nights it independently discards 113/113 and 114/114
windows. (So on this firmware the gate is the second lock. See the sequencing note below.)
so no gap ever exceeds
rsaGapToleranceSand [hrv] classifyCoverage has a ceiling and no floor, so an under-covered night reads as plausible #977 is blind to it.A firmware that changes its record period moves a stream from one geometry to the other without warning.
Known limitation, pinned deliberately
The gate detects banking by its symptom — coarse, repeated timestamps — and that symptom is a
transport artifact that is trivially repairable. Re-timing each record's beats by cumsum from the
record's own timestamp moves the fraction 0.0246 / 0.0235 → 0.875 / 0.863, defeating both this gate
and #977 — while the estimate stays 13.3333 and stays unchanged under shuffling. A well-meant decoder
change that distributes beat timestamps within a record would silently switch respiration back on for a
stream carrying no breathing information.
testRetimingABankedStreamDefeatsTheGate_knownLimitationfails the day that happens, and says what todo: re-base the gate on provenance (was this stream banked?) rather than on timestamp shape.
Sequencing vs #877
#877 is what first routes a ring night into the estimator (
detectSleepcurrently bails on agravity-less owner, so
matchedis empty andrespRateBpmcomes out nil). Landing this first coversthat path on day one — worth doing, and free.
Honest caveat on the urgency: on this firmware the exposure window is empty. #977's splice skip
already returns NaN on both nights, so merging #877 first would still yield nil. The ordering is
belt-and-braces, not a live bug — but the geometry argument above is exactly why I would not rely on
that staying true.
Verification
StrandAnalytics1270 / 0 (RespRateRsaTests7/7, 3 new).compileFullDebugKotlinclean;testFullDebugUnitTest3554 tests / 3 failed — all 3(
AiCoachContextTest, 2×StandardHrSensorFormatTestlocale) reproduce on cleanupstream/main,pre-existing.
four fixtures.
Still untested
Neither capture contains a beat-accurate R-R stream (
rrIntervalis 100% Oura; the WHOOP rows areimport-only with no R-R), so the gate's false-positive direction — refusing a stream it should
accept — is still covered only by the synthetic fixtures. A WHOOP night streamed rather than imported
would close that.