Skip to content

fix(oura): gate RSA respiration to no-data when the R-R stream is banked - #1127

Merged
ryanbr merged 1 commit into
ryanbr:mainfrom
pipiche38:oura-resp-rsa-beat-accuracy-gate
Aug 8, 2026
Merged

fix(oura): gate RSA respiration to no-data when the R-R stream is banked#1127
ryanbr merged 1 commit into
ryanbr:mainfrom
pipiche38:oura-resp-rsa-beat-accuracy-gate

Conversation

@pipiche38

Copy link
Copy Markdown

Re-land of #883, which was closed in the 2026-07-31 batch rather than on merit. Reworked onto
HRVAnalyzer.beatAccurateFraction (#1108 landed the canonical definition after #883 was written), and
re-validated on two fresh nights — which corrected the original writeup in three places.

The problem

An Oura night reports a respiratory rate that looks entirely normal, while a WHOOP strap worn on the
same nights measures ~16.4. Respiration is derived by RSA from the R-R stream, and the ring's banked
overnight IBI cannot support RSA: it stores R-R values against coarse per-record timestamps, so the
beat-to-beat variation RSA reads is not in the data.

The fix

respRateFromRR (both platforms) refuses a stream that is not beat-accurate, calling
HRVAnalyzer.beatAccurateFraction / beatValuesAreTrustworthy instead of carrying a second copy of the
same judgement and its two constants. That is the collapse #1108's body promised — one boundary, one
definition, so a threshold change moves respiration and SDNN together instead of letting them drift.
WHOOP R-R and the synthetic RSA fixtures measure ~100% and pass unchanged.

Measured, 2026-08-07, two real nights

31,460 and 30,754 in-bed beats, through the real SleepStager.respRateFromRR (temporary probe
against Packages/StrandAnalytics), plus a port validated against it — both synthetic fixtures
reproduce (15.0000 / 10.9091) and both nights reproduce NaN — used to switch the gate off, since the
Swift constants are static let.

Night 1 (08-05→08-06) Night 2 (08-06→08-07)
beatAccurateFraction 0.0246 0.0235
shipped path NaN NaN
gate OFF, splice-skip ON NaN NaN
both disabled 13.33 bpm 13.33 bpm

Three corrections to the original #883 writeup — all of which strengthen it

1. The number is 13.33, not "~7–10". It sits inside respPlausibleRangeBpm (8–25) and inside the
real adult sleeping range, so the range clamp is no protection whatsoever. Only 1 of 113 and 0 of
114
windows fall below 10 bpm. This also answers @vishk23's reading in
#883 (comment) — the exposed band is
not 8–10 with the sub-8 part already clamped; it is dead centre. Everything else in that comment holds,
and the consequence chain it traced (Charge inflation via the lower-is-better recovery term →
contaminated "resp" baseline → second-order illness FP, no same-night false positive) is the right
one. It is just worse than it looked: a fabricated 13.3 is indistinguishable by eye from a real reading.

2. The estimate carries zero information. Shuffling or reversing the night's R-R values returns the
same 13.3333 to four decimals, on both nights. The tachogram's breath band is a flat 1/f shelf with
no peak (neighbouring bins at 0.86–0.96 of the maximum); 13.33 is the peak-picker's own floor on the
4 Hz grid, given its 2.5 s minimum distance. That is the #194 bar failed outright — hence a gate on
banked-ness, not on whether the output looks sane.

3. The mechanism in the old comment was wrong. The time axis is not corrupted: sum(R-R) over wall
span is 1.030 / 1.008, so beat-time reconstructs the night to 1–3%. What is unusable is the interval
values — the ~6.6 s record decomposed into ~6 intervals whose sum is right to ~1% while the
individuals are not beat-to-beat measurements, the same decomposition already documented on
beatValuesAreTrustworthy. Proof: the batched fixture, run ungated, returns the correct 15.0, because
cumsum recovers relative beat timing. Banked timestamps alone do not break RSA.

Why both this and #977's splice skip are kept

They catch opposite banking geometries, and neither subsumes the other — now pinned by a test:

A firmware that changes its record period moves a stream from one geometry to the other without warning.

Known limitation, pinned deliberately

The gate detects banking by its symptom — coarse, repeated timestamps — and that symptom is a
transport artifact that is trivially repairable. Re-timing each record's beats by cumsum from the
record's own timestamp moves the fraction 0.0246 / 0.0235 → 0.875 / 0.863, defeating both this gate
and #977 — while the estimate stays 13.3333 and stays unchanged under shuffling. A well-meant decoder
change that distributes beat timestamps within a record would silently switch respiration back on for a
stream carrying no breathing information.

testRetimingABankedStreamDefeatsTheGate_knownLimitation fails the day that happens, and says what to
do: re-base the gate on provenance (was this stream banked?) rather than on timestamp shape.

Sequencing vs #877

#877 is what first routes a ring night into the estimator (detectSleep currently bails on a
gravity-less owner, so matched is empty and respRateBpm comes out nil). Landing this first covers
that path on day one — worth doing, and free.

Honest caveat on the urgency: on this firmware the exposure window is empty. #977's splice skip
already returns NaN on both nights, so merging #877 first would still yield nil. The ordering is
belt-and-braces, not a live bug — but the geometry argument above is exactly why I would not rely on
that staying true.

Verification

  • Swift StrandAnalytics 1270 / 0 (RespRateRsaTests 7/7, 3 new).
  • Android compileFullDebugKotlin clean; testFullDebugUnitTest 3554 tests / 3 failed — all 3
    (AiCoachContextTest, 2× StandardHrSensorFormatTest locale) reproduce on clean upstream/main,
    pre-existing.
  • Both platforms carry the same gate, the same constants (now via one shared definition), and the same
    four fixtures.
  • Analytics-only; no BLE path touched, so no hardware behaviour changes.

Still untested

Neither capture contains a beat-accurate R-R stream (rrInterval is 100% Oura; the WHOOP rows are
import-only with no R-R), so the gate's false-positive direction — refusing a stream it should
accept — is still covered only by the synthetic fixtures. A WHOOP night streamed rather than imported
would close that.

An Oura night reports a respiratory rate that looks entirely normal while a WHOOP
strap worn on the SAME nights measures ~16.4. Respiration is derived by RSA from
the R-R stream, and the ring's banked overnight IBI cannot support RSA: it stores
R-R VALUES against coarse per-record timestamps, so the beat-to-beat variation RSA
reads is not in the data.

Fix: respRateFromRR (both platforms) refuses a stream that is not beat-accurate,
reusing HRVAnalyzer.beatAccurateFraction / beatValuesAreTrustworthy (ryanbr#1108) rather
than carrying a second copy of the same judgement and its two constants. That is the
collapse ryanbr#1108's PR body promised: one boundary, one definition, so a threshold
change moves respiration and SDNN together. WHOOP R-R and the synthetic RSA fixtures
measure ~100% and pass unchanged.

MEASURED 2026-08-07 on two real nights (31,460 and 30,754 in-bed beats; the real
SleepStager.respRateFromRR, plus a validated port to switch the gate off):

  beatAccurateFraction        0.0246 / 0.0235   (threshold 0.50)
  shipped path                NaN / NaN
  both protections disabled   13.33 bpm / 13.33 bpm

Three things this corrects in the original ryanbr#883 writeup, all of which strengthen it:

1. The number is 13.33, not the "~7-10" first reported. It sits inside
   respPlausibleRangeBpm (8-25) and inside the real adult sleeping range, so the
   range clamp is no protection whatsoever. Only 1 of 113 and 0 of 114 windows fall
   below 10 bpm.

2. The estimate carries zero information. SHUFFLING or REVERSING the night's R-R
   values returns the same 13.3333 to four decimals, on both nights. The tachogram's
   breath band is a flat 1/f shelf with no peak; 13.33 is the peak-picker's own floor
   on the 4 Hz grid. This is the ryanbr#194 bar failed outright, which is why the gate is on
   BANKED-ness and not on whether the output looks sane.

3. The mechanism in the old comment was wrong. The time AXIS is not corrupted:
   sum(R-R) over wall span is 1.030 / 1.008, so beat-time reconstructs the night to
   1-3%. What is unusable is the interval VALUES - the ~6.6 s record decomposed into
   ~6 intervals whose sum is right to ~1% while the individuals are not beat-to-beat
   measurements, the same decomposition documented on beatValuesAreTrustworthy.

Also documents, and pins with a test, that this gate and ryanbr#977's splice skip catch
OPPOSITE banking geometries and neither subsumes the other: ryanbr#977 catches banking that
TILES time (the real ring - it independently discards 113/113 and 114/114 windows on
these nights), this catches banking that COMPRESSES it (the batched fixture, where no
gap ever exceeds rsaGapToleranceS and ryanbr#977 is blind).

And one known limitation, pinned deliberately: the gate detects banking by its
symptom, and that symptom is repairable. Re-timing each record's beats by cumsum from
the record's own timestamp moves the fraction 0.0246/0.0235 -> 0.875/0.863, defeating
both this gate and ryanbr#977 - while the estimate stays 13.3333 and stays unchanged under
shuffling. A well-meant decoder change would silently switch respiration back on for a
stream carrying no breathing information. testRetiming...knownLimitation fails the day
that happens, and says what to do: re-base the gate on provenance, not timestamp shape.

Tests: Swift StrandAnalytics 1270/0 (RespRateRsaTests 7/7 incl. 3 new); Android
compileFullDebugKotlin clean, 3554 tests / 3 failed - all 3 (AiCoachContextTest, 2x
StandardHrSensorFormatTest locale) reproduce on clean upstream/main, pre-existing.
No hardware behaviour changes: this is analytics-only, no BLE path touched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XXdjctbkxqo359NuuJascK
@ryanbr
ryanbr merged commit bdc350b into ryanbr:main Aug 8, 2026
14 checks passed
@pipiche38
pipiche38 deleted the oura-resp-rsa-beat-accuracy-gate branch August 8, 2026 18:24
@ryanbr

ryanbr commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Cross-ref: this gate has an unintended interaction with WHOOP 4.0 — see #1331.

WHOOP 4.0's R-R is over-counted (#1008 stamps several R-R in one coarse second), which is timing-indistinguishable from an Oura banked stream, so the beat-accuracy gate here catches 4.0 too and blanks respiratory rate on every 4.0 night (regression since V10). The gate itself is correct — 4.0's over-counted beats genuinely can't support RSA (beat-time is rebuilt from the R-R values, so duplication reads the rate low). The real fix is the #1118 two-channel R-R de-dup, after which 4.0 passes this gate legitimately. Noting here so the interaction is on record.

ryanbr added a commit that referenced this pull request Aug 15, 2026
… fix (#1331/#1008/#1118) (#1352)

* diag(hrv): shadow-log a deduped R-R stream to validate the over-count fix (#1331/#1008/#1118)

The WHOOP 4.0 R-R over-count (#1008 — each beat emitted 2-3x, an exact dup plus
a two-optical-channel twin ~34ms off) corrupts HRV (#1008/#1118) AND, via
#1127's RSA beat-accuracy gate, blanks respiratory rate on 4.0 since V10 (#1331).
The fix is a de-dup, but it changes a physiological signal, so it must be
validated against ground truth before it becomes the read path.

Stage 1 (this PR) — instrument, do not change behaviour:
- HRVAnalyzer/HrvAnalyzer.collapseOverCount: pure, byte-identical de-dup that
  collapses same-second beats within ~40ms of one already kept (the exact dups +
  the ~34ms channel twins). Returns the deduped (tsSec, rrMs).
- On an over-count night the always-on hrv diag appends a `hrv dedup` line with
  the DEDUPED stream's rmssd/sdnn/meanNN/coverage/beatAccurate beside the raw.
  Shipped HRV/resp is UNCHANGED — this only logs a candidate.

So a strap log now shows, per 4.0 over-count night, whether de-dup drives
coverage→~1.0 and beatAccurate high (which would pass #1127's RSA gate → resp
returns, the #1331 fix) and rmssd/sdnn toward physiological — validated against
WHOOP's own numbers and @artemc's Polar H10 (#1118) before Stage 2 flips it live.

Tests: HrvCollapseOverCount{Test,Tests} pin a synthetic 3x over-count collapsing
180→60 beats, coverage ~3x→~1.0, clean streams untouched. compileFullDebugKotlin
+ the Kotlin test green; StrandAnalytics via swift-packages; IntelligenceEngine
shadow is app-target → app-build gate.

Refs #1331, #1008, #1118, #1127.

* diag(hrv): log an exact-dup reference beside the ~40ms de-dup candidate (#1331)

Re-review caught that the ~40ms same-second collapse can over-merge two REAL
neighbouring beats whose R-R values sit within 40ms (normal at sleeping HR),
so its coverage undershoots 1.0 even when de-dup is working — confounding the
validation signal. Add the EXACT-duplicate collapse (rrTolMs=0, same ts AND
value, provably no real-beat loss) as a safe reference line, so the shadow logs
the honest spectrum: covExact (safe floor, the unambiguous exact-dup magnitude)
alongside cov40/beatAcc40/rmssd40 (aggressive upper bound). The real de-dup sits
between them. Test pins the exact-dup floor (180->120, coverage ~2x, twins kept).

* fixup: reattach collapsedCoverage doc (collapseOverCount was inserted between doc + fn)

doc_comment_lint flagged HrvAnalyzer.kt:503 — collapseOverCount landed between
collapsedCoverage's #550 KDoc and its declaration, detaching the doc. Move
collapseOverCount above the #550 doc so it reattaches. No code change.
simoncad7 pushed a commit to simoncad7/noop that referenced this pull request Aug 17, 2026
…ked (ryanbr#1127)

An Oura night reports a respiratory rate that looks entirely normal while a WHOOP
strap worn on the SAME nights measures ~16.4. Respiration is derived by RSA from
the R-R stream, and the ring's banked overnight IBI cannot support RSA: it stores
R-R VALUES against coarse per-record timestamps, so the beat-to-beat variation RSA
reads is not in the data.

Fix: respRateFromRR (both platforms) refuses a stream that is not beat-accurate,
reusing HRVAnalyzer.beatAccurateFraction / beatValuesAreTrustworthy (ryanbr#1108) rather
than carrying a second copy of the same judgement and its two constants. That is the
collapse ryanbr#1108's PR body promised: one boundary, one definition, so a threshold
change moves respiration and SDNN together. WHOOP R-R and the synthetic RSA fixtures
measure ~100% and pass unchanged.

MEASURED 2026-08-07 on two real nights (31,460 and 30,754 in-bed beats; the real
SleepStager.respRateFromRR, plus a validated port to switch the gate off):

  beatAccurateFraction        0.0246 / 0.0235   (threshold 0.50)
  shipped path                NaN / NaN
  both protections disabled   13.33 bpm / 13.33 bpm

Three things this corrects in the original ryanbr#883 writeup, all of which strengthen it:

1. The number is 13.33, not the "~7-10" first reported. It sits inside
   respPlausibleRangeBpm (8-25) and inside the real adult sleeping range, so the
   range clamp is no protection whatsoever. Only 1 of 113 and 0 of 114 windows fall
   below 10 bpm.

2. The estimate carries zero information. SHUFFLING or REVERSING the night's R-R
   values returns the same 13.3333 to four decimals, on both nights. The tachogram's
   breath band is a flat 1/f shelf with no peak; 13.33 is the peak-picker's own floor
   on the 4 Hz grid. This is the ryanbr#194 bar failed outright, which is why the gate is on
   BANKED-ness and not on whether the output looks sane.

3. The mechanism in the old comment was wrong. The time AXIS is not corrupted:
   sum(R-R) over wall span is 1.030 / 1.008, so beat-time reconstructs the night to
   1-3%. What is unusable is the interval VALUES - the ~6.6 s record decomposed into
   ~6 intervals whose sum is right to ~1% while the individuals are not beat-to-beat
   measurements, the same decomposition documented on beatValuesAreTrustworthy.

Also documents, and pins with a test, that this gate and ryanbr#977's splice skip catch
OPPOSITE banking geometries and neither subsumes the other: ryanbr#977 catches banking that
TILES time (the real ring - it independently discards 113/113 and 114/114 windows on
these nights), this catches banking that COMPRESSES it (the batched fixture, where no
gap ever exceeds rsaGapToleranceS and ryanbr#977 is blind).

And one known limitation, pinned deliberately: the gate detects banking by its
symptom, and that symptom is repairable. Re-timing each record's beats by cumsum from
the record's own timestamp moves the fraction 0.0246/0.0235 -> 0.875/0.863, defeating
both this gate and ryanbr#977 - while the estimate stays 13.3333 and stays unchanged under
shuffling. A well-meant decoder change would silently switch respiration back on for a
stream carrying no breathing information. testRetiming...knownLimitation fails the day
that happens, and says what to do: re-base the gate on provenance, not timestamp shape.

Tests: Swift StrandAnalytics 1270/0 (RespRateRsaTests 7/7 incl. 3 new); Android
compileFullDebugKotlin clean, 3554 tests / 3 failed - all 3 (AiCoachContextTest, 2x
StandardHrSensorFormatTest locale) reproduce on clean upstream/main, pre-existing.
No hardware behaviour changes: this is analytics-only, no BLE path touched.


Claude-Session: https://claude.ai/code/session_01XXdjctbkxqo359NuuJascK

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
simoncad7 pushed a commit to simoncad7/noop that referenced this pull request Aug 17, 2026
… fix (ryanbr#1331/ryanbr#1008/ryanbr#1118) (ryanbr#1352)

* diag(hrv): shadow-log a deduped R-R stream to validate the over-count fix (ryanbr#1331/ryanbr#1008/ryanbr#1118)

The WHOOP 4.0 R-R over-count (ryanbr#1008 — each beat emitted 2-3x, an exact dup plus
a two-optical-channel twin ~34ms off) corrupts HRV (ryanbr#1008/ryanbr#1118) AND, via
ryanbr#1127's RSA beat-accuracy gate, blanks respiratory rate on 4.0 since V10 (ryanbr#1331).
The fix is a de-dup, but it changes a physiological signal, so it must be
validated against ground truth before it becomes the read path.

Stage 1 (this PR) — instrument, do not change behaviour:
- HRVAnalyzer/HrvAnalyzer.collapseOverCount: pure, byte-identical de-dup that
  collapses same-second beats within ~40ms of one already kept (the exact dups +
  the ~34ms channel twins). Returns the deduped (tsSec, rrMs).
- On an over-count night the always-on hrv diag appends a `hrv dedup` line with
  the DEDUPED stream's rmssd/sdnn/meanNN/coverage/beatAccurate beside the raw.
  Shipped HRV/resp is UNCHANGED — this only logs a candidate.

So a strap log now shows, per 4.0 over-count night, whether de-dup drives
coverage→~1.0 and beatAccurate high (which would pass ryanbr#1127's RSA gate → resp
returns, the ryanbr#1331 fix) and rmssd/sdnn toward physiological — validated against
WHOOP's own numbers and @artemc's Polar H10 (ryanbr#1118) before Stage 2 flips it live.

Tests: HrvCollapseOverCount{Test,Tests} pin a synthetic 3x over-count collapsing
180→60 beats, coverage ~3x→~1.0, clean streams untouched. compileFullDebugKotlin
+ the Kotlin test green; StrandAnalytics via swift-packages; IntelligenceEngine
shadow is app-target → app-build gate.

Refs ryanbr#1331, ryanbr#1008, ryanbr#1118, ryanbr#1127.

* diag(hrv): log an exact-dup reference beside the ~40ms de-dup candidate (ryanbr#1331)

Re-review caught that the ~40ms same-second collapse can over-merge two REAL
neighbouring beats whose R-R values sit within 40ms (normal at sleeping HR),
so its coverage undershoots 1.0 even when de-dup is working — confounding the
validation signal. Add the EXACT-duplicate collapse (rrTolMs=0, same ts AND
value, provably no real-beat loss) as a safe reference line, so the shadow logs
the honest spectrum: covExact (safe floor, the unambiguous exact-dup magnitude)
alongside cov40/beatAcc40/rmssd40 (aggressive upper bound). The real de-dup sits
between them. Test pins the exact-dup floor (180->120, coverage ~2x, twins kept).

* fixup: reattach collapsedCoverage doc (collapseOverCount was inserted between doc + fn)

doc_comment_lint flagged HrvAnalyzer.kt:503 — collapseOverCount landed between
collapsedCoverage's ryanbr#550 KDoc and its declaration, detaching the doc. Move
collapseOverCount above the ryanbr#550 doc so it reattaches. No code change.
ryanbr added a commit that referenced this pull request Aug 17, 2026
…w candidate (#1331/#1118)

The #1331 4.0 logs read `crossSecondOverCount` every night: the same-second shadow
collapses (#1352's exact + 40 ms candidates) can't reach the duplicate twins
because they straddle the second boundary, so `cov40` stays ~1.7-2.0 on the heavy
nights and RSA-respiration stays blanked by #1127's beat-accuracy gate.

Add a THIRD shadow candidate, `xsec` — the 40 ms collapse widened to a 1-second
window — logged beside the others so we can SIZE how much of a night is
cross-second (does coverage fall to ~1.0, does beat-accuracy clear the 0.5 gate?).
`collapseOverCount` gains a `windowSec` param (default 0 = same-second, byte-
identical for every existing caller).

Instrumentation ONLY — the shipped HRV/resp path is unchanged. The window is a
strict UPPER BOUND, deliberately NOT a shippable de-dup: a steady real HR has
near-identical intervals one second apart, so a cross-second collapse over-merges
real beats (the test pins this: a 10-beat steady stream drops to 5 at window=1).
The real fix is density/timeline-based and must be validated against ground truth
(@artemc's H10) before it becomes the read path — this just quantifies the target.

Both platforms (byte-identical `hrv dedup` line + twin tests).
ryanbr added a commit that referenced this pull request Aug 17, 2026
…w candidate (#1331/#1118) (#1399)

The #1331 4.0 logs read `crossSecondOverCount` every night: the same-second shadow
collapses (#1352's exact + 40 ms candidates) can't reach the duplicate twins
because they straddle the second boundary, so `cov40` stays ~1.7-2.0 on the heavy
nights and RSA-respiration stays blanked by #1127's beat-accuracy gate.

Add a THIRD shadow candidate, `xsec` — the 40 ms collapse widened to a 1-second
window — logged beside the others so we can SIZE how much of a night is
cross-second (does coverage fall to ~1.0, does beat-accuracy clear the 0.5 gate?).
`collapseOverCount` gains a `windowSec` param (default 0 = same-second, byte-
identical for every existing caller).

Instrumentation ONLY — the shipped HRV/resp path is unchanged. The window is a
strict UPPER BOUND, deliberately NOT a shippable de-dup: a steady real HR has
near-identical intervals one second apart, so a cross-second collapse over-merges
real beats (the test pins this: a 10-beat steady stream drops to 5 at window=1).
The real fix is density/timeline-based and must be validated against ground truth
(@artemc's H10) before it becomes the read path — this just quantifies the target.

Both platforms (byte-identical `hrv dedup` line + twin tests).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants