Skip to content

feat: separate experimental data from MCP, REST and UI defaults - #439

Merged
wpfleger96 merged 2 commits into
mainfrom
wpfleger96/mcp-experimental-gate
Nov 1, 2026
Merged

wpfleger96 merged 2 commits into
mainfrom
wpfleger96/mcp-experimental-gate

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

SNORE's default surfaces mix its own heuristics with device data. A default get_nightly_summary call returns 11 experimental fields, and the server instructions call computed metrics primary and raw device data an escape hatch. Day detail, the Dashboard, Stats and reports show the same FL and RERA numbers, plus SNORE's effectiveness and trend judgements, next to what the machine reported. docs/fl-rera-validation.md doesn't support that placement: the RERA proxy runs at about 1.6x chance precision and about 5% sensitivity against device RE events.

This PR moves that data to separately named MCP tools, tagged REST operations and separate UI pages, so an agent or a person only gets it by asking. Nothing is deleted, no detection algorithm changes, and analysis still runs on import. The line is OSCAR's: device data plus the kind of arithmetic OSCAR does stays default, and SNORE's heuristics and invented measures move. It controls what is shown, not what is computed, and it is opt-in by naming, not access control.

Default surface Moved to
get_nightly_summary experimental_get_nightly_metrics: rera_index, rdi, FL fields, rera_proxy_*, periodic_breathing_pct, and segmented Ti and I:E as segmented_ti_median_s and segmented_ie_ratio
get_events experimental_get_event_context: MV slope, stability index, flow-derived MV
compare_epochs experimental_compare_epochs: breath distributions, flow classes, rera_proxy_count
get_breath_table, find_windows Renamed experimental_get_breath_table, experimental_find_windows
Day detail and DayDetail: fl_class_ge4_pct, rera_index, rera_count and their reasons /days/:date/experimental and GET /days/{date}/experimental
Dashboard, Stats, TherapySummary, report templates: effectiveness, ahi_trend_direction Therapy Assessment card on the Analysis page and GET /stats/experimental
Event Explorer's match section Event Comparison card on the Analysis page

On MCP, a parameter would have left one tool with two contracts and a server flag would hide data with no way to ask, so calling an experimental_* tool is the opt-in. REST URLs stay put, and operations that return experimental data carry a FastAPI experimental tag. UI routes stay put too, and the sidebar group "Analysis" is renamed "Experimental".

Default paths also stop reading breath rows, so they answer the same whether or not analysis has run. That needed more than hiding fields: get_nightly_summary, compare_epochs, GET /days/{date} and the leak stat all depended on analysis before. get_nightly_summary gains device-scored obstructive_apneas, central_apneas, hypopneas and reras, because removing the proxy would otherwise leave it with no RERA figure.

Tests hold the boundary on each surface:

  • tests/unit/test_mcp_provenance.py: every tool is in DEFAULT_TOOLS or prefixed, and no default model carries an Experimental field at any depth.
  • tests/integration/test_mcp_device_waveform_aggregates.py: every default tool runs with SQL capture to prove it reads no breath rows.
  • tests/unit/test_provenance_tags.py: no untagged REST operation can reach an Experimental field. Tagging is required on numeric, boolean, enum and Literal fields, and plain strings aren't checked.
  • ui/src/tests/experimentalBoundary.test.ts: no default page may call an experimental wrapper, import an experimental view or component, or reach an experimental path through the raw API client. It finds the wrappers by calling every ui/src/api export against a fake adapter, then scans all of ui/src except api, tests and types. cancelAnalysisJob is the one exception, since ImportView needs it and it reads no metrics.

ui/src/tests/experimentalOperations.json is the single tagged-operation list, read by both the REST and UI tests, so a route can't be tagged to escape the check.

Event matching

GET /sessions/{id}/events/match reported 0 matched. It added the stored timestamp_start to each SNORE event's relative time and compared the result with absolute device times, but timestamp_start is 0.0 on stored analysis results, so nothing lined up. On every analyzed session I checked locally it matched nothing, which made every device event a false negative and every SNORE event a false positive. /waveforms/compare had the right time base but used any-match false negatives, counted every device event type, had no matched count, and hardcoded the 5 s tolerance.

This PR deletes /events/match and rewrites compare_events on match_events_by_start_time with EVENT_MATCH_TOLERANCE_SECONDS, pooling apneas and hypopneas on both sides so each event is in at most one pair. RERAs join the pool only when the device scored some, the same rule validate_against_machine_events uses, so unmatched SNORE RERAs aren't counted against ground truth that doesn't exist. The result gains matched_events and false_positives_rera, and Event Explorer becomes a device-only page. The Event Comparison card shows Matched and Sensitivity, names its detection mode, follows the mode toggle without flicker, keeps each mode's result until the next run, and loads after a run from the page. It hides only when the selected mode has no comparison (404), and any other failure shows an error with a retry.

Removed: /events/match, EventMatchResult, EventService.match_events, get_machine_apnea_hypopnea_times, MATCHABLE_MACHINE_EVENT_TYPES, split_by_tolerance_match, HasStartTime and getEventMatch.

Changes that show up in numbers

MCP:

  • leak_above_24_pct is now the percent of leak samples at or above 24 L/min, which is OSCAR's time over the default redline. It used to be a percent of segmented breaths, and it was tagged Experimental.
  • ti_median_s and ie_ratio_median in get_nightly_summary now come from the device STR values, so they're null on APAP nights (0 of 529 AirSense sessions have them). Device Ti can be coarse (1.0 or 2.0 s on my AirCurve), because that's how the device reports it. The STR header gives I:E a physical dimension of %, its exact definition is unconfirmed, and it isn't comparable to segmented_ie_ratio. The field descriptions say both.
  • Default compare_epochs covers every enabled session instead of only analyzed ones, drops metrics, and reports n_samples and a true calendar n_nights. The old night count counted sessions.
  • Epoch comparison no longer treats a session with no recorded settings as a prescription change. Before, any epoch crossing the point a device started reporting settings was nulled. An epoch with no sessions reports no_data_in_range on both epoch tools, and experimental_compare_epochs returns it at the top level for an owned device_id with no sessions.
  • A corrupt leak blob gives a null leak_above_24_pct with the new reason channel_corrupt. get_events returns a night with a corrupt pressure, leak or MV blob, with null context for the affected channel. Corrupt fl and snore blobs still report channel_absent.
  • get_events fills spo2_drop_pct and peak_flow_limitation from the event table when present. No parser fills either yet.
  • A Day row no longer picks up channel stats from another device on the same date.

Stored and API values:

  • STR leak values are now converted from L/s to L/min. That fixes leak_percentile_70 on every ResMed session, and leak_median, leak_95th and leak_max on sessions with no leak waveform.
  • Tidal volume now reads in mL on every surface. OSCAR stores mL and ResMed STR stores litres, and nothing accounted for that consistently: tv_mean_ml and the session page multiplied OSCAR values by 1000, snore stats and snore session show printed litres as mL (about 500 mL showed as 1 mL), and avg_tidal_volume averaged the two units together. All of them now go through tidal_volume_ml, so SessionDetail.statistics.tidal_volume_* and avg_tidal_volume are mL for every import source. The Statistics columns themselves keep their mixed units.
  • db scrub-demo converts OSCAR-sourced tidal volume to litres when it copies statistics, because it rewrites import_source to demo and would otherwise lose the unit.
  • effectiveness, ahi_trend_direction, MV slope and MV stability are Experimental, and effectiveness and ahi_trend_direction are enums in the API.
  • /stats/experimental takes no days_limit and always returns 200. An empty range gives {"effectiveness": "unknown", "ahi_trend_direction": null}.
  • /stats/summary, /stats/periods, /stats/trends and /stats/records return 422 for days_limit outside 0 to 36500. Very large values used to return 500 and negative ones an empty result. No client sends either.
  • device_id on /days and /days/{date} and the session id on /sessions/{id}/waveforms* return 422 above 2**63 - 1, which SQLite can't bind.

Event comparison:

  • Sensitivity goes from 0% to its real value. Across the 172 analyzed sessions in my local database that's 773 of 2,575 device apneas and hypopneas, about 30%, with 1,802 false negatives and 8,851 false positives. It's still SNORE's experimental detection scored against device events, not a clinical measurement.
  • machine_event_count on the card and in snore waveform compare now counts only apneas and hypopneas, plus RERAs when the device scored some, so it can drop on a session with other device event types. It doesn't on those 172 sessions, and none of them have device RERAs. The card and the CLI both say so.
  • False-negative and false-positive totals can shift under one-to-one matching. On those sessions they didn't move.
  • A missed device hypopnea reports type H. It used to be unknown.
  • snore analysis Missed and Extra lists are one-to-one and time-ordered, so they can be longer than before. snore analysis and snore waveform compare now apply the same RERA rule, so they agree.

UI and CLI:

  • The Therapy Assessment card covers all recorded data and says so. The old Stats badge followed the Range toggle.
  • snore stats drops the Effectiveness line and the trend direction in the chart title.
  • DayDetailView reloads when the date changes and drops stale responses.
  • False-positive and false-negative tables page at 25 rows with PaginationBar, driven by a shared usePagedRows, which Events by Mode and Event Explorer use too. The pager reads "1–25 of 30".
  • A browser tab left open across the deploy shows an undefined effectiveness badge until reload.

Breaking for clients

  • MCP renames and moved fields break saved prompts and any client holding the old names. That's deliberate, and it includes ie_ratio, which is now ie_ratio_median in get_nightly_summary (the companion stays ie_ratio_reason).
  • In docs://schemas, compare_epochs_response holds the default schema and experimental_compare_epochs_response holds the experimental one.
  • /waveforms/compare returns matched_events and false_positives_rera, and /events/match is gone.
  • SessionDetail.statistics.tidal_volume_* is mL instead of the stored value.
  • Database errors on default tools no longer mention breath data or tell the client to run analysis.
  • page is capped at 1,000,000 on every paged tool, including experimental_get_breath_table, and non-finite thresholds, min_duration, context_seconds and flattening_threshold are rejected.

Rollout

No migration, re-analysis or recompute-days is needed. Stored leak_percentile_70 values stay in L/s until a session is re-parsed with import --force. So do stored leak_median, leak_95th and leak_max on sessions with no leak waveform, along with the Day rows built from them, and a new no-waveform session can average with an old one on the same Day. Don't run import --force on a multi-device profile until #449 is fixed.

Left as is

  • The default nightly path deserializes a third waveform blob per session on the event loop, and an 88-night call takes about 0.2 s locally. The embedded HTTP server shares that loop, and default epochs have no length cap now that the analysis gate is gone. fix(mcp): bound memory and event-loop cost of epoch and nightly tools #451 tracks both, along with the same cost on the experimental epoch path.
  • DayExperimentalMetrics has no device_id, and DayDetailView has no device query, so on a two-device date the page shows the lowest device id, the same row GET /days/{date} picks by default.
  • The leak "Assessment" line in snore stats (well controlled or elevated) is also a SNORE judgement. It wasn't part of this split.
  • Apple Health stays on the therapy pages, and Records and Rx best/worst stay default.

Related: #332 (its plan to expose csr_pct isn't adopted, the STR value is unusable on these devices), #408 (min_usage_hours touches the tools split here), #405 and #407 (RERA proxy caveats), #431 (rdi inherits the recount gap), #450 (the 1970 analysis_results timestamps).

@wpfleger96
wpfleger96 force-pushed the wpfleger96/mcp-experimental-gate branch from aaf2a2d to 02825f1 Compare November 1, 2026 19:08
wpfleger96 added a commit that referenced this pull request Nov 1, 2026
`GET /sessions/{id}/events/match` reports 0 matched. It adds the stored
`timestamp_start` to each SNORE event's relative time and compares the
result with absolute device times, but `timestamp_start` is `0.0` on
stored analysis results, so nothing lines up. On all 205 analyzed
sessions I checked locally it matched nothing, which is the 0 matched
and 0% sensitivity on Event Explorer, and it also made every device
event a false negative and every SNORE event a false positive.

`/waveforms/compare` has the right time base but disagrees with it in
other ways: it uses any-match false negatives, counts every device event
type in `machine_event_count`, has no matched count, and hardcodes the 5
s tolerance.

This PR makes `compare_events` the one implementation and deletes
`/events/match`. It now pools apneas and hypopneas on both sides and
uses `match_events_by_start_time` with `EVENT_MATCH_TOLERANCE_SECONDS`,
the batch-validation matcher, so each event is in at most one pair. The
result gains `matched_events`, the name `EventValidationResult` already
uses. Matched and Sensitivity move to the Event Comparison card on the
Analysis page, and Event Explorer drops its match section, so it's now a
device-only page. The card names its detection mode, follows the mode
toggle, loads after a run from the page, and hides when the selected
mode has no comparison. `compare_events` now finds its row through
`latest_analysis_row`.

**Removed:** `/events/match`, `EventMatchResult`,
`EventService.match_events`, `get_machine_apnea_hypopnea_times`,
`MATCHABLE_MACHINE_EVENT_TYPES`, `split_by_tolerance_match`,
`HasStartTime` and `getEventMatch`. `convert_machine_events` already
keeps only the matchable types.

**Changes that show up in numbers:**
- Sensitivity goes from 0% to its real value. Across 200 analyzed local
sessions that's 347 of 1,132 device apneas and hypopneas, about 31%.
- `machine_event_count` on the card and in `snore waveform compare` now
counts only apneas and hypopneas, so it drops (1,149 to 1,132 on those
sessions). The card says so, because the summary card on the same page
uses the same label for all device event types.
- False-negative and false-positive totals can shift under one-to-one
matching. On those sessions the totals didn't move (785 and 5,289), and
new counts equal the old matching logic with the time base corrected on
all 205 sessions.
- A missed device hypopnea now reports type `H`. It used to be
`unknown`, because `HypopneaEvent` has no `event_type`.
- `snore waveform compare` prints the matched count, and its Machine
line says "(apneas and hypopneas only)".
- `snore analysis` Missed and Extra lists are now one-to-one and
time-ordered, so they can be longer than before. It still includes RERAs
when the device scored them and `snore waveform compare` doesn't, so the
two can differ on those sessions.
- False-positive and false-negative tables now page at 25 rows through a
shared `TablePager`, which Events by Mode uses too.

These local analysis rows are `STALE_VERSION`, so the exact percentages
will move after re-analysis. The sensitivity number is still SNORE's
experimental detection scored against device events, not a clinical
measurement.

`ui/src/tests/experimentalBoundary.test.ts` closes the gap the REST test
can't see: a default page calling an experimental route. It calls every
`ui/src/api` export against a fake adapter to find the wrappers that hit
experimental operations, then scans all of `ui/src` except `api`,
`tests` and `types`. The operation list lives in
`ui/src/tests/experimentalOperations.json`, and
`EXPERIMENTAL_OPERATIONS` reads the same file, so a new tagged route is
covered on both sides. `cancelAnalysisJob` is the one exception, since
`ImportView` needs it and it reads no metrics.

No migration, re-analysis, re-parse or `recompute-days` is needed.

Stack: #439 → #440 → this PR
wpfleger96 added a commit that referenced this pull request Nov 1, 2026
Day detail, the Dashboard, Stats and reports show SNORE's own FL and
RERA numbers and its effectiveness and trend judgements next to device
data. `GET /days/{date}` also runs the breath lookup on every load to
produce them. This PR moves that data to separate pages and endpoints,
so a person only sees it by going there. Nothing is deleted, and
analysis still runs on import.

| Default surface | Moved to |
|---|---|
| Day detail and `DayDetail`: `fl_class_ge4_pct`, `rera_index`,
`rera_count` and their reasons | `/days/:date/experimental` and `GET
/days/{date}/experimental` |
| Dashboard, Stats, `TherapySummary`, report templates: `effectiveness`,
`ahi_trend_direction` | Therapy Assessment card on the Analysis page and
`GET /stats/experimental` |

REST URLs stay put. Operations that return experimental data now carry a
FastAPI `experimental` tag: the analysis and validation routers,
`/events/match`, `/waveforms/compare` and the two new endpoints.
`tests/unit/test_provenance_tags.py` rejects any Experimental field
reachable from an untagged operation, and pins the tagged list in
`EXPERIMENTAL_OPERATIONS` so a route can't be tagged to escape the
check. Tagging is now required on enum and `Literal` response fields
too, so a judgement like `effectiveness` can't go untagged. Plain string
fields still aren't checked. The sidebar group "Analysis" is renamed
"Experimental".

**Changes that show up:**
- `effectiveness` and `ahi_trend_direction` are retagged from Derived to
Experimental, and their API types are now enums. They're SNORE's
judgements, not arithmetic on device data.
- The Therapy Assessment card covers all recorded data and says so. The
old Stats badge followed the Range toggle.
- `GET /days/{date}` no longer touches breath rows, so it answers the
same whether or not analysis has run.
- `snore stats` drops the Effectiveness line and the trend direction in
the chart title.
- `/stats/experimental` takes no `days_limit` and always returns 200. An
empty range gives `{"effectiveness": "unknown", "ahi_trend_direction":
null}`.
- `/stats/summary`, `/stats/periods`, `/stats/trends` and
`/stats/records` return 422 for `days_limit` outside 0 to 36500. Very
large values used to return 500 and negative ones an empty result. No
client sends either.
- `ApneaEvent.event_type` is now tagged Experimental.
- `DayDetailView` reloads when the date changes and drops stale
responses, like the new page.
- A browser tab left open across the deploy shows an undefined
effectiveness badge until reload.

**Left as is:**
- `DayExperimentalMetrics` has no `device_id`, and `DayDetailView` has
no device query, so on a two-device date the page shows the lowest
device id, the same row `GET /days/{date}` picks by default.
- The leak "Assessment" line in `snore stats` (`well controlled` or
`elevated`) is also a SNORE judgement. I left it for now since it wasn't
part of this split.
- Event Explorer still calls `/events/match`. The next PR in the stack
removes that.
- `ImportView` still calls `DELETE /analysis/jobs/{job_id}`, which is
tagged `experimental` with the rest of the analysis router. It only
cancels a job and reads no metrics.

Apple Health stays on the therapy pages, and Records and Rx best/worst
stay default.

No migration, re-analysis, re-parse or `recompute-days` is needed.

Stack: #439 → this PR → #441
@wpfleger96 wpfleger96 changed the title feat(mcp): gate experimental metrics behind separate tools feat: separate experimental data from MCP, REST and UI defaults Nov 1, 2026
@wpfleger96
wpfleger96 force-pushed the wpfleger96/mcp-experimental-gate branch from b202cc0 to a566195 Compare November 1, 2026 20:42
SNORE's default surfaces mixed its own heuristics with device data, so an
agent or a person could not tell what the machine reported. The validation doc
does not support that placement: the RERA proxy runs near 1.6x chance
precision and about 5% sensitivity against device RE events.

Experimental data now lives in experimental_* MCP tools, tagged REST
operations and separate UI pages, so calling or opening one is the opt-in.
Default paths no longer read breath rows, so they answer the same with or
without analysis. Tests enforce the boundary on all three surfaces.

Event matching merges into the Analysis page because /events/match compared
relative times with absolute ones and never matched anything.

Breaking for MCP clients: get_breath_table and find_windows are renamed, and
nightly, event and epoch fields moved. STR leak values are now stored in
L/min, so stored leak_percentile_70 stays in L/s until import --force.
Tidal volume reads in mL on every surface, whichever parser stored it.
@wpfleger96
wpfleger96 force-pushed the wpfleger96/mcp-experimental-gate branch from a566195 to 9e7f758 Compare November 1, 2026 22:52
wpfleger96 added a commit that referenced this pull request Nov 1, 2026
@wpfleger96

Copy link
Copy Markdown
Owner Author

Screenshots

All of these use a mocked API, so the numbers are illustrative and not real results.

Event Comparison card

The card on the Analysis page shows Matched and Sensitivity and names its detection mode. The tables page at 25 rows with PaginationBar, and SNORE RERAs join the false-positive table when the device scored RERAs.

01-event-comparison

Event Comparison on mobile

The pager uses the shared PaginationBar, so it gets the mobile tap-target sizing that TablePager didn't have.

02-event-comparison-mobile

Event Comparison failure

A failed comparison request now shows an error with a retry. Only a 404 hides the card.

03-event-comparison-error

Therapy Assessment

effectiveness and the AHI trend moved off the Dashboard and Stats into this card on the Analysis page, under the sidebar group renamed "Experimental".

04-therapy-assessment

Experimental day metrics

FL and RERA proxy values moved off Day detail to /days/:date/experimental.

05-day-experimental

Event Explorer

The match section is gone, so this is a device-only page.

06-event-explorer

Tidal volume

Session detail shows mL for every import source. The values come from the API in mL, so the page no longer multiplies by 1000.

07-session-tidal-volume

@wpfleger96
wpfleger96 merged commit 4aca31c into main Nov 1, 2026
9 checks passed
@wpfleger96
wpfleger96 deleted the wpfleger96/mcp-experimental-gate branch November 1, 2026 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant