Structured EHR: dataset scoping then standard-battery replication (tracking)

[sebasmos]

Tracking issue for the Structured EHR (tabular) lane. Unlike the other datasets this one needs a dataset + shortcut design first, because a plausible shortcut must exist for the cascade to be meaningful.

Scoping (do first, this is a design question)

• Post the top 3 open-source structured-EHR options in the group chat, with a one-line note on what plausible shortcut each admits
• Pick one with @sebasmos; it must support a logical spurious feature (a column a benchmark could reward)
• Build the loader/adapter (Structured EHR loader for resource-constraint scenarios #49)

Then replicate (real data + real API, results under experiments/ehr/results/)

• Solo shortcut susceptibility + noise floor
• Cascade: confident wrong seed contagion
• Referee detection: naive gate vs targeted vs deployable

Keep it apples-to-apples with the other lanes where the tabular format allows.

[sebasmos]

Still needs the dataset picked before the Core can start. Could you post the three candidate open datasets you're considering (with a plausible tabular shortcut) so we can pick one and get moving?

[MohShahin]

Still needs the dataset picked before the Core can start. Could you post the three candidate open datasets you're considering (with a plausible tabular shortcut) so we can pick one and get moving?

Here are three candidates, each genuinely independent of MIMIC (which is
already the imaging+text lane via MIMIC-CXR) and of each other, with
links:

1. eICU Collaborative Research Database — hospital-site shortcut.
https://physionet.org/content/eicu-crd/2.0/ (docs: https://eicu-crd.mit.edu/)
208 hospitals / 335 ICU units, PhysioNet-hosted BigQuery. Shortcut:
hospitalid as a spurious predictor of mortality/readmission/discharge
disposition -- the most direct tabular analogue to Zech et al. [15],
already cited in our own related work: a literal labeled hospital-site
column standing in for the inferred acquisition-artifact shortcut in
imaging. Requires a separate PhysioNet DUA (same CITI training likely
covers it, but it's a distinct signed agreement, not automatic from
MIMIC-IV access).

2. Diabetes 130-US Hospitals — race/payer_code shortcut on 30-day
readmission.
https://archive.ics.uci.edu/dataset/296/diabetes+130-us+hospitals+for+years+1999-2008
~101,766 encounters, 130 hospitals, Health Facts/Cerner extraction (not
PhysioNet at all). Fully open, zero credentialing (ucimlrepo package).
Shortcut: race or payer_code as a spurious predictor of readmission --
the standard testbed in the algorithmic-fairness literature for exactly
this shortcut (Strack et al. 2014, reused across many fairness-
benchmarking papers since).

3. SUPPORT2 (Study to Understand Prognoses and Preferences for
Outcomes and Risks of Treatments) — race/income/insurance shortcut on
mortality.
https://archive.ics.uci.edu/dataset/880/support2
(docs: https://hbiostat.org/data/repo/supportdesc)
9,105 seriously ill hospitalized adults across 5 US academic medical
centers, 1989-1994, Vanderbilt/Knaus et al. 1995. Fully open, zero
credentialing (ucimlrepo, id=880). Task: in-hospital or 180-day
mortality prediction from admission-baseline covariates. Shortcut: race
or income as a spurious predictor -- both are literal columns in the
dataset, and SUPPORT2 is itself a landmark, heavily-cited clinical
prognosis dataset (not a fairness-specific derivative like #2), so a
demographic shortcut here reads as "a real clinical model would pick this
up" rather than "we imported a bias-testbed dataset." Also multi-center
(5 institutions), so a secondary dzgroup/site-adjacent shortcut is
possible too if the primary one doesn't pan out.

My lean, unchanged: eICU as primary for the closest fit to our own
cited related work and reusable BigQuery tooling from #49/#340, with
SUPPORT2 as a strong zero-friction fallback (better thematic fit than
Diabetes-130, since it's a general mortality-prognosis study rather than
a single-disease fairness testbed) if the eICU DUA takes a while. Should
I start the eICU DUA now in parallel?

[sebasmos]

Just one first, SUPPORT2. Treat eICU/Diabetes-130 as nice-to-have if time allows after.

[Agastya191]

Reopening: the replication half is not done.

f4dd74f merged #354 with Resolves #297 in the merge commit, which closed this automatically. But #354 is implementation only, and all three boxes under Then replicate are still unchecked. @sebasmos asked for exactly this in review on #354 ("until it lands this is implementation only, so I would rather not have it read as closing part of #297"), so the close was an artifact of the merge message rather than a decision.

Answering the open question from that review: I am running it. @MohShahin is the assignee here and offered to take the run, but the imaging lane is ahead of it in his queue, so I am doing the run now against the manifest committed in #354 rather than leaving the lane parked. Starting now, results land under experiments/support2/results/.

Two notes on how the results will differ from the letter of this issue:

• Path. This issue asks for experiments/ehr/results/. The results go to experiments/support2/results/, because ehr is a separate credentialed MIMIC-IV entry in staging.SOURCES and folding an open UCI dataset into that path would conflate two different access regimes. Same lane, different directory.
• Plausibility. status.py currently lists plausibility="pending" for support2. The plausibility arm is the plausible-vs-implausible distractor contrast (experiments/medqa/plausible_distractor.py), and SUPPORT2's outcome is binary, so there is exactly one wrong option and no distractor gradient to vary. That field can never reach "done" as written. Changing it to not applicable, which is what the imaging lane already carries for the same structural reason. The three boxes on this issue are unaffected.

Will close this when the three runs are committed and reproduce from cache.

[MohShahin]

Reopening: the replication half is not done.

f4dd74f merged #354 with Resolves #297 in the merge commit, which closed this automatically. But #354 is implementation only, and all three boxes under Then replicate are still unchecked. @sebasmos asked for exactly this in review on #354 ("until it lands this is implementation only, so I would rather not have it read as closing part of #297"), so the close was an artifact of the merge message rather than a decision.

Answering the open question from that review: I am running it. @MohShahin is the assignee here and offered to take the run, but the imaging lane is ahead of it in his queue, so I am doing the run now against the manifest committed in #354 rather than leaving the lane parked. Starting now, results land under experiments/support2/results/.

Two notes on how the results will differ from the letter of this issue:

• Path. This issue asks for experiments/ehr/results/. The results go to experiments/support2/results/, because ehr is a separate credentialed MIMIC-IV entry in staging.SOURCES and folding an open UCI dataset into that path would conflate two different access regimes. Same lane, different directory.
• Plausibility. status.py currently lists plausibility="pending" for support2. The plausibility arm is the plausible-vs-implausible distractor contrast (experiments/medqa/plausible_distractor.py), and SUPPORT2's outcome is binary, so there is exactly one wrong option and no distractor gradient to vary. That field can never reach "done" as written. Changing it to not applicable, which is what the imaging lane already carries for the same structural reason. The three boxes on this issue are unaffected.

Will close this when the three runs are committed and reproduce from cache.

Thanks for picking this up @Agastya191 was queued behind MIMIC-imaging on my end, appreciate you not waiting. Let me know if you want a second pair of hands on anything.

[Agastya191]

Status on the replication half. All three arms are run on real data and real API, and committed, but nothing is merged yet, so I am leaving this open and the boxes unticked until review lands.

• Solo susceptibility: run, n=120
• Cascade contagion: run, n=120
• Referee detection: run, n=120

Two PRs, stacked:

• results(#297): SUPPORT2 tabular lane, real run on n=120 (solo null, cascade at ceiling) #366 the three runs. 1,266 calls, results plus call cache committed, whole lane replays from cache with no key.
• methodology(#297): the four #366 follow-ups, and a stimulus bug they exposed #369 the four follow-ups results(#297): SUPPORT2 tabular lane, real run on n=120 (solo null, cascade at ceiling) #366 identified, plus a stimulus bug they exposed. 1,270 further calls.

Results are consolidated in experiments/support2/results/lane_findings.json.

The lane does not reproduce the shape the imaging and text lanes show, and that is the finding rather than a caveat on one:

• Solo is a null, and stays a null after correcting the comparator. Against a proper temperature-0 whitespace control (0.140), unit_rescale (+0.093) and field_order (+0.074) clear it nominally but nothing survives BH across the six-cue family. The useful part is the power: three of the six cues were tested at power 0.03 to 0.09 and could not have detected an effect, so they were never evidence of absence. n=120 is the binding constraint here, which is a statement about the design and not about the model.
• The cascade is saturated at 1.000 and the ceiling belongs to the model, not the stimulus. A full 2x3 over peer count and board style moves adoption only between 0.976 and 1.000, non-monotone and nothing significant. Halving the peers and hedging their confidence to an explicit "we could easily be wrong" does not shift it.
• The referee ranking reproduces but is entailed by that ceiling. At 100% adoption targeted becomes mathematically identical to naive_gate and deployable to oracle_audit, so four referees collapse into two tiers. Consistent with the other lanes, but not independent evidence for them.

One correction to the record: the cascade and referee arms measure adoption of bare votes, not of a reasoned argument. run_board computed the case-anchored rationale and then dropped it before building the prompt, so the seed_style label was wrong. Detail on #366 and the fix in #369.

@MohShahin the eICU and Diabetes-130 options from your comparison are untouched, so if the "nice-to-have if time allows" is still live, note that a tabular lane at n=120 is underpowered for effects this size; a second dataset would want a larger cohort from the start.

Remaining before this can close: review and merge of both PRs. Separately open, not blocking: a larger solo cohort, a different holdout model to get off the cascade ceiling, and whether the dropped-rationale pattern exists in other lanes.