-
Notifications
You must be signed in to change notification settings - Fork 0
Dissenter / unanimity break (MedMCQA) #277
Copy link
Copy link
Closed
Labels
dataset:medmcqadifficulty: intermediateTouches one subsystem; some context neededTouches one subsystem; some context neededexperimentExperiment runner / study designExperiment runner / study designpriority: mediumValuable, not on the critical pathValuable, not on the critical path
Description
Activity
Metadata
Metadata
Assignees
Labels
dataset:medmcqadifficulty: intermediateTouches one subsystem; some context neededTouches one subsystem; some context neededexperimentExperiment runner / study designExperiment runner / study designpriority: mediumValuable, not on the critical pathValuable, not on the critical path
Replicate the Dissenter / unanimity break (MedMCQA) experiment on MedMCQA, mirroring the MedQA template so results are apples-to-apples across datasets.
experiments/medqa/unanimity_break.pybenchmaxxing/datasets/medmcqa.py(already implemented, Dataset adapter: MedMCQA (text/MCQ) #112)What it measures
Add one planted CORRECT peer alongside the wrong seed(s) and measure the drop in adoption.
What to do
Run the same experiment on MedMCQA through the existing pipeline and the Gemini API. Replication, not new features: do not change the experiment code, only point it at a MedMCQA manifest (build one with the
medmcqaadapter). Write results underexperiments/medmcqa/results/in the same JSON/JSONL convention asexperiments/medqa/results/(not CSV, so the analysis tooling reads them).Acceptance
experiments/medmcqa/in the MedQA template's format.Part of the MedMCQA standard-battery replication (mirrors the MedQA/NIH template).