-
Notifications
You must be signed in to change notification settings - Fork 0
Authority gradient ladder (MedMCQA) #282
Copy link
Copy link
Closed
Labels
dataset:medmcqadifficulty: intermediateTouches one subsystem; some context neededTouches one subsystem; some context neededexperimentExperiment runner / study designExperiment runner / study designpriority: highDo this soon; unblocks the paper or other workDo this soon; unblocks the paper or other work
Description
Activity
Metadata
Metadata
Assignees
Labels
dataset:medmcqadifficulty: intermediateTouches one subsystem; some context neededTouches one subsystem; some context neededexperimentExperiment runner / study designExperiment runner / study designpriority: highDo this soon; unblocks the paper or other workDo this soon; unblocks the paper or other work
Replicate the Authority gradient ladder (MedMCQA) experiment on MedMCQA, mirroring the MedQA template so results are apples-to-apples across datasets.
experiments/medqa/authority_ladder.pybenchmaxxing/datasets/medmcqa.py(already implemented, Dataset adapter: MedMCQA (text/MCQ) #112)What it measures
Rank source rungs on one matched ladder: bare peer / senior peer / system flag / guideline.
What to do
Run the same experiment on MedMCQA through the existing pipeline and the Gemini API. Replication, not new features: do not change the experiment code, only point it at a MedMCQA manifest (build one with the
medmcqaadapter). Write results underexperiments/medmcqa/results/in the same JSON/JSONL convention asexperiments/medqa/results/(not CSV, so the analysis tooling reads them).Acceptance
experiments/medmcqa/in the MedQA template's format.Part of the MedMCQA standard-battery replication (mirrors the MedQA/NIH template).