Skip to content

feat(bench): add LEK (Lekarski Egzamin Końcowy) MCQ loader - #33

Open
bartoszkobylinski wants to merge 1 commit into
slayerlabs:mainfrom
bartoszkobylinski:feat/mcq-loader-lek
Open

feat(bench): add LEK (Lekarski Egzamin Końcowy) MCQ loader#33
bartoszkobylinski wants to merge 1 commit into
slayerlabs:mainfrom
bartoszkobylinski:feat/mcq-loader-lek

Conversation

@bartoszkobylinski

Copy link
Copy Markdown

Co to dodaje

Nowy natywnie polski benchmark MCQ — LEK (Lekarski Egzamin Końcowy) — jako loader load_lek w bench/bench_mcq.py (zadanie #4 „Dodaj loader MCQ").

Źródło: amu-cai/medical-exams-LEK-PL-2008-2024 (pierwotnie CEM).

  • 4312 pytań, natywnie PL, publiczny + ungated, 5 opcji A–E
  • gold zbalansowany: A=773 / B=847 / C=918 / D=896 / E=878
  • mapuje się 1:1 jak load_pes (opcje wklejone w treść pytania + gold z litery) → zero nowej maszynerii w harnessie
  • zarejestrowany jako bench "lek" (lang pl), dopisany do listy w docstringu modułu

Testy

Nowy plik tests/test_bench_mcq.py (pytest jako dev-dependency w pyproject.toml):

  • 9 testów offlinedatasets.load_dataset jest monkeypatchowane, więc działają bez sieci / w CI (mapowanie liter→indeks, normalizacja wielkości/spacji, odrzucanie złych etykiet i pustych pytań, kontrakt itemu, próbkowanie, determinizm seeda, rejestracja w BENCHES)
  • 1 test live bramkowany RUN_HF_TESTS=1 — potwierdza 4312 wierszy i rozkład gold
.venv/bin/python -m pytest                          # offline (CI-safe)
RUN_HF_TESTS=1 .venv/bin/python -m pytest -k live   # + sieć vs HF

Katalog

Wpis lek w public/data/benchmarks.json (źródło prawdy dla /bench-explorer) oraz public/data/datasety.json i wiersz na stronie /datasety.

Transparentność (do oceny przez maintainerów)

  • Licencja: w karcie HF brak zadeklarowanej licencji (license=None), źródło CEM — ta sama postawa co już zaakceptowany pes. Odnotowane w polu uwagi_review.
  • Domena: medyczna, jak pes, ale to inny egzamin — LEK = ogólny egzamin końcowy lekarza (licencyjny), PES = egzamin specjalizacyjny. Nie jest to trywialny wariant istniejącego benchu.

Wybór i powyższe zastrzeżenia zgłoszone wcześniej na Discordzie (#4).

@vercel

vercel Bot commented Jun 16, 2026

Copy link
Copy Markdown

@bartoszkobylinski is attempting to deploy a commit to the kwikiel's projects Team on Vercel.

A member of the Team first needs to authorize it.

Native-PL medical final-exam benchmark from amu-cai/medical-exams-LEK-PL-2008-2024 (4312 questions, native Polish, public/ungated, balanced A–E gold). load_lek mirrors load_pes (options inlined in the prompt, gold from the answer letter) and is registered as bench "lek".

Adds offline unit tests (monkeypatched HF, CI-safe) plus a gated live test (RUN_HF_TESTS=1), and catalog entries in benchmarks.json/datasety.json and the datasets page.
@dzienisz
dzienisz requested a review from kwikiel June 18, 2026 11:48
@xfaang-ci

Copy link
Copy Markdown
Contributor

🤖 Automated code review (Xavier / Xfaang), requested by @zientara in #tasks. Read-only analysis — findings for you to weigh.

Verdict: APPROVE WITH NITS — solid, well-scoped. Mirrors load_pes, the gold-index mapping is correct, tests are good.

Gold / answer index mapping — VERIFIED CORRECT (this is the #20 footgun): bench/bench_mcq.py:53 L2I = {c:i for i,c in enumerate("ABCDE")} → 0-based. Cross-checked vs the live dataset: answer is a single letter A–E (row 0 = "E" → gold 4), options pre-embedded in question_w_options. The eval path parse_letter does ord(letter)-65, so predicted and gold are in the same 0-based space. No off-by-one (unlike INCLUDE in #20, which did int(a)-1).

Findings (all nits, non-blocking):

  • bench_mcq.py:62load_lek's guard if lab in L2I and q and q.strip(): is actually stricter than load_pes's; consider back-porting the whitespace check to load_pes for parity.
  • decontamination/dedup unaudited — LEK 2008–2024 is public CEM material with likely overlap with Polish web/medical corpora, and CEM items repeat year-to-year (possible exact dups across editions). load_pes doesn't dedup either, so not a blocker, but worth running decon_audit.py against the training mix before treating LEK as a clean signal, and noting it in uwagi_review.
  • determinism — sample_strat seeds random and is tested; loader pins no dataset revision= (consistent with sibling loaders, optional hardening).
  • tests — offline fixture returns a plain list rather than a real HF Dataset; the gated live test (RUN_HF_TESTS=1) with the exact gold distribution compensates well.
  • catalog — benchmarks.json marks status:"verified" + licencja:"publiczny" while uwagi_review notes the HF card declares no license. Matches the accepted pes posture, but the CEM-copyright/“verified” call is a maintainer decision (@kwikiel).

Registration in BENCHES and both catalog JSONs is consistent. Safe to approve once the license posture is confirmed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants