fix(bench): błędy inferencji nie liczone jako odpowiedzi modelu (czystość pomiaru) - #76
Open
xfaang-ci wants to merge 1 commit into
Open
fix(bench): błędy inferencji nie liczone jako odpowiedzi modelu (czystość pomiaru)#76xfaang-ci wants to merge 1 commit into
xfaang-ci wants to merge 1 commit into
Conversation
Sentinel '__ERR__'/'' z ask()/chat() po wyczerpaniu retry byl traktowany jak realna
odpowiedz -> skazenie accuracy/F1 (czystosc pomiaru).
- bench_mcq.py: __ERR__/'' wykrywane przed parse_letter (errors+=1; continue),
osobne pola errors/n_scored, accuracy liczona po n_scored. Bez tego parse_letter
('__ERR__',5)==4 ('E') liczylo blad sieci jak odpowiedz 'E' (PES, 5-opcji).
- bench_poquad.py: pred '__ERR__' wykluczony z ans_n/judged/F1, pole infer_errors;
guard judged_accuracy gdy n==0.
- bench_gsm8k.py: ollama 200 + {'error':...} (np. brak modelu) -> twardy RuntimeError,
zamiast maskowac jako ciche 0%.
Zweryfikowane: parse_letter('__ERR__',5)=4 (bug potwierdzony), bramka go lapie,
metryka mcq 54%/100 (skazona) -> 60%/90 (czysta), gsm8k brak modelu -> hard fail.
Closes slayerlabs#39
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@xfaang-ci is attempting to deploy a commit to the kwikiel's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #39
Problem
Po wyczerpaniu retry
ask()/chat()zwracają sentinel__ERR__{e}lub'', a downstream traktował to jak realną odpowiedź → skażenie accuracy/F1.bench_mcq:parse_letter('__ERR__...', 5) == 4(literaEzERR) → dla PES (5 opcji) błąd sieci liczony jak odpowiedźE, dla 4-opcyjnych-1. PES to metryka decydująca, multi-seed na obciążonym GPU.bench_poquad:__ERR__szło do sędziego i F1 na śmieciowym tekście, bez licznika.bench_gsm8k:{'error':...}z ollamy (brak modelu) łapane jak timeout → ciche 0%.Fix
__ERR__/''wykrywane przedparse_letter(errors+=1; continue), osobne polaerrors/n_scored, accuracy pon_scored.__ERR__wykluczony zans_n/judged/F1, poleinfer_errors; guard gdyn==0.ask()pojson.loadssprawdza'error'→ twardyRuntimeErrorzamiast cichego 0%.Weryfikacja
Pełnego benchu nie odpalałem (ollama+modele+GPU), ale logika sentinela, parse i metryk pokryta.
py_compile3/3 OK.Zgodnie z CONTRIBUTING: zasada czystości pomiaru (błąd ≠ odpowiedź) i fail-loud (brak modelu).
🤖 Generated with Claude Code