PL-MT: morphology evaluator v2 (levels, robust scoring, 4×5 orchestration) - #34
Draft
kwiscion wants to merge 12 commits into
Draft
PL-MT: morphology evaluator v2 (levels, robust scoring, 4×5 orchestration)#34kwiscion wants to merge 12 commits into
kwiscion wants to merge 12 commits into
Conversation
Polish free-generation language-quality benchmark, Layer A: self-hosted LanguageTool (pinned erikvl87/languagetool:6.5) scoring errors/100tok in morphosyntax/spelling/style buckets, typographic noise disabled. Shared common.py (ollama/openrouter/gemini HTTP, MODELS, schema), PolNative decon (decon_plgen.py) + EVAL_SOURCES registration. Deps: language-tool-python, pytest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Blind LLM judge of free-form Polish. Ships a single DeepSeek-V4-Pro in 'guided' mode: phenomena-anchored, strict-proofreader prompt + truncation- ignore. The open 3-judge panel (Llama/Mistral/Command-A) under-detected Polish and rubber-stamped (198/200 pass) so it's parked as a commented diagnostic; guided+DeepSeek discriminates (validated vs human gold: 56% exact / 100% +-1-class). Krippendorff-alpha kept for multi-judge use (N/A for a single judge). Includes parked Stanza agreement-checker spike (_spike_stanza.py) used by anno_cli --audit-la. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bench_plgen.py: --stage gen|score|aggregate|all over 4 subject models, per-item live diag streaming, aggregates-only output + paste-ready matrix section (Layer A/B kept as separate sub-scores, never averaged). anno_cli.py: blind human-gold annotation with inline LT/Stanza markers + --audit-la mode (Layer-A FP/FN audit). validate_judges.py: per-judge agreement vs human gold. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PLGEN.md card (shipped LT+DeepSeek config, honest caveats, next steps) and first reference run: 50-prompt/domain subset, seed 42, judge DeepSeek-V4-Pro guided. Judge discriminates (panel_score spread 5.8-37.5: gemma4 37.5 > Bielik 29.5 > Qwen3.5 12.2 > Qwen3.6 5.8); Layer A a flat low-recall floor. Cost ~$0.79. Aggregates only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… run The generated section advertised '≥3 judges / 3 seeds'; the shipped reference run is a single DeepSeek-V4-Pro (guided) judge on one seed (s42) subset. Make the protocol/note honest + derive judge name from the report. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-ran the full 193 held-out prompts (single seed s42), judge DeepSeek-V4-Pro guided. Panel_score: gemma-4-31B-it 32.8 ≈ Bielik-11B-v3 31.4 > Qwen3.5-27B instr 18.9 > Qwen3.6-27B 11.3 (same ranking as the 50-subset). Cost ~$2.82. - Merge the section into public/results/matrix.json (official_for renamed to the spec name 'bledy-jezykowe-pl'; updated 2026-06-16). - Drop the IJA row + fix protocol string (single judge → IJA N/A; full-193/1-seed). - Refresh PLGEN.md reference table, cost, caveats. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The full-193 run accidentally used the stale 400 default → ~84% truncated (median 169 tok), the confound we'd fixed at 1024. Make 1024 the default so runs are complete by default; regenerating the full set at 1024. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A human verdict is tied to a specific, non-reproducible (temp>0) generation; storing only (id,model,seed) orphaned the gold when gen files were regenerated. Now anno_cli writes the exact scored 'ans' + a sha into each gold/audit record, and validate_judges judges that stored text (gen-file fallback only for legacy). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-ran the full 193 at 1024 (the earlier run wrongly used 400 → 84% truncated, which flattered the weaker models). Truncation now 34%. panel_score: gemma-4 32.4 > Bielik 29.6 > Qwen3.5 13.5 > Qwen3.6 6.7. Cost $3.64. Refresh plgen_v1.json, matrix section, card. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ation Polish morphology benchmark evaluator. Extends @lizzy-606's v0.1 evaluator: - level field (1-7) encoding the error mechanism — per-level diagnostics, not a flat score - word-boundary matching (was substring): 'lepiej' no longer passes 'najlepiej', 'duże' no longer false-flags in 'dużej' - acceptable_all AND-match for 'list all cases' items — phrasing/connector-agnostic - randomized order WITHIN each level (seeded); level order fixed (no positional gaming) - early stopping OFF by default (levels aren't monotonic in difficulty) - reuses bench/plgen/common (ask/MODELS): same 4 models + OpenRouter/ollama backends, seeds - response-cache run loop: gold edits re-score with 0 LLM calls; only new ids generate - cross-seed/cross-model aggregate (mean±std per level) Data is eval_only and lives in the private datasets repo (data/eval/plmt); the evaluator reads a gitignored local copy (slayer-data/plmt) registered in decon_audit.EVAL_SOURCES. decon_audit: handle whole-file .json task sets (not just JSONL). Benchmark + level taxonomy authored by @lizzy-606. Co-Authored-By: lizzy-606 <lizzy-606@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@kwiscion is attempting to deploy a commit to the kwikiel's projects Team on Vercel. A member of the Team first needs to authorize it. |
Scorer no longer substring-matches multi-answer items — fixes over- AND under-counting: - match='cases': 'list all cases' items require the EXACT case set (rejects an extra biernik/narzędnik = overgeneration); order/connector irrelevant. SYNCRETISM_001/004. - match='forms': full paradigms — all required forms present AND none forbidden (e.g. plural forms when singular asked). Accepts 'ja piszę / ty piszesz / on pisze', rejects a full-paradigm dump. VERB_CONJ_004. - aggregate() excludes is_reformulation_of variants from the level matrix (A/B items). - dump_answers.py: <question + all model/seed answers> file for manual analysis (0 LLM). Effect (0-LLM re-score of cache): VERB_CONJ_004 bielik 5/5->0/5 (overgeneration now fails), gemma/qwen35 4/5->5/5 (label formatting now passes); SYNCRETISM_001 bielik 1/5->0/5 (extra-case false-pass removed). Ranking unchanged; bielik L1 0.72->0.65. Incorporates @lizzy-606's analysis (Type-A instruction vs Type-B morphology) and her 3 reformulated variants (PASSIVE_002b, CONS_ALT_009b, IMPERATIVE_007b; originals kept). Co-Authored-By: lizzy-606 <lizzy-606@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reports per model: morphology (non-ID items), instruction+morphology (all), ID-only, and Δ(morf−ID). Reads instruction_dependent flags from the task set; reformulation variants excluded. Honest guard in output: the raw Δ is CONFOUNDED by difficulty composition — ID flags correlate with easy L1-L3 while the non-ID bucket holds all of L5/L6/L7, so Δ mostly reflects bucket difficulty, not instruction load. The clean item-matched instruction measure is the reformulation A/B delta (PASSIVE_002→_002b). Co-Authored-By: lizzy-606 <lizzy-606@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PL-MT — Polish morphology benchmark (evaluator)
The code half of PL-MT. Pairs with the data PR (private): slayerlabs/datasets#3.
Primary contributor: @lizzy-606 — author of the benchmark and the difficulty-level
taxonomy (v0.1). This PR extends her evaluator.
What's here
bench/plmt/evaluator_v2.py— the evaluatorbench/plmt/PLMT.md— benchmark card (methodology + reference matrix; no answer keys)bench/decon_audit.py— register the PL-MT task set; handle whole-file.jsonsourcesChanges over v0.1
level(1–7) encodes the error mechanism → per-level diagnostics, not a flat46/71lepiejno longer passesnajlepiej;dużeno longer false-flags inside
dużejacceptable_allAND-match for "wymień wszystkie przypadki" items — passes on thecomplete set of case names regardless of phrasing/connector (
ivsoraz)bench/plgen/common— same 4 models, OpenRouter/ollama backends, seed supportData placement (eval_only)
Tasks are not in this repo. Master in
datasets/data/eval/plmt/; evaluator reads agitignored local copy at
slayer-data/plmt/; registered indecon_audit.EVAL_SOURCES.Reference (4 models × 5 seeds, avg-levels macro)
gemma4 0.74 · bielik 0.68 · qwen35 0.58 · qwen36 0.53 — same ranking as PL-GEN. Bielik leads
the Polish-specific levels (L3 suppletivism, L5 collective numerals, L4 noun syncretism).
Verification
--selftestpasses; decon extracts 683 strings from the task set (was 0 before the.jsonloader fix); aggregate reproduces from cache with 0 LLM calls.
🤖 Generated with Claude Code