PL-MT: morphology evaluator v2 (levels, robust scoring, 4×5 orchestration)

[kwiscion]

PL-MT — Polish morphology benchmark (evaluator)

The code half of PL-MT. Pairs with the data PR: slayerlabs/datasets#3.

Base = plgen-benchmark, not main (stacked PR). The evaluator reuses
bench/plgen/common (ask / MODELS), which isn't on main yet. Retarget to main
once plgen-benchmark merges. The diff here is plmt-only.

Primary contributor: @lizzy-606 — author of the benchmark and the difficulty-level
taxonomy (v0.1). This PR extends her evaluator.

What's here

• bench/plmt/evaluator_v2.py — the evaluator
• bench/plmt/PLMT.md — benchmark card (methodology + reference matrix; no answer keys)
• bench/decon_audit.py — register the PL-MT task set; handle whole-file .json sources

Changes over v0.1

• level (1–7) encodes the error mechanism → per-level diagnostics, not a flat 46/71
• Word-boundary matching (was substring): lepiej no longer passes najlepiej; duże
  no longer false-flags inside dużej
• acceptable_all AND-match for "wymień wszystkie przypadki" items — passes on the
  complete set of case names regardless of phrasing/connector (i vs oraz)
• Randomized order within each level (seeded); level order fixed — no positional gaming
• Early stopping OFF by default — levels are not monotonic in difficulty
• Reuses bench/plgen/common — same 4 models, OpenRouter/ollama backends, seed support
• Response-cache run loop — gold edits re-score with 0 LLM calls; only new IDs generate
• Cross-seed / cross-model aggregate (mean±std per level)

Data placement (eval_only)

Tasks are not in this repo. Master in datasets/data/eval/plmt/; evaluator reads a
gitignored local copy at slayer-data/plmt/; registered in decon_audit.EVAL_SOURCES.

Reference (4 models × 5 seeds, avg-levels macro)

gemma4 0.74 · bielik 0.68 · qwen35 0.58 · qwen36 0.53 — same ranking as PL-GEN. Bielik leads
the Polish-specific levels (L3 suppletivism, L5 collective numerals, L4 noun syncretism).

Verification

--selftest passes; decon extracts 683 strings from the task set (was 0 before the .json
loader fix); aggregate reproduces from cache with 0 LLM calls.

🤖 Generated with Claude Code

[kwiscion]

Superseded by slayerlabs#34 (same branch, targeting the org repo).