Skip to content

PL-GEN: Polish free-generation language-quality benchmark (extracted to standalone repos) - #25

Open
kwiscion wants to merge 13 commits into
slayerlabs:mainfrom
kwiscion:plgen-benchmark
Open

PL-GEN: Polish free-generation language-quality benchmark (extracted to standalone repos)#25
kwiscion wants to merge 13 commits into
slayerlabs:mainfrom
kwiscion:plgen-benchmark

Conversation

@kwiscion

@kwiscion kwiscion commented Jun 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

PL-GEN — Polish free-generation language quality (LT + DeepSeek judge)

Mierzy, czy model pisze naturalną, poprawną polszczyznę w dłuższym, swobodnym tekście —
jakość emergentna, nie znajomość pojedynczych form. Dwie warstwy raportowane OSOBNO:
Warstwa A = LanguageTool (błędy/100 tok, dolne ograniczenie), Warstwa B = sędzia LLM
(DeepSeek-V4-Pro, tryb guided), zwalidowany vs złoto ludzkie (n=30: 52% exact / 93% ±1-klasa).

⚠️ Zmiana architektury — kod wydzielony z monorepo

Żeby odsprzęgnąć ewaluacje od aplikacji Next.js i umożliwić granularny dostęp/współpracę,
harness PL-GEN wyprowadzono z tego repo do dedykowanych repozytoriów:

repo widoczność zawartość
kwiscion/plgen publiczne kod/harness (stdlib-only, pip install -e)
kwiscion/plgen-data prywatne held-out prompty + złoto (eval_only)
HF gated kwiscion/plgen (planowane) mirror tylko-do-publikacji dla biegów z HF_TOKEN

Ten PR (po amendzie) nie wprowadza już bench/plgen/ do monorepo. Zamiast tego slayer:

  • konsumuje harness jako zależność git (plgen @ git+https://github.com/kwiscion/plgen.git);
  • trzyma tylko wyniki publiczne (public/results/plgen_v1.json + sekcja w matrix.json);
  • zachowuje rejestrację eval_only w decon_audit.py:EVAL_SOURCES (bramka anty-wyciekowa po
    stronie treningu — ścieżka danych, nie kod).

Uruchomienie z env slayera:

PLGEN_DATA=<klon plgen-data> uv run python -m plgen.bench_plgen --stage all --seeds 42,43,44

Wyniki referencyjne (pełne 193, 1 seed s42, sędzia po rekalibracji progu mixed/fail)

model LT morpho ↓ LT spelling ↓ LT style ↓ panel_score 0–100 ↑ naturalność 1–5 ↑
gemma-4-31B-it 0.050 0.929 0.030 49.4 4.08
Bielik-11B-v3 0.061 0.716 0.023 47.2 3.88
Qwen3.5-27B instr 0.082 0.869 0.028 29.9 3.32
Qwen3.6-27B 0.111 1.476 0.030 19.3 2.96

Sygnał = rozrzut (19.3–49.4), nie poziom bezwzględny. Warstwa A płaska (dolne ograniczenie),
zgodnie z projektem. Karta metody: PLGEN.md.

Znane pułapki (uczciwie)

  • 1 seed (s42); 3 ziarna TODO (wariancja generacji niezmierzona).
  • Sędzia walidowany na n=30 złota; "mixed" to najtrudniejsza klasa.
  • Warstwa A = dolne ograniczenie o niskim recall (~29% morfoskładnia).

kwiscion and others added 4 commits June 16, 2026 13:30
@kwiscion @claude
PL-GEN: Layer A (LanguageTool) + generation/decon scaffolding
9148d46 
Polish free-generation language-quality benchmark, Layer A: self-hosted
LanguageTool (pinned erikvl87/languagetool:6.5) scoring errors/100tok in
morphosyntax/spelling/style buckets, typographic noise disabled. Shared
common.py (ollama/openrouter/gemini HTTP, MODELS, schema), PolNative decon
(decon_plgen.py) + EVAL_SOURCES registration. Deps: language-tool-python, pytest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion @claude
PL-GEN: Layer B judge — DeepSeek-V4-Pro (guided)
407d189 
Blind LLM judge of free-form Polish. Ships a single DeepSeek-V4-Pro in
'guided' mode: phenomena-anchored, strict-proofreader prompt + truncation-
ignore. The open 3-judge panel (Llama/Mistral/Command-A) under-detected
Polish and rubber-stamped (198/200 pass) so it's parked as a commented
diagnostic; guided+DeepSeek discriminates (validated vs human gold: 56%
exact / 100% +-1-class). Krippendorff-alpha kept for multi-judge use (N/A
for a single judge). Includes parked Stanza agreement-checker spike
(_spike_stanza.py) used by anno_cli --audit-la.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion @claude
PL-GEN: orchestrator, annotation CLI, judge validation
6fd29a5 
bench_plgen.py: --stage gen|score|aggregate|all over 4 subject models,
per-item live diag streaming, aggregates-only output + paste-ready matrix
section (Layer A/B kept as separate sub-scores, never averaged). anno_cli.py:
blind human-gold annotation with inline LT/Stanza markers + --audit-la mode
(Layer-A FP/FN audit). validate_judges.py: per-judge agreement vs human gold.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion @claude
PL-GEN: benchmark card + reference results
ea2b04e 
PLGEN.md card (shipped LT+DeepSeek config, honest caveats, next steps) and
first reference run: 50-prompt/domain subset, seed 42, judge DeepSeek-V4-Pro
guided. Judge discriminates (panel_score spread 5.8-37.5: gemma4 37.5 >
Bielik 29.5 > Qwen3.5 12.2 > Qwen3.6 5.8); Layer A a flat low-recall floor.
Cost ~$0.79. Aggregates only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercel Bot commented Jun 16, 2026

Copy link
Copy Markdown

@kwiscion is attempting to deploy a commit to the kwikiel's projects Team on Vercel.

A member of the Team first needs to authorize it.

@kwiscion
kwiscion marked this pull request as draft June 16, 2026 11:51
kwiscion and others added 5 commits June 16, 2026 13:59
@kwiscion @claude
PL-GEN: fix matrix-section protocol to reflect shipped 1-judge/1-seed…
2fae192 
… run

The generated section advertised '≥3 judges / 3 seeds'; the shipped reference
run is a single DeepSeek-V4-Pro (guided) judge on one seed (s42) subset. Make
the protocol/note honest + derive judge name from the report.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion @claude
PL-GEN: full-193 single-seed reference run + merge matrix section
9b3053b 
Re-ran the full 193 held-out prompts (single seed s42), judge DeepSeek-V4-Pro
guided. Panel_score: gemma-4-31B-it 32.8 ≈ Bielik-11B-v3 31.4 > Qwen3.5-27B
instr 18.9 > Qwen3.6-27B 11.3 (same ranking as the 50-subset). Cost ~$2.82.

- Merge the section into public/results/matrix.json (official_for renamed to the
  spec name 'bledy-jezykowe-pl'; updated 2026-06-16).
- Drop the IJA row + fix protocol string (single judge → IJA N/A; full-193/1-seed).
- Refresh PLGEN.md reference table, cost, caveats.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion @claude
PL-GEN: default num_predict 400→1024 (400 truncated ~84% of outputs)
7052032 
The full-193 run accidentally used the stale 400 default → ~84% truncated
(median 169 tok), the confound we'd fixed at 1024. Make 1024 the default so
runs are complete by default; regenerating the full set at 1024.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion @claude
PL-GEN: make gold self-contained (store ans+ans_sha)
18e68ed 
A human verdict is tied to a specific, non-reproducible (temp>0) generation;
storing only (id,model,seed) orphaned the gold when gen files were regenerated.
Now anno_cli writes the exact scored 'ans' + a sha into each gold/audit record,
and validate_judges judges that stored text (gen-file fallback only for legacy).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion @claude
PL-GEN: corrected full-193 results @num_predict=1024
da6aa4c 
Re-ran the full 193 at 1024 (the earlier run wrongly used 400 → 84% truncated,
which flattered the weaker models). Truncation now 34%. panel_score:
gemma-4 32.4 > Bielik 29.6 > Qwen3.5 13.5 > Qwen3.6 6.7. Cost $3.64.
Refresh plgen_v1.json, matrix section, card.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion kwiscion mentioned this pull request Jun 17, 2026
Draft
@kwiscion
kwiscion marked this pull request as ready for review June 21, 2026 19:36
@kwiscion @claude
PL-GEN: recalibrate judge mixed/fail threshold + refresh validation (…
1ee15a2 
…n=30)

A single minor tested-phenomenon violation is now 'mixed', not auto-'fail'.
Validated on the self-contained n=30 gold: DeepSeek-guided exact 43→52%,
±1 93%, and human 'mixed' now maps to 'mixed' 9/15 (was 5/15) — the over-fail
on the middle is fixed. Card updated with the n=30 recalibrated table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion
kwiscion requested a review from kwikiel June 22, 2026 15:54
kwiscion and others added 3 commits June 22, 2026 17:59
@kwiscion @claude
Merge upstream/main into plgen-benchmark
dcf69c7 
Resolve public/results/matrix.json: keep upstream's sections (incl. renamed
official_for) + re-append the PL-GEN bledy-jezykowe-pl section.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion @claude
PL-GEN: re-score full-193 with recalibrated judge
10b36c1 
Re-judged the frozen @1024 gens with the recalibrated guided threshold
(no regeneration — gens/gold untouched). panel_score rises (fewer auto-fails),
ranking unchanged: gemma-4 49.4 ≈ Bielik 47.2 > Qwen3.5 29.9 > Qwen3.6 19.3.
Refresh plgen_v1.json, matrix section, card. Cost $2.64.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion @claude
PL-GEN: consume harness as external dependency (extracted to kwiscion…
1369e02 
…/plgen)

The plgen benchmark code moved to its own public repo (kwiscion/plgen) with
held-out data in a private companion (kwiscion/plgen-data). slayer no longer
vendors bench/plgen/; it depends on the package via git and keeps only the
published results (public/results/) + the eval_only decon registration.

- remove bench/plgen/ (code+tests now live in kwiscion/plgen)
- pyproject: drop language-tool-python (was plgen-only), add plgen git dep
- run:  PLGEN_DATA=<plgen-data> uv run python -m plgen.bench_plgen --stage all

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kwiscion kwiscion changed the title PL-GEN: Polish free-generation language-quality benchmark (LT + DeepSeek judge) PL-GEN: Polish free-generation language-quality benchmark (extracted to standalone repos) Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kwikiel kwikiel Awaiting requested review from kwikiel

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kwiscion