2026-09-05_posttraining-reverse-kl - #13
Open
adamskrodzki wants to merge 8 commits into
Open
Conversation
- JudgeGPT: verbatim gpt.py trunk + pooling + 3 heads (meter/mode/type) - train_judge.py: char-level, windowed bodies, tune_id leak-free split, GPU/bf16 - benchmark_judge.py: domain-filtered expert benchmark, failures=0, score@coverage - benchmark_ood.py: expert x domain transfer matrix (home-bias, coverage) - domains.json: per-checkpoint domain config; judge_v2 canonical - findings: judge ranking agrees with PPL; headers don't steer OOD (home-bias 0.9), melody context half-steers; small models bend easiest; reels->jigs best OOD cells - renderer.py: MIDI->WAV CLI renderer with diagnostics - REPORT.md/README/docs updated with E-JUDGE results and conclusions
- train_rkl.py: KL student↔nauczyciel na rolloutach studenta (per-token, okna kontekstowe stride 96/32, α-mixing, rotacja domena×zadanie cont/scratch, eval per komórkę: kl/tlp/ent, zapis best+last) - train_gpt.py: --max-iters (argparse, pozycyjne zgodne z dotychczasowymi) - prepare_data.py: --mixed (cały korpus, M: z wiersza) - nauczyciele *_sh_ckpt.pt: wspólny słownik 54 znaki (VOCAB_FROM=universalist) - universalist_ckpt.pt: baseline 47k melodii (--max-iters 8000) - studenty RKL: cont-only oraz cont+scratch; wyniki w docs/Posttraining-ReverseKL-Eksperyment.md — student cont+scratch ściaga się do poziomu specjalistów na ich przekątnych bez surowych danych domen (waltz 0.20→0.54 scratch, Pareto-dominacja nad universalistą)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Posttraining: reverse KL z rotującym nauczycielem
Problem: perplexity nie mierzy jakości muzyki
Idea: klasyfikator jako sędzia (classifier-as-judge) w Reverse KLD RL
Implementacja: Posttraining: reverse KL z rotującym nauczycielem (eksperyment E-RKL)
Inspiracja MiniLLM: On-Policy Distillation of Large Language
Models
i odrobinę:
RLHF