[napetrov]

Problem

The skill-novelty headline numbers (added / known / degraded, placebo-gated) were graded by a single judge (gpt-5.6-luna). How much do they depend on that judge?

Change

• docs/judge-panel-study.md: re-grades 2,664 sampled answers (plus 268 fresh no-skill repeats for the noise floor) with 7 judges from 7 families using the same GRADE_PROMPT. It reports per-judge headlines, pairwise kappa, self-preference, aggregation rules (majority, mean, median, unanimous, any-correct, Dawid-Skene) and panel size, and had a judge-panel CLI/config proposal at its end, now removed (AGENTS.md keeps design plans out of the tree).
• CSV tables and two SVG figures in docs/assets/judge-panel/.
• Scripts in research/judge-panel/.
• Register records X031, F021, F022, F023.

Key results

• The original judge is the most lenient, so the paper's lenient added shares are conservative. Haiku's gated added share is 0.10 under the original judge and 0.32 under the 7-judge majority.
• On the lenient placebo-gated metric, judge choice moves an answer model's added share by up to 0.23 (haiku-4.5); the known share moves by up to 0.32. Under that metric nemotron is first and opus last under every judge; the middle ranks change.
• Lenient kappa is 0.70–0.88. Self-preference is ≤0.04.
• Majority and binary Dawid-Skene agree on 99.3% of rows and differ by at most 0.016 on the gated added share. Two judges cut the SD of the gated added share across judge subsets from 0.039 to 0.026; a third changes it by 0.001. Three judges from three families agree with the full panel on 96.6% of rows (one judge: 94.1%).

Local self-review: a fresh-context check of every number against the CSVs and of the analysis script. Round 2 applied four claim fixes. Round 3 (after the bot review) pins the study analysis to the seven-judge panel, replaces exec with an importlib load in gen.py, purges the grades of dropped repeats (with a test), and corrects the any-correct, Dawid-Skene and noise-floor claims. The OpenAI leniency delta is now computed on the shared rows (−0.025 and −0.027, were −0.042 and −0.043); generation_kappa.csv and the page follow.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Rerunning repeats overwrites answers while retaining stale verdicts, silently corrupting noise-floor calculations, and several documented statistical claims remain inaccurate or unsupported.

4 open issues

• Retrying repeats overwrites answers but reuses stale verdicts · New
• Middle-model rankings change across judges · New
• Noise comparison uses the wrong metric and unsupported models · New
• Known-share spread exceeds the bound; endpoint claim needs qualification · New

What changed in this PR

7 new findings.

This PR studies how judge choice affects Dokimi’s skill-novelty results and uses the evidence to propose multi-model judging support. It adds research tooling and documentation, not product CLI changes.

Changes:

• Compares seven new judges with the original judge, including agreement, self-preference, aggregation rules, and panel sizes.
• Adds sampling, grading, analysis, and plotting scripts with CSV tables and SVG figures.
• Publishes the study and records its experiment and findings.

File	Description
dokimi/​research/​records/​finding/​F021.md	Records aggregation-rule sensitivity.
dokimi/​research/​records/​finding/​F020.md	Records judge sensitivity and ranking conclusions.
dokimi/​research/​records/​experiment/​X030.md	Summarizes the experiment and results.
dokimi/​research/​judge-panel/​judge_panel.py	Samples answers, generates repeats, and collects grades.
dokimi/​research/​judge-panel/​fig.py	Generates judge-comparison figures.
dokimi/​research/​judge-panel/​analyze.py	Computes headline metrics and panel statistics.
dokimi/​mkdocs.yml	Adds the study to documentation navigation.
dokimi/​docs/​judge-panel-study.md	Presents results and proposed panel configuration.
dokimi/​docs/​assets/​judge-panel/​self_preference.csv	Stores judge offsets by answer model.
dokimi/​docs/​assets/​judge-panel/​per_judge_headline.csv	Stores per-judge headline and noise-floor metrics.
dokimi/​docs/​assets/​judge-panel/​panel_size.csv	Reports stability across panel sizes.
dokimi/​docs/​assets/​judge-panel/​pairwise_kappa.csv	Reports pairwise judge agreement.
dokimi/​docs/​assets/​judge-panel/​headline_range.csv	Summarizes headline ranges across judges.
dokimi/​docs/​assets/​judge-panel/​gated_added_by_judge.svg	Visualizes lenient gated-added shares.
dokimi/​docs/​assets/​judge-panel/​gated_added_by_judge_strict.svg	Visualizes strict gated-added shares.
dokimi/​docs/​assets/​judge-panel/​disagreement_by_group.csv	Breaks down disagreement by answer characteristics.
dokimi/​docs/​assets/​judge-panel/​agreement_with_ds.csv	Compares judges and rules with Dawid-Skene consensus.
dokimi/​docs/​assets/​judge-panel/​aggregation_rules.csv	Compares headline metrics across aggregation rules.

_{Review effort: Balanced}

[exai-bot]

OpenCodeReview: Commit c414bdb was not reviewed. This PR has reached its review limit (6 reviews).

CI at `c414bdb`: 13 still running, 8 passing, 2 skipped

Check	Status
Build docs (strict)	still running
CodeQL (python)	still running
External catalog smoke test (uxlfoundation/skills)	still running
Linux Python 3.11	still running
Linux Python 3.14	still running
Linux Wheels Python 3.11	still running
Linux Wheels Python 3.14	still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com...	still running
benchmark	still running
conda-forge dependency solve (py3.11)	still running

3 more not listed.

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (4de58a2): no findings</summary>

Open threads from the bot: 0

</details>

[exai-bot]

Must-fix. Impact: the reported generation leniency delta mixes a judge effect with a change in answer composition. Trigger: compare the original gpt-5.6-luna with either GPT-6 judge in this study. Evidence: lenient_rate() uses each judge's available rows independently; the original has 2,664 sample grades while the newer judges also grade 268 no-skill repeats, as documented in section 6. Thus these deltas are not comparisons on the same answers (unlike kappa(), which intersects coverage). Workaround: restrict both rates to keys graded by both judges. Medium severity: a bounded but material distortion of the generation-comparison statistic. Retain overall per-judge rates if useful, but calculate pair deltas on shared keys; likewise use shared complete cells for headline differences when coverage differs.

[napetrov]

Waiting on a decision before I change this: the 2,664 vs 2,932 row mismatch is real, so the OpenAI leniency delta mixes row sets (the doc now says so). Options: (1) regenerate the generation deltas on shared keys, which needs the grades (not on this box); (2) keep the caveat. I recommend (1).

[napetrov]

Decision: recompute the OpenAI generation deltas on the rows both judges graded (author's choice). The grades that recompute needs are not on this box, so it is pending: tracked in #585. The two OpenAI rows of the generation table are marked pending recompute in b06cd6d. This thread stays open until the recompute lands.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.72%. Comparing base (c1d7f5a) to head (c414bdb).
⚠️ Report is 18 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #482      +/-   ##
==========================================
- Coverage   95.73%   95.72%   -0.01%     
==========================================
  Files         408      415       +7     
  Lines       54644    55363     +719     
  Branches     9404     9516     +112     
==========================================
+ Hits        52311    52994     +683     
- Misses       1442     1466      +24     
- Partials      891      903      +12     

see 37 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at a441f94. The scripts and tables hold up, and the repeat fixes are right. The study's text still states several things its own tables do not support, and gen.py still fails on partial grades. Seven points inline; I'd fix them before merge.

What I ran, on a Linux review host at a441f94:

• make ci-local: 32 passed, 0 failed, 13 skipped; changed-line coverage passes
• Reproductions for the script findings, on small synthetic grade sets, with no model calls.
• A mutation run on do_repeat / purge_grades: 23 mutants, 8 survive the new tests; one of the 8 is equivalent. The ones that matter are in the inline comment on use_cache.
• Every number in the study and the records checked against the CSVs. The tables match; the problems are in the prose built on them.

Checked, no issues

• Pins: the repeat-keeps-saved-answers and purge-dropped-grades tests kill the rerun-all, drop-kept and no-purge mutants.
• Numbers: per-judge headline, kappa range (0.70–0.88 lenient), self-preference (≤0.04), 99.3% agreement and the 0.23 / 0.32 spreads all match the CSVs.
• Commits: all non-merge commits carry Signed-off-by.

[antonwolfy]

The merge still waits for the OpenAI leniency delta. The one commit since my last review marks the two OpenAI cells in the section 6 table as pending, but the same delta is still reported unmarked elsewhere. No new findings.

Still open from my earlier review:

• dokimi/research/judge-panel/gen.py:95: lenient_rate_delta still subtracts rates computed on each judge's own rows (2,664 for gpt-5.6-luna, 2,932 for the GPT-6 judges). The new "pending" mark is on two table cells only. These still state the delta without it: generation_kappa.csv (-0.0422, -0.0428, next to n 2664), the bullet "OpenAI and DeepSeek got stricter (−0.04, −0.10)", the "36% … 27%" strict rates, and the "both OpenAI pairs lower mean κ" claim (judge_mean_kappa.csv also compares 2,664 rows with 2,932). On shared rows the lenient delta can differ in sign: with 9.1% of the new judge's rows being the 268 repeats, it turns positive if the new judge's rate on those repeats is below about 0.29 (derived from judge_leniency.csv, not run, since the grades are not available). Strict still falls on shared rows by the same bound. Mark every place that quotes the OpenAI delta, or recompute on shared keys (the author's stated plan, tracked in #585).

Open bot points I confirm:

• I agree with exai-bot that the generation leniency delta still mixes row sets. New evidence: generation_kappa.csv still reports -0.0422 and -0.0428 for the OpenAI pairs next to n 2664, and the study's leniency bullet still says "OpenAI and DeepSeek got stricter (−0.04, −0.10)" with no caveat.

The author has chosen to recompute on shared rows, so the choice is made; what is left is the recompute and the unmarked places above.

What I ran, on a Linux review host at b06cd6d:

• make ci-local: 32 passed, 0 failed, 13 skipped; changed-line coverage passes. The earlier failures in the repo-reach tests did not recur.
• The one new commit, b06cd6d, read against every place the OpenAI delta appears in the docs, records and CSVs.

Checked, no issues

• Commits: all 30 non-merge commits carry Signed-off-by.
• Rest of the delta: the merges from main bring in unrelated files; none of the judge-panel files changed except the two table cells.
• Same-row tables: generation_added_share.csv and generation_panels.csv use the 2,664 existing cells only, so the added-share columns are unaffected.

[napetrov]

Re: review by antonwolfy

The OpenAI delta is quoted unmarked elsewhere, so the merge still waits
Fixed in 9e32d6e: the CSV pointer, the leniency bullet and the mean-kappa claim say pending #585.

[vchamarthi]

Review at 1e73815. 6 new findings: 5 must-fix, 1 nit. In the line comments. Adds a seven-judge re-grade of the skill-novelty headline numbers: a sample, a no-skill repeat for the noise floor, grades, the scripts that tabulate them, figures and records. The repeat and grade steps and their tests hold, but the sampler can exceed its per-model count, the analysis script crashes when a study judge has no grades, gen.py computes the OpenAI lenient delta over mismatched answer sets and writes kappa 1.0 for constant labels, and the study page links a missing file and carries a plan that AGENTS.md keeps out of the tree.

Verified

• Changed tests on pr: The 13 tests in dokimi/tests/test_judge_panel_gen.py and test_judge_panel_repeat.py pass on 1e73815.
• Changed tests on main: The same 13 tests fail on main, where the scripts they load do not exist.
• Merge: The PR merges onto main with no conflict.
• Whole test run: The whole test run on the PR merged onto main adds no failure beyond the three tests that also fail on main.
• Gen partial coverage: On 1e73815, gen.py names the judge and the answer model it lacks and exits without a traceback; the two tests that pin this pass.
• Gen constant labels: On 1e73815, gen.py writes kappa_old_new 1.0 for two judges that give one label on every row, and test_gen_runs_when_two_judges_agree_on_every_row asserts that value.
• Purge of dropped repeats: In do_repeat, replacing purge_grades({key(r) for r in dropped}) with purge_grades(set()) fails two tests, so the purge is pinned.
• Sample with an empty model: On 1e73815, do_sample with a model that has no complete cell writes only the other model's rows and does not stop.

Not verified

• do_sample rewrites sample.jsonl with new answers, while repeats.jsonl and grades.jsonl keep answers under keys with no digest of the answer text. If probe_results.jsonl changes between runs in the same JP_OUT, an old grade could be attached to new answer text: Store a digest of the answer in each repeat and grade row, drop rows whose digest no longer matches, or start each study in a fresh JP_OUT.

[napetrov]

Re: review by vchamarthi

Changes recommended: 5 must-fix, 1 nit
Fixed in dc59656: each point is in its thread's commit; the open one is asked.

[xaleryb]

• Ran: the head merged with current main, plus the two commits below. Full suite, ruff, mypy, bandit, growth and comment-block gates, and docs check --strict: all pass.
• Re-sample (residual of the "Not verified" point in an earlier review): an answer in probe_results changes, then sample and grade run again in the same JP_OUT -> 119 of 120 grade jobs; the old verdict stays attached to the new answer -> expected: the changed key is re-graded.
• rate_delta test gap: repeat rows that only the new judge grades -> a mutant that counts them on both sides passes every test -> expected: a red test.
• Pushed 2 commits (144c0c6..6402b29):
  • a5f00ee: test that rows only the new judge grades move neither rate.
  • 6402b29: do_sample purges grades for keys whose answer changed or that left the sample (reuses purge_grades), with a test.
• Left for the author: resolve the two fixed gen.py delta threads, and close Recompute the OpenAI generation deltas on the rows both judges graded #585 now that the recompute has landed.

[xaleryb]

Re-checked at 144c0c6 + a5f00ee + 6402b29 merged with main (Python 3.13): full suite, ruff, mypy, bandit, growth and comment-block gates, and docs check --strict pass. A changed answer in probe_results is now re-graded after sample, and a test pins that rows only the new judge grades move neither rate in lenient_rate_delta. Since then only main (#737) was merged in; the PR's own diff is unchanged, and CI on 1c19928 is green. The two fixed gen.py delta threads can be resolved.

[xaleryb]

@vchamarthi every point of your changes-requested review is fixed at the head, and its original input was re-run and passes (details in the comments above). Could you re-review so the changes-requested state clears?

[xaleryb]

Pushed bab16fc (1c19928..bab16fc): merges main. The dokimi/mkdocs.yml conflict keeps both nav entries: skill-novelty-evidence.md, then judge-panel-study.md, then grading-tiers.md. On the merged tree the full suite (19950 passed), ruff, mypy, bandit, docs check --strict, the CLI snapshot step and mkdocs build --strict pass. If #627 lands first, the judge-panel link in the old docs/README.md index moves to docs/documentation-index.md.

[antonwolfy]

Review at bab16fc. The fix for the OpenAI leniency delta is right, and it covers the point of my open thread: rate_delta takes both rates on the answers both judges grade, and the new tests pin it. One risk on a test gap in the pairwise kappa of analyze.py, and two nits.

What I ran, on a Linux review host at bab16fc:

• make ci-local: 29 passed, 3 failed, 14 skipped. Two of the failures are the gitleaks install and the commit scan that depends on it, the download of the tool returned HTTP 504, so neither step read the PR; GitHub's own check scans it. The third is the full test run: 19,967 tests passed, 19 skipped, 2 errors. The two errors are 60 s pytest timeouts in test_audit_test_quality.py, which this PR does not touch; the file passes alone (172 passed in 19 s), so they are timeouts of a loaded run, not the PR. Changed-line coverage: the PR changes no measured line (all changes are research scripts and docs).
• A mutation run on the changes since my last review: 27 mutants, 7 survive. rate_delta's shared-row logic, the gen.py kappa guard, the no-grades exit in analyze.py and the do_sample purge are all pinned by killed mutants. The survivors are in the inline comments.
• Reproductions on synthetic grade sets, with no model calls, for do_sample's purge and for gen.py / analyze.py with constant labels and disjoint judges.
• The numbers in section 6 of the study, the three records and the generation CSVs, re-derived from the CSVs: all match, including the shared-row deltas (-0.025, -0.027) and the mean kappa direction counts.

Checked, no issues

• Delta: rate_delta and kappa use the same key set, and kappa runs first and exits with a message when two judges share no answer, so rate_delta never divides by zero.
• Constant labels in gen.py: an empty kappa cell reaches gen_summary.json and generation_kappa.csv without a crash, and judge_mean_kappa.csv skips it.
• Docs: no place still quotes the old OpenAI delta or the "pending" mark; the mkdocs.yml nav and docs/README.md entries are present.
• Commits: all 40 non-merge commits carry Signed-off-by.

Bot points I checked that no longer apply at this head (1)

• exai-bot's point on the generation leniency delta does not apply: rate_delta in gen.py now computes both rates on the keys both judges graded, and a test pins that rows only the new judge grades move neither rate.

[antonwolfy]

No test reaches the pairwise kappa of run_study(), so the constant-label rule here is unpinned. gen.py's copy of this rule is pinned, this one is not.

Mutation check, each passes all 17 tests in test_judge_panel_gen.py and test_judge_panel_repeat.py:

• None if pe >= 1 changed to 1.0 if pe >= 1 (the old behaviour: two judges that give one label to every shared answer report kappa 1.0 in pairwise_kappa.csv)
• pe > 1 instead of pe >= 1
• _round returning 0 for None
• the mean at lines 247-251 counting a None kappa as 1.0 or as 0, which moves mean_kappa_lenient in the summary

The only analyze test, test_analyze_names_a_study_judge_with_no_grades, stops at the no-grades exit. test_gen_skips_optional_judges_that_share_no_answer only checks that a judge name appears in a CSV.

Suggestion: add a test that runs analyze.py on a fixture with all seven study judges and the original judge, where two judges give one label on every row. Assert that the pair's cell in pairwise_kappa.csv is empty and that mean_kappa_lenient for those two judges equals the mean over the other pairs only.

[vchamarthi]

Fixed in c414bdb: Added a test that runs analyze.py with two judges giving one label on every row: the pair's cells in pairwise_kappa.csv are empty and mean_kappa_lenient for glm-5 equals the mean over its defined pairs. tests/test_judge_panel_gen.py::test_analyze_leaves_a_constant_label_pair_without_kappa pins it and fails without the change.

[antonwolfy]

Nit: The new purge covers the sample's keys only, so the grades of no_skill_repeat rows survive a re-sample until repeat runs again. A repeat's key (p|no_skill_repeat|m) is not in the old sample.jsonl, so purge_grades never sees it.

Case: sample, repeat, grade; then sample again with a smaller sample (cell p9 leaves), then grade and gen.py without repeat. repeats.jsonl still has the p9 repeat and do_grade grades it. gen.py then stops with KeyError: 'no_skill' at analyze.py:105, because the cell has a repeat arm and no baseline. The documented order (sample, repeat, grade) avoids it, since do_repeat drops those rows at line 144.

Suggestion: have do_sample also drop the repeat rows and grades of cells that left the sample, or have do_grade skip repeat rows whose cell is not in sample.jsonl.

[vchamarthi]

Fixed in c414bdb: do_sample also drops repeats of cells that left the sample and purges their grades. tests/test_judge_panel_repeat.py::test_resample_drops_repeats_of_cells_that_left_and_their_grades pins it and fails without the change.

[antonwolfy]

Nit: The rounding of rate_delta is not pinned. Changing round(rate[new] - rate[old], 4) to round(..., 1) passes both new delta tests. They compare the output with round(..., 4) of the same expression, and the fixture's deltas are round numbers, so one decimal gives the same value. With the study data this would print -0.0 for the OpenAI pairs (-0.0248, -0.027).

Suggestion: give the fixture a shared-row set whose delta needs four decimals (for example 7 shared rows with one verdict flipped, so the delta is 0.1429) and assert that literal value.

[vchamarthi]

Fixed in c414bdb: Added a case with one of 60 shared answers differing, asserting the literal -0.0167. tests/test_judge_panel_gen.py::test_gen_lenient_delta_keeps_four_decimals pins it and fails without the change.

[vchamarthi]

Pushed 1 commit to research/judge-panel (bab16fc..c414bdb).

• mean_score>=0.4 rounds the mean before comparing.
• analyze.py exits naming the judge and answer model when a judge has no complete no_skill/full/placebo triplet.
• agree_lenient is empty for a pair with no shared answer.
• do_sample drops repeats of cells that left the sample, with their grades.

Threads: 3 fixed, 0 answered as not a defect.
Run: the full suite with main merged in, 20495 passed.

From the comments:

• no test reaches the pairwise kappa constant-label rule in analyze.py: fixed, covered by the reply on that thread.
• do_sample leaves repeat grades of cells that left: fixed, covered by the reply on that thread.
• rate_delta rounding not pinned: fixed, covered by the reply on that thread.