Repository navigation
Conversation
Re-grade 2,932 study answers with eight judges from seven families and report per-judge added/known/gated shares, noise floor, pairwise kappa, self-preference, six aggregation rules and panel size. Review how Dokimi configures judge panels today and propose --judges, --aggregate and a judge family policy for arms run. Register entries X030, F020, F021. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
--judge PROVIDER/MODEL, repeatable, is the panel; --judge-aggregate
{median,majority,unanimous} turns the verdicts into one. Records carry a
judge_panel block (per-judge 2x2, added-share range, judge_sensitive) and
every run keeps each judge's verdict. Warns on same-vendor judges; the
estimate counts the extra judges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
…ing covers the panel Verdicts are keyed by model name, so the same model on two providers collapsed into one vote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): ~ `dokimi skill novelty analyze` parameters changed:
+ added option `--judge`
+ added option `--judge-aggregate`
~ `dokimi skill novelty probe` parameters changed:
+ added option `--judge`
+ added option `--judge-aggregate`Updated on every push; the |
Seven more judges (kimi-k2.5, haiku-4.5, deepseek-r1, gpt-6-sol, opus-5.5, nemotron-super-3, minimax-m2.5) re-grade the same 2,932 rows. Within a family, generations agree at kappa 0.61-0.72, about like two vendors; one added share moves up to 0.15; a three-family panel keeps it within 0.07. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
…e-panel-cli Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/docs/reference/cli-interface.json # dokimi/dokimi/commands/novelty.py # dokimi/dokimi/novelty/analyze.py # dokimi/research/records/experiment/X030.md # dokimi/research/records/finding/F020.md
Move the panel helpers the command used (panel clients, judge refs, role models, the judge2-style stamp of the panel block) into novelty_run.py, and move panel_record into measure.py, so panel.py no longer imports measure. measure keeps the one lazy edge it needs out of panel at module level. Lowers the novelty.py growth baseline to the actual 881 lines. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Make the three shebang scripts executable (EXE001), mark the one exec of this directory's own analyze.py for bandit (B102), shorten three comment blocks to the two-line limit, and allow the sibling llm and run_probes imports in deptry (DEP001), as the audit_mocks precedent does. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…m the index Indent the nested lists so mkdocs reads them the same as GitHub, and mark the proposed arms run CLI as proposed text, not a command the doc checker must resolve. Link judge-panel-study.md from docs/README.md. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment blocks of three lines or more fail the comment check (AGENTS.md). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The list reflow put a paragraph break inside the sentence, so the site rendered two fragments. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… name The DEP001 ignore is by module name for the whole scan, so a bare llm would hide a real missing dependency elsewhere. The study scripts are not shipped, so the path exclude is the narrower fix. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… the license check The CSV and SVG files under docs/assets/judge-panel are written by research/judge-panel. A header would break the CSV readers, and regeneration would drop it, so they are ignored like the other data files. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #489 +/- ##
========================================
Coverage 95.71% 95.72%
========================================
Files 415 417 +2
Lines 55346 55577 +231
Branches 9514 9560 +46
========================================
+ Hits 52977 53199 +222
- Misses 1466 1473 +7
- Partials 903 905 +2
🚀 New features to boost your workflow:
|
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 6 selected item(s). CI at Review details
|
| "lenient_rate_delta": round(out["per_judge"][new]["lenient_rate"] | ||
| - out["per_judge"][old]["lenient_rate"], 4), |
There was a problem hiding this comment.
Must-fix — Impact: the reported generation leniency delta mixes a judge change with a change in answer composition. Trigger: run gen.py on the documented study containing fresh no-skill repeats. Evidence: judge_leniency.csv reports 2,664 grades for gpt-5.6-luna versus 2,932 for gpt-6-luna/sol; generation_kappa.csv nevertheless labels their unmatched rate differences (-0.0422/-0.0428) with n=2,664. The extra 268 rows are exclusively no-skill repeats from three answer models, not a comparable sample of all six arms. Compute both rates on the same intersection of keys used for each pair's kappa. Workaround: exclude repeats from both judges before calculating generation rate deltas. Severity justification: medium because this produces a confounded research statistic for the actual supported dataset, but the defect is bounded to the generation comparison.
There was a problem hiding this comment.
Partly fixed in 91ad8bd: gen.py computes the lenient-rate delta of each pair on the answers both judges graded. Not changed: generation_kappa.csv and the section 6 numbers are not regenerated, since the study data is not in the repository.
There was a problem hiding this comment.
Same as the population thread: a is fixed in 9329068; b (regenerating the tables) waits for the author.
Waiting on a decision before I change this: regenerate generation_kappa.csv and section 6 (A), or drop the rate-delta column until a re-run (B)? My recommendation: B for this PR, A as a follow-up, since the study data is not in the repository. The thread stays open until then.
There was a problem hiding this comment.
🟡 Changes recommended
Critical defects allow duplicate-model voting, stale-grade reuse, and mismatched analysis populations, making panel results unreliable and causing analysis crashes.
7 open issues
What changed in this PR
10 findings identified.
This PR adds multi-model judging to Dokimi’s skill-novelty analysis, building on the judge-panel study in #482 to expose how judge choice affects measured results.
Changes:
- Adds repeatable
--judgeoptions and median, majority, or unanimous aggregation. - Records individual verdicts, per-judge outcomes, sensitivity warnings, and additional judge costs.
- Includes the supporting study, research scripts, data tables, figures, documentation, and tests.
| File | Description |
|---|---|
dokimi/tests/test_novelty_tiers.py |
Updates role-model helper imports. |
dokimi/tests/test_novelty_panel.py |
Tests aggregation, recording, validation, and failures. |
dokimi/scripts/growth_baseline.json |
Lowers the novelty command’s size baseline. |
dokimi/research/records/finding/F023.md |
Records judge-dependent headline sensitivity. |
dokimi/research/records/finding/F022.md |
Records judge-generation effects. |
dokimi/research/records/finding/F021.md |
Records aggregation-rule effects. |
dokimi/research/records/experiment/X031.md |
Summarizes the judge-panel experiment. |
dokimi/research/judge-panel/judge_panel.py |
Samples answers, generates repeats, and grades them. |
dokimi/research/judge-panel/gen.py |
Compares judge generations. |
dokimi/research/judge-panel/fig.py |
Generates judge-comparison figures. |
dokimi/research/judge-panel/analyze.py |
Computes study metrics and tables. |
dokimi/pyproject.toml |
Excludes external-checkout study scripts from dependency checks. |
dokimi/mkdocs.yml |
Adds study navigation. |
dokimi/dokimi/novelty/panel.py |
Implements aggregation and panel warnings. |
dokimi/dokimi/novelty/measure.py |
Grades with multiple judges and summarizes outcomes. |
dokimi/dokimi/novelty/analyze.py |
Integrates panels into the analysis pipeline. |
dokimi/dokimi/commands/novelty.py |
Wires panel options into analyze and probe. |
dokimi/dokimi/commands/novelty_run.py |
Adds panel parsing, clients, pricing, and metadata. |
dokimi/dokimi/cli_flags.py |
Registers the aggregation flag. |
dokimi/docs/skill-novelty.md |
Documents panel usage and output. |
dokimi/docs/reference/cli-interface.json |
Updates the CLI interface snapshot. |
dokimi/docs/README.md |
Links the study. |
dokimi/docs/judge-panel-study.md |
Presents study methods and conclusions. |
dokimi/docs/assets/judge-panel/self_preference.csv |
Stores judge self-preference measurements. |
dokimi/docs/assets/judge-panel/per_judge_headline.csv |
Stores per-judge headline outcomes. |
dokimi/docs/assets/judge-panel/panel_size.csv |
Compares panel sizes. |
dokimi/docs/assets/judge-panel/pairwise_kappa.csv |
Stores pairwise judge agreement. |
dokimi/docs/assets/judge-panel/headline_range.csv |
Summarizes headline ranges across judges. |
dokimi/docs/assets/judge-panel/generations/judge_mean_kappa.csv |
Stores mean agreement by judge. |
dokimi/docs/assets/judge-panel/generations/judge_leniency.csv |
Stores judge acceptance rates. |
dokimi/docs/assets/judge-panel/generations/generation_panels.csv |
Compares old and new panels. |
dokimi/docs/assets/judge-panel/generations/generation_kappa.csv |
Stores generation-pair agreement and rate shifts. |
dokimi/docs/assets/judge-panel/generations/generation_added_share.csv |
Stores generation-dependent added-share changes. |
dokimi/docs/assets/judge-panel/gated_added_by_judge.svg |
Visualizes lenient gated-added shares. |
dokimi/docs/assets/judge-panel/gated_added_by_judge_strict.svg |
Visualizes strict gated-added shares. |
dokimi/docs/assets/judge-panel/disagreement_by_group.csv |
Groups disagreement by arm and answer characteristics. |
dokimi/docs/assets/judge-panel/agreement_with_ds.csv |
Compares judges and rules with inferred consensus. |
dokimi/docs/assets/judge-panel/aggregation_rules.csv |
Stores outcomes for each aggregation rule. |
dokimi/changelog.d/20261108_120000_napetrov_judge-panel-cli.md |
Announces novelty judge-panel support. |
.github/.licenserc.yaml |
Exempts generated study assets from headers. |
Review effort: Balanced
| "lenient_rate_delta": round(out["per_judge"][new]["lenient_rate"] | ||
| - out["per_judge"][old]["lenient_rate"], 4), |
There was a problem hiding this comment.
Partly fixed in 91ad8bd: gen.py computes the rate delta on the answers both judges graded. Not changed: generation_kappa.csv and the section 6 numbers are not regenerated, since the study data is not in the repository.
There was a problem hiding this comment.
a: gen.py computes the rate delta on the answers both judges graded. b: generation_kappa.csv and section 6 are not regenerated here; the study data is not in the repository.
Waiting on a decision before I change this: regenerate generation_kappa.csv and section 6 (A), or drop the rate-delta column until a re-run (B)? My recommendation: B for this PR, A as a follow-up, since the study data is not in the repository. The thread stays open until then.
Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com> # Conflicts: # dokimi/docs/README.md # dokimi/docs/reference/cli-interface.json # dokimi/dokimi/commands/novelty.py # dokimi/dokimi/novelty/analyze.py # dokimi/dokimi/novelty/measure.py # dokimi/mkdocs.yml # dokimi/scripts/growth_baseline.json # dokimi/tests/test_novelty_tiers.py
…nts, saved verdicts - `--judge` refuses one model under two spellings (`same_model`); `check_judges` refuses it for Python callers before any call. - `--judge2` is checked against every panel judge. - `--estimate-only` refuses an unusable `--judge` and counts the extra judges' calls. - `judge_panel` range is taken on the pairs every judge graded; moved to `panel_record.py`. - `analyze` writes `<record>.judge_verdicts.json`; the report names the panel. - `shared_vendors` knows grok/xai. - a spend stop in the middle of a panel, or on a truncated judge reply, leaves that run ungraded instead of taking one judge's vote. - `SpendCapReached` declares `panel_votes` so the type check passes. - study scripts: `analyze.py` pinned to its seven judges, `gen.py` rate delta on shared answers and undefined kappa, `judge_panel.py` fingerprints graded answers. - docs/records: ranking claim limited to the extremes, noise-floor comparison qualified. Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
|
|
Pushed 2 commits to
Threads: 19 fixed, 0 answered as not a defect. The PR description needs these corrections:
Questions for @napetrov1. Can you re-run
|
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
The merge waits for points @xaleryb raised that are still open at this head, and for the author's answer on the regenerated study tables. I found three risks of my own: the unanimous wording, a stale verdicts file and an unpinned estimate. The research-script points are nits.
What I ran, on a Linux review host at 9dabb8b:
make ci-local: 29 passed, 3 failed (host: gitleaks download, two pytest timeouts that pass alone), 14 skipped; 19983 tests; changed-line coverage passes- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run on
tests/test_novelty_panel.py: 27 mutants, 13 of which pass the full suite. The survivors are the USD estimate, theadd_panel_callswiring, theanalyzeandprobepass-through ofpanel=andaggregate=, the default rule, and theprobepre-flight self-judge loop.
I agree with @xaleryb's points on the veto in the per-judge 2×2, the vendor warning for later judges, a failed judge that is never warned about, the --judge-model hint, and the missing probe / analyze tests. New evidence: with a dead third judge, two judges that disagree completely give range None and judge_sensitive False.
@vchamarthi's question on re-running gen.py and committing the regenerated generation_kappa.csv waits for the author.
Checked, no issues
- Flags and defaults:
--judge,--judge-aggregate(defaultmedian), the refusal of--judgewith--judge-model, duplicates by model identity, and the zero-paid-call self-judge refusal over every panel member inanalyzeall behave as the docs say. - Aggregation:
aggregate()matches its docstring for even panels, ties,wrong_reversedand strict majority. - Estimate:
--estimate-onlycounts each extra judge once: 12 judge calls for one judge, 36 for three. - Repo rules:
cli-interface.jsonandcli_flags.pyare updated, the study page is in themkdocs.ymlnav, the changelog fragment is present, every commit has aSigned-off-by, andjudge_panelis an additive key that the open top level ofnovelty.v1.jsonaccepts.
Bot points I checked that no longer apply at this head (19)
- exai-bot's point on the ranking claim in the study summary does not apply: the summary and the section 1 heading now say the extremes are stable and the middle is not.
- exai-bot's point on the noise-floor comparison does not apply: the paragraph now qualifies the shift by model and metric.
- exai-bot's point on duplicate
--judgespellings does not apply: duplicates are compared by model identity andopenrouter/openai/gpt-4ois refused next toopenai/gpt-4o. - exai-bot's point on
--judge2replaying a panel member does not apply:--judge2is checked against every panel judge before a paid call, inanalyzeandprobe. - exai-bot's point on
analyzelosing the individual verdicts does not apply:analyzewrites them beside the record (the stale-file case is a separate inline comment). - exai-bot's point on
--estimate-onlycall counts does not apply: three judges report judge=36 calls against 12 for one, and the USD figure counts each extra judge once. - exai-bot's point on one model given twice through
run_measurementdoes not apply:check_judgesrefuses it before any call. - exai-bot's point on
added_share_rangeover unequal samples does not apply: the range is taken on the pairs every judge graded in both arms. - exai-bot's point on
FAM[j]raising aKeyErrordoes not apply:analyze.pyis pinned to the judges inFAMand the study test passes. - exai-bot's point on the generation leniency delta does not apply to the code, which now uses the shared answers; the regenerated tables wait for the author's answer to the open question on
generation_kappa.csv. - exai-bot's point on stale grades after
repeatdoes not apply: resume keeps drawn repeats and grades carryanswer_sha. - exai-bot's point on the ranking claim in the research records does not apply: X031 and F023 now limit it to the extremes of the gated share.
- Copilot's point on duplicate
--judgespellings does not apply: duplicates are detected by model identity. - Copilot's point on
analyze.pycrashing atFAM[j]does not apply: the study calculations are pinned to the declared panel. - Copilot's point on mixed populations in the leniency delta does not apply to the code, which now uses shared answers; the regenerated tables wait for the author's answer to the open question.
- Copilot's point on stale grades after
repeatdoes not apply: resume keeps drawn repeats and grades carryanswer_sha. - Copilot's point on estimated call counts does not apply: the extra judges' grading calls are in
by_role.judge,probe_calls, both totals andexpected_calls. - Copilot's point on the report naming one judge does not apply: the report names the panel and the rule when
judge_panelis present. - Copilot's point on a zero-division in
kappa()does not apply togen.py: it returnsNonefor an empty or fully chance-agreed pair (the consumers still fail, see the inline comment).
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/docs/skill-novelty.md
A missing vote is no datum, the veto applies to each judge's own 2x2, the answerer and failed-judge warnings name each panel judge, and a single-judge run removes the panel's verdict file. The unanimous help no longer calls the rule conservative; the estimate tests pin the panel's calls and the self-judge refusal at the analyze command. Thread: PRRT_kwDOVBg7g863cJ2x Thread: PRRT_kwDOVBg7g863eJks Thread: PRRT_kwDOVBg7g863eJk3 Thread: PRRT_kwDOVBg7g863eJk0 Thread: PRRT_kwDOVBg7g863eJk4 Thread: IC_kwDOVBg7g88AAAABmcYFzQ#1 Thread: IC_kwDOVBg7g88AAAABmcYFzQ#2 Thread: IC_kwDOVBg7g88AAAABmcYFzQ#3 Thread: IC_kwDOVBg7g88AAAABmcYFzQ#6 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
analyze.py reads only the judges of its panel and skips a grade whose answer_sha is not the answer now stored under its key. Undefined kappa is None, not a crash, and gen.py rates each judge pair on the answers both judges graded. Thread: PRRT_kwDOVBg7g863cJ21 Thread: PRRT_kwDOVBg7g863eJkv Thread: PRRT_kwDOVBg7g863cJ22 Thread: PRRT_kwDOVBg7g863eJkw Thread: PRRT_kwDOVBg7g863cJ26 Thread: PRRT_kwDOVBg7g863eJkz Thread: PRRT_kwDOVBg7g863cJ28 Thread: PRRT_kwDOVBg7g863Sppg Thread: PRRT_kwDOVBg7g863SlmW Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The warning covers the families in the client's vendor table, not every model family, so the doc names them. Thread: PRRT_kwDOVBg7g863eJk6 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
These points were fixed in 9dabb8b or earlier on this branch; the pins for them are in the commits above. The reviewer's 91ad8bd is not on the branch. Thread: PRRT_kwDOVBg7g863Sll9: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863SlmA: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863SlmD: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863SlmE: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863SlmH: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863SlmJ: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863SlmL: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863SlmO: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863SlmS: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863Slma: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863Slmg: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863SpoQ: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863Spo9: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863SpqK: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863Spqq: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863Spq-: already fixed in 9dabb8b Thread: PRRT_kwDOVBg7g863SprP: already fixed in 9dabb8b Thread: IC_kwDOVBg7g88AAAABmcxO4A: already fixed in 9dabb8b Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
@top.posted.md |
| - Lenient added share moves by at most 0.067 (haiku-4.5 answers, 0.27 → 0.20), within the | ||
| noise floor. On opus-5 and grok-4.6 answers it does not move. |
There was a problem hiding this comment.
Must-fix — Impact: this treats the Haiku generation shift as ordinary repeat variability without a matching noise-floor measurement. Trigger: interpreting the 0.067 decrease as evidence that the three-family panel absorbs changes within repeat noise. Evidence: generation_panels.csv confirms Haiku's 0.267→0.200 shift, but section 1 gives Haiku repeat-added floors of 0.02–0.03 (0.03 for the seven-judge majority). gen.py exports no repeat floor for these generation panels, and the old panel cannot grade repeats because the original judge has no repeat grades. Workaround: retain the measured shift but replace “within the noise floor” with an explicit statement that these panels' repeat floors are unmeasured, or measure a matching floor before making the comparison. Severity justification: medium, a bounded but material error in the research evidence supporting panel robustness.
| if raw is None: | ||
| return ERRORED | ||
| return resolve_verdict(raw, run.checks)[0] if run.checks else raw |
There was a problem hiding this comment.
Must-fix — Impact: a failed judge can enter the supposedly fully graded shared sample and create a false judge-sensitivity warning. Trigger: an explicit ERRORED vote on an answer with a failed veto. Evidence: only None returns early here; resolve_verdict(ERRORED, [{"role": "veto", "passed": False}]) returns wrong_other. For one pair with baseline votes {j1: wrong_other, j2: error} plus that veto, and full votes {j1: correct_complete, j2: wrong_other}, panel_record reports range=1.0 and judge_sensitive=True, although j2 never graded the baseline and there are no pairs graded by every judge. Preserve both absent and explicitly errored votes as missing before applying checks, and extend the missing-vote/veto regression test to cover error. Workaround: exclude pairs containing raw errored votes when interpreting panel diagnostics. Severity justification: medium because this materially corrupts bounded per-judge and sensitivity statistics.
Check run: inconclusive, the result is not shown as a pass.
Suggestion:
| if raw is None: | |
| return ERRORED | |
| return resolve_verdict(raw, run.checks)[0] if run.checks else raw | |
| if raw is None or raw == ERRORED: | |
| return ERRORED | |
| return resolve_verdict(raw, run.checks)[0] if run.checks else raw |
| po = sum(x == y for x, y in zip(a, b, strict=True)) / n | ||
| ca, cb = Counter(a), Counter(b) | ||
| pe = sum(ca[l] * cb[l] for l in labels) / n / n | ||
| return (po - pe) / (1 - pe) if pe < 1 else 1.0 |
There was a problem hiding this comment.
Must-fix. Impact: pairwise_kappa.csv and mean_kappa_lenient report perfect chance-corrected agreement where that statistic is undefined. Trigger: both judges assign the same single class to every shared answer, as in the all-correct regression fixtures. Evidence: pe=1 makes the kappa denominator zero, but this branch returns 1.0; gen.py correctly returns None for that same case. Return None here too, letting the existing rounded() and mean_kappa() handling omit undefined values, and assert the empty cells in a constant-label test. Workaround: exclude constant-label pairs from interpretation. Severity justification: medium—a bounded statistical correctness error in the reported agreement metrics.
Check run: inconclusive, the result is not shown as a pass.
Suggestion:
| return (po - pe) / (1 - pe) if pe < 1 else 1.0 | |
| return (po - pe) / (1 - pe) if pe < 1 else None |
| sd = np.mean([np.std(v) for v in spreads.values()]) | ||
| rg = np.mean([max(v) - min(v) for v in spreads.values()]) |
There was a problem hiding this comment.
Must-fix. Impact: partial grading aborts analysis before panel_size.csv and summary.json are written. Trigger: keep original grades for all five models, but give a new judge complete arm triplets for only one model. Evidence: headline() returns gated=None for the other models; spreads retains those values and np.std(v) raises TypeError (max(v) - min(v) has the same assumption). The disjoint-row regression gives each judge coverage of every model, so it misses this case. Handle unavailable shares explicitly and emit undefined statistics when no usable values remain. Workaround: complete every judge/model's grading before analysis. Severity justification: medium—a bounded loss of reporting functionality on output the resumable grader can produce.
| (r["rule"], r["answer_model"]): float(r["gated"]) | ||
| for r in csv.DictReader((T / "aggregation_rules.csv").open()) |
There was a problem hiding this comment.
Must-fix. Impact: figure generation aborts instead of plotting the available judge results. Trigger: run fig.py on tables from the existing disjoint-judges regression fixture. Evidence: those two new judges never share a row, so analyze.py's two-vote quorum yields undefined aggregation headlines and writes gated=""; float("") here raises ValueError, even in strict mode where aggregation markers are not used. The per-judge conversion below has the same missing-value assumption. Skip undefined values, omit unavailable majority markers, and handle models with no valid dots. Workaround: complete overlapping grading before generating figures. Severity justification: medium because valid resumable-study output causes a bounded loss of figure-generation functionality.
| out["panels"][name] = {"judges": panel, "added": {}} | ||
| for mode, good in (("lenient", LEN), ("strict", STR)): | ||
| for m in MODELS: | ||
| h = headline(panel_get(panel, good), m, good) |
There was a problem hiding this comment.
Must-fix; the panel-comparison portion of the earlier population finding remains unaddressed. Impact: generation_panels.csv can show a generation effect caused solely by missing grades. Trigger: two probes have identical old/new decisions, one added and one known, but a new-panel member lacks the known probe's placebo grade. Evidence: panel_get() checks only its own panel, so this loop reports old added=0.5 (n=2) and new added=1.0 (n=1) despite no changed decisions. jget() now intersects judge-pair coverage, but these panel comparisons do not. Require complete grades from both panels for every compared cell, and add an asymmetric-missing-grade regression. Workaround: restrict inputs to cells fully graded by all six panel members. Severity justification: medium—a bounded but materially confounded research comparison.
| A three-family median panel cuts the generation shift to ≤0.07 lenient (≤0.11 strict) and | ||
| keeps the order of the extreme answer models. |
There was a problem hiding this comment.
Must-fix — Impact: the finding implies that the extreme-model ranking survives the panel upgrade on both reported scales, but it does not on strict scoring. Trigger: using this record's ranking claim for a strict-scored comparison. Evidence: generation_panels.csv gives the new panel strict added shares of 0.452 for Grok and 0.433 for Nemotron, whereas the old panel puts Nemotron highest (0.533). Workaround: qualify “keeps the order” as applying to the lenient added share only; apply the same qualification to section 6's “In both panels” bullet in docs/judge-panel-study.md. Severity justification: medium because this is a bounded but material overstatement of the research conclusion about ranking stability.
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 4919509. The delta since 9dabb8b fixes most of my earlier points: the veto and the missing vote in each judge's own 2×2, the stale verdicts file, the estimate test, the unanimous help text, and the new answerer-vendor and failed-judge warnings all hold. Three things are worth fixing before merge: the unanimous sentence that is still in the docs, and two ranking and noise-floor claims in the study that its own tables do not support. Six nits follow, five inline and one below. The merge also waits for the author's answer on the regenerated generation tables.
What I ran, on a Linux review host at 4919509:
make ci-local: 31 passed, 2 failed, 14 skipped. The two failures are the gitleaks download (HTTP 504) and the commit scan that needs it; GitHub's own check scans the PR. Changed-line coverage passes.- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run on
tests/test_novelty_panel.py(36 tests): 21 mutants of the new panel record, warnings and verdict-file code, of which 7 pass the full suite. Only the answerer-model pass-through below decides whether a warning shows.
Finding outside the inline comments:
Nit: The vendor list in the docs omits xai. docs/skill-novelty.md:283-285 names claude, gpt and gemini/mistral families, and line 345 says "the families named above" print a warning. _VENDOR_MARKERS in client.py also knows grok and xai, so two Grok judges, or a Grok answerer with a Grok judge (the docs' own example at line 317 uses grok-4.6), warn although the docs do not say so. Suggestion: add "grok and xai another" to the list at line 283.
Open bot points I confirm:
- exai-bot's point on the generation table stands:
generation_kappa.csvstill holds rate deltas computed on unmatched populations (-0.0422on n=2,664 against 2,932 rows):gen.pyis fixed, but the stored table and section 6 are not regenerated, so the published numbers still come from the old calculation. - Copilot's point on the generation table stands:
generation_kappa.csvand the section 6 claims have to be regenerated or the rate-delta column dropped; the author asked which, and the answer is open.
Still open from my earlier review:
docs/skill-novelty.md:331: the sentence namingunanimousfor the 0.20 move is still there (inline below).
@vchamarthi's question on re-running gen.py and committing the regenerated tables waits for the author, who handed the choice back.
Checked, no issues
- Verdict file: a one-judge
analyzeinto a directory that held a panel run removes the old<stem>.judge_verdicts.json; deleting theunlinkfails the new test. - Own 2×2: a judge's verdict goes through
resolve_verdictwith the run's checks, and a judge with no vote isERRORED; removing either fails a test. - Warnings: the answerer-vendor warning fires for a routed
openrouter/anthropic/claude-xagainstclaude-sonnet-5, and the failed-judge count matchesERROREDvotes.analyzeandprobeboth pass the answerer model on. - Repo rules: every non-merge commit has a
Signed-off-by, the study page is in themkdocs.ymlnav,cli-interface.jsonandcli_flags.pymatch the flags,tests/test_growth_ratchet.pypasses, and the new study-script tests leave the tree clean.
Bot points I checked that no longer apply at this head (1)
- exai-bot's point on an explicit
errorvote under a failed veto does not apply: the headline resolves an errored vote under a failed veto towrong_otheras well, and the range of 1.0 in that case comes from the two judges disagreeing on the full arm, not from the error.
| In the study, median and majority matched Dawid-Skene on 99.3% of rows. Unanimous and | ||
| any-correct moved the headline by up to 0.20. A judge whose grade fails is left out of the |
There was a problem hiding this comment.
The docs still say unanimous moved the headline by up to 0.20. The help text is fixed, but this sentence and the one at dokimi/dokimi/novelty/panel.py:10 are not. In judge-panel-study.md the unanimous rule gives .32 for Haiku answers against .32 under majority, and the largest move of unanimous against majority is Nemotron's .43 to .52 (0.09). Only any-correct, which the CLI does not offer, gives the 0.20. A user who picks --judge-aggregate unanimous to be safe reads a number that does not describe it. By reading the study table.
Suggested fix: "Any-correct moved the headline by up to 0.20; unanimous raised the added share by up to 0.09 over majority." Change panel.py:10 the same way.
| - Lenient added share moves by at most 0.067 (haiku-4.5 answers, 0.27 → 0.20), within the | ||
| noise floor. On opus-5 and grok-4.6 answers it does not move. |
There was a problem hiding this comment.
I agree with exai-bot here; "within the noise floor" has no floor to compare with. Haiku's lenient shift is 0.267 to 0.200 (0.067) in generation_panels.csv. Section 1 gives Haiku's repeat-added floor as .02-.03 for every judge and .03 for the seven-judge majority, and line 462 says the old panel cannot grade repeats. The 0.10 used elsewhere is the maximum over all models. By reading the CSV and section 1.
Suggested fix: "Haiku's lenient shift (0.067) is larger than its measured repeat floor (.03), and no floor is measured for these two panels."
| - A newer judge sits no closer to the panel consensus. | ||
|
|
||
| A three-family median panel cuts the generation shift to ≤0.07 lenient (≤0.11 strict) and | ||
| keeps the order of the extreme answer models. |
There was a problem hiding this comment.
I agree with exai-bot here; "keeps the order of the extreme answer models" holds for the lenient share only. In generation_panels.csv the old panel's strict added share puts Nemotron highest (0.533), while the new panel gives Grok 0.452 and Nemotron 0.433. Opus is lowest everywhere, so the "least" half holds. The bullet "In both panels, opus-5 answers gain least and nemotron-nano answers gain most" at judge-panel-study.md:450 has the same gap. By reading the CSV.
Suggested fix: "keeps the order of the extreme answer models on the lenient share; on the strict share Grok and Nemotron swap", and the same qualifier in the section 6 bullet.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…nel record Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to c6b0fb9, which fix the six nits from my review:
fbacfe7:docs/skill-novelty.md:283: the vendor list now namesgrokandxaias another family.937c77f:research/judge-panel/analyze.py:224:kappareturns None when chance agreement is 1, the docstring says so, and the disjoint-rows test gains a constant-label case.644e3e4:research/judge-panel/analyze.py:421: the panel-size spreads keep only shares that are not None, so a model with no value is skipped.3ef9e97:research/judge-panel/fig.py:26: both dicts are built only from rows with a non-emptygated, and a model with no dot (or no majority entry) is skipped.9e3c003:research/judge-panel/gen.py:73:panel_gettakes the other panel asalsoand returns None unless both panels graded the cell.c6b0fb9:tests/test_novelty_panel.py: a new test runsprobeandanalyzewith a mocked transport and assertssame_vendor_as_answerer == ["claude-opus-5"]in the written record.
make ci-local at c6b0fb9: 32 passed, 1 failed (guidance self-scan, caused by an untracked results/ directory left by an early run of the new test, not by a commit), 14 skipped.
Not approving yet. Still open: the three risks from my review; @vchamarthi on generation_kappa.csv and section 6 still carry the rate deltas from mixed populations; the code is fixed but the stored tables are not regenerated (or the column dropped); @vchamarthi on choice between regenerating generation_kappa.csv and section 6 (A) or dropping the rate-delta column until a re-run (B); the author handed it back.
Implements the judge panel the study in #482 recommends, for
skill novelty analyzeandskill novelty probe. Stacked on #482: this branch mergesresearch/judge-panel, so the diff shows the study too until #482 lands.--judge PROVIDER/MODELis repeatable, and two or more judges form a panel. It is the canonical spelling from CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94. Mixing it with--judge-model/--judge-provideris refused, as is a model given twice. The first judge is the primary judge and the default extractor.--judge-aggregate {median,majority,unanimous}, defaultmedian, registered incli_flags.pyand thecli-interface.jsonsnapshot.dokimi/novelty/panel.py:judge_panelrecord block, which has each judge's own paired 2×2,added_share_rangeandjudge_sensitive(lenient range > 0.10).run_measurementgrades each answer with every judge and storesjudge_verdicts. A failed judge is left out of the vote. Every panel judge gets the self-judge check, and cost and estimate include every judge.Tests:
tests/test_novelty_panel.py(21). Ruff is clean. The parity, cli-surface and symlink failures on Windows also fail on origin/main.Follow-ups:
eval score --judgepanels, a Dawid-Skene rule, and a wider vendor map.🤖 Generated with Claude Code