Judge-comparison core: joint scorer and panel-vs-single-judge analysis

[andalibmalit]

Judge-comparison core on top of partner mode: a joint scorer variant and the panel-vs-single-judge analysis.

• humanebench/joint_scorer.py: scores all 8 principles in one call per turn (vs. the overseer's one call per principle), with per-slug parsing, invalid-handling, and aggregation mirroring the per-principle overseer so joint-vs-decomposed comparisons aren't confounded by pipeline differences. Principle order is randomized per turn and recorded in score metadata for position-bias analysis.
• src/partner_rejudge_joint_task.py: replay task using the joint scorer; run without --model.
• scripts/compare_panel_vs_single_judge.py: compares the 3-judge panel against a dataset's original judgments under pre-committed analysis rules — ordinal-first (per-cell panel medians), flags-based denominators, per-stratum reporting, two Holm-corrected confirmatory contrasts with everything else exploratory, NaN-attrition accounting, and __rep2/__rep3 reliability slices.
• Tests: tests/test_joint_scorer.py, tests/test_compare_panel.py (synthetic fixtures throughout).

Second in the stack. Part of #90

🤖 Generated with Claude Code

[ErikaOnFire]

Comment only — no formal verdict yet. My read is approve-worthy once two decisions get made explicitly; neither is a bug. Stack context (mergeability, the absence of CI, and what could and could not be verified mechanically — including that the 675 lines of tests here were read but not run) is in my review on #90.

9 files, +2,008/−102:

• humanebench/joint_scorer.py — 343 lines, scores all 8 principles in one call per turn, with per-turn randomized principle order seeded from the sample id
• scripts/compare_panel_vs_single_judge.py — 741 lines
• 675 lines of tests

The contract against humanebench/scorer.py holds

I checked the joint scorer against the single scorer point by point, because a parallel scorer that diverges quietly is the worst kind of bug here:

• individual_scores is a list of per-judge severities in both (scorer.py:226).
• The strict any-judge-fails ⇒ NaN rule is mirrored per-slug rather than collapsed to a single global failure.
• _extract_json and InvalidOverseerResponse are imported rather than reimplemented — so the parsing behaviour cannot drift between the two scorers. This is the right call and it's the one I'd have worried about most.
• HUMANE_PATTERNS keys equal the pattern ids, so the prompt's JSON template and the parser agree by construction rather than by convention.
• load_cells correctly treats both NaN-valued scores and the early-return failure path as NaN cells. That second one is easy to miss: the early return at scorer.py:196-204 omits individual_scores entirely rather than filling it with NaN, so a reader who only handles the NaN-valued case gets a KeyError or a silently-dropped cell. You handle it.

Seeding the shuffle from the sample id is also the right choice over a global RNG — it makes the ordering reproducible per sample without coupling to iteration order.

Review point 1 — the confirmatory contrast pairs a one-principle treatment with an all-principles control

contrast_fhr_deescalation builds the treatment arm as c.slug == FHR_SLUG and c.orig_severity == -1.0, but builds the RTM control arm as c.flags.get("is_positive_extreme") and c.orig_severity == 1.0 — with no slug filter. So contrast (a) compares an FHR-only treatment against a pooled, cross-principle control.

The FHR-only control counts (ctrl_fhr_n, ctrl_fhr_regressed) are computed and printed as a descriptive, so the information is there. But the one-sided Fisher test uses the pooled control.

For a claim the PR itself frames as "a one-principle claim and is tested as one," pairing it with a cross-principle control should be an explicit reviewer decision, not an absorbed implementation detail. It may well be the right choice — a pooled control has more power and the regression rate may not vary much by principle — but that argument should be on the record before the number goes in a paper, because it's the first thing a referee will ask. Either filter the control to FHR, or state in the docstring why pooling is defensible here.

Review point 2 — holm() shrinks the pre-registered family from m=2 to m=1 when one contrast is empty

NaN p-values are dropped and the survivor is corrected over the remaining finite tests only. The docstring argues for this well, and the argument is correct as far as it goes: a NaN sorted first would force every valid contrast to 1.0, which is worse. A test pins the behaviour, so it's deliberate rather than incidental.

The thing I'd want said out loud: the effect is weaker FWER control than the pre-registration implies. If you pre-registered a family of two and one comes back empty, correcting the survivor at m=1 means it faces α rather than α/2 — which is a different test than the one that was registered. That's a legitimate position (you can argue the empty contrast was never a test), but it's a reviewer's call and it belongs in the analysis write-up, not only in a docstring.

Both of these are statistical rather than mechanical, and both are documented choices rather than oversights. I'm raising them because they're exactly the two places where "the code does what it says" and "the claim is the one that was registered" can come apart.