Conversation-level judging and joint-vs-decomposed analyzer

[andalibmalit]

Conversation-level judging and the joint-vs-decomposed analyzer.

• Scorers accept an optional prompt-template override, slot-validated at call time so a template missing a required placeholder fails loudly.
• src/convlevel_templates.py: conversation-level templates derived from the single-turn templates by replacing exactly the framing sentences; each replacement asserts its target occurs exactly once, so any upstream rubric edit that would invalidate the derivation fails at import time. tests/test_convlevel_templates.py pins byte-identity of global rules, severity scale, and output format.
• src/partner_convlevel_task.py / src/partner_convlevel_joint_task.py: judge a full transcript (per-principle and joint call structures respectively); output shapes match the single-turn tasks so existing parsers work unchanged.
• tests/test_partner_task_parity.py: freeze-guard pinning the identical judge-panel block across all four partner task files, so a panel edit can't silently confound cross-arm comparisons.
• scripts/convert_partner_results.py: fresh-audit datasets (stored conversations with no original judgments) convert only via explicit --fresh-audit opt-in; optional extra_metadata pass-through (analysis-only, never shown to judges); deterministic --sample-turns subsampling for pre-registering a joint slice.
• scripts/compare_joint_vs_decomposed.py + tests/test_joint_vs_decomposed.py: three-arm joint-vs-decomposed comparison with position-bias diagnostics from the recorded per-turn principle order.

Top of the stack. Closes #90

🤖 Generated with Claude Code

[ErikaOnFire]

Comment only — no formal verdict yet. Stack context (mergeability, the absence of CI, and what could and could not be verified mechanically — including that the 432 lines of tests here were read but not run) is in my review on #90.

10 files, +1,248/−23:

• a template= override on both overseer and joint_overseer, with a fail-fast slot check at scorer-construction time rather than at first call
• src/convlevel_templates.py, deriving conversation-level templates from single-turn ones by exact-string replacement, with an import-time assertion that each target occurs exactly once
• two conversation-level tasks
• scripts/compare_joint_vs_decomposed.py — 518 lines
• 432 lines of tests, including a freeze-guard pinning the judge-panel block across all four partner tasks

What I verified, and what I like

The derivation works against the current template. _SINGLE_INTRO and _SINGLE_EVAL_LINE in convlevel_templates.py match humanebench/scorer.py:59 and :78 exactly, so the _replaced_once assertions pass at import rather than blowing up on someone at runtime. I checked this specifically because derive-by-string-replacement is fragile, and the import-time exactly once assertion is the right mitigation — it converts a silent wrong-prompt into a loud startup failure. Same for putting the slot check at scorer construction instead of first judge call. Both are good instincts.

The freeze-guard on the judge panel across all four partner tasks is also a nice touch — that block drifting between tasks is precisely the kind of thing no one notices until two result sets disagree.

Minor: the slot validation added to overseer checks {issue_display_name}, {issue_description}, {prompt} and {response} but not {issue_key}. Harmless — a template omitting {issue_key} still formats cleanly with the extra kwarg present — so this is a completeness note, not a defect.

Blocking, for me: the two conversation-level tasks ship with no way to produce their input

The dataset format requires packing every turn up to the final user message into input, and the final assistant response into metadata.ai_output. Per your own comment on #90, the builder for that "exists as an engagement-specific internal script" and is deliberately deferred.

So src/partner_convlevel_task.py and src/partner_convlevel_joint_task.py land as code no one outside the engagement can run. It's disclosed rather than hidden, and I do appreciate that you said so plainly — but for a public repo that's the definition of unfinished. Someone clones this, finds two tasks with a documented input format and no producer, and either writes a second incompatible builder or files an issue.

The ask: either the builder, or an explicit note in docs/partner-mode.md. Right now the doc changes in this PR don't mention conversation-level mode at all, so there isn't even a place where a reader would learn that the gap is intentional. A short "conversation-level mode requires a dataset builder not included in this repo; the expected format is X" section would be enough for me.

Cross-cutting: conceptual duplication with #94 (the Rust CLI)

Posting this on both PRs, because it's a coordination decision for the maintainer rather than a defect in either one.

There is no file-level conflict. #94 is 22 files entirely under cli/ (cli/src/**, cli/Cargo.*, cli/rubric/, its own cli/.gitignore); none of your PRs touch cli/, and #94 touches nothing outside it. Those two can merge in either order.

But there is real duplication in two places:

1. A third rubric text, with different global rules. cli/rubric/judge_prompt_v3.md restates the 8 principles with 4 global rules, against humanebench/scorer.py:58's OVERSEER_TEMPLATE with 7 — and they are not the same 4. The CLI's rule 3 makes parasocial behaviour an automatic −1.0 on Foster Healthy Relationships; the Python rule 3 states the prohibition without mandating a score. The principle vocabularies don't even join: cli/src/judge/mod.rs:134 declares PRINCIPLES = ["respect_attention", "meaningful_choices", ...] in snake_case, while Python uses respect-user-attention, enable-meaningful-choices in kebab-case. #94's own cli/rubric/README.md calls this out under a "Known gap" heading. Merging #92 adds a fourth prompt lineage on the Python side while #94 adds the Rust one, and nothing is reconciling them.

2. Conversation-level judging is being built twice, independently. cli/src/judge/rollup.rs:27 has rubric_preamble(), which produces a conversation-level prompt by splitting the rubric at "Now, evaluate the following" and discarding the per-turn framing. This PR's src/convlevel_templates.py produces one by replacing the per-turn framing sentences in the Python templates. Same construction, same intent, two languages, two rubric texts, zero shared code or tests, and neither PR references the other.

If both merge as-is, "HumaneBench conversation-level score" means two different numbers depending on which entry point produced it. Worth a decision about which lineage is canonical before either lands, rather than after someone compares results across them.