Partner mode: re-judge external prompt/response datasets

[andalibmalit]

Adds partner mode: re-judging an external dataset of stored prompt/response pairs with the HumaneBench judge panel, with no generation anywhere in the path.

• src/dataset_source.py + task changes: the three benchmark tasks accept -T dataset=/abs/path.jsonl; default behavior unchanged. Deliberately no environment-variable override — a stale variable could silently redirect a benchmark run.
• src/partner_rejudge_task.py: replays metadata.ai_output via the strict replay solver and scores it with the standard overseer ensemble; run without --model, so anything attempting generation errors loudly.
• scripts/curate_production_pairs.py: tags rows (synthetic/QA traffic, duplicate clusters, trivial labels, language heuristic) without ever dropping any; tags are analyst-only, never shown to judges.
• scripts/convert_partner_results.py: fans each judged turn out to one sample per principle, carrying original judgments in metadata for post-run comparison.
• scripts/select_comparison_subset.py: deterministic stratified subset selection, including repeat slices (__rep2 within-run, __rep3 between-run) for panel reliability measurement.
• docs/partner-mode.md: ground rules (external data never enters git; provider-egress sign-off; HumaneScore comparability caveat) and the pipeline walkthrough.
• tests/test_partner_scripts.py: synthetic-fixture coverage of curation, conversion, and subset selection.

First of a 3-PR stack. Part of #90

🤖 Generated with Claude Code

[andalibmalit]

Known gap / possible follow-up after the stack lands

The conversation-level tasks in the third PR ship without a public dataset builder: producing their input JSONL (packing every turn up to the final user message into input, replaying the final assistant response as metadata.ai_output) currently requires hand-rolling the transcript packing. Two candidate additions, both parameterized out of internal run tooling rather than lifted as-is:

1. Conversation-level dataset builder — turn-indexed rows in, conv-level samples out (per-principle and joint variants), with the same loud-failure contracts as the converter (non-empty response required, schema-collision rejection).
2. Rerun-recovery builder — given a directory of .eval logs from a partially failed run (provider errors, NaN scores), emit a minimal rerun dataset covering only the samples without valid verdicts, unioning valid verdicts across all prior logs per config.

Both exist as engagement-specific internal scripts; upstreaming them is a deliberate parameterization rewrite, not a scrub, so they're deferred until there's demand. Leaving this note so the gap is tracked rather than rediscovered.

[ErikaOnFire]

Comment only — no formal verdict yet, but I'll say up front that this one reads as approve-worthy to me, with two notes below. 11 files, +933/−33 is a reviewable PR and it shows.

Context for this stack (stating it once here rather than in #91 and #92 too)

• All seven of your open PRs are currently MERGEABLE with behind_by=0 against main at d94963d.
• This repo has no CI at all. gh api repos/buildinghumanetech/humanebench/actions/workflows returns total_count: 0 and there is no .github/workflows. So a BLOCKED status here means review-required (required_approving_review_count: 1 plus require_code_owner_reviews), not a failing check.
• No test suite could be executed during this review — inspect_ai, numpy, scipy, pandas and pytest are all unimportable in the environment I reviewed from, so the 675 lines of tests in #91 and the 432 in #92 were read, not run. What was verified mechanically is that every changed .py file parses under ast.parse: 9/8/10 files across #90/#91/#92, zero syntax errors. That's a floor, not a test pass.

Integration points verified against main

• use_pregenerated_output_strict exists at src/pregenerated_solver.py:47.
• The nested-metadata convention state.metadata["metadata"]["ai_output"] matches src/pregenerated_solver.py:63-65.
• The hardcoded judge panel in partner_rejudge_task.py is byte-identical to src/baseline_task.py:33-40.
• The refactor of the three benchmark tasks preserves default behaviour.

All four line up. Nice.

Note 1 — the membership flags are internally inconsistent in this PR, and #91 rewrites them

select_comparison_subset.py emits curation.flags whose stated purpose is getting correct analysis denominators, but the predicates don't match the strata they're named for:

• is_negative is min(sev) <= -0.5, while the negative stratum is -1.0 < worst <= -0.5 — the flag swallows the -1.0 cases the stratum excludes.
• is_positive_extreme is max(sev) >= 1.0, while the stratum also requires worst > -1.0.
• There is no is_positive flag at all, despite a positive stratum existing.

#91 fixes exactly this — membership_flags becomes an actual partition and gains is_positive. So as a step inside this stack it's safe, and I'm not asking you to change it here.

The reason I'm flagging it anyway: anyone who runs the selector off #90 alone and analyzes by flags gets wrong denominators for three of the seven strata, silently. If #90 can land on main before #91 is ready — and it can, its base is main — then there's a window where that's the published behaviour. A one-line note in the flags docstring saying they're not yet a partition, or just landing the two together, would close it.

Note 2 — the .gitignore append will collide with #84

This is an inference from the hunk headers, not a merge I performed. #90's hunk header is @@ -255,3 +255,9 @@ and #84's is @@ -255,3 +255,37 @@. Both take the file's last three lines as context, so whichever lands second will conflict there.

Worth stating explicitly: GitHub reports both as MERGEABLE because it computes each against main independently, so the UI will not warn you about this. It's a trivial conflict to resolve by hand — I'm noting it only so it isn't a surprise at merge time.

Minor, not a defect

*.eval as a bare global ignore pattern in the root .gitignore is broader than it probably means to be. No .eval files are currently tracked, so nothing breaks today — but a bare glob will also swallow one somebody deliberately wants committed later, with no error.

One thing I appreciated

Your own comment on this PR documenting the two deferred builders is a genuinely useful disclosure. It's the thing that let me scope #92's review correctly rather than guessing at why a dataset format had no producer. More of that, please.