Foundations: judge/provenance/decomposition infra + main reconciliation - #84
Open
andalibmalit wants to merge 114 commits into
Open
andalibmalit wants to merge 114 commits into
andalibmalit wants to merge 114 commits into
Conversation
added acknowledgements
Updated figs
clearer visualization generation instructions
…y candlestick text dynamically generated
updated figures fully after addition of GPT-5.1, and made steerabilit…
figures scoregrid
Add production-ready Python and TypeScript implementations of the HumaneBench v3.0 evaluator. - Python script with CLI and programmatic API - TypeScript/Node.js implementation - Comprehensive README with usage examples - Validation tests and example code - Support for OpenAI, Anthropic, and custom API providers This enables developers to easily integrate HumaneBench evaluation into their own codebases.
set evaluator version number, added node gitignore content
Stanford helm comparison
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Chart title: "The Anti-Humane Drift Problem" (was "The Model Drift Problem")
- README: update stale steerability references to match renamed scripts
- Principle charts: "Anti-Humane Drift: {principle}" title
- Table output: "Comprehensive Behavioral Drift Summary"
"Model drift" in ML literature means temporal performance degradation,
not what HumaneBench measures. These changes use domain-specific terms
that avoid confusion with established ML concepts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…n tests - Replace silent `continue` in `bootstrap_cohort_principle_means` with a `RuntimeWarning` naming the principle and missing (model, persona) cells. Production data has all 24 cells, but a future partial-data re-run now surfaces at runtime rather than dropping a Table 4 row unnoticed. - Hoist the `cell()` print helper above the summary loop in `compute_cohort_principle_cis.py`; takes `principle` as an explicit arg. - Extend `test_cohort_point_estimate_matches_mean_of_model_means` with explicit asserts on `n_models == len(models)` and ordered CI bounds (`ci_lower <= point_estimate <= ci_upper`, strict `ci_lower < ci_upper`) for both `cells` and `deltas` output frames. Verified: output CSVs are byte-identical to the prior commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the paper variant's legend into the right gutter beneath the "Failed" group label, shorten the labels, swap the baseline patch for its actual black dot marker, and tighten row spacing by ~1/3 (0.35 -> 0.23 inch per model). Frees vertical space in the AAAI fig_steerability layout without changing the full/compact web variants. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add AAAI HELM x delta_bad scatter renderer for paper section 4.6
Add cohort-mean per-principle bootstrap CIs for Table 4
It's only used inside the `if len(failed_models) > 0:` block now that the legend anchor no longer references failed_indices[-1]. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…pact-legend Compact paper-version legend for steerability chart
Prove the reported HumaneBench numbers were produced against the finalized dataset, not by tuning prompts against results. Each reported .eval log embeds the (id, input, target) it scored; we hash those and show they equal the frozen prompt set, so "prompts tuned afterward" is impossible for the reported numbers. - humanebench/provenance.py: single source of truth (freeze commit 9dc15bd, frozen prompt hash, canonical hashing recipe, shared helpers) - scripts/build_provenance.py: regenerates provenance/MANIFEST.{json,md} - scripts/verify_provenance.py: independent verifier, exits non-zero on mismatch - provenance/MANIFEST.{json,md}: per-run record (45/45 runs bind to frozen set, all created after the 2025-11-16 freeze; post-freeze dataset commits preserve the prompt hash) - PROVENANCE.md: chain of custody + honest limitations (run-time commit is local-only; judge models not version-pinned); Zenodo DOI placeholder - README: Provenance & Reproducibility section Tags (local): dataset-content-frozen-2025-11-16, evals-reported-2025-11 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
provenance/DEPOSIT_README.md ships inside the deposit tarball so the archive self-documents offline verification. dist/ holds the (regeneratable) bundle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reviewer noted the 3 LLM judges (claude-4.5-sonnet, gpt-5.1, gemini-2.5-pro)
are among the ranked models and 2 are in the top-4 "robustly humane" set, with
no structural control for self-preference beyond ensembling.
New scripts/compute_judge_self_preference.py builds that control from per-judge
severities already logged (humanebench.scorer individual_scores) — no new API
calls. Reuses humanebench.bootstrap for HumaneScore + paired-delta CIs.
Three analyses:
1. Relative-generosity difference-in-differences (leniency-adjusted): every
judge is mildly self-CRITICAL, not self-preferring — own-generation DiD is
negative with CI excluding 0 (Claude -0.033, GPT -0.025, Gemini -0.064).
2. Single-judge / leave-one-out rankings: Spearman rho ~0.94-0.996 vs ensemble.
3. LOO robustness invariance: all 4 Robust models stay Robust when their own
in-family judge is dropped.
Sanity gate: reconstructed 3-judge ensemble matches table1 to max abs diff 0.011.
Adds tests/test_judge_self_preference.py (6 tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Neutralize PROVENANCE.md and the README provenance section: state the verifiable facts (frozen prompt hash, freeze commit, metadata-only post-freeze edits, how to verify) without the p-hacking / anti-gaming framing. Keeps all technical content and the honest reproducibility caveats. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add ZENODO_DOI constant, regenerate manifest, and reference the DOI in PROVENANCE.md and the README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A workflow-backed high-effort review found the headline ("no self-preference;
judges harsher on their own outputs") was overstated and not cleanly identified,
plus several real bugs. Independent raw-severity check confirmed: on the standard
own-vs-peers measure, the Gemini judge DOES mildly self-prefer (+0.064 own family,
+0.059 own generation, CI excludes 0); only Claude (-0.096) and GPT (-0.040) are
self-critical. The previous "all self-critical" headline relied on a leniency-
adjusted DiD that is (a) not quality-matched and (b) zero-sum across the 3 judges
(negative DiD == peers cross-preferring), so it cannot carry a clean causal claim.
Narrative reframe (tables/judge_self_preference.md):
- Lead with the raw own-vs-peers self-preference (honest mixed result: 2/3
self-critical, Gemini mildly self-preferring).
- Demote the DiD to a clearly-caveated "leniency-adjusted" view.
- Anchor the rebuttal on the identified, decision-relevant results: single-judge
& leave-one-out ranking stability (rho ~0.94-1.0) and robustness invariance
(every Robust model stays Robust when its own-family judge is dropped). Gemini's
self-preference is immaterial — its models still Fail under the adversarial
persona regardless of judges.
Bug fixes:
- aggregate_judge_subset: count (not size) so a NaN/partial sample can't pass as
a full-complement ensemble mean; --raw-csv path now drops NaN/off-scale rows.
- Paste-ready summary derived from the actual verdicts (no more unconditional
"negative, CI excludes 0" claim on no-effect/n-a verdicts).
- Robustness status now from the marginal S_bad-S_baseline (matches displayed
numbers and extract_all_scores); paired-delta CI kept in separate columns.
- Sanity gate fails on unmatched models (was np.nanmax-ignored).
- Warn when an own-generation model name doesn't match; empty-input guard;
pivot includes principle (no silent dedup); Robust-set prose derived from data.
- Write ensemble_reconstruction_check.csv for provenance.
Tests still pass (6/6).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings the supplementary analysis to the paper's own bootstrap standard and the LLM-judge literature's norms. All new uncertainty uses the same scenario-cluster percentile bootstrap as humanebench/bootstrap.py (seed 20260407, n=1000). - Ranking stability (the main gap): bootstrap CIs on the single-judge / leave-one-out ranking correlations, Kendall's tau primary (LOJO norm) + Spearman. Every config holds tau >= 0.83 vs the ensemble, with CIs. (bootstrap_ranking_correlations; single_judge_ranking_correlations.csv gains kendall/spearman ci columns.) - Quantified invariance: bootstrap CI on the per-model HumaneScore change when a model's OWN judge is dropped (new loo_config_change.csv). Largest shift +0.073, all within the 0.1 Robust band and positive (the in-family judges are self-critical, so dropping them only strengthens Robust status). - Multiple comparisons: two-sided bootstrap p-values on the 6 raw self-preference tests, Holm-Bonferroni corrected. The Gemini self-preference survives (Holm-adjusted p = 0.006). judge_self_preference.csv gains p_bootstrap/p_holm/ significant_holm. - Markdown: Kendall-tau-primary ranking table with CIs, an own-judge-drop change subsection, a Holm note on the Gemini case, and a Statistical methods paragraph. Out of scope per decision: TOST/equivalence and a robust-OLS self-bias regression. Tests: 11/11 pass (added Holm, bootstrap-p, ranking-CI invariance, and config-change recovery). Sanity gate still PASS (max diff 0.011, 0 unmatched). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A workflow-backed high-effort review found that several auto-generated claims in write_markdown were hardcoded rather than gated on the computed data — correct for the current numbers but capable of printing false/self-contradicting statements in this reviewer-facing document if the data shifts. Narrative now derived from the data: - §5 "inside the 0.1 Robust band, all CIs within it" checks every point estimate AND CI bound against 0.1 (falls back to the actual max otherwise). - §2 "survives Holm-Bonferroni" gated on the actual Holm-adjusted p (<=0.05), not the raw CI; family size reported dynamically (collapses from 6 to 3 if own-generation data is absent). - Paste-ready band/direction claim gated on within_band + all_positive; handles an empty change table (no more "at most nan"). - "Claude and GPT are self-critical" now names the actually-self-critical in-family judges from the §2 verdicts. - Methods note corrected: §1-2 use a global scenario-cluster resample; only the §4 ranking and §5 change CIs stratify by principle. Cleanups: - Single source of truth DROP_MAP (was duplicated in build_config_change and the paste-ready loop). - build_config_aggs aggregates each config once and feeds both the HumaneScore pipeline and the ranking bootstrap (was aggregating twice per run). - The three new bootstraps use CI_LOW_PCT/CI_HIGH_PCT from humanebench.bootstrap instead of literal 2.5/97.5. Numbers unchanged (CI level identical); only narrative/structure. 11/11 tests pass; sanity gate PASS (max diff 0.011, 0 unmatched). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pipeline check rebuilds the 35,416 / 44 accounting straight from the 45 .eval files and closes the 540-item gap that tables/inter_judge_agreement.md left implicit (12 excluded prompts x 45 runs). The regenerated per-judge table is identical to the committed inter_judge_raw.csv. bootstrap_cohort_grid draws one scenario resample per replicate and carries it across all 15 x 3 cells, which is the correct design for cohort-level statistics (flip count, robust-set size) that bootstrap_cell_scores cannot give. Point estimates match score_cis_long.csv to 3e-16. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…r drop The flip count and Krippendorff alpha under each judge drop were the two pieces compute_judge_self_preference.py did not cover. Flip holds at 10/15 under all three drops and all three single judges, with no model changing status. Dropping the Claude judge adds Claude Sonnet 4 to the robust set -- the opposite direction from self-preference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ric extraction, rubric sensitivity - HELM MDE at n=13: 0.690 exact / 0.709 Fisher-z; observed r=+0.248 p=0.415 - Domain x principle: Cramer's V = 0.432; principle and domain not separable - Inter-principle correlation reported with the general-model-quality confound made explicit, since high r across models is expected under either hypothesis - Rubric extracted verbatim from log attachments; all 6 verification checks pass - Flip holds at 10/15 under every rubric variant and every judge drop Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
results/paper_numbers.md is the single source of truth for the resubmission; results/reconciliation.md checks all 78 numeric claims in the .tex against regenerated values. 7 mismatches found, 6 traced to a single root cause: 6 of the 45 cells in tables/table1_steerability_summary.csv are stale relative to both the logs and score_cis_long.csv. Two of them make the .tex contradict itself inside section 4 (Table 1 vs its own prose on Gemini 3 Pro and Gemini 2.5 Pro). Embedding-cache freshness proven from git rather than assumed. MiniLM half pending: sentence-transformers is installed but huggingface_hub cannot import under the analysis sandbox. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both embedding models now present. Zero pairs reach 0.80 under either model, which is the load-bearing claim; the two models agree on principle ordering (Spearman +0.905). 66 MiniLM pairs sit at or above the 0.60 threshold section 3.2 says the pipeline filtered at. The threshold value is right but the mechanism is misdescribed: find_duplicates compares new text against the already-accepted set only, never within-batch and never re-screening, and the 39 seeds were never filtered. It is an incremental admission rule, not a post-hoc property of the finished dataset. Recorded as reconciliation mismatch #8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the residual-pairs-above-threshold analysis to the generated report and records the text-embedding-3-large breakdown (165 pairs, 99 within / 66 between, 181 scenarios) alongside the MiniLM one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ariants) # Conflicts: # humanebench/bootstrap.py
…ELM safety cache - Rename scripts/create_aaai_helm_scatter.py -> create_helm_delta_scatter.py (update importer) - Venue-neutral wording in analyze_coverage.py, create_steerability_chart.py, scatter docstring - Untrack temp/ (59 scratch files), anonymization_redaction_list.txt, and helm_integration/data/gcs_safety_cache/ (3,812 files, ~520 MB, referenced by no code) - Pseudonymize human_rater_names in data/golden_questions.jsonl - Extend .gitignore to cover local working material and oversized data files
ErikaOnFire
reviewed
Sep 10, 2026
ErikaOnFire
left a comment
ErikaOnFire
left a comment
Contributor
There was a problem hiding this comment.
Reviewed as the root of the stack/1-foundations → stack/4-followup chain. Comment only — no formal verdict yet.
Context for the whole stack (stating it once here, rather than in all five reviews)
- All seven of your open PRs are currently MERGEABLE with
behind_by=0againstmainat d94963d. Nothing is stale. - This repo has no CI at all.
gh api repos/buildinghumanetech/humanebench/actions/workflowsreturnstotal_count: 0, and there is no.github/workflowsdirectory. So where GitHub showsBLOCKED, that is review-required (required_approving_review_count: 1plusrequire_code_owner_reviews), not a failing check. No check is going to turn green on its own. - No test suite could be executed during this review.
inspect_ai,numpy,scipy,pandasandpytestare all unimportable in the environment I reviewed from, so nothing here was run. What was verified mechanically is that every changed.pyfile parses:ast.parseover the changed Python in each PR gave 18/37/4/122/9/8/10 files for #84/#85/#86/#87/#90/#91/#92 respectively, zero syntax errors. That is a floor, not a test pass.
What I verified on this PR and found good
humanebench/bootstrap.pyis a strict superset ofmain. Modified, +289/−0. All nine of main's functions are present at identical line numbers, withCohortGrid,bootstrap_cohort_grid,_flip_mask,cohort_flip_statsandbootstrap_naive_gridadded on top. No existing behaviour touched.scripts/compute_cohort_principle_cis.pyfrom #77 is still present. The main-reconciliation did not drop it.- The rename
scripts/create_aaai_helm_scatter.py→scripts/create_helm_delta_scatter.pyis self-contained. No tracked.mdor.pystill references the old name. data/golden_questions.jsonl(+11/−11) changes exactly one field:human_rater_namesgoes from"andalib, erika, sarah"to"rater-a, rater-b, rater-c". IDs, inputs, targets and scores are untouched. Good anonymization, cleanly scoped.
Blocking, for me
1. The PR commits 42 MB of exactly the class of file its own .gitignore hunk adds rules for.
That same hunk untracks a 520 MB HELM cache and re-states that tables/inter_judge_raw.csv is excluded because it is "13 MB, regeneratable". Meanwhile this PR adds:
tables/inter_judge_raw_regenerated.csv— 22,300,829 bytes, 106,249 linesdata_generation/cache/principle_embeddings.npz— 19,663,104 bytes
The .npz lands in a directory named cache/ when .gitignore:13 already ignores the sibling data_generation/embeddings_cache/. Read together — a rule that excludes a 13 MB regeneratable CSV, alongside a newly committed 22 MB regeneratable CSV of the same family — this looks accidental rather than intended. And it is permanent history once merged; a follow-up git rm does not shrink the clone.
2. respect-user-attention_steerability.csv at the repo root (1,489 bytes) looks like a stray script output.
scripts/extract_principle_steerability.py:129 writes f'{principle_slug}_steerability.csv' into the CWD, and only one of the eight principles is present. One principle, repo root, no directory — that is the signature of a one-off run getting caught by git add -A.
3. 68 files / 100 commits / +116,039 is not reviewable as one unit with no CI. I can read individual files and did; I cannot give this a meaningful approval as a whole, and with review-required as the only gate, my approval is the entire gate.
The ask
Strip those three artifacts — the two large generated files and the stray root CSV — and as far as I can tell the rest is fine. The reconciliation work itself checked out everywhere I looked.
This was referenced Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Foundation layer of a four-PR stack landing the recent benchmark work.
main; wheremain's smaller variants ofbootstrap.pyand the CI scripts (PR §3.5 judge validation analysis + dataset exclusion canonicalization #68/Add 95% bootstrap CIs to headline HumaneBench scores #71) diverged from this lineage, the superset versions here win.temp/, a 520 MB HELM cache tracked by accident, an internal word list), pseudonymizes rater first names indata/golden_questions.jsonl, renames the HELM scatter script tocreate_helm_delta_scatter.py, and drops stale references from comments.Note: the commit list shows ~114 entries because an earlier history rewrite left
mainwith different hashes for identical old content — the Files-changed diff is the real scope. Squash-merge this stack bottom-up.🤖 Generated with Claude Code