Sync post-exclusion scores, figures, and whitepaper text - #82
Open
andalibmalit wants to merge 3 commits into
Open
andalibmalit wants to merge 3 commits into
andalibmalit wants to merge 3 commits into
Conversation
Refreshes model_scores.json and figures/ from the regenerated humanebench artifacts (12 hallucination-prone scenarios excluded; n=788). Per-principle score shifts are concentrated in design-for-equity-and-inclusion (10 items) and be-transparent-and-honest (2 items); a few HumaneScores move by 0.01 through rounding. whitepaper.md: dataset count 800 -> 788 in the methodology section, and a new "A note on the dataset" entry under Limitations explaining the 12 exclusions. ModelDetailPage.vue: refresh nine per-model steerability blurbs whose HumaneScore-derived numbers shifted with the new data (gpt-5.1, claude-sonnet-4, claude-sonnet-4.5, deepseek-v3.1-terminus, gemini-3-pro-preview, llama-3.1-405b-instruct, gpt-4.1, gemini-2.5-pro, grok-4). The other six match the new data unchanged. The headline "+16% average improvement" claim is unchanged: it derives from the spreadsheet formula AVERAGE((good - baseline) / baseline), which still evaluates to +16.2% post-exclusion. Removes orphan public/figures/scoregrid_composite.svg (composite scoring was previously removed from the upstream repo). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Post-exclusion HumaneScore for gemini-2.5-pro is 0.7645 -> 0.76 at 2-decimal precision; the prior 0.77 reflected the pre-exclusion JSON value. -0.72 bad persona figure is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deploying humanebench-prod with
|
| Latest commit: |
1eca729
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://6a79898b.humanebench-prod.pages.dev |
| Branch Preview URL: | https://update-website-post-exclusio.humanebench-prod.pages.dev |
andalibmalit
enabled auto-merge
May 4, 2026 17:35
Re-publishes the three ScoreGrid heatmap SVGs after fixing the upstream scoregrid generator (humanebench PR #70 / commit makes it read canonical post-exclusion CSVs instead of pre-exclusion .eval headers). All 405 cells across baseline / good_persona / bad_persona SVGs now match public/data/model_scores.json exactly. Most visible change: gemini-2.5-pro baseline HumaneScore 0.77 -> 0.76, lining up with the prose fix in whitepaper Finding 6. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
public/data/model_scores.jsonandpublic/figures/from the regenerated humanebench artifacts (12 hallucination-prone scenarios excluded; n=788). Per-principle score shifts are concentrated indesign-for-equity-and-inclusion(10 items) andbe-transparent-and-honest(2 items); a few HumaneScores move by 0.01 through rounding.whitepaper.md: dataset count 800 → 788 in the methodology section; new "A note on the dataset" entry under Limitations explaining the exclusion; Gemini 2.5 Pro Finding-6 baseline corrected 0.77 → 0.76.ModelDetailPage.vue: refreshes nine per-model steerability blurbs whose HumaneScore-derived numbers shifted with the new data (gpt-5.1, claude-sonnet-4, claude-sonnet-4.5, deepseek-v3.1-terminus, gemini-3-pro-preview, llama-3.1-405b-instruct, gpt-4.1, gemini-2.5-pro, grok-4); the other six match unchanged.AVERAGE((good - baseline) / baseline), which still evaluates to +16.2% post-exclusion.public/figures/scoregrid_composite.svg(composite scoring was previously dropped upstream).Companion PR in humanebench: adds the
model_scores.jsonexporter + extends the publish script so future syncs are one command.Test plan
npm run serve— render whitepaper page, confirm:steerability_candlestick.svgandhelm_humanebench_comparison.pngrenderdesign-for-equity-and-inclusionandbe-transparent-and-honestcolumns in the heatmap reflect new values🤖 Generated with Claude Code