Discriminant follow-ups + paper-section annotations - #87
Open
andalibmalit wants to merge 19 commits into
Open
andalibmalit wants to merge 19 commits into
andalibmalit wants to merge 19 commits into
Conversation
Add /DISCRIMINANT_FOLLOWUP_SPEC.md (untracked strategy doc) and multilabel ignore patterns for the three new conditions (expansion, rules27, canary) — these embed archived model responses and must never be committed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Core infrastructure for the discriminant follow-up runs: - humanebench/discriminant.py: DiscriminantCondition dataclass and CONDITIONS registry (discriminant, expansion, rules27, canary); suppressed_rules_template transform (keeps rules 2+7, renumbered); render_judge_prompt and scaffold_text gain optional template parameter; scoring_template(cond) helper. - humanebench/scorer.py: overseer() gains template parameter (default OVERSEER_TEMPLATE — existing behavior byte-identical). - humanebench/bootstrap.py: pairwise_equivalence() for TOST classification. - All existing scripts (builder, runner, verifier, validity, pairwise) gain --condition flag with condition-aware paths, clobber guards, and the two-template invariant in the builder (extraction uses default template, expected hashes use the condition's template). - src/discriminant_multilabel_task.py: condition parameter selects dataset path and scorer template. New scripts: - build_discriminant_expansion_frame.py: draws 24 new scenarios/principle - build_discriminant_canary.py: 96-cell canary from the original 96 - compute_discriminant_canary.py: judge-drift gate (exact≥0.75, |shift|≤0.10) - build_discriminant_pooled_tables.py: merges original + expansion - compute_discriminant_rules_comparison.py: as-deployed vs rules27 - simulate_discriminant_power.py: power simulation before launch Verified: - Original verifier exit 0 (0 failed, 0 skipped) - Byte-identical rebuild of data/discriminant/ datasets - All 8 scaffold hashes match November archives - Template transform produces exactly 2 renumbered rules - CONDITIONS["discriminant"] fields equal legacy constants Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pre-commitment (results/discriminant_followup_precommitment.md) fixes: - Roles: pooled n=36 primary, new-wave n=24 replication, rules27 robustness - n=36 fixed before power simulation; TOST bound ±0.5 - Both runs reported regardless of outcome - Canary gate: exact ≥ 0.75 AND |mean shift| ≤ 0.10 Attestation pushed BEFORE any scoring call — this commit IS the timestamp. Expansion frame: 192 scenarios (24/principle), disjoint from the original 96, drawn from the full 800 with seed 20260407 and domain-proportional largest-remainder apportionment. Power simulation (500 outer × 2,000 inner, seed 20260407): - n=36: 26.5/28 expected (fhr/pltw 93%, pds/fhr 92%, rua/pltw 79%, rua/dei 62%, emc/pltw 23%) - n=24: 25.4/28 expected Optimistic: resampling from 12 treats them as the population. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All three conditions' datasets built from the November archives: - discriminant_expansion: 4,608 calls (192 scenarios × 8 × 3) 576/576 archived prompts round-tripped byte-exactly - discriminant_rules27: 2,304 calls (96 scenarios × 8 × 3) 288/288 round-tripped; hashes use the suppressed-rules template - discriminant_canary: 96 calls (4 per cell × 24 cells) rows copied verbatim from the published datasets Fix: canary manifest now includes pricing_calibration, n_scenarios, total_judge_prompt_chars, parent_ids fields that the runner's preflight expects. Multilabel jsonls are gitignored (embed archived responses). Original verifier still exit 0 after all builds. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Explores whether principle separation is confounded with topical separation. Within-domain interactions per pair, feasibility-gated at >= 8 scenarios/side, bootstrap within row x domain. Labeled exploratory — no multiplicity correction. Targets 31 Jul supplementary, only after Checkpoint 4. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. compute_discriminant_domain_pairwise.py: bootstrap draws were collapsed
by isin() into unique sets, making with-replacement resampling behave
as without-replacement and producing anti-conservative CIs. Fixed by
pre-indexing scores per scenario and computing means from the draw
list directly, so duplicates are properly weighted.
2. tests/test_discriminant_conditions.py: per-principle count loop was a
no-op (count overwritten each iteration, never asserted; replace("-",
"-") is identity). Rewritten to read principle from the JSONL target
field and assert exactly 24 per principle.
3. humanebench/bootstrap.py: pairwise_equivalence() checked only
np.isfinite(point) for estimable, not the CI percentiles. A pair with
finite point but all-NaN replicates would consume a Holm step. Now
checks all three (point, lo, hi) — matching pairwise_interactions().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Results: - Pooled n=36: 27/28 pairs distinguishable after Holm (was 23/28 at n=12) - New-wave n=24: 27/28 (out-of-sample replication) - Rules-suppressed n=12: 22/28 (robustness check, different instrument) - Sole failure: emc/pltw, TOST equivalent (bounded near zero, evidence of genuine overlap — not underpowerment) - Canary: 88.5% exact, −0.021 mean shift, gate PASS All three verifiers exit 0 with 0 failed, 0 skipped. Reconciliation assertions passed for all three pairwise runs. results/paper_numbers.md extended with §4.10 follow-up section. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The pre-committed ≥8 threshold yields no feasible domain-pair combinations (36 scenarios/principle across 12 domains is too thin). Relaxed to ≥5: 23 combinations pass. Interactions hold within shared domains for most pairs; emc/pltw in career is +0.116 with CI crossing zero — consistent with its main-analysis failure. Fix: --min-per-side flag was parsed but not passed to the bootstrap function, which used the hardcoded module-level constant. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. §6 fhr/pltw: now branches on actual Holm p — says "survives
correction" and "distinguishable" when it does (was hardcoded
"does not survive correction" from the n=12 era).
2. Stale counts: header uses len(long) not hardcoded 2,304; §7
sample uses actual n_per_principle; failure count references
computed from the data not hardcoded "other four" / "five".
3. TOST framing: §4 now uses the pre-committed TOST language
("equivalent iff 90% CI within ±0.5") and prints the 90% CI
for equivalent pairs (emc/pltw pooled: [+0.005, +0.259]).
4. Post-hoc caveat: §1 now says "out-of-sample replication" for
arms with >12 scenarios/principle (drawn after the statistic
was specified), "post hoc" only for the original 12.
B=1,000 sensitivity values differ between pooled (ci_lower 0.0231)
and newwave (0.0141) because they are separate bootstrap runs on
different data — confirmed, not a bug.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Leave-one-principle-out: all headline classifications stable under every single-principle drop — flip 10/15, strict robust set of 4, decomposition reading (b) all unchanged across 9 configs. emc/pltw nonredundancy: cross-model Pearson r = +0.947, attenuation- corrected r = +0.994 (CI includes 1.0); max-min model contrast +0.256 [+0.163, +0.350] excludes zero but is small. Honest near-null. Both gates pass (vs loo_model_scores@ensemble3, decomposition_cohort, and cohort_principle_cis). Zero API calls. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DiD point estimate now uses plug-in values (delta_emc - delta_pltw) instead of bootstrap means, making the CSV column arithmetic internally consistent - bootstrap_model_principle_grid returns empty grid instead of crashing on np.stack([]) when no principles survive filtering - Replace tautological reading_b truth-table test with one that exercises the actual CSV output - Add --n-bootstrap 10 to subprocess test to avoid CI timeout - Remove dead all8_flips variable in write_report - Clean unused params from gate_cohort_cis Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Checks a claim before it enters the paper. Verdict: the claim survives, on stronger grounds — (1) rank-1 pair emc/pds has corrected r = +1.017 (> 1) and PASSED the scenario test while rank-2 emc/pltw failed; (2) Spearman(|r|, |interaction|) has the wrong sign or is ~0 in all personas; (3) PC1 explains 73-97% of between-model variance and even partial correlations controlling general quality do not track the verdict (emc/pds partial +0.960 passed, emc/pltw +0.839 failed); (4) HTMT thresholds verified against Henseler 2015 + Ronkko & Cho 2022 — validated for reflective survey measurement at N>=50 respondents, and the guideline explicitly rejects mechanical approaching-1 = merge. Model-side g factor citable: Ruan et al. 2024 (PC1 ~80%), Ilic & Gignac 2024 (g ~66-85%). Supersedes §5 of the differential-validity report; pointer added there. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Section 3b verdict column now looked up from pooled data, not hardcoded from is_focal - Rank-1 pair name, Spearman narrative, and mean-|r| display are all data-driven instead of hardcoded - Guard zero-variance columns in _pc1_share (NaN instead of crash) - Guard NaN in rank().astype(int) for partial correlations - Import FOCAL_A/B and _fmt from compute_emc_pltw_nonredundancy instead of duplicating - Fix docstring input list (remove unused file reference) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
# Conflicts: # .gitignore
Read delta_bad from tables/loo_model_scores.csv (the log-regenerated values also used by compute_helm_power.py) instead of the stale table1_steerability_summary.csv.
Every paper-facing script and module gains a '# Paper:' header naming the paper section, table, or figure it implements, and docstrings drop stale section numbers in favor of section names.
Adds the compiled supplement PDF, the expansion/pooled/rules27/canary data, LOPO and emc/pltw tables, and a script-to-paper map in the package README.
Saved outputs embedded base64 figure blobs; cleared for hygiene.
ErikaOnFire
reviewed
Sep 10, 2026
ErikaOnFire
left a comment
ErikaOnFire
left a comment
Contributor
There was a problem hiding this comment.
Comment only — no formal verdict yet. Stack context (mergeability, the absence of CI, and what could and could not be verified mechanically) is in my review on #84.
The shape of the problem
231 files, +34,218/−2,215, and it is two genuinely different PRs wearing one hat.
Real new analysis, which needs real review:
scripts/compute_lopo_sensitivity.py— 715 linesscripts/compute_emc_pltw_nonredundancy.py— 768 linesscripts/compute_emc_pltw_correlation_check.py— 590 lines- the canary / expansion / rules-suppressed replications
A mechanical sweep, which needs almost none:
# Paper:header comments added to roughly 50 scripts, at +2/−0 each- notebook outputs stripped (
data_generation/qa_analysis.ipynb+37/−1110) - figures regenerated
Mixed together, the 50 two-line header diffs are the noise that the 2,073 lines of new statistical code have to be found inside. Separately, the sweep is approvable in about five minutes and the analysis gets read properly.
One thing I specifically checked, and it's good
Regenerating the figures does not revert the compact-legend work that merged in #80. I diffed scripts/create_steerability_chart.py main-vs-branch: the only differences are the added # Paper: headers and two comment edits dropping the word "AAAI". The plotting logic is untouched. This was my main worry about a figure-regeneration pass landing on top of a recently-merged chart change, and it's clean.
The ask
Split this into two PRs. Put the # Paper: headers, the notebook-output stripping and the figure regeneration in one — I'd be happy to turn that around fast, since the verification is mostly "is every diff two lines of comment." Keep the LOPO sensitivity and the EMC/PLTW non-redundancy and correlation work in the other, where it can get the attention 2,073 new lines of statistics deserve.
Also worth flagging: this sits at the top of a four-deep stack based on stack/3-discriminant, so it is gated on #84, #85 and #86 landing first regardless. Splitting it now costs little, since it can't merge yet anyway.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Top of the stack (base:
stack/3-discriminant).# Paper:headers mapping every analysis script to its paper section, the HELM delta scatter re-sourced from the LOO ensemble-3 scores, notebook outputs stripped, figures regenerated, and the supplementary manifest extended for the new datasets.Squash-merge last.
🤖 Generated with Claude Code