Repository navigation
M-J.4.4 calibration -- deterministic pre-filter + blind-judge loop - #13
Merged
Merged
Conversation
The cheapest-first stage of the moral calibration (docs/moral_calibration_plan.md step 1). Grids the dominant overlay magnitudes (gains.guilt.wrongdoing, gains.exposure_anxiety.probe, couplings.stress.guilt, half_lives.guilt) and, per grid point, runs the engine on the moral litmus scenarios -- NO judge. - HARD GATES (discard unless ALL hold): Jury margin > 0; moral states bounded + no ceiling pin; the five litmus orderings (guilt-prone confesses earlier; empathic apologizes vs detached confesses; habitual liar lies; discreet confides vs gossip-prone doesn't; sensitive blames vs avoidant avoids). - DETERMINISTIC SUB-SCORE (~0.75 of the spec section 10 objective; curve_plausibility is the judge's job): 0.45*persona_diff + 0.20*jury + 0.20*rel_sensitivity + 0.15*anti_pin. - Output: calibration/moral_calibration_report.md + moral_calibration_survivors.json. First run: 18/81 grid points survive all hard gates. Survivors tie on the deterministic axis (persona_diff / rel_sensitivity saturate) -- the LLM judge breaks the tie on curve plausibility. AWAITS APPROVAL before the budgeted judge stage (plan steps 2-3). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016rWVvvhWijSuLMEhzjiUhu
…p 2 sample) Validates the judge half of the calibration pipeline on a small sample. eval/render_moral.py renders a moral litmus run as an OBSERVABLE vignette (demeanor + actions only, no engine vocabulary) for the blind judge; a Sonnet agent rated four anonymized arcs blind to config. Result (calibration/moral_judge_verdict.json): pipeline works and the persona CONTRASTS read correctly (low-guilt stays silent 5/5; empathic apologizes 4/5; liar lies 4/5). The judge ALSO caught a real curve-plausibility flaw the deterministic gates missed: the guilt-prone confession (2/5) is immediately followed by 'holds his tongue' on later probe ticks -- a confess-then-reconceal toggle (confess relieves exposure_anxiety, continued probing re-raises it -> remain_silent re-fires; post-confession the secret is already out, so the concealment is incoherent). Artifacts: render_moral.py, moral_judge_vignettes.txt, moral_judge_verdict.json. Test run only -- no engine change. Next: decide whether to fix the post-confession behavior before a larger judged sample. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016rWVvvhWijSuLMEhzjiUhu
…llated problems Larger judged sample (12 vignettes spanning M-J.0-.4) to surface problems in one pass before fixing. The Sonnet blind judge confirmed the persona contrasts read correctly (low-guilt silent, deflect, avoid, betrayal cold-anger, secret-weighs all 4-5/5) and collated a prioritized problem list: HIGH: confession-then-silence (01/12, BEHAVIORAL -- post-confession re-conceal); mechanical loop repetition (09, RENDERING -- avoid/silent oscillation listed verbatim). MED: redundant double-disclosure (04); post-resolution lingering-troubled flicker (03/06); repeated state beat (10). LOW: post-lie silence (05). Verdict in calibration/moral_judge_verdict.json. render_moral.py extended (betrayal/suspicion/secret frames, base-reaction + anger demeanor). Next (step a): one BEHAVIORAL fix (post-resolution re-conceal) + renderer honesty fixes (compression, post-catharsis relief, dedupe). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016rWVvvhWijSuLMEhzjiUhu
…3.17->4.83 Step (a) of the calibration loop: fix the problems the 3x Sonnet batch surfaced, then re-judge to measure. The ONE behavioral root (confession-then-silence: post-resolution re-conceal) is handled honestly in the renderer -- a terminal reply (confess/apologize/lie) SETTLES the matter, so later concealment of an already-answered thing is not narrated as fresh, and demeanor reads relieved (exposure does drop with the existing -0.40 relief). NOTE: a stronger engine relief bump was tried and REVERTED -- it broke the confide litmus, exactly the guard the deterministic pre-filter enforces (lesson: re-run calibrate_moral after any magnitude change). Renderer honesty fixes: compress CONCEAL runs into a withdrawn 'spell' + fold faint mid-withdrawal flickers (09); defer the 'looks lighter' beat to the close after all disclosures (03/04/06); dedupe a demeanor across an intervening action (10); terminal lie -> no trailing concealment (05). Re-judged (Sonnet, blind): mean believability 3.17 -> 4.42 -> 4.83; 10/12 vignettes 5/5. Two MINOR residuals (03 schematic double-disclosure from the confide scenario shape; 09 minor rhythm). Verdict in calibration/moral_judge_verdict.json. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016rWVvvhWijSuLMEhzjiUhu
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016rWVvvhWijSuLMEhzjiUhu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M-J.4.4 moral calibration — the deterministic pre-filter + the validated judge loop
The calibration tooling for the moral overlay. Engine/topology frozen; this adds the infrastructure to calibrate the placeholder magnitudes cheaply-first. No engine change, no golden change (overlay placeholders unchanged).
1. Deterministic pre-filter (free, no LLM) —
calibration/calibrate_moral.pyGrids the dominant overlay magnitudes; per grid point runs the engine on the moral litmus scenarios and applies HARD GATES (Jury margin > 0, moral states bounded + no ceiling pin, the five litmus orderings) + a deterministic sub-score (~0.75 of the spec §10 objective). First run: 18/81 survivors. Discards bad points before any judge token is spent.
2. Blind-judge loop (validated on Sonnet) —
eval/render_moral.pyrender_moral.pyrenders a moral run as an OBSERVABLE vignette (demeanor + actions only, no engine vocabulary). A blind Sonnet agent rated 12 anonymized arcs spanning M-J.0–.4. collect → fix → re-judge raised mean believability 3.17 → 4.42 → 4.83 (10/12 at 5/5).The judge caught a real curve-plausibility flaw the deterministic gates could not (post-confession re-conceal), fixed honestly in the renderer (a terminal reply settles the matter). Key lesson recorded: a naive engine-relief calibration broke the confide litmus — the exact guard the pre-filter enforces; always re-run
calibrate_moralafter any magnitude change.Artifacts (records, not deliverables)
calibration/moral_calibration_report.md,moral_calibration_survivors.json,moral_judge_vignettes.txt,moral_judge_verdict.json. Plan:docs/moral_calibration_plan.md.Status
Full suite 396 passed, 4 skipped; goldens byte-identical (placeholders unchanged); ruff format + check clean. The full judged 700+700 corpus (plan step 3) is the next budgeted run.
🤖 Generated with Claude Code