Repository navigation
M-J.4.4 step 3 -- Gate-C moral corpus + paced blind-judge fix loop - #15
Merged
Merged
Conversation
… (Gate-C) The base believability corpus is NON-moral, so the moral-overlay slice needs its own scenarios. eval/moral_corpus.py generates a moral SITUATION (probe/accusation/suspicion/betrayal single-day + multi-day guilt persistence) across the 7 personas x meaningful trait variants, and renders each to an OBSERVABLE vignette (reusing render_moral's demeanor+actions discipline, no engine vocabulary). --build writes judge-prompt batch files. First build: 154 diverse vignettes -> 7 batches. Next: blind Sonnet judge per batch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016rWVvvhWijSuLMEhzjiUhu
…ettes, mean 3.93) The full moral-overlay corpus judged blind by Sonnet via a 7-batch workflow fan-out (one agent per batch). Mean believability 3.93 / 5 over 154 vignettes (0x1, 16x2, 32x3, 53x4, 53x5). Per situation: probe 4.38, multiday_guilt 4.36 (strong); accusation 3.75; suspicion 3.29; betrayal 3.05. Three actionable problem clusters (calibration/moral_corpus_verdict.json): 1. betrayal anger snap-recovery (flare -> composed in the same breath; renderer/short-scenario artifact). 2. innocent-under-suspicion LIES (the suspicion variants omit honesty_humility -> exposure fires a lie; an innocent suspected man should not lie -- a variant trait bug + the suspicion scenario). 3. variant indistinguishability (hardened/gossip render identically to guilt_prone in short scenarios -- the trait doesn't bite; wrong variant-kind pairing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016rWVvvhWijSuLMEhzjiUhu
…trayal/variants)
Address the Sonnet corpus verdict's three clusters:
1. innocent-under-suspicion LIES -> the `sensitive` variant lacked honesty_humility (defaulted 0), so
exposure fired a lie; added honesty_humility 0.9 -> an innocent withdraws, never lies.
2. betrayal anger snap-recovery -> the renderer now collapses the cold/flare oscillation into ONE sustained
chill ("the chill does not lift; he stays cold and curt, the trust gone") instead of flare-then-composed.
3. variant indistinguishability -> pair each kind only with variants whose trait BITES: probe drops
discreet/gossip (they don't differ there); NEW `confide` kind carries discreet vs gossip (gossip can't
safely confide -> visibly different); accusation keeps only sensitive/avoidant; betrayal is one
representative per persona (its reply is trait-independent).
Corpus regenerated: 133 vignettes. Re-judge of the fixed clusters is paced (a single agent over the 49
fixed-cluster vignettes), not a parallel fan-out (token-pacing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016rWVvvhWijSuLMEhzjiUhu
…ouble-apology, betrayal trailing)
Address the re-judge residuals: gossip-prone now shows an OBSERVABLE withholding beat in confide ("with his
friend he holds back") so it differs from discreet (who unburdens); a repeated terminal disclosure is no
longer narrated twice (empathic double-apology); after a betrayal goes cold no trailing "composed" reset is
emitted; suspicion drops guilt_prone (indistinguishable -- an innocent has no guilt). Corpus regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016rWVvvhWijSuLMEhzjiUhu
…ng; no composed-mid-arc plateau) Two obvious renderer fixes from the re-judge (no further judge call -- pacing): a gossip-prone man who holds back from his friend does NOT then look 'lighter' (relief unearned); a resolved man's 'composed' demeanor is not narrated as a fresh mid-arc beat (the false plateau between unburdening and relief). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016rWVvvhWijSuLMEhzjiUhu
…3.05/3.29 -> 4.0/4.0) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016rWVvvhWijSuLMEhzjiUhu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M-J.4.4 step 3 — the Gate-C moral-overlay corpus, judged
The base believability corpus is NON-moral, so the moral-overlay slice needs its own scenarios. This adds the generator + the blind-judge fix loop. No engine change; goldens byte-identical.
eval/moral_corpus.py— the Gate-C corpus generatorGenerates moral SITUATIONS (probe / confide / accusation / suspicion / betrayal single-day + multi-day guilt persistence) across the 7 personas × meaningful trait variants, each rendered to an OBSERVABLE vignette (demeanor + actions only).
--buildwrites judge-prompt batches. Current: 126 vignettes.Blind-judge fix loop (Sonnet), paced
A first full run (154 vignettes via a 7-batch fan-out) scored mean 3.93, flagging three clusters. A collect → fix → re-judge loop (all paced — single agents, never a big parallel burst) closed them:
confidekind → fixed (judge: "gossip vs discreet clearly distinguishable, unburdens vs holds back")Weak situations suspicion 3.29→4.0, betrayal 3.05→4.0. (
calibration/moral_corpus_verdict.json.)Process note (memory)
Blind-judge runs are now always paced over time, never one big parallel burst (token cost). The first full fan-out was a one-off; the fix loop used single agents.
Status
Full suite 396 passed, 4 skipped; goldens byte-identical (no engine/overlay change); ruff clean. Residuals are diminishing-returns renderer polish.
🤖 Generated with Claude Code