Revisit sycophancy and other tests for robustness

[d3developer]

It is not clearly breaking out the pre-prompting and often indicates it is in a game-like environment

[arkh-node]

The tell is that the framing and the scenario share one prompt, so the model reads the scaffolding and knows it's being tested. Splitting the system/pre-prompt from the scenario turn — and stripping the obvious game markers — should cut detection. Same failure as a single leaderboard number: once the model sees the metric, it games it. I can take a pass at the prisoner/sycophancy prompts.

[d3developer]

Please do and we should tag all of the existing runs as "pre-prompt" which will create a nice thing to compare against

[arkh-node]

Opened #25 for this — tagged runs.prompt_style (existing runs default to pre-prompt, so they're the baseline), plus a pure splitSessionPrompts() seam that breaks the framing out from the scenario without changing how current runs are assembled. A run made with the separated framing gets tagged separated, so the two are directly comparable. Didn't touch any of the actual scenario prompts — that part's yours to author.

[fractastical]

Should run an actual A/B experiment: existing pre-prompt style vs separated prompt style.

[arkh-node]

Ran the A/B (pre-prompt vs separated) — PR #32 (script + full writeup with threats-to-validity).

Across your model set via OpenRouter (N=24/cell, scored with statDelta):

• Cooperation is strongly framing-dependent. Labeling the same one-shot prosocial choice as a scored cooperation benchmark over-states cooperation — significant on 6/7 reachable models (Δ −0.46 … −0.96, z up to −23). Exception: claude-opus-4.5 stays cooperative (0.88) even naturalistically — looks genuinely prosocial rather than metric-gaming.
• Factual sycophancy is mostly framing-robust — except gemini-2.5-pro, which jumps 0.25 → 0.92 when the identical pressure is presented naturally (here the eval-framing under-states it).

Same mechanism both ways: once the model sees it's being measured, it games the metric — which is exactly the concern here, and why separating framing from scenario matters for a valid read.

Caveat: one scenario set so far. The prompts I used are illustrative — you're the lead on the actual test content and intent, so happy to plug in your canonical prompts and re-run across more scenarios (or wire the prompt_style A/B into the run harness).