kir — keep it real

A schema-level deferral channel for concise LLM dialogue.

Research code, data, and pre-registered protocols for the paper Keep It Real: A Schema-Level Deferral Channel for Concise LLM Dialogue.

The idea in one paragraph

Chat models monologue. The standard explanation is training pressure toward length; the hypothesis tested here is narrower and more actionable — a model dumps everything because nothing survives the turn. Give the unsaid thing a guaranteed place to live and brevity becomes safe. Concretely: every reply is structured output with two channels — say (what the user sees) and park (a private one-line list of deferred threads that persists across turns and can be resurfaced later). No fine-tuning, no decoding changes, no length penalty. Only the shape of the output schema changes.

What we found

1. The deferral channel alone cuts length, with no cap enforced. 400 conversations, 3 model families, 5 scenarios, 10 seeds per cell, pre-registered analysis:

Model	Scenarios supported	Words/turn reduction
claude-sonnet-5	5 / 5	41–54%
gpt-5.6-sol	3 / 3	31–42%
glm-5.2	1 / 3	up to 30% (one scenario slightly longer)

2. The mechanism is permission, not storage. A placebo park — the prompt claims the list persists, but contents are silently discarded and the injected list is always empty — reproduces the full effect. The model shortens because it believes it can come back to things, not because it actually can. (50 conversations, pre-registered as §12/P12 before running.)

3. Humans confirm the felt difference, but not as a preference. Blinded pairwise study, n = 46 Prolific raters, pre-registered H1–H4:

• "Which one dumped too much on you?" → 86% name the verbose arm. 95% CI [0.79, 0.92], excludes chance. This is the headline.
• "Which would you rather talk to?" → 51.5%, CI [0.44, 0.59]. A coin flip overall, splitting by genre (concise wins on advice, loses on explanation).

The honest claim is therefore "people feel less dumped on" — not "shorter is better." Both results are reported at equal prominence.

Reproducing

uv venv .venv --python 3.12
uv pip install -p .venv/bin/python -r requirements.txt
# ANTHROPIC_API_KEY (and OPENROUTER_API_KEY for the sweep) go in .env

.venv/bin/python tests_offline.py                     # 165 offline checks, no API key
.venv/bin/streamlit run harness.py                    # interactive two-pane harness
.venv/bin/python run_sweep.py --phase all             # the 400-conversation sweep
.venv/bin/python analyze_sweep.py                     # pre-committed C1–C5 only
.venv/bin/python analyze_study2.py --self-test        # human-study analysis

Every sweep conversation is committed under sweep/ as raw JSONL plus a transcript — the full record, not just the aggregates.

How this repo is organized

The protocols were frozen before each experiment ran, and the commit history is the evidence: predictions are filed in one commit and scored in a later one. Nothing was decided after seeing the data; where a gap appeared mid-run, it was handled by a dated amendment committed before proceeding.

File	Job
BRIEF.md	Ground truth: theory, settled decisions, open dilemmas, amendments A1–A15.
FINDINGS.md	The lab notebook: every session, its numbers, and the predictions ledger P1–P13 with each prediction scored right or wrong.
SWEEP-PLAN.md	Frozen replication protocol: model matrix, scenarios, seeds, operational rules, pre-committed analysis.
STUDY2-PLAN.md	Frozen human-study protocol: blinded pairwise design against the known rater length-bias, mechanical pair selection, H1–H4.
PRIOR-ART.md	Two fact-checked prior-art sweeps. The parts exist; the whole doesn't.
PAPER.md	The route to publication and the credibility checklist.
paper/	Manuscript source, figures, and the arXiv package.

park/            the mechanism (UI-agnostic)
  core.py        turn loop: cap -> call -> format check -> commit
  schema.py      two-channel output schema (field order is an experiment variable)
  caps.py        dialogue-act regex -> sentence budget
  store.py       thread persistence, decay, dedupe
  prompts.py     the system prompt
sweep/           400 conversations: raw JSONL + transcripts + analysis.json
study2/          human-study instrument, stimuli, and analysis
logs/            one JSONL per interactive session

Data availability

Sweep and placebo data are complete and committed. Raw human-study responses are not published — they carry Prolific participant identifiers. The committed study2/analysis.json contains the full derived results, and analyze_study2.py reproduces every reported number from a raw export.

Limitations

Reduction is measured in words per turn on 10-turn simulated dialogues, not on real users over long horizons. Coverage is measured by an LLM judge. The preference result is a null. glm-5.2 shows the effect is not universal across model families. Full list in the paper's Discussion.

Citation

See CITATION.cff.

License

Code is MIT (see LICENSE). The manuscript text and figures under paper/ are CC BY 4.0.

Acknowledgments

Built with substantial assistance from Claude Fable 5 and Claude Opus 4.8; the extent of that assistance is disclosed in the paper.