Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,3 +193,19 @@ The task lifecycle and the invariants a change must not break are in [CLAUDE.md]
## License

[MIT](LICENSE).

### Read evaluation with Codex

The experiment runner selects the tested host and judge independently. Pass
`--host codex --judge-host codex` and explicit `--model` / `--judge-model` values
for a Codex-only run. Omitting `--judge-host` retains the Claude Code judge and
its historical default model. `calibrate` and `regrade` also accept `--judge-host`.
Use `calibrate --cases <labelled-cases.json> --output <calibration.json>` to retain
individual votes and distinguish transport failures from label disagreements.

`run --observe-reads` retains bounded exam host output and CLI/read evidence in
`observations/`, outside store truth. Observation is off by default; missing or
truncated evidence is not proof of no tool calls. `run.json` fixes both host/model
pairs, configuration, source stores, code revision and episode identity. Replay
with `--reuse-stores` and a separate workspace for each configuration. Small panels
check execution and exploratory behavior, not a statistically established improvement.
17 changes: 17 additions & 0 deletions docs/design/evidence-sufficiency.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Premise-aware adaptive read

The host assesses whether answering plausibly requires prior personal, project, or session
state. Self-contained tasks retain the ordinary answer path. For memory-dependent tasks the
host searches, selectively reads full entries, and checks support for the precise requested
facts and relationships. Partial evidence warrants one focused search for the missing facts;
unresolved specifics remain explicitly unknown. Search and full reads have fixed budgets.

The shared prompt module owns this policy for the native agentic exam and generated skill.
RecallConfig's master switch disables both adaptive guidance and the earlier evidence gate,
recovering the mainline exam policy for paired comparisons. Configuration fingerprints record
the switch and budgets. Fixed exams retain their existing separate framing.

This is host guidance, not an enforced runtime controller or model judge. Passive observation
records command rounds and results without modifying retrieval. Retrieval ranking, write,
manage, lifecycle and raw storage retain their behavior. Tests prove delivery and ablation;
answer quality requires paired replays at one revision over indexed copies of identical truth.
Loading
Loading