feat: add provenance-based raw evidence reading - #28
Open
faj-design5260 wants to merge 4 commits into
Open
faj-design5260 wants to merge 4 commits into
faj-design5260 wants to merge 4 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Latest update (2026-09-18)
tools/read_exposure_gate.pyto verify copied indexes, a known retrieval query,and nonempty reads through each intended path, while detecting incomplete observations
and direct store access during a small pilot.
CI, Ruff, mypy, and 68 relevant local tests pass on
81dd43b3. The next evaluation stepis a fresh 12-question pilot with these checks before expanding the sample or interpreting
the score comparison.
Problem
A Memory can retain the right source while omitting a detail needed by a later question.
The existing read path exposes the Memory but provides no bounded way to inspect the exact
messages it cites.
Change
mem --json read <name>exposes the Memory's provenance without expanding Raw.mem --json trace <name> --pointer 'sessions/<session>#<start>-<end>'reads one cited rangeor a valid subrange; tracing without a selector reads all cited sources.
Memory validity metadata, and legacy provenance support.
is sufficient, otherwise inspect its bound evidence.
The harness can now select the tested host and judge independently. Claude Code remains the
default judge with its historical model and voting behavior. Codex judging runs ephemerally in
a read-only sandbox and accepts only the final-message file. Optional
--observe-readsevidencecaptures bounded exam, host-attempt, Recall, Read, Trace, empty-result, and error events without
changing CLI results or store truth. Run metadata fixes both host/model pairs, config, source
stores, episode identity, and code revision.
Compatibility
Store.trace()list return type remain compatible.--judge-hostand--observe-readsretains the existing Claude evaluation path.Validation
malformed or missing Raw, legacy formats, historical Memory, symlinks, and read-only behavior.
Codex votes, transport failures, run identity, opt-in bounded evidence, and observation-off
byte equivalence.
gpt-5.6-soljudge agreed with all 25 labelled calibration cases.readfollowed bytrace --pointerand recovered thecited value.
Base:
origin/mainat34d12a2f8678d5561aba27bd8ff73c5ae4b6a258.