Skip to content

feat: add provenance-based raw evidence reading - #28

Open
faj-design5260 wants to merge 4 commits into
mainfrom
feat/memory-raw-evidence-read
Open

faj-design5260 wants to merge 4 commits into
mainfrom
feat/memory-raw-evidence-read

Conversation

@faj-design5260

@faj-design5260 faj-design5260 commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Latest update (2026-09-18)

  • Reused stores now project their Memory and Raw indexes before an exam begins.
  • Added tools/read_exposure_gate.py to verify copied indexes, a known retrieval query,
    and nonempty reads through each intended path, while detecting incomplete observations
    and direct store access during a small pilot.
  • Added regression tests and documented the pilot checks for read-side comparisons.

CI, Ruff, mypy, and 68 relevant local tests pass on 81dd43b3. The next evaluation step
is a fresh 12-question pilot with these checks before expanding the sample or interpreting
the score comparison.

Problem

A Memory can retain the right source while omitting a detail needed by a later question.
The existing read path exposes the Memory but provides no bounded way to inspect the exact
messages it cites.

Change

  • mem --json read <name> exposes the Memory's provenance without expanding Raw.
  • mem --json trace <name> --pointer 'sessions/<session>#<start>-<end>' reads one cited range
    or a valid subrange; tracing without a selector reads all cited sources.
  • Trace preserves source session, original message index, role, time, per-message reference,
    Memory validity metadata, and legacy provenance support.
  • Missing, inconsistent, unbound, path-traversing, and redirected sources fail explicitly.
  • Normal Recall and Read remain memory-only, and Trace does not mutate truth or usage state.
  • The agentic exam and generated skill use the same memory-first policy: stop when the Memory
    is sufficient, otherwise inspect its bound evidence.

The harness can now select the tested host and judge independently. Claude Code remains the
default judge with its historical model and voting behavior. Codex judging runs ephemerally in
a read-only sandbox and accepts only the final-message file. Optional --observe-reads evidence
captures bounded exam, host-attempt, Recall, Read, Trace, empty-result, and error events without
changing CLI results or store truth. Run metadata fixes both host/model pairs, config, source
stores, episode identity, and code revision.

Compatibility

  • Text Read output and the Python Store.trace() list return type remain compatible.
  • Historical Memory access keeps its existing explicit-name behavior and is labelled as such.
  • The new selector cannot access evidence outside the chosen Memory's provenance.
  • Omitting --judge-host and --observe-reads retains the existing Claude evaluation path.

Validation

  • CI, Ruff, and mypy pass on the latest commit; 68 relevant local tests pass.
  • Raw-evidence tests cover distill-to-trace, multiple and overlapping sources, subranges,
    malformed or missing Raw, legacy formats, historical Memory, symlinks, and read-only behavior.
  • Harness tests cover independent host/judge selection, historical Claude defaults, isolated
    Codex votes, transport failures, run identity, opt-in bounded evidence, and observation-off
    byte equivalence.
  • Codex gpt-5.6-sol judge agreed with all 25 labelled calibration cases.
  • A live synthetic Codex smoke observed read followed by trace --pointer and recovered the
    cited value.

Base: origin/main at 34d12a2f8678d5561aba27bd8ff73c5ae4b6a258.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant