Skip to content

feat: add premise-aware adaptive memory reads - #35

Open
faj-design5260 wants to merge 4 commits into
mainfrom
feat/evidence-sufficiency-gate
Open

faj-design5260 wants to merge 4 commits into
mainfrom
feat/evidence-sufficiency-gate

Conversation

@faj-design5260

@faj-design5260 faj-design5260 commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Problem

Relevant retrieval cannot help when an agent does not enter the memory path. A first relevant Memory may also leave a requested fact or relationship unsupported.

Design

  • A host-facing premise check distinguishes tasks that plausibly require prior state from self-contained tasks.
  • Memory-dependent tasks use focused L0 Recall and selective full Read, with at most two Recall rounds and four full Reads.
  • The host checks fact-level sufficiency and may issue one targeted follow-up query for a missing answer slot. It stops with an explicit unresolved answer if evidence remains insufficient.
  • The master recall.adaptive_read_enabled switch turns the new protocol and this PR's evidence guidance off together. Off restores the mainline agentic exam policy at the same code revision.
  • mem recall --round initial|follow-up only labels optional observation events; it does not change retrieval results.

Compatibility

No new model client, Write or Manage change, ranking/index change, or Memory schema change. Vector and the independent Raw Evidence PR remain separate. The generated skill and agentic exam share one authoritative policy source; fixed exams are unchanged.

Validation

453 tests passed; total coverage 93%; Ruff and mypy passed. The two-case mechanics smoke observed real initial Recall and full Read on both cases and a tagged follow-up Recall on the multi-fact case. All 12 paired Store copies passed index/Recall/full-Read preflight and post-run truth-hash checks.

12 × 2 E2E

At one clean code revision, using the same fixed 12 questions, canonical Store truth, Codex gpt-5.6-sol Host and independent Judge, and identical settings except the master switch:

Baseline (off) Treatment (on)
Correct 11/12 5/12
Recall exposure 12/12 11/12
Full Read exposure 11/12 9/12
Tagged follow-up Recall 0 4
New full Read after follow-up 0 0
Median exam latency 43.56 s 48.15 s

Paired outcomes: 0 wrong→right, 6 right→wrong, 5 same-correct, 1 same-wrong. The treatment passed the prespecified exposure gate, but its follow-up rounds never expanded full-read evidence. This run does not support an E2E accuracy improvement. The baseline already entered Memory for this subset.

Flip analysis

The six regressions comprised a related-fact transfer, a count/time-scope synthesis error, two questions whose exact evidence was in archived session material reached only by the baseline's deep path, a temporal relation left unresolved after repeat follow-up hits, and a six-event chronology where the bounded read set omitted needed evidence. See the privacy-preserving per-flip summary.

Limits and privacy

This is one fixed 12-case live-model replay, not a broad accuracy estimate. Interruptions and timeouts caused retries; first successful attempt per question and arm, in file order, was selected independently of correctness. Reliable token counts were unavailable. No claim is made about Vector or the Raw Evidence PR.

The public PR contains only implementation, tests, and an aggregate/per-flip sanitized report. Raw Stores, questions, gold answers, session transcripts, and Host observation logs are retained locally for owner audit and are not committed or pushed.

The older 2026-09-14 three-arm evidence-gate run is invalid for feature-effect attribution because copied Store indexes were empty and there was no actual Memory exposure. Its scores are not used here.

@faj-design5260 faj-design5260 changed the title feat: add evidence sufficiency guidance for memory answers feat: add premise-aware adaptive memory reads Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant