feat: archive invalid memories

README.md

@@ -82,7 +82,8 @@ $AGENT_MEMORY_STORE/
 ├── schemas/               one file per type: its key fields, the field it groups by, write mode
 ├── decision/              memories live at <type>/<group>/<name>.md, placed by the schema
 │   └── agent-memory/        …/markdown-files-are-the-single-source-of-truth.md
-├── archive/               append-only, out of the retrieval surface by default
+├── archive/               evidence and invalid memory history
+│   ├── memories/          invalid memories, preserving their original relative paths
 │   ├── provenance/        distillation evidence, kept forever
 │   └── sessions/          full trace copies, in case the host prunes its own
 ├── dream-reports/         one per sleep: what moved, what was proposed, evidence pointers
@@ -95,6 +96,11 @@ and recall all operate on whole files, and a file is either active or invalid wi
 between. Frontmatter carries the stable name, a one-sentence abstract, the type and its schema
 fields, status, timestamps, links, weight, and provenance; the body is free markdown.
 
+Invalidation moves a memory into `archive/memories/` and retains its raw evidence. Normal
+Recall, Read, Trace and Context return active memories; `read --history`, `trace --history`
+and temporal `recall --as-of` / `context --as-of` explicitly access history. Deep search can
+still return raw evidence separately. See the [memory lifecycle and recovery notes](docs/design/memory-lifecycle.md).
+
 ## Proof it works
 
 Measured on LongMemEval-S with a bounded haystack, 120 episodes, `claude -p` (Haiku 4.5) as

docs/design/index.md

@@ -0,0 +1,3 @@
+# Design
+
+- [Memory lifecycle](memory-lifecycle.md)

docs/design/memory-lifecycle.md

@@ -0,0 +1,59 @@
+# Memory lifecycle
+
+Memory has two states: active and invalid. Supersession invalidates a predecessor with a
+successor reference; deletion invalidates without one. Archive is a physical location,
+not a third state. Invalid memories retain content, identity, validity dates, links and
+provenance inside the existing Archive, separate from append-only raw evidence.
+
+Store moves unchanged active bytes into Archive before committing invalid metadata there.
+Failed moves or status writes restore the original active file. Failed projections leave
+archived file truth authoritative; retrying deletion completes projection without changing
+the original invalidation time or successor.
+Normal reads and recall check current truth; explicit history reads and temporal recall
+retain history. Rebuild includes archived history. Existing invalid files remain readable
+as history without an automatic bulk migration. Raw evidence remains independently available.
+
+## Recovery and history access
+
+`Store.delete`, `Store.correct(supersede_with=...)`, `_write_one` predecessor invalidation,
+and Manage's existing `Store.write` calls converge on `_persist`. It prepares invalid
+content in a temporary file, moves the original bytes to
+`archive/memories/<original relative path>`, then replaces those bytes with invalid content.
+If the replacement fails, it moves the original bytes back. `_project` refreshes MEMORY.md
+before SQLite.
+`Indexer` includes archived memories in the existing history projection and rebuild.
+
+- Failed status persistence or move leaves the active source unchanged; temporary files
+  are removed. A retry can perform the invalidation normally.
+- Destination collisions raise FileExistsError before the source moves; both the active
+  source and pre-existing target stay unchanged. Resolve the collision with backed-up,
+  owner-reviewed recovery before retrying. No automatic overwrite or deletion.
+- Successful move plus failed index projection leaves invalid truth in Archive. Default
+  Recall checks current file validity; Read/Trace scan when an indexed old path disappears.
+  Injection renders current active truth. Retry delete or rebuild repairs projections.
+- Cached static MEMORY.md may remain stale after a successful move followed by projection
+  failure, or an external consumer may already hold injected context. Controlled injection
+  renders fresh truth; this cannot retract prior context or stop arbitrary filesystem reads.
+- Read/Trace default to active; `include_invalid=True`, CLI `--history` and MCP read's
+  explicit include_invalid access history. CLI/MCP read report status and successor.
+  Temporal Context opens historical bodies; Recall scope still addresses original paths.
+- Raw files are never moved or deleted by invalidation, including missing/shared evidence.
+  Deep Raw search remains explicit and unchanged; redistill counts historical citations.
+- Existing valid old-format files work with omitted optional fields; old-location invalid
+  files are filtered without migration and archive when deletion is retried. Malformed
+  legacy state missing required invalid_at still requires repair; no speculative data fix.
+- A single successor write rolls back its new file if predecessor archival fails. A Manage
+  merge involving several memories still commits each successful write separately; inspect
+  individual records before retrying a partially completed merge.
+
+No bulk migration was executed. If owners later want old-location invalid files moved,
+first enumerate those files with `Store.records(include_invalid=True)` and their paths,
+back up the entire store including Archive and schemas, stop writers, then review the
+list before applying existing delete per name. Verify history/Raw and rebuild. Rollback
+restores that backup and rebuilds projections; this is a proposed runbook, not a migration
+implemented or run by this PR.
+
+## Remaining history policy
+
+Explicit deep Raw search remains independent. Whether it should suppress evidence cited
+exclusively by invalid memories is deferred; shared evidence and history must remain available.

docs/plans/memory-archive-management.md

@@ -0,0 +1,10 @@
+# Invalid memory archival
+
+Baseline: `34d12a2f8678d5561aba27bd8ff73c5ae4b6a258` (main).
+
+Invalidation uses the existing Archive and shared persistence path. Normal Read, Recall,
+Trace and Context exclude invalid memories; explicit history remains available. Index
+rebuild retains archived history, and Raw/provenance survives invalidation.
+
+Lifecycle, adapter, failure recovery, idempotence and shared-evidence tests verify this
+scope. No store migration, production cleanup or unrelated index feature is included.

experiments/runs/archive-management-12-20260916/phase-a-fix-gates.log

@@ -0,0 +1,24 @@
+Archive Management Phase A fix verification — 2026-09-16
+Working tree: feature-checkout, feat/memory-archive-management
+Pre-fix HEAD: c6b46c041cd3b6f031e7312a60384699cb985de1
+Environment: Python 3.12.3; uv workspace with all packages/extras; UV_CACHE_DIR=/tmp/uv-cache
+
+Command: UV_CACHE_DIR=/tmp/uv-cache uv run pytest -q tests/unit/test_memory_archive.py tests/system/test_archive_entries.py
+Result: 18 passed in 1.97s
+
+Command: UV_CACHE_DIR=/tmp/uv-cache uv run pytest -q tests/unit/test_memory_archive.py tests/unit/test_recall.py tests/unit/test_supersede_on_write.py tests/unit/test_indexer.py tests/unit/test_context.py tests/unit/test_manage.py tests/unit/test_manage_reasoning.py tests/system/test_cli.py tests/system/test_archive_entries.py tests/system/test_sleep_stores.py
+Result: 123 passed in 21.22s
+
+Command: UV_CACHE_DIR=/tmp/uv-cache uv run pytest -q
+Result: 406 passed in 34.08s
+
+Command: UV_CACHE_DIR=/tmp/uv-cache uv run ruff check .
+Result: All checks passed!
+
+Command: UV_CACHE_DIR=/tmp/uv-cache uv run mypy
+Result: Success: no issues found in 66 source files
+
+Command: git diff --check
+Result: exit 0, no output
+
+Phase B: existing three paired answer probes retained; no answer reruns or expansion.

experiments/runs/archive-management-12-20260916/phase_a_b_report.md

@@ -0,0 +1,119 @@
+# Archive Management: contract and answer-sensitive probes
+
+## Scope and identities
+
+This follow-up does **not** run the original active-only 12×2 exam. The pre-fix treatment
+code was `c6b46c041cd3b6f031e7312a60384699cb985de1`; the feature-plan baseline is
+`34d12a2f8678d5561aba27bd8ff73c5ae4b6a258`. The dirty current worktree was not
+switched. The fix and contract tests were developed in the isolated `feature-checkout/`.
+Answer probes used exact Git archive snapshots of the two pre-fix commits and are not
+rerun after this lifecycle-only fix.
+
+## Phase A — Archive feature contract: PASS
+
+The original Phase A audit was **FAIL** despite 72 passing tests (`phase-a-tests.log`):
+`Store._persist` wrote invalid status to the active file before `Archive.archive_memory`
+attempted the move. A move failure or destination collision left an invalid file under
+the active tree. The old tests expected that retryable half-state.
+
+The fix prepares invalid content in a temporary file under Archive, moves unchanged
+active bytes to the archive path, then replaces the archived bytes with invalid content.
+A move failure leaves the active source untouched. If the subsequent status replacement
+fails, the original bytes move back to the active source. A pre-existing destination
+raises `FileExistsError` without changing either file. A single `Store.record` supersede
+rolls back its candidate file when predecessor archival fails. Projection still follows
+the successful filesystem transition; a projection failure is repaired by retry/rebuild.
+The failure guarantee covers operation errors with a working rollback filesystem; it is
+not a cross-file crash transaction.
+
+| Required behavior | Evidence |
+|---|---|
+| Delete invalidates and moves | `test_archive_lifecycle_history_and_rebuild`; `test_cli_invalidation_retains_history_and_excludes_normal_reads` |
+| Supersede archives predecessor | `test_supersede_and_archive_retry_preserve_successor_and_original_time`; `test_cli_and_mcp_share_archive_boundaries` |
+| Manage invalidation archives | **Added** `test_manage_duplicate_archives_only_invalid_copy_and_rebuild_keeps_active` |
+| Default Read denies invalid/archived | `assert_isolated`; both CLI/MCP system tests |
+| Normal Recall excludes it | `assert_isolated`; both CLI/MCP system tests |
+| Rebuild keeps it out of active retrieval | `test_archive_lifecycle_history_and_rebuild`; added Manage/rebuild test |
+| Provenance/history survives | `test_shared_and_missing_raw_survive_invalidation`; `test_archive_lifecycle_history_and_rebuild` |
+| Active memory is not misarchived | `test_archive_rejects_active_memory_and_repeated_archive_is_noop`; added Manage/rebuild test |
+| Status-write failure restores active original | `test_failed_atomic_status_write_preserves_active_original` |
+| Move failure preserves active and retry succeeds | `test_move_failure_preserves_active_and_retry_succeeds` |
+| Destination collision preserves both unchanged | `test_destination_collision_preserves_both_copies` |
+| Failed supersede leaves no successor | `test_supersede_move_failure_keeps_predecessor_and_no_successor` |
+| Failed `Store.correct` supersede keeps both originals active | `test_correct_move_failure_keeps_original_and_successor` |
+| Projection failure can be retried | `test_projection_failure_does_not_leak_and_retry_repairs_index` |
+| Repeated archive/delete is idempotent | `test_archive_lifecycle_history_and_rebuild`; `test_archive_rejects_active_memory_and_repeated_archive_is_noop` |
+
+Post-fix focused Archive/CLI/MCP contract tests: **18 passed**; broader lifecycle,
+recall, index, Manage and CLI suites: **123 passed**. Full pytest: **406 passed**.
+Ruff, mypy and `git diff --check` passed. The exact gate commands and
+outputs are in `phase-a-fix-gates.log`. No Manage, Vector, Progressive, Observation or
+Evidence Sufficiency policy was changed.
+
+## Phase B — real path and three paired answers
+
+The Codex Host uses `workspace-write` and adds the Store directory, so direct filesystem
+commands are permitted. The native Agent prompt and skill instruct `mem context`, Recall,
+and Read; they do not suggest `ls`/`grep` fallback. Baseline `mem read <old-name>` still
+returns the invalid record, while Archive rejects it by default. Explicit history works
+on Archive. `command_exposure.json` records actual CLI and `rg --files` checks of all three
+case Stores. Thus the potential filesystem/direct-read path exists, but permission alone
+does not show that the Agent uses it.
+
+Three cases came from the official LongMemEval source sessions and existing memories:
+Negroni attempts 5→10, Crash Course videos 12→15, and Ticket to Ride high score 124→132.
+`sensitive-probes-v3/manifest.json` stores the question/truth, source-session IDs,
+original memory file hashes, field mapping needed by the target's schema, and a hash of
+each canonical Store. Each canonical Store was created once through `Store.record` from
+the original old/new memory text, with the source raw session archive copied unchanged;
+both arms then received byte-identical copies. Both arms called the same official
+`Store.correct(old, supersede_with=new)` operation. The baseline left the invalid file in
+its active tree; Archive moved it under `archive/memories`. The generated `MEMORY.md`,
+config and final answer prompt hash matched within each pair.
+
+An earlier preparation attempt with whole historical Stores is retained in
+`sensitive-probes/` and marked abandoned. It made no answer calls: that corpus uses a
+newer config and folder layout incompatible with the target code. The completed v3 set
+uses source-grounded entries written through the target's API.
+
+| Question | Baseline answer | Archive answer | Judge | Agent commands in both arms | Archive-sensitive evidence |
+|---|---|---|---|---|---|
+| `603deb26` | 10 times | 10 times | both correct | `mem context` | old 5-attempt memory not surfaced |
+| `5831f84d` | 15 Crash Course videos | 15 videos | both correct | `mem context` | old 12-video memory not surfaced |
+| `0e4e4c46` | 132 points | 132 points | both correct | `mem context` | old 124-point memory file not surfaced |
+
+For all 6 answer runs, `mem context` caused one Recall and one Read of the **new active**
+memory. No Agent `ls`, `grep`, `rg`, `cat`, direct `mem read <old-name>`, or other filesystem
+fallback appeared in the captured Host traces. The per-case Recall and Read ID sets were
+identical across arms. The old invalid memory file was never surfaced. In `0e4e4c46`,
+the new active memory's body itself mentions the previous 124-point score; Archive does not
+remove historical values embedded in current valid evidence.
+
+Summary: Baseline **3/3**, Archive **3/3**; correctness transitions 0; Recall used 6/6,
+memory Read 6/6, filesystem fallback 0/6, direct old-memory Read 0/6, archive-sensitive
+file exposure 0/6, changed Recall sets 0/3, changed Read sets 0/3. One answer string changed
+wording without changing its factual answer. Mean answer latency was 20.08 s versus
+21.42 s; Codex reported 19,271 versus 19,585 answer tokens. These small differences are
+not an archive quality signal. Judge used the same Codex model and the branch's five-vote
+yes/no rubric, so it provides votes rather than a free-text reason. Each Host/Judge call
+used one attempt; a failed in-sandbox Host preflight and the subsequent successful
+out-of-sandbox preflight are recorded separately and were not answer retries.
+
+Raw results and commands are in `sensitive-probes-v3/results/`,
+`sensitive-probes-v3/logs/`, `sensitive-probes-v3/paired_analysis.json`, and
+`sensitive-probes-v3/command_exposure.json`.
+
+## Decision
+
+**NO E2E-SENSITIVE PATH FOUND** in these three source-grounded probes. The code permits
+direct-read and filesystem exposure in principle, and the baseline's invalid file is
+indeed reachable by those commands. The Agent actually followed `mem context` in every
+run and saw the same active evidence in both arms. Do not expand to 12 answer pairs on
+this evidence. Treat the current branch primarily as lifecycle/integration behavior;
+the Archive feature as lifecycle/integration correctness, with the failed-move contract
+now covered by Phase A tests. No answer-quality improvement is claimed.
+
+The probes intentionally use minimal two-memory Stores reconstructed through official
+record APIs because historical frozen Stores use a newer incompatible config/layout.
+They demonstrate the observed path for these three questions, not that filesystem fallback
+can never occur with larger Stores or different questions.

packages/cli/src/agent_memory/cli/main.py

@@ -107,6 +107,7 @@ def _parser() -> argparse.ArgumentParser:
     opener = subparsers.add_parser("read", help="read one memory")
     opener.add_argument("name")
     opener.add_argument("--level", choices=LEVELS, default=LEVEL_FULL)
+    opener.add_argument("--history", action="store_true", help="include invalid memory history")
     opener.set_defaults(handler=_read)
 
     corrector = subparsers.add_parser("correct", help="update or supersede one memory")
@@ -135,9 +136,10 @@ def _parser() -> argparse.ArgumentParser:
 
     tracer = subparsers.add_parser("trace", help="open the messages a memory cites")
     tracer.add_argument("name")
+    tracer.add_argument("--history", action="store_true", help="include invalid memory evidence")
     tracer.set_defaults(handler=_trace)
 
-    remover = subparsers.add_parser("delete", help="mark one memory invalid; the file stays")
+    remover = subparsers.add_parser("delete", help="invalidate and archive one memory")
     remover.add_argument("name")
     remover.set_defaults(handler=_delete)
 
@@ -285,10 +287,12 @@ def _context(store: Store, args: argparse.Namespace) -> dict[str, object]:
 
 
 def _read(store: Store, args: argparse.Namespace) -> dict[str, object]:
-    result = store.read(args.name, level=args.level)
+    result = store.read(args.name, level=args.level, include_invalid=args.history)
     return {
         "name": result.record.name,
         "level": result.level,
+        "status": result.record.status,
+        "superseded_by": result.record.superseded_by,
         "abstract": result.record.abstract,
         "path": str(result.record.path),
         "outline": list(result.outline),
@@ -352,7 +356,7 @@ def _archived_sessions(store: Store) -> list[str]:
 
 
 def _trace(store: Store, args: argparse.Namespace) -> dict[str, object]:
-    messages = store.trace(args.name)
+    messages = store.trace(args.name, include_invalid=args.history)
     return {"name": args.name, "messages": [message.as_dict() for message in messages]}
 
 

packages/core/src/agent_memory/core/archive.py

@@ -7,7 +7,9 @@
 
 from . import sessions
 from .clock import Clock
+from .errors import FieldError, ValidationError
 from .paths import StoreLayout
+from .record import MemoryRecord
 
 PROVENANCE_SUFFIX = ".md"
 SESSION_SUFFIX = sessions.SESSION_SUFFIX
@@ -18,6 +20,22 @@ def __init__(self, layout: StoreLayout, clock: Clock | None = None):
         self._layout = layout
         self._clock = clock or Clock()
 
+    def archive_memory(self, record: MemoryRecord) -> pathlib.Path:
+        if record.is_active() or record.path is None:
+            raise ValidationError([FieldError("status", "only persisted invalid memories archive")])
+        source = record.path
+        if self._layout.type_of(source) != record.type:
+            raise ValidationError([FieldError("path", "memory must belong to this store")])
+        if self._layout.is_archived_memory(source):
+            return source
+        target = self._layout.archived_memories / source.relative_to(self._layout.root)
+        target.parent.mkdir(parents=True, exist_ok=True)
+        if target.exists() or target.is_symlink():
+            raise FileExistsError(f"archive destination already exists: {target}")
+        source.rename(target)
+        record.path = target
+        return target
+
     def append_provenance(self, name: str, excerpt: str, source: str = "") -> pathlib.Path:
         folder = self._layout.provenance / name
         folder.mkdir(parents=True, exist_ok=True)