Capture Claude/Codex harness events as Basic Memory producer envelopes

[phernandez]

Summary

Add a safe, opt-in "harness WAL" path for the Claude Code and Codex plugins that normalizes supported hook/transcript events into Basic Memory producer envelopes and writes useful agent-memory artifacts.

Design note: memory://basic-memory-7020de4e925843c68c9056c60d101d9e/basic-memory-llc/product/ideas/harness-wal-capture-river-ingestion

Why

Claude Code and Codex sessions generate a stream of useful work context: prompts, tool calls, file changes, tests, blockers, decisions, handoffs, and compaction points. Today we capture some high-signal checkpoints, but the broader session stream still mostly evaporates.

The idea is to treat the harness output like an append-only event log:

Claude/Codex hook event
  -> normalized producer envelope
  -> durable inbox/raw event record
  -> coalesced artifact writer
  -> SessionNote / ToolLedger / RecallQuery / DecisionNote candidates
  -> consolidation or memory_routine follow-up

This implements the producer side of SPEC-55 and feeds SPEC-61 memory routines without turning Basic Memory into an agent runtime.

Scope for v0

• Add a shared event/envelope helper for Claude Code and Codex plugins.
• Capture supported lifecycle hooks into a normalized envelope shape.
• Include source, session id, turn id when available, cwd, project hint, timestamp, hook name, and safe payload summary.
• Coalesce events into existing artifact types from SPEC-55, especially SessionNote and ToolLedger.
• Keep hooks fast; expensive summarization and consolidation should happen out of band.
• Add idempotency keys so repeated hooks do not duplicate notes.
• Add opt-in configuration and clear redaction controls.
• Emit provenance observations such as [source] codex/<session_id> or [source] claude-code/<session_id>.

Possible normalized event types

• session_started
• user_prompted
• assistant_message
• tool_called
• tool_result
• file_changed
• test_ran
• decision_observed
• blocked
• handoff_requested
• compaction_imminent
• session_ended

V0 should only capture events exposed through supported harness hooks and visible transcript artifacts. It should not try to capture private model reasoning.

Privacy and safety constraints

• Capture must be opt-in or clearly tied to plugin installation/configuration.
• Do not capture hidden chain-of-thought or private model reasoning.
• Prefer summaries, metadata, and selected visible turns over raw transcript dumps.
• Redact obvious secrets before writing artifacts.
• Provide deny rules for paths, tools, environment-like values, and large payloads.
• Preserve source offsets or hook event ids so artifacts can be audited without storing everything forever.
• Fail fast on missing Basic Memory project mapping rather than writing to the wrong project.

Related specs

• SPEC-55: Agent Memory Pipeline (Producer Framework)
• SPEC-61: Event-Driven Memory Routines
• SPEC-56: Memory Consolidation (Dream Mode)
• SPEC-74: Semantic Diff for Basic Memory Notes

Open questions

• Should raw envelopes be stored locally in an inbox first, or should plugin hooks write directly through existing Basic Memory MCP/API calls for v0?
• Which hook events are available and stable enough across Claude Code and Codex today?
• Should Codex and Claude plugins share one helper package/script, or duplicate thin adapters with a shared envelope schema?
• What is the minimum useful artifact: session checkpoint, tool ledger, recall query, or all three?
• How should users configure redaction and excluded paths?
• Should the first version only run on PreCompact and SessionEnd, or also capture PostToolUse events?
• Where should raw event retention live once the local daemon exists?

[claude]

Claude finished @phernandez's task in 1m 55s —— View job

---

Issue Triage

• Load tools
• Explore existing plugin/hook structure
• Check for duplicate issues
• Classify issue type, priority, component, complexity
• Apply labels
• Provide triage summary

---

Classification

Dimension	Assessment
Type	Feature request — new functionality
Priority	Medium — high strategic value but no existing functionality is broken
Component	Plugins (plugins/claude-code/, plugins/codex/), with cross-cutting impact on integrations/openclaw/
Complexity	Complex — architectural work across multiple plugin packages

Labels applied: enhancement

---

No Duplicates Found

Searched for "harness WAL", "producer envelope", "SPEC-55", and "hook events" — no prior issues. This is a new initiative.

---

Current State of the Harness

The existing hook landscape is minimal but real:

Claude Code plugin (plugins/claude-code/hooks/):

• SessionStart → session-start.sh — briefs Claude from the knowledge graph
• PreCompact → pre-compact.sh — checkpoints the session before compaction

Codex plugin (plugins/codex/hooks/):

• Same two hooks, with matchers (startup|resume|compact and manual|auto)

OpenClaw integration (integrations/openclaw/hooks/capture.ts):

• Already has a post-turn capture handler that indexes conversation turns as notes via client.indexConversation()
• Filters by minimum char count, extracts last user+assistant turn — this is a working example of the producer side

The issue is asking for a more structured, normalized version of this pattern shared across Claude Code and Codex (not just OpenClaw), with defined envelope types, idempotency, opt-in config, and redaction.

---

Complexity Assessment

This is a complex feature for the following reasons:

1. Shared helper across two plugin ecosystems: Claude Code uses shell scripts + bash; Codex uses Python hooks (pre-compact.py, session-start.py) and shell. A shared envelope schema that works across both (and possibly OpenClaw's TypeScript) requires careful design.

2. Open architecture questions: The issue itself lists ~7 open design questions (inbox vs. direct write, which hooks are stable, where raw retention lives, etc.) — these need resolution before implementation.

3. Privacy/redaction infrastructure: Deny rules for paths, tools, env-like values, large payloads is non-trivial and easy to get wrong.

4. New event types: 12 proposed normalized event types, most of which have no current hook support (e.g. PostToolUse, file_changed, test_ran) — availability depends on what Claude Code and Codex actually expose.

5. SPEC-55 / SPEC-61 dependencies: This is described as the "producer side" of SPEC-55 and feeds SPEC-61 routines. Implementation correctness depends on those specs being stable.

---

Recommended Scope for v0

Based on the existing code, a minimal v0 could be:

1. Share the envelope shape — a simple JSON schema (or markdown spec) defining the normalized envelope fields (source, session_id, turn_id, cwd, project, timestamp, hook_name, payload_summary)
2. Hook PreCompact only — both plugins already have this; extend the checkpoint scripts to emit a structured envelope alongside the existing session note
3. Reuse the OpenClaw capture.ts pattern — adapt it for the shell/Python hooks
4. Idempotency key = sha256(session_id + hook_name + timestamp) — write to a note only if that key isn't already present

This deferral of PostToolUse and the more exotic event types (file_changed, test_ran) is consistent with the "V0 should only capture events exposed through supported harness hooks" note in the issue.

---

Open Questions to Resolve Before Implementation

The issue lists several — the two most blocking are:

1. Inbox vs. direct write: Direct write through existing MCP calls is simpler for v0; an inbox adds durability at the cost of a new persistence layer. Recommend direct write for v0.
2. Hook availability: Claude Code exposes PostToolUse — confirm whether Codex does, and whether it's stable enough to depend on.

[sourrris]

I am working on this now.

[hegu-1]

This producer-envelope direction is very close to how I’d expect a personal memory vault to ingest agent work traces.

One design distinction I’d make explicit: the raw harness stream should not itself become “memory.” It is trace material. Durable memory should be a promoted artifact derived from that trace.

A useful pipeline shape could be:

raw_event          = append-only, short/medium retention, replay/debug
tracelet           = normalized visible event with provenance
session_note       = compressed continuity summary
tool_ledger        = what was called, why, and what changed
candidate_memory   = proposed durable fact/decision/preference
vault_memory       = user-owned, promoted, editable markdown source-of-truth

That keeps the vault from becoming a transcript landfill while still preserving enough evidence to audit how a memory was formed.

For v0, I’d consider making the envelope carry a few fields that support later promotion:

source_agent
source_tool
project_hint
visibility: private | project | public
event_type
provenance_ref
redaction_status
promotion_status: raw | summarized | candidate | accepted | rejected

The promotion_status field matters because agents should be able to propose memory, but not silently turn every observed session detail into durable user memory.

For personal vault usage, the ideal contract is: raw traces are available for audit; generated summaries are replaceable; authored/promoted vault notes are the source of truth. That distinction is what lets memory stay user-owned across Claude, Codex, and future tools.

I’ve been exploring a small Personal AI OS / Memory Vault pattern around this same boundary: continuity across tools without making any single agent runtime the owner of memory.

Related public projects:

• https://github.com/hegu-1/personal-memory-vault-starter
• https://github.com/hegu-1/personal-ai-os

[phernandez]

Direction update after two implementation rounds (#998, #1064), a survey of prior art (claude-hook-utils, codex-honcho, graphify, recall), and re-alignment with SPEC-55. The architecture in SPEC-55 stands; one call changes: the async producer front door moves into the bm CLI itself. Recorded as a dated revision on SPEC-55 (2026-07-15).

The revised shape:

• New bm hook <event> command group in core (basic_memory/hooks/): parses the harness hook JSON on stdin (per-harness adapters for Claude Code and Codex), builds the producer envelope, applies redaction, and writes it to the SPEC-55 inbox (~/.basic-memory/inbox/, atomic tmp+rename). No daemon required to capture — envelopes accumulate from day one.
• Plugins ship zero logic: hook manifests plus one-line shims. bm resolution order: BM_BIN → basic-memory/bm on PATH (preferred, keeps the hook consistent with the user's MCP server) → uvx "basic-memory>=<floor>" as the zero-install default, with the floor bumped by release tooling. uv is the documented per-platform prerequisite the plugin checks for. No vendoring, no shell heredocs — hook logic gets normal tests, typing, and releases in core.
• Envelope contract additions: actor, caused_by, and promotion_status (raw → summarized → candidate → accepted/rejected) so raw trace never silently becomes durable memory. Projected artifacts carry created_by + caused_by_event provenance alongside the required [source] observation.
• Phase 2a scope: capture for session-start / pre-compact, an interim deterministic projector (bm hook flush — SessionNote skeletons and ToolLedger entries, no LLM), and bm hook install / remove / status with surgical, ownership-tagged config merging per harness. Execution is fail-open (never disrupt a session); the capture gate is fail-closed (captureEvents must be boolean true). The SPEC-55 ingestion worker supersedes flush when the SPEC-54 daemon lands.

This supersedes PRs #998 and #1064. The envelope internals proven there — the contract shape, idempotency keying, recursive redaction, and the fail-closed capture gate — carry into basic_memory/hooks/ with commit credit to @sourrris. Implementation PR to follow.