Skip to content

feat(ingest): extract Records from completed Codex, Kimi and Pi turns (#369) - #375

Merged
entelecheia merged 9 commits into
mainfrom
feat/369-ingest-records
Nov 11, 2026
Merged

entelecheia merged 9 commits into
mainfrom
feat/369-ingest-records

Conversation

@entelecheia

@entelecheia entelecheia commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Context

amem ingest <codex|kimi|pi|qwen> wrote deterministic capture rows without parsing Records, so the SessionStart brief showed decisions and fixes from Claude sessions only (#313 item 7). This change runs the existing Record parser over each reader's positively completed final assistant text and stages the result through the existing staging, receipt and replay path.

Replaces task 220. Person authorized #369, then #368, then #370.

Change

  • amem ingest <codex|pi|kimi> extracts canonical Record rows from complete final assistant text only, reusing parseRecordLines with the existing staging, receipt, replay, screening, anchor and project-binding path. Subagent frames, tool output, malformed lines and partial turns never yield Records.
  • Positive completion evidence per reader, as the native files actually record it: Codex phase=final_answer with a main SessionMeta, Pi stopReason=stop, Kimi a main-agent step.end whose finishReason is end_turn. No fixture invents a finish field.
  • Qwen and Jcode stage no Records: their pinned native formats carry no completion marker, so their candidates are counted and refused as finality-unavailable, and their ordinary capture, Title and Compact behaviour is unchanged. Their AC1 coverage stays open by ruling.
  • recordsEnabled(base, cwd) gates extraction; with records: false ingest behaves as before.
  • amem ingest --json gains an additive per-reader records receipt (supported, found, staged, duplicatesSkipped, refused by reason). No new ingest kind or CLI surface is added.
  • Non-Claude Record rows publish as format: 4 under hook-v4-, the same envelope, receipt and dedupe path. A running 0.24.0 matches only hook-v[23]-, so it leaves those files pending instead of retiring them as corrupt; the new reader drains formats 2, 3 and 4. All existing Claude and native publication bytes are unchanged and no format floor is written.
  • No cursor reset, no history backfill and no new cursor state: extraction skips offsets behind an existing source cursor, and a Kimi step split across ingest windows is refused as finality-unavailable.

Review round 1 (2026-11-11)

All three important findings are fixed at f61261b, with README.md and docs/contracts.md naming the same values at e16dff6.

  • I1, Kimi finality. Real Kimi 1.5 wire files complete a step with finishReason: "end_turn" and never with "completed", so the gate refused every real Kimi Record. Counted over 669 local agents/main/wire.jsonl files: end_turn 348, tool_use 7026, error 37, interrupted 4, completed 0. The reader now accepts end_turn and refuses the other three exactly as before. test/ingest-records.test.ts covers all four values, and the fixture's default completion value is now the native one.
  • I2, Codex header screen. session_meta was screened as a whole line, and a real header carries about 21.8 KB of payload.base_instructions, above the standalone screen's 16 KiB single-string bound, so the screen refused the header, the session scope became unknown, and every Record in the file was refused. The header line is never appended, and the reader consumes only identity and scope fields, so headerScreen now screens exactly those, both in the main-header gate and in the per-line scope gate. A test writes a 21,866-byte base_instructions with a git remote and still stages its Record.
  • I3, jcode false capture-loss notice. A jcode Record refused as finality-unavailable was written as a capture drop, and the SessionStart notice filtered only reasons starting with record_, so every later brief showed capture dropped: finality unavailable (N). One predicate, recordRefusalReason in src/hook-record.ts, now names Record refusals for both the metrics and the notice; the metric itself is unchanged. Covered in test/hook-record-instruction.test.ts, which also keeps a real loss (spool_full) visible.
  • N1 (nit), acknowledged, not fixed. The ingest drain gets a fresh replay budget per turn and drains oldest first, so when a re-staged Record's spool file is only drained in a later turn, the appended row's already result arrives without this turn's staged map and the count reads staged instead of duplicatesSkipped. Dedupe itself holds and no row is duplicated; an honest count would need the staged receipts persisted across turns, which is not part of this round.

Acceptance criteria

  • AC1: valid, malformed and subagent Records from Codex, Pi and Kimi yield only the valid main-session Records. Covered per reader in test/ingest-records.test.ts and test/ingest.test.ts. Qwen and Jcode deliberately extract nothing and report the refusal (ruling in feat(ingest): extract Records from Codex, Kimi, Pi, Qwen and Jcode transcripts #369's second comment), so this criterion stays open for them.
  • AC2: re-running ingest adds no duplicate Record rows, and a duplicate native message identity in one window is reported as a duplicate.
  • AC3: records: false in the home or repository config disables extraction.
  • AC4: ingested Records appear in the SessionStart brief under the existing Record slot and byte caps.
  • AC5: amem ingest --json reports the per-reader counts.

Per the contract this keeps the issue open: Refs #369.

Verification

Review round 1 on the hidden remote runner, focused scope only, no heavy run on the Mac:

  • f61261b: install, tsc --noEmit clean, boundary 173 source files, the eight Records-focused files (116 pass / 0 fail), the registry-integrity-verified published 0.24.0 hook-v4 compatibility test (1 pass), exit 0.
  • e16dff6 (docs only): check-docs OK with the four pre-existing stale stamps, readme-facts OK, tsc --noEmit clean, exit 0.
  • Hosted CI covers the full suite at the pushed head.

Earlier full pipeline at 10e8beb (base 6f9ade6): the nine Records-focused files (112 pass / 0 fail / 909 assertions), the published-0.24.0 hook-v4 test (1 pass), the full suite (3732 pass / 1 fail / 27 skip across 341 files) and npm pack readiness. The single failure was a healthy second-process writer can finish while a synchronous reader waits for its short intent in test/public-prune-live-advance.test.ts, a file this branch does not touch, with public prune view unavailable; it passed in the previous run and 15 pass / 0 fail on three consecutive isolated runs at that head, so it is read as a load flake rather than a regression and is not claimed green in the full suite.

Two full-suite failures that predated the last fix commit, and were invisible because the pipeline used to stop at the focused stage, are corrected in this branch: the ingest JSON fixture now carries the additive records receipt, and the spool compatibility fixture simulates its unknown newer format as 5 now that 4 is real.

Rulings honored

  • hook-v4 only for non-Claude Record rows; no Claude or native format bytes change.
  • No Records from Qwen or Jcode; counted as finality-unavailable.
  • No cursor reset and no backfill.
  • A Kimi step split across ingest windows is refused as finality-unavailable, with a ponytail note and a test.
  • No format floor write.

PendingHookCapture.backlog reports that older project work was pending around the
publication, not that this call added nothing, so a brand new Record published
behind a backlog was counted as a duplicate. That is why the Codex and Kimi
backpressure tests reported staged 0 / duplicates 2 instead of staged 1 /
duplicates 1 after the resume.

Set a transient duplicate witness only in publishCapture's identical existing
body branch and drive the record counters and map.fresh from it. Spool envelope
bytes, nonce, receipt and backlog scheduling semantics are unchanged.

Refs #369
…ixtures

Two assertions in the full suite had never run on this branch, because the
verification pipeline stopped at the focused stage. The ingest JSON fixture is
compared as exact compact bytes and now carries the additive records receipt,
and the spool compatibility fixture simulated an unknown newer format as 4,
which this branch made real, so the unknown format moves to 5.

Both failed with the pre-fix sources as well, so neither is a regression of the
duplicate witness change.

Refs #369
Copilot AI balanced review requested due to automatic review settings November 11, 2026 08:23
@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-11T08:23:14.918779Z
We couldn’t complete this review. Comment @codex review to try again.
a32bebb PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Kimi 1.5 ends a completed step with finishReason "end_turn"; 669 local wire
files never carry "completed", so every real Kimi Record was refused as
finality-unavailable. Accept the native value and refuse tool_use, error and
interrupted as before.

A Codex session_meta carries about 21.8 KB of base_instructions, above the
standalone screen's 16 KiB single-string bound, so screening the whole header
line refused the session and every Record in it. The header line is never
published and the reader consumes only identity and scope fields, so those are
what the screen judges now.

A jcode Record refused as finality-unavailable was recorded with a drop reason
that does not start with record_, so every later SessionStart brief showed a
false "capture dropped: finality unavailable" notice. One predicate now names
the Record refusals for metrics and for that notice.

Refs #369
The extraction sentences described a "completed step", which does not name the
value the reader checks. Real Kimi 1.5 wire files complete with end_turn, a Codex
header is screened as its identity and scope fields because base_instructions is
above the screen's single-string bound, and a Record refusal never reaches the
capture-loss notice.

Refs #369
@entelecheia
entelecheia merged commit fb23782 into main Nov 11, 2026
9 checks passed
@entelecheia
entelecheia deleted the feat/369-ingest-records branch November 11, 2026 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants