Repository navigation
feat(ingest): extract Records from completed Codex, Kimi and Pi turns (#369) - #375
Merged
Merged
Conversation
PendingHookCapture.backlog reports that older project work was pending around the publication, not that this call added nothing, so a brand new Record published behind a backlog was counted as a duplicate. That is why the Codex and Kimi backpressure tests reported staged 0 / duplicates 2 instead of staged 1 / duplicates 1 after the resume. Set a transient duplicate witness only in publishCapture's identical existing body branch and drive the record counters and map.fresh from it. Spool envelope bytes, nonce, receipt and backlog scheduling semantics are unchanged. Refs #369
…ixtures Two assertions in the full suite had never run on this branch, because the verification pipeline stopped at the focused stage. The ingest JSON fixture is compared as exact compact bytes and now carries the additive records receipt, and the spool compatibility fixture simulated an unknown newer format as 4, which this branch made real, so the unknown format moves to 5. Both failed with the pre-fix sources as well, so neither is a regression of the duplicate witness change. Refs #369
# Conflicts: # CHANGELOG.md
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Kimi 1.5 ends a completed step with finishReason "end_turn"; 669 local wire files never carry "completed", so every real Kimi Record was refused as finality-unavailable. Accept the native value and refuse tool_use, error and interrupted as before. A Codex session_meta carries about 21.8 KB of base_instructions, above the standalone screen's 16 KiB single-string bound, so screening the whole header line refused the session and every Record in it. The header line is never published and the reader consumes only identity and scope fields, so those are what the screen judges now. A jcode Record refused as finality-unavailable was recorded with a drop reason that does not start with record_, so every later SessionStart brief showed a false "capture dropped: finality unavailable" notice. One predicate now names the Record refusals for metrics and for that notice. Refs #369
The extraction sentences described a "completed step", which does not name the value the reader checks. Real Kimi 1.5 wire files complete with end_turn, a Codex header is screened as its identity and scope fields because base_instructions is above the screen's single-string bound, and a Record refusal never reaches the capture-loss notice. Refs #369
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
amem ingest <codex|kimi|pi|qwen>wrote deterministic capture rows without parsing Records, so the SessionStart brief showed decisions and fixes from Claude sessions only (#313 item 7). This change runs the existing Record parser over each reader's positively completed final assistant text and stages the result through the existing staging, receipt and replay path.Replaces task 220. Person authorized #369, then #368, then #370.
Change
amem ingest <codex|pi|kimi>extracts canonicalRecordrows from complete final assistant text only, reusingparseRecordLineswith the existing staging, receipt, replay, screening, anchor and project-binding path. Subagent frames, tool output, malformed lines and partial turns never yield Records.phase=final_answerwith a mainSessionMeta, PistopReason=stop, Kimi a main-agentstep.endwhosefinishReasonisend_turn. No fixture invents a finish field.finality-unavailable, and their ordinary capture,TitleandCompactbehaviour is unchanged. Their AC1 coverage stays open by ruling.recordsEnabled(base, cwd)gates extraction; withrecords: falseingest behaves as before.amem ingest --jsongains an additive per-readerrecordsreceipt (supported,found,staged,duplicatesSkipped,refusedby reason). No new ingest kind or CLI surface is added.format: 4underhook-v4-, the same envelope, receipt and dedupe path. A running 0.24.0 matches onlyhook-v[23]-, so it leaves those files pending instead of retiring them as corrupt; the new reader drains formats 2, 3 and 4. All existing Claude and native publication bytes are unchanged and no format floor is written.finality-unavailable.Review round 1 (2026-11-11)
All three important findings are fixed at
f61261b, withREADME.mdanddocs/contracts.mdnaming the same values ate16dff6.finishReason: "end_turn"and never with"completed", so the gate refused every real Kimi Record. Counted over 669 localagents/main/wire.jsonlfiles:end_turn348,tool_use7026,error37,interrupted4,completed0. The reader now acceptsend_turnand refuses the other three exactly as before.test/ingest-records.test.tscovers all four values, and the fixture's default completion value is now the native one.session_metawas screened as a whole line, and a real header carries about 21.8 KB ofpayload.base_instructions, above the standalone screen's 16 KiB single-string bound, so the screen refused the header, the session scope becameunknown, and every Record in the file was refused. The header line is never appended, and the reader consumes only identity and scope fields, soheaderScreennow screens exactly those, both in the main-header gate and in the per-line scope gate. A test writes a 21,866-bytebase_instructionswith a git remote and still stages its Record.finality-unavailablewas written as a capture drop, and the SessionStart notice filtered only reasons starting withrecord_, so every later brief showedcapture dropped: finality unavailable (N). One predicate,recordRefusalReasoninsrc/hook-record.ts, now names Record refusals for both the metrics and the notice; the metric itself is unchanged. Covered intest/hook-record-instruction.test.ts, which also keeps a real loss (spool_full) visible.alreadyresult arrives without this turn's staged map and the count readsstagedinstead ofduplicatesSkipped. Dedupe itself holds and no row is duplicated; an honest count would need the staged receipts persisted across turns, which is not part of this round.Acceptance criteria
test/ingest-records.test.tsandtest/ingest.test.ts. Qwen and Jcode deliberately extract nothing and report the refusal (ruling in feat(ingest): extract Records from Codex, Kimi, Pi, Qwen and Jcode transcripts #369's second comment), so this criterion stays open for them.records: falsein the home or repository config disables extraction.amem ingest --jsonreports the per-reader counts.Per the contract this keeps the issue open:
Refs #369.Verification
Review round 1 on the hidden remote runner, focused scope only, no heavy run on the Mac:
f61261b: install,tsc --noEmitclean, boundary173 source files, the eight Records-focused files (116 pass / 0 fail), the registry-integrity-verified published 0.24.0hook-v4compatibility test (1 pass), exit 0.e16dff6(docs only):check-docsOK with the four pre-existing stale stamps,readme-factsOK,tsc --noEmitclean, exit 0.Earlier full pipeline at
10e8beb(base6f9ade6): the nine Records-focused files (112 pass / 0 fail / 909 assertions), the published-0.24.0hook-v4test (1 pass), the full suite (3732 pass / 1 fail / 27 skip across 341 files) andnpm packreadiness. The single failure wasa healthy second-process writer can finish while a synchronous reader waits for its short intentintest/public-prune-live-advance.test.ts, a file this branch does not touch, withpublic prune view unavailable; it passed in the previous run and 15 pass / 0 fail on three consecutive isolated runs at that head, so it is read as a load flake rather than a regression and is not claimed green in the full suite.Two full-suite failures that predated the last fix commit, and were invisible because the pipeline used to stop at the focused stage, are corrected in this branch: the ingest JSON fixture now carries the additive
recordsreceipt, and the spool compatibility fixture simulates its unknown newer format as 5 now that 4 is real.Rulings honored
hook-v4only for non-Claude Record rows; no Claude or native format bytes change.finality-unavailable.finality-unavailable, with aponytailnote and a test.