feat(sources): split collapsed Claude Code attachment sidecar events - #3423
Conversation
Problem: the Claude Code JSONL parser collapsed all attachment.type subtypes into one claude_attachment session_event, so a real referenced file and a hook_success ping were structurally indistinguishable at the event_type level. A full-corpus enumeration (~11.7K session files, ~/.claude/projects, 2026-07-31) found 38 distinct subtypes (up from the 20 the prior pass estimated), of which 4 carry zero information content in every occurrence observed (total_tokens_reminder is a constant string, todo_reminder is always empty, context_tip/companion_intro are UI/novelty chatter unrelated to session evidence). Solution: _attachment_sidecar_event dispatches on the nested attachment.type to one of ~22 semantically grouped event_types (real file/reference content gets its own type per structurally distinct artifact; hook lifecycle pings share one type distinguished by outcome; mode transitions, capability deltas, task/todo evidence, and small single-field signals are grouped analogously). An unrecognized future subtype routes to claude_attachment_unclassified rather than silently merging into a known bucket (FAIL LOUD). queued_command (an attachment subtype) reuses the existing claude_queue_operation event_type since it is the same message-queue entity as the top-level queue-operation record type, just reached through a different provider code path. Also renamed _SKIPPED_SIDECAR_RECORD_TYPES -> _NON_MESSAGE_SIDECAR_RECORD_TYPES: 13 of its 15 members persist as session_events today, so "skipped" has described the pre-polylogue-pbuh behavior for several PRs. Hard rename, no compat alias. Re-verified init/mode (previously dismissed as transient) against the current corpus: init still has 0 occurrences, mode is still 100% "normal" (21,545/21,545) -- disposition unchanged but now backed by a second independent measurement instead of the original single pass. No new row-count inflation: total attachment session_events emitted goes from 87,539 (all one type) to 81,845 across ~22 types (5,694 confirmed- transient records now correctly emit nothing instead of noise). Verification: devtools test tests/unit/sources/test_claude_code_sidecar_evidence.py tests/unit/sources/test_parsers_claude_code_artifacts.py tests/unit/sources/test_assembly_claude_code_history.py (68 passed); devtools verify --quick (ruff format/check, mypy --strict, render all --check, all quick gates) exit 0. Co-Authored-By: Claude <noreply@anthropic.com>
… payloads The previous commit's comments claimed deferred_tools_delta, mcp_instructions_delta, agent_listing_delta, skill_listing, invoked_skills, and diagnostics were bounded (names/counts, not full text), but the implementation still fell through to the generic dict(attachment) pass-through -- an unverified claim the diff didn't support. Added dedicated bounded builders (_bounded_delta_payload, _bounded_capability_snapshot_payload, _bounded_diagnostics_payload) so the event payload matches what the comments actually describe: names/counts for capability deltas and skill listings, per-file finding counts for diagnostics, matching the existing bounded-summary precedent in _tool_execution_result_payload (hunk counts, not full diffs). Verification: devtools test tests/unit/sources/test_claude_code_sidecar_evidence.py (28 passed, 3 new tests pin the bounded shapes); ruff format --check / ruff check / mypy --strict on the touched file all clean. Co-Authored-By: Claude <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 8 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8944ebd0d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Real file/reference content -- each subtype is a structurally distinct | ||
| # artifact, so each gets its own event_type (no collapsing these into | ||
| # each other, let alone into the hook/capability buckets below). | ||
| "file": "claude_attachment_file", # 711: a real referenced file's full content |
There was a problem hiding this comment.
When an existing archive upgrades, these new event types will not be materialized for previously ingested Claude Code sessions: this commit changes parser output from claude_attachment to subtype-specific events, but leaves INDEX_SCHEMA_VERSION at 47 with no SEMANTIC_REPARSE declaration, while the live parser fingerprint also remains unchanged. Consequently, unchanged source files retain the old rows indefinitely, producing a mixed taxonomy where only newly changed or manually reprocessed sessions expose the feature. Add a declared semantic-reparse index bump so existing raw evidence is replayed on upgrade.
AGENTS.md reference: AGENTS.md:L169-L175
Useful? React with 👍 / 👎.
…nts (#3437) ## Summary Restores master to green. Two independently-correct PRs — #3419 (per-type sidecar coverage) and #3423 (attachment sidecar split) — produced a broken combination, and the merge that joined them was verified against a narrower test selection than the change warranted. ## Problem `devtools test -k "sidecar or code_parser"` on master: **7 failed, 138 passed**. All seven are `test_attachment_*` in `tests/unit/sources/test_claude_code_sidecar_evidence.py`. They assert exact equality on `parsed.session_events`; #3419 unconditionally appends a `claude_parse_coverage` event, so every expected list gained an unexpected member: ``` Left contains one more item: ('claude_parse_coverage', {'sidecar_seen': {'attachment': 1}, 'sidecar_persisted': {}, ...}) ``` That payload also exposes a real defect rather than only a test-reconciliation problem: `sidecar_persisted` is empty while `sidecar_seen` counts the record. The attachment branch appends a `session_event` but never sets `persisted_this_record`, so per-type coverage understated persistence for every attachment record — the exact metric #3419 exists to report. ## Solution - `polylogue/sources/parsers/claude/code_parser.py` — the attachment branch sets `persisted_this_record = True` when it emits an event, matching every sibling branch in the same dispatch. - `tests/unit/sources/test_claude_code_sidecar_evidence.py` — the seven exact-equality assertions exclude `claude_parse_coverage`. It is a parse-level report orthogonal to the attachment evidence each test is about; filtering keeps each test pinned to its own subject rather than loosening the assertion. ## Verification ``` devtools test -k "sidecar or code_parser" before: 7 failed, 138 passed after: 145 passed ``` `devtools test tests/unit/sources/test_claude_code_sidecar_evidence.py` → 30 passed. ## Note The conflict resolution that produced this was verified with `devtools test tests/unit/sources/test_assembly_claude_code_history.py` (14 passed) — a file that does not exercise the attachment path. A cross-feature merge deserves the affected-area selection, not the file the conflict happened to sit in. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 participant
Summary
Closes the three gaps flagged in the
_SKIPPED_SIDECAR_RECORD_TYPESdisposition audit (polylogue-pbuh follow-up): theattachmentrecord type's 20+ subtypes were collapsed into oneclaude_attachmentsession_event;init/modewere dismissed as transient without a fresh re-check; and the frozenset's own name ("skipped") had been wrong since most of its members started persisting assession_events.Problem
Gap 1 (attachment collapse).
code_parser.py's own comment conceded it: "20 distinct attachment.type payloads incl. real files, edited-file records, diagnostics; per-subtype fidelity split is a follow-up, not this pass." A full-corpus enumeration (~/.claude/projects, ~11,700 session files, 2026-07-31) found 38 distinct subtypes, 87,539 total occurrences — a real referenced file's content and ahook_successping were both landing asevent_type="claude_attachment", distinguishable only by JSON-inspectingpayload["type"].Gap 2 (init/mode dismissal).
permission-modewas kept ("real operational signal, varies") while its siblingmodewas dropped as transient, on the strength of a single earlier pass. That needed independent re-verification, not a repeated assertion.Gap 3 (misleading name).
_SKIPPED_SIDECAR_RECORD_TYPESsaid "skipped." 13 of its 15 members are persisted assession_eventstoday (polylogue-pbuh). The name described the pre-polylogue-pbuh behavior only.Solution
Gap 1 — attachment subtype dispatch (
_attachment_sidecar_event,_ATTACHMENT_SUBTYPE_EVENT_TYPES)Full per-subtype disposition (live corpus,
~/.claude/projects, 2026-07-31; counts are subtype occurrences, not full-corpus row totals — see Verification for the total):fileclaude_attachment_fileedited_text_fileclaude_attachment_edited_filenested_memoryclaude_attachment_nested_memoryplan_file_referenceclaude_attachment_plan_referencecompact_file_referenceclaude_attachment_file_referencedirectoryclaude_attachment_directory_listinghook_successclaude_hook_eventhook_non_blocking_errorclaude_hook_eventhook_blocking_errorclaude_hook_eventhook_cancelledclaude_hook_eventhook_system_messageclaude_hook_eventhook_additional_contextclaude_hook_eventauto_modeclaude_agent_mode_eventauto_mode_exitclaude_agent_mode_eventplan_modeclaude_agent_mode_eventplan_mode_exitclaude_agent_mode_eventplan_mode_reentryclaude_agent_mode_eventdeferred_tools_deltaclaude_capability_deltamcp_instructions_deltaclaude_capability_deltaagent_listing_deltaclaude_capability_deltaskill_listingclaude_capability_snapshotinvoked_skillsclaude_capability_snapshotoutput_styleclaude_output_stylecommand_permissionsclaude_command_permissionsqueued_commandqueue-operation, reused typeclaude_queue_operationtask_statusclaude_task_statustask_reminderclaude_task_reminderdiagnosticsclaude_diagnosticsgoal_statusclaude_agent_goal_statusdate_changeclaude_date_changeread_truncation_noticeclaude_read_truncation_noticeultrathink_effortclaude_agent_effortstructured_outputclaude_structured_outputmax_turns_reachedclaude_max_turns_reachedtotal_tokens_reminder"<total_tokens>Infinite tokens left</total_tokens>"in every sampletodo_reminder{"content": [], "itemCount": 0}in every occurrence observedcontext_tipcompanion_introGrouping rule: subtypes reporting the same real-world entity (a hook firing, an agent-mode transition, a capability-surface delta) share one
event_typedistinguished by payload fields — not 38 near-empty types. Real file/reference content gets its ownevent_typeper subtype since each is a structurally distinct artifact.An unrecognized future subtype (39th+) routes to
claude_attachment_unclassifiedrather than silently merging into a known bucket — FAIL LOUD, independent of whether #3419'sclaude_parse_coverageevent lands first.Deliberately not wired into
ParsedAttachment/the blob store this pass:_acquire_attachment_blobinstorage/sqlite/archive_tiers/write.pyraises oninline_bytesunless routed through an archive-owned blob publisher, andwrite.pyis out of this lane's scope (owned by #3419's neighboring work). Real file content (file,edited_text_file,nested_memory,plan_file_reference) is kept at full fidelity assession_eventspayloads instead — promoting these to first-classattachmentsrows is a bigger, separately-scoped change through the blob-publisher path.Capability-delta/skill-listing/diagnostics payloads are bounded (names/counts, not full injected instruction text or full LSP message text) — same precedent as the existing
_tool_execution_result_payload(hunk counts, not full diffs).Gap 2 — init/mode re-verification
Re-ran the corpus scan independently (not just re-asserting the prior comment):
init— 0 occurrences (confirmed twice now);mode— 21,545/21,545 (100%) literal"normal"(up from 20,779 total on the prior pass, same zero-variance result). Disposition unchanged; the comment now cites two independent measurements instead of one, and explicitly contrasts withpermission-mode(5 distinct values, kept).Gap 3 — rename
_SKIPPED_SIDECAR_RECORD_TYPES→_NON_MESSAGE_SIDECAR_RECORD_TYPES. Hard rename, no compat alias (repo policy). Updated the one other reference (origin_specs.py's admission-declaration comment) and an illustrative event-type mention inapi/archive.py's docstring.Verification
devtools test tests/unit/sources/test_claude_code_sidecar_evidence.py tests/unit/sources/test_parsers_claude_code_artifacts.py tests/unit/sources/test_assembly_claude_code_history.py tests/unit/sources/test_source_laws.py— 204 passed.devtools verify --quick(ruff format/check, mypy --strict, render all --check, topology/layering/closure-matrix/schema-versioning/schema-promotion gates) — exit 0.ruff format --check/ruff check/mypyon the touched file individually — clean.session_eventsemitted goes from 87,539 (allevent_type="claude_attachment") to 81,845 across ~22 event_types, i.e. a net reduction of 5,694 rows (the 4 confirmed-transient subtypes now correctly emit nothing instead of noise).init/modedisposition is unchanged (still 0 / still dropped) so no additional rows from Gap 2.Out of scope (explicitly, per instructions)
progressrecord type's disposition (onlyagent_progresspersists; the other six subtypes are superseded streaming ticks) — untouched, per instructions.file,edited_text_file,nested_memory,plan_file_reference) to first-classParsedAttachment/blob-store rows — needs the archive-owned blob-publisher wiring instorage/sqlite/archive_tiers/write.py, which this lane was told to avoid. Filing a follow-up bead is reasonable next-session work if the operator wants that fidelity bump.#3419'sclaude_parse_coverageseen/persisted counter — independent PR, not merged as of this branch; this PR's fail-loud mechanism (claude_attachment_unclassified) works standalone and composes cleanly if feat(sources): report per-type sidecar coverage for Claude Code parse #3419 lands first or second.