refactor(pipeline): route comparison identity through typed constructors - #3405
Conversation
Problem: session/message/attachment/event comparison identity was built by slicing a hash-stable payload dict against a tuple of allowed field names (_MESSAGE_IDENTITY_FIELDS, _ATTACHMENT_IDENTITY_FIELDS, _EVENT_BASE_IDENTITY_FIELDS). Two real defects were exactly this shape going wrong: polylogue-bu1i folded attachment acquisition state into identity, and polylogue-nuec folded a provider-reported measurement into event identity. Both were fixed by hand, per field, in #3401 -- nothing made a THIRD occurrence structurally impossible, since any code could still call hash_payload() on an arbitrary dict and call it "identity". What changed: pipeline/ids.py now exposes four fixed keyword-only constructors -- message_identity_hash(*, id), attachment_identity_hash(*, message_id, name, mime_type), event_base_identity_hash(*, event_type, source_message_provider_id), event_canonical_identity_hash(*, base_identity, content_hash) -- as the sole path into each comparison-identity value. Passing any other field (or spreading a whole payload dict as **kwargs) is a TypeError at the call boundary, not a value a reviewer has to remember to exclude. session_revision_projection now calls these instead of the removed _message_identity_payload/_attachment_identity_payload/ _event_base_identity_payload dict-slice helpers. Every stored/pinned hash stays byte-identical: the constructors build the exact same dict shapes the removed helpers did, and test_session_revision_projection_golden_hashes (pinned digests) plus the independent-recomputation test both still pass unchanged. This is a pure re-plumbing, not a semantic/schema change. docs/plans/hash-boundary-registry.yaml is NOT retired. It governs all 198 hashlib/core.hashing call sites across polylogue/ (58+ files: blob storage content-addressing, HMAC signatures, redaction digests, cache/dedup keys, ...), the overwhelming majority of which are unrelated to session/message/ attachment/event comparison identity and were never part of the bu1i/nuec defect pattern. Registered the four new call sites and removed the four occurrences of session_revision_projection they replaced; lint is clean (devtools verify hash-boundary-census). Ref polylogue-aggz. Filed polylogue-ubwg to evaluate whether any of the registry's other 91 'identifier'-classified sites share the aggz failure shape and would benefit from the same constructor pattern before the registry itself could be retired. Verification: - devtools test tests/unit/pipeline/test_pipeline_ids.py tests/unit/archive/test_session_revision_membership.py -> 56 passed - uv run mypy --strict polylogue/pipeline/ids.py polylogue/archive/session_revision_membership.py tests/unit/pipeline/test_pipeline_ids.py -> Success, no issues - devtools verify --quick -> exit_code 0 (includes hash-boundary-census) Co-Authored-By: Claude <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 8 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
# Conflicts: # .beads/issues.jsonl
#3454) ## Summary Consolidates three independently-duplicated "skip this write as stale" freshness-tie checks (`archive_tiers/write.py`, `pipeline/services/ingest_batch/_core.py`, `archive_tiers/revision_governance.py`) into one `should_skip_stale_replace()` in `archive_tiers/ingest_precedence.py`. No behavior change: the comparison body is unchanged, just no longer copy-pasted three times. ## Problem `polylogue-t83e` investigated 6 live `session_id` collisions where a Claude Code transcript the operator uploaded into an AI Studio conversation (re-downloaded via Drive sync into `~/.local/share/polylogue/drive-cache/gemini/<uuid>.jsonl.txt.json`) was winning the archive row over the genuinely local `~/.claude/projects/...` transcript of the *same* conversation — even though the drive copy is a strict byte-prefix of the local file (confirmed by direct diff: 213 vs 214 Claude Code JSONL messages for one collision, the local file has exactly one trailing `summary` record the drive copy lacks). **Before/after collision table** (read-only queries against `/realm/db/polylogue/{source,index}.db`): | native_id | drive raw msgs | local raw msgs | `sessions.raw_id` today | `raw_session_memberships.decision` | relation under current code | |---|---|---|---|---|---| | 0213d48f-5b7a-4241-b77a-eb714672dc3b | 213 | 214 | drive (wrong) | `ambiguous` (stale, decided 2026-07-30T05:05Z) | `a_contains_b` (local dominates) | | 063a6885-8d6a-4f91-80b2-7f67fa06d680 | — | — | drive | `ambiguous` (stale) | not individually re-simulated; same code path | | 705f1fcb-8953-4b8b-92f1-9244fcf9db91 | — | — | drive | `ambiguous` (stale) | same | | 8c9f8c3d-4859-44cf-be9c-338803a8e7de | — | — | drive | `ambiguous` (stale) | same | | a952ffa4-73b0-48bd-a212-ebe5b9772d1e | — | — | drive | `ambiguous` (stale) | same | | cf3404fa-89e0-400a-af3e-ff1450eecef4 | — | — | drive | `ambiguous` (stale) | same | (15 drive-cache raws total, all genuinely Claude-Code-shaped by content, parsing into 12 sessions — 6 collide with a local raw as above, 6 do not.) **Root cause, precisely**: this is a revision-arbitration bug, not an identity/detection bug. `session_id = origin || ':' || native_id` correctly computes the same identity for both raws because they *are* the same conversation — that's not a defect. Which raw should win is `archive/session_revision_membership.py`'s job (a content-only set relation: `equal`/`a_contains_b`/`b_contains_a`/`conflict`, polylogue-aggz). Directly simulating that relation against the real raw bytes with **current** code (`parse_stream_payload` + `session_revision_projection`) gives `a_contains_b` (local strictly dominates: 0 drive-only messages, 1 local-only message, 0 content mismatches on the 213 shared identities) — i.e. **the fix already exists**. The live archive's `raw_session_memberships` decision rows for these 6 cohorts (`decided_at_ms` ≈ 2026-07-30T05:05–05:13Z) simply predate the commits that fixed the comparison logic (`9fc5220ef` #3401 "collapse revision comparison into a content-only relation", 15:55Z same day, and `a9f2f307d` #3405, 17:50Z) — stale data, not a live bug. **This self-heals without further code changes.** `daemon/bulk_rebuild.py`'s own docstring documents that the daemon routes a bulk-scale backlog (which `polylogue ops reset --index` creates) into `rebuild_index_from_source` → `backfill_historical_revision_evidence` → `classify_membership_revisions` — the current, already-fixed relation — "with zero operator involvement". The operator's already-planned `ops reset --index && polylogued run` will recompute these 6 cohorts correctly. **No `SEMANTIC_REPARSE` delta declaration is needed**: no detection or parsing semantics changed in this PR, and the relevant fix already shipped (and presumably was already declared) in #3401/#3405. **Record correction**: PR #3436 verified these 12 drive-cache/gemini rows are content-shape-correct ("naming coincidence, not a bug") — that remains true, the content-shape classification was never wrong. It did not check `native_id` collisions against local raws, so it didn't catch that 6 of the 12 were silently shadowing a fuller local transcript. That shadowing was a data-staleness artifact of a bug fixed the same day this bead's forensics ran, not a defect in #3436's own change. **Two approaches were tried and reverted** before landing on this conclusion (session-identity gating in `sources/origin_specs.py` + `pipeline/services/ingest_worker.py`, refusing session admission for drive-cache Claude-Code-shaped paths). Both worked mechanically but were wrong: they would have permanently suppressed the 6 *non-colliding* drive-cache raws (real content with no local counterpart to supersede them, so they'd become inert un-queryable `raw_artifacts` rows instead of correctly-attributed real content) and added unneeded machinery for the 6 colliding ones, which the already-fixed relation resolves once the data catches up. Full trail recorded in `polylogue-t83e`'s closing comment. ## Solution What remained a real, independently-valuable defect, unrelated to the collision-resolution question above: the freshness-tie comparison itself (`existing_updated_at_ms is not None and incoming_freshness_ms < existing_updated_at_ms`) was hand-duplicated in three call sites that could silently drift apart. Extracted `should_skip_stale_replace()` into `archive_tiers/ingest_precedence.py` (the module that already owns browser-capture write precedence) and call it from all three: - `polylogue/storage/sqlite/archive_tiers/write.py` (`write_parsed_session_to_archive`) - `polylogue/pipeline/services/ingest_batch/_core.py` (`_write_session`) - `polylogue/storage/sqlite/archive_tiers/revision_governance.py` (`_write_parsed_precedence_result`) Each call site keeps its own surrounding guard conditions (`force_write`/`force_replace`, browser-capture precedence, append-only, revision-authority membership, `source_index` gating) — those decide *whether* the check applies at all, not the comparison. The function's docstring documents that this is a narrow per-write timestamp fallback, not where content-subset supersession is decided (that's revision membership, upstream, and takes precedence when it has classified a cohort). ## Verification - `devtools test tests/unit/pipeline/test_archive_write.py::test_older_full_replace_does_not_overwrite_newer_session_body tests/unit/pipeline/test_ingest_batch.py::test_write_session_force_write_replaces_older_freshness` — 2 passed (exercises `write.py` and `ingest_batch/_core.py`'s edited branches directly) - `devtools test tests/unit/storage/test_revision_replay.py tests/unit/storage/test_raw_revision_authority.py tests/unit/sources/test_revision_backfill.py` — 103 passed, 1 failed. The failure (`test_parse_one_still_replays_real_claude_code_sessions_with_no_path_rule`) is pre-existing on `origin/master`: zero diff in the files it exercises (`archive/artifact_taxonomy/`, `sources/revision_backfill.py`, `sources/origin_specs.py`), reproduces identically in isolation, and is about an unrelated `analysis/`-path content-gate regression. - `devtools verify --seed-testmon --skip-slow` — ruff format, ruff check, mypy --strict, render all --check, topology/layering/closure-matrix, schema-versioning policy, schema-promotion audit all green (see step-by-step `run.json`, exit 0 through step 18); the full pytest seed pass was still mid-sweep when the verification budget for this session ran out — unrelated to this change's narrow surface, already covered by the targeted runs above. - `devtools verify --quick` — green via the pre-push hook (format/lint/mypy/render). Ref polylogue-t83e (closed this session with the full investigation trail in its final comment) Co-authored-by: Claude <noreply@anthropic.com>
…ty (#3472) ## Summary Fixes the dominant remaining cause of ambiguity in claude-code-session revision membership: fork/resume/usage-limit-quirk boundary records that Claude Code stamps with an ancestor session's sessionId were composing that ancestor's bare id as their own identity, colliding every such carryover fragment onto the ancestor's logical_source_key. ## Problem Measured while verifying polylogue-oycw's set-containment fix (#3401/#3405) against real ambiguous cohorts: only 56/185 (30.3%) of claude-code-session cohorts resolved cleanly, far below chatgpt-export/claude-ai-export (>90%). Root cause verified directly against real ~/.claude/projects files: Claude Code re-stamps a handful of records with an ancestor's sessionId even inside a file that is otherwise entirely a different session's own content (a leading resume/fork boundary record, or a mid-file usage-limit/exit quirk). ## Solution dispatch.py's Claude Code grouping now identifies each file's own real content ("primary") and detects a carryover run via structural signals (occurs before primary's first record, or its parentUuid resolves into a uuid primary already produced). A carryover run's identity is qualified so siblings off one ancestor stay distinct; the ancestor id becomes parent_session_id, routing through session_links/lineage instead of colliding revision membership. A new explicit trust_fallback_id flag scopes this override to only dispatch-proven carryover fragments. SEMANTIC_REPARSE index bump to v53. ## Verification devtools verify --quick green. devtools test across the affected surface: 320 passed, 2 pre-existing failures confirmed unrelated via a throwaway origin/master worktree (filed as polylogue-yl8t, polylogue-ihro). New regression test proves two sibling fork files off one ancestor no longer collide identity. Ref polylogue-jc4q
…aggz comparison identity nuec, d8al, hith, c429, bu1i — all five census beads' failure modes are structurally eliminated by the merged set-based (identity, content) comparison architecture (PRs #3401/#3405): event-payload allowlist, content-derived attachment identity, order-insensitive message comparison, acquisition-state exclusion. Live censuses: 554/587 claude-ai (94.4%) and 119/135 chatgpt (88.1%) ambiguous cohorts resolve, 0 regressions. Retroactive verdict healing remains polylogue-9dxn (lane in flight). Co-Authored-By: Claude <noreply@anthropic.com>
Summary
Makes it structurally impossible for non-content data to enter a session's
comparison-identity value: message/attachment/event identity now go through
four fixed keyword-only constructors instead of a dict-key-list projection.
Problem
session_revision_projection(polylogue-aggz Invariant 1, landed in #3401)built comparison identity by slicing an already-constructed hash-stable
payload dict against a tuple of allowed field names
(
_MESSAGE_IDENTITY_FIELDS,_ATTACHMENT_IDENTITY_FIELDS,_EVENT_BASE_IDENTITY_FIELDS). Two real, expensive defects were exactly anon-content field reaching that value: polylogue-bu1i folded attachment
acquisition state (
inline_bytes/size_bytes) into identity (151aistudio-drive cohorts wrongly ambiguous, 14 sessions absent, a day of
investigation), and polylogue-nuec folded a provider-reported measurement
(
elapsed_duration_ms) into event identity (135 chatgpt cohorts read asdivergent despite byte-identical content). Both were fixed by hand, per
field. Nothing prevented a third occurrence: any code could still call
hash_payload()on an arbitrary dict and call it "identity" — theallowlist was a convention encoded in a tuple of strings, not enforced by
the type system.
docs/plans/hash-boundary-registry.yaml— a machine-checked census of everyhash-producer call site in
polylogue/— exists as a workaround for themissing chokepoint: since anyone could compute an identity hash anywhere,
the repo keeps a list of where they did.
Solution
polylogue/pipeline/ids.pynow exposes the sole path into eachcomparison-identity value as a fixed keyword-only function:
message_identity_hash(*, id: str) -> bytesattachment_identity_hash(*, message_id, name, mime_type) -> bytesevent_base_identity_hash(*, event_type, source_message_provider_id) -> bytesevent_canonical_identity_hash(*, base_identity, content_hash) -> bytesPassing any other field — or spreading a whole payload dict as
**kwargs—is a
TypeErrorat the call boundary, not a value someone has to rememberto strip.
session_revision_projectionnow calls these instead of theremoved
_message_identity_payload/_attachment_identity_payload/_event_base_identity_payloaddict-slice helpers, which are deleted alongwith their backing field-name tuples.
Every stored/pinned hash stays byte-identical. The constructors build
the exact same dict shapes the removed helpers did (same keys, same values,
hash_payloadsorts keys so order is irrelevant), andtest_session_revision_projection_golden_hashes(which pins exact digestbytes) and the independent-recomputation test both pass unchanged with no
golden updates. This is a pure re-plumbing of an existing correct
computation, not a semantic or schema change — the "if you find yourself
editing the goldens, stop" tripwire never fired.
docs/plans/hash-boundary-registry.yamlis NOT retired. It governs all198
hashlib/core.hashingcall sites acrosspolylogue/(58+ files:blob-store content-addressing, HMAC signatures, redaction digests,
cache/dedup keys, judgment-run identifiers, sinex material adapters, ...),
of which only ~12 (in
pipeline/ids.py) are the session/message/attachment/event comparison-identity axis this bead targets. Claiming the
registry's whole purpose is now covered would not survive a diff-grep. The
registry's
pipeline/ids.pyentries were updated to match the new callsites (four new
<module>.<constructor>entries replacing foursession_revision_projectionoccurrences);devtools verify hash-boundary-censusis clean. Filed polylogue-ubwg to audit whether any ofthe registry's other 91
identifier-classified sites share the aggzfailure shape (a mutable/acquisition-state field folded into a value used
for equality/dedup comparison) and would benefit from the same constructor
pattern — only once every site is provably covered or provably out of class
can the registry itself be deleted.
Tests
Added to
tests/unit/pipeline/test_pipeline_ids.py:test_message_identity_hash_rejects_non_content_fields— passingtextraises
TypeError.test_attachment_identity_hash_rejects_acquisition_state— passingsize_bytes/inline_content_hashraisesTypeError.test_attachment_identity_hash_rejects_full_payload_spread— simulates afuture parser adding a brand-new field to the hash-stable payload and
spreads the entire payload dict (including that new field) into the
constructor; rejected outright, proving a new field can't silently enter
identity by construction, not by someone remembering to exclude it.
test_event_base_identity_hash_rejects_measurement_fields— passingpayload={"elapsed_duration_ms": ...}raisesTypeError.test_identity_constructors_ignore_new_payload_fields_when_called_correctly— two attachments differing only in acquisition state produce identical
identity.
test_event_canonical_identity_hash_folds_base_and_content— sanity checkon the fold.
Verification
devtools test tests/unit/pipeline/test_pipeline_ids.py tests/unit/archive/test_session_revision_membership.py→ 56 passeduv run mypy --strict polylogue/pipeline/ids.py polylogue/archive/session_revision_membership.py tests/unit/pipeline/test_pipeline_ids.py→ Success, no issues founddevtools verify --quick→ exit_code 0 (includesverify hash-boundary-census)devtools verify --all/ full integration suite (out of scopeper repo convention — per-PR CI skips the heavy
testsuite; targeteddevtools testabove covers every affected module).Ref polylogue-aggz.
Co-Authored-By: Claude noreply@anthropic.com