Skip to content

refactor(pipeline): route comparison identity through typed constructors - #3405

Merged
Sinity merged 2 commits into
masterfrom
feature/refactor/typed-identity-constructor
Jul 30, 2026
Merged

refactor(pipeline): route comparison identity through typed constructors#3405
Sinity merged 2 commits into
masterfrom
feature/refactor/typed-identity-constructor

Conversation

@Sinity

@Sinity Sinity commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Summary

Makes it structurally impossible for non-content data to enter a session's
comparison-identity value: message/attachment/event identity now go through
four fixed keyword-only constructors instead of a dict-key-list projection.

Problem

session_revision_projection (polylogue-aggz Invariant 1, landed in #3401)
built comparison identity by slicing an already-constructed hash-stable
payload dict against a tuple of allowed field names
(_MESSAGE_IDENTITY_FIELDS, _ATTACHMENT_IDENTITY_FIELDS,
_EVENT_BASE_IDENTITY_FIELDS). Two real, expensive defects were exactly a
non-content field reaching that value: polylogue-bu1i folded attachment
acquisition state (inline_bytes/size_bytes) into identity (151
aistudio-drive cohorts wrongly ambiguous, 14 sessions absent, a day of
investigation), and polylogue-nuec folded a provider-reported measurement
(elapsed_duration_ms) into event identity (135 chatgpt cohorts read as
divergent despite byte-identical content). Both were fixed by hand, per
field. Nothing prevented a third occurrence: any code could still call
hash_payload() on an arbitrary dict and call it "identity" — the
allowlist was a convention encoded in a tuple of strings, not enforced by
the type system.

docs/plans/hash-boundary-registry.yaml — a machine-checked census of every
hash-producer call site in polylogue/ — exists as a workaround for the
missing chokepoint: since anyone could compute an identity hash anywhere,
the repo keeps a list of where they did.

Solution

polylogue/pipeline/ids.py now exposes the sole path into each
comparison-identity value as a fixed keyword-only function:

  • message_identity_hash(*, id: str) -> bytes
  • attachment_identity_hash(*, message_id, name, mime_type) -> bytes
  • event_base_identity_hash(*, event_type, source_message_provider_id) -> bytes
  • event_canonical_identity_hash(*, base_identity, content_hash) -> bytes

Passing any other field — or spreading a whole payload dict as **kwargs
is a TypeError at the call boundary, not a value someone has to remember
to strip. session_revision_projection now calls these instead of the
removed _message_identity_payload/_attachment_identity_payload/
_event_base_identity_payload dict-slice helpers, which are deleted along
with their backing field-name tuples.

Every stored/pinned hash stays byte-identical. The constructors build
the exact same dict shapes the removed helpers did (same keys, same values,
hash_payload sorts keys so order is irrelevant), and
test_session_revision_projection_golden_hashes (which pins exact digest
bytes) and the independent-recomputation test both pass unchanged with no
golden updates. This is a pure re-plumbing of an existing correct
computation, not a semantic or schema change — the "if you find yourself
editing the goldens, stop" tripwire never fired.

docs/plans/hash-boundary-registry.yaml is NOT retired. It governs all
198 hashlib/core.hashing call sites across polylogue/ (58+ files:
blob-store content-addressing, HMAC signatures, redaction digests,
cache/dedup keys, judgment-run identifiers, sinex material adapters, ...),
of which only ~12 (in pipeline/ids.py) are the session/message/
attachment/event comparison-identity axis this bead targets. Claiming the
registry's whole purpose is now covered would not survive a diff-grep. The
registry's pipeline/ids.py entries were updated to match the new call
sites (four new <module>.<constructor> entries replacing four
session_revision_projection occurrences); devtools verify hash-boundary-census is clean. Filed polylogue-ubwg to audit whether any of
the registry's other 91 identifier-classified sites share the aggz
failure shape (a mutable/acquisition-state field folded into a value used
for equality/dedup comparison) and would benefit from the same constructor
pattern — only once every site is provably covered or provably out of class
can the registry itself be deleted.

Tests

Added to tests/unit/pipeline/test_pipeline_ids.py:

  • test_message_identity_hash_rejects_non_content_fields — passing text
    raises TypeError.
  • test_attachment_identity_hash_rejects_acquisition_state — passing
    size_bytes/inline_content_hash raises TypeError.
  • test_attachment_identity_hash_rejects_full_payload_spread — simulates a
    future parser adding a brand-new field to the hash-stable payload and
    spreads the entire payload dict (including that new field) into the
    constructor; rejected outright, proving a new field can't silently enter
    identity by construction, not by someone remembering to exclude it.
  • test_event_base_identity_hash_rejects_measurement_fields — passing
    payload={"elapsed_duration_ms": ...} raises TypeError.
  • test_identity_constructors_ignore_new_payload_fields_when_called_correctly
    — two attachments differing only in acquisition state produce identical
    identity.
  • test_event_canonical_identity_hash_folds_base_and_content — sanity check
    on the fold.

Verification

  • devtools test tests/unit/pipeline/test_pipeline_ids.py tests/unit/archive/test_session_revision_membership.py → 56 passed
  • uv run mypy --strict polylogue/pipeline/ids.py polylogue/archive/session_revision_membership.py tests/unit/pipeline/test_pipeline_ids.py → Success, no issues found
  • devtools verify --quick → exit_code 0 (includes verify hash-boundary-census)
  • Not run: devtools verify --all / full integration suite (out of scope
    per repo convention — per-PR CI skips the heavy test suite; targeted
    devtools test above covers every affected module).

Ref polylogue-aggz.

Co-Authored-By: Claude noreply@anthropic.com

Problem: session/message/attachment/event comparison identity was built by
slicing a hash-stable payload dict against a tuple of allowed field names
(_MESSAGE_IDENTITY_FIELDS, _ATTACHMENT_IDENTITY_FIELDS,
_EVENT_BASE_IDENTITY_FIELDS). Two real defects were exactly this shape going
wrong: polylogue-bu1i folded attachment acquisition state into identity, and
polylogue-nuec folded a provider-reported measurement into event identity.
Both were fixed by hand, per field, in #3401 -- nothing made a THIRD
occurrence structurally impossible, since any code could still call
hash_payload() on an arbitrary dict and call it "identity".

What changed: pipeline/ids.py now exposes four fixed keyword-only
constructors -- message_identity_hash(*, id), attachment_identity_hash(*,
message_id, name, mime_type), event_base_identity_hash(*, event_type,
source_message_provider_id), event_canonical_identity_hash(*, base_identity,
content_hash) -- as the sole path into each comparison-identity value.
Passing any other field (or spreading a whole payload dict as **kwargs) is a
TypeError at the call boundary, not a value a reviewer has to remember to
exclude. session_revision_projection now calls these instead of the removed
_message_identity_payload/_attachment_identity_payload/
_event_base_identity_payload dict-slice helpers.

Every stored/pinned hash stays byte-identical: the constructors build the
exact same dict shapes the removed helpers did, and
test_session_revision_projection_golden_hashes (pinned digests) plus the
independent-recomputation test both still pass unchanged. This is a pure
re-plumbing, not a semantic/schema change.

docs/plans/hash-boundary-registry.yaml is NOT retired. It governs all 198
hashlib/core.hashing call sites across polylogue/ (58+ files: blob storage
content-addressing, HMAC signatures, redaction digests, cache/dedup keys,
...), the overwhelming majority of which are unrelated to session/message/
attachment/event comparison identity and were never part of the bu1i/nuec
defect pattern. Registered the four new call sites and removed the four
occurrences of session_revision_projection they replaced; lint is clean
(devtools verify hash-boundary-census). Ref polylogue-aggz. Filed
polylogue-ubwg to evaluate whether any of the registry's other 91
'identifier'-classified sites share the aggz failure shape and would
benefit from the same constructor pattern before the registry itself could
be retired.

Verification:
- devtools test tests/unit/pipeline/test_pipeline_ids.py
  tests/unit/archive/test_session_revision_membership.py -> 56 passed
- uv run mypy --strict polylogue/pipeline/ids.py
  polylogue/archive/session_revision_membership.py
  tests/unit/pipeline/test_pipeline_ids.py -> Success, no issues
- devtools verify --quick -> exit_code 0 (includes hash-boundary-census)

Co-Authored-By: Claude <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@Sinity, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 8 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 6c902d71-553f-4030-be3c-4a1e5e4bd73d

📥 Commits

Reviewing files that changed from the base of the PR and between 32266af and 8253c56.

📒 Files selected for processing (4)
  • docs/plans/hash-boundary-registry.yaml
  • polylogue/archive/session_revision_membership.py
  • polylogue/pipeline/ids.py
  • tests/unit/pipeline/test_pipeline_ids.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Sinity
Sinity merged commit a9f2f30 into master Jul 30, 2026
3 checks passed
@Sinity
Sinity deleted the feature/refactor/typed-identity-constructor branch July 30, 2026 17:50
Sinity added a commit that referenced this pull request Jul 31, 2026
#3454)

## Summary

Consolidates three independently-duplicated "skip this write as stale"
freshness-tie checks (`archive_tiers/write.py`,
`pipeline/services/ingest_batch/_core.py`,
`archive_tiers/revision_governance.py`) into one
`should_skip_stale_replace()` in `archive_tiers/ingest_precedence.py`.
No behavior change: the comparison body is unchanged, just no longer
copy-pasted three times.

## Problem

`polylogue-t83e` investigated 6 live `session_id` collisions where a
Claude Code transcript the operator uploaded into an AI Studio
conversation (re-downloaded via Drive sync into
`~/.local/share/polylogue/drive-cache/gemini/<uuid>.jsonl.txt.json`) was
winning the archive row over the genuinely local
`~/.claude/projects/...` transcript of the *same* conversation — even
though the drive copy is a strict byte-prefix of the local file
(confirmed by direct diff: 213 vs 214 Claude Code JSONL messages for one
collision, the local file has exactly one trailing `summary` record the
drive copy lacks).

**Before/after collision table** (read-only queries against
`/realm/db/polylogue/{source,index}.db`):

| native_id | drive raw msgs | local raw msgs | `sessions.raw_id` today
| `raw_session_memberships.decision` | relation under current code |
|---|---|---|---|---|---|
| 0213d48f-5b7a-4241-b77a-eb714672dc3b | 213 | 214 | drive (wrong) |
`ambiguous` (stale, decided 2026-07-30T05:05Z) | `a_contains_b` (local
dominates) |
| 063a6885-8d6a-4f91-80b2-7f67fa06d680 | — | — | drive | `ambiguous`
(stale) | not individually re-simulated; same code path |
| 705f1fcb-8953-4b8b-92f1-9244fcf9db91 | — | — | drive | `ambiguous`
(stale) | same |
| 8c9f8c3d-4859-44cf-be9c-338803a8e7de | — | — | drive | `ambiguous`
(stale) | same |
| a952ffa4-73b0-48bd-a212-ebe5b9772d1e | — | — | drive | `ambiguous`
(stale) | same |
| cf3404fa-89e0-400a-af3e-ff1450eecef4 | — | — | drive | `ambiguous`
(stale) | same |

(15 drive-cache raws total, all genuinely Claude-Code-shaped by content,
parsing into 12 sessions — 6 collide with a local raw as above, 6 do
not.)

**Root cause, precisely**: this is a revision-arbitration bug, not an
identity/detection bug. `session_id = origin || ':' || native_id`
correctly computes the same identity for both raws because they *are*
the same conversation — that's not a defect. Which raw should win is
`archive/session_revision_membership.py`'s job (a content-only set
relation: `equal`/`a_contains_b`/`b_contains_a`/`conflict`,
polylogue-aggz). Directly simulating that relation against the real raw
bytes with **current** code (`parse_stream_payload` +
`session_revision_projection`) gives `a_contains_b` (local strictly
dominates: 0 drive-only messages, 1 local-only message, 0 content
mismatches on the 213 shared identities) — i.e. **the fix already
exists**. The live archive's `raw_session_memberships` decision rows for
these 6 cohorts (`decided_at_ms` ≈ 2026-07-30T05:05–05:13Z) simply
predate the commits that fixed the comparison logic (`9fc5220ef` #3401
"collapse revision comparison into a content-only relation", 15:55Z same
day, and `a9f2f307d` #3405, 17:50Z) — stale data, not a live bug.

**This self-heals without further code changes.**
`daemon/bulk_rebuild.py`'s own docstring documents that the daemon
routes a bulk-scale backlog (which `polylogue ops reset --index`
creates) into `rebuild_index_from_source` →
`backfill_historical_revision_evidence` →
`classify_membership_revisions` — the current, already-fixed relation —
"with zero operator involvement". The operator's already-planned `ops
reset --index && polylogued run` will recompute these 6 cohorts
correctly. **No `SEMANTIC_REPARSE` delta declaration is needed**: no
detection or parsing semantics changed in this PR, and the relevant fix
already shipped (and presumably was already declared) in #3401/#3405.

**Record correction**: PR #3436 verified these 12 drive-cache/gemini
rows are content-shape-correct ("naming coincidence, not a bug") — that
remains true, the content-shape classification was never wrong. It did
not check `native_id` collisions against local raws, so it didn't catch
that 6 of the 12 were silently shadowing a fuller local transcript. That
shadowing was a data-staleness artifact of a bug fixed the same day this
bead's forensics ran, not a defect in #3436's own change.

**Two approaches were tried and reverted** before landing on this
conclusion (session-identity gating in `sources/origin_specs.py` +
`pipeline/services/ingest_worker.py`, refusing session admission for
drive-cache Claude-Code-shaped paths). Both worked mechanically but were
wrong: they would have permanently suppressed the 6 *non-colliding*
drive-cache raws (real content with no local counterpart to supersede
them, so they'd become inert un-queryable `raw_artifacts` rows instead
of correctly-attributed real content) and added unneeded machinery for
the 6 colliding ones, which the already-fixed relation resolves once the
data catches up. Full trail recorded in `polylogue-t83e`'s closing
comment.

## Solution

What remained a real, independently-valuable defect, unrelated to the
collision-resolution question above: the freshness-tie comparison itself
(`existing_updated_at_ms is not None and incoming_freshness_ms <
existing_updated_at_ms`) was hand-duplicated in three call sites that
could silently drift apart. Extracted `should_skip_stale_replace()` into
`archive_tiers/ingest_precedence.py` (the module that already owns
browser-capture write precedence) and call it from all three:

- `polylogue/storage/sqlite/archive_tiers/write.py`
(`write_parsed_session_to_archive`)
- `polylogue/pipeline/services/ingest_batch/_core.py` (`_write_session`)
- `polylogue/storage/sqlite/archive_tiers/revision_governance.py`
(`_write_parsed_precedence_result`)

Each call site keeps its own surrounding guard conditions
(`force_write`/`force_replace`, browser-capture precedence, append-only,
revision-authority membership, `source_index` gating) — those decide
*whether* the check applies at all, not the comparison. The function's
docstring documents that this is a narrow per-write timestamp fallback,
not where content-subset supersession is decided (that's revision
membership, upstream, and takes precedence when it has classified a
cohort).

## Verification

- `devtools test
tests/unit/pipeline/test_archive_write.py::test_older_full_replace_does_not_overwrite_newer_session_body
tests/unit/pipeline/test_ingest_batch.py::test_write_session_force_write_replaces_older_freshness`
— 2 passed (exercises `write.py` and `ingest_batch/_core.py`'s edited
branches directly)
- `devtools test tests/unit/storage/test_revision_replay.py
tests/unit/storage/test_raw_revision_authority.py
tests/unit/sources/test_revision_backfill.py` — 103 passed, 1 failed.
The failure
(`test_parse_one_still_replays_real_claude_code_sessions_with_no_path_rule`)
is pre-existing on `origin/master`: zero diff in the files it exercises
(`archive/artifact_taxonomy/`, `sources/revision_backfill.py`,
`sources/origin_specs.py`), reproduces identically in isolation, and is
about an unrelated `analysis/`-path content-gate regression.
- `devtools verify --seed-testmon --skip-slow` — ruff format, ruff
check, mypy --strict, render all --check,
topology/layering/closure-matrix, schema-versioning policy,
schema-promotion audit all green (see step-by-step `run.json`, exit 0
through step 18); the full pytest seed pass was still mid-sweep when the
verification budget for this session ran out — unrelated to this
change's narrow surface, already covered by the targeted runs above.
- `devtools verify --quick` — green via the pre-push hook
(format/lint/mypy/render).

Ref polylogue-t83e (closed this session with the full investigation
trail in its final comment)

Co-authored-by: Claude <noreply@anthropic.com>
Sinity added a commit that referenced this pull request Jul 31, 2026
…ty (#3472)

## Summary
Fixes the dominant remaining cause of ambiguity in claude-code-session revision membership: fork/resume/usage-limit-quirk boundary records that Claude Code stamps with an ancestor session's sessionId were composing that ancestor's bare id as their own identity, colliding every such carryover fragment onto the ancestor's logical_source_key.

## Problem
Measured while verifying polylogue-oycw's set-containment fix (#3401/#3405) against real ambiguous cohorts: only 56/185 (30.3%) of claude-code-session cohorts resolved cleanly, far below chatgpt-export/claude-ai-export (>90%). Root cause verified directly against real ~/.claude/projects files: Claude Code re-stamps a handful of records with an ancestor's sessionId even inside a file that is otherwise entirely a different session's own content (a leading resume/fork boundary record, or a mid-file usage-limit/exit quirk).

## Solution
dispatch.py's Claude Code grouping now identifies each file's own real content ("primary") and detects a carryover run via structural signals (occurs before primary's first record, or its parentUuid resolves into a uuid primary already produced). A carryover run's identity is qualified so siblings off one ancestor stay distinct; the ancestor id becomes parent_session_id, routing through session_links/lineage instead of colliding revision membership. A new explicit trust_fallback_id flag scopes this override to only dispatch-proven carryover fragments. SEMANTIC_REPARSE index bump to v53.

## Verification
devtools verify --quick green. devtools test across the affected surface: 320 passed, 2 pre-existing failures confirmed unrelated via a throwaway origin/master worktree (filed as polylogue-yl8t, polylogue-ihro). New regression test proves two sibling fork files off one ancestor no longer collide identity.

Ref polylogue-jc4q
Sinity added a commit that referenced this pull request Jul 31, 2026
…aggz comparison identity

nuec, d8al, hith, c429, bu1i — all five census beads' failure modes are
structurally eliminated by the merged set-based (identity, content)
comparison architecture (PRs #3401/#3405): event-payload allowlist,
content-derived attachment identity, order-insensitive message comparison,
acquisition-state exclusion. Live censuses: 554/587 claude-ai (94.4%) and
119/135 chatgpt (88.1%) ambiguous cohorts resolve, 0 regressions.
Retroactive verdict healing remains polylogue-9dxn (lane in flight).

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant