From 410879996bf33bbcbfc92721b9b9bc1b0a21244e Mon Sep 17 00:00:00 2001 From: Sinity Date: Thu, 30 Jul 2026 19:23:13 +0200 Subject: [PATCH] refactor(pipeline): route comparison identity through typed constructors Problem: session/message/attachment/event comparison identity was built by slicing a hash-stable payload dict against a tuple of allowed field names (_MESSAGE_IDENTITY_FIELDS, _ATTACHMENT_IDENTITY_FIELDS, _EVENT_BASE_IDENTITY_FIELDS). Two real defects were exactly this shape going wrong: polylogue-bu1i folded attachment acquisition state into identity, and polylogue-nuec folded a provider-reported measurement into event identity. Both were fixed by hand, per field, in #3401 -- nothing made a THIRD occurrence structurally impossible, since any code could still call hash_payload() on an arbitrary dict and call it "identity". What changed: pipeline/ids.py now exposes four fixed keyword-only constructors -- message_identity_hash(*, id), attachment_identity_hash(*, message_id, name, mime_type), event_base_identity_hash(*, event_type, source_message_provider_id), event_canonical_identity_hash(*, base_identity, content_hash) -- as the sole path into each comparison-identity value. Passing any other field (or spreading a whole payload dict as **kwargs) is a TypeError at the call boundary, not a value a reviewer has to remember to exclude. session_revision_projection now calls these instead of the removed _message_identity_payload/_attachment_identity_payload/ _event_base_identity_payload dict-slice helpers. Every stored/pinned hash stays byte-identical: the constructors build the exact same dict shapes the removed helpers did, and test_session_revision_projection_golden_hashes (pinned digests) plus the independent-recomputation test both still pass unchanged. This is a pure re-plumbing, not a semantic/schema change. docs/plans/hash-boundary-registry.yaml is NOT retired. It governs all 198 hashlib/core.hashing call sites across polylogue/ (58+ files: blob storage content-addressing, HMAC signatures, redaction digests, cache/dedup keys, ...), the overwhelming majority of which are unrelated to session/message/ attachment/event comparison identity and were never part of the bu1i/nuec defect pattern. Registered the four new call sites and removed the four occurrences of session_revision_projection they replaced; lint is clean (devtools verify hash-boundary-census). Ref polylogue-aggz. Filed polylogue-ubwg to evaluate whether any of the registry's other 91 'identifier'-classified sites share the aggz failure shape and would benefit from the same constructor pattern before the registry itself could be retired. Verification: - devtools test tests/unit/pipeline/test_pipeline_ids.py tests/unit/archive/test_session_revision_membership.py -> 56 passed - uv run mypy --strict polylogue/pipeline/ids.py polylogue/archive/session_revision_membership.py tests/unit/pipeline/test_pipeline_ids.py -> Success, no issues - devtools verify --quick -> exit_code 0 (includes hash-boundary-census) Co-Authored-By: Claude --- .beads/issues.jsonl | 6 +- docs/plans/hash-boundary-registry.yaml | 28 ++-- .../archive/session_revision_membership.py | 2 +- polylogue/pipeline/ids.py | 97 +++++++++----- tests/unit/pipeline/test_pipeline_ids.py | 126 ++++++++++++++++++ 5 files changed, 208 insertions(+), 51 deletions(-) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 3745f36a99..18406ce1a4 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,4 +1,4 @@ -{"_type":"issue","id":"polylogue-b508","title":"21% of index sessions are metadata sidecars materialized as conversations (agent-*.meta, toolu_*, wf_*)","description":"## What the data shows\n\nClassifying every `claude-code-session` session_id in the live index by the shape\nof its native_id:\n\n 8,586 (52.6%) parent:agent-* real subagent transcripts\n 4,945 (30.3%) \u003cagent\u003e.meta SIDECAR METADATA, not a conversation\n 2,762 (16.9%) uuid real top-level sessions\n 7 wf_* workflow ids\n 3 toolu_* TOOL-USE ids\n 16,312 total\n\nContent of the suspicious classes:\n\n .meta 4,945 sessions 0 with messages 4,286 with events\n toolu_ 3 sessions 0 with messages 0 with events\n wf_ 7 sessions 0 with messages 0 with events\n\nThe `.meta` rows originate from\n`~/.claude/projects/\u003cproject\u003e/\u003csession-uuid\u003e/subagents/agent-\u003cid\u003e.meta.json` --\na per-subagent metadata sidecar. 5,053 raws come from `*.meta.json` paths.\n\n**The real subagent transcript is separately and correctly ingested.** Sampled\n300 `.meta` sessions and looked for the corresponding `%:agent-\u003cid\u003e` session:\n300 of 300 found. So these are not the only record of anything; they are\nduplicate phantom rows standing beside the real session.\n\nNet effect: 4,945 of 23,230 index sessions -- **21% of the archive's session\ncount** -- are metadata sidecars materialized as conversations.\n\n## Why this matters beyond a wrong count\n\nThis is the same pathology as the hook-event inflation already fixed once\n(83,286 -\u003e 18,391 sessions, `write_hook_event`, PR #3265): a per-session sidecar\nrecord ingested as a standalone session. A different sidecar type, the identical\nbug class, and it survived that repair because the fix was specific to hook\nevents rather than to the category.\n\nConsequences that are not merely cosmetic:\n\n- Every per-session aggregate -- counts, cost rollups, activity timelines,\n \"how many sessions did I have\" -- is inflated by 21% for claude-code.\n- 659 of them carry neither messages nor events, so they are pure empty rows.\n- Search and read surfaces can return a `.meta` session that has no content to\n show.\n- `toolu_*` sessions mean a TOOL-USE id was promoted to a session identity,\n which indicates identity derivation falling back to whatever id it found\n rather than failing loudly.\n\n## Hypothesis for the mechanism (needs confirming before fixing)\n\nProvider detection / payload lowering treats any JSON document under a\n`subagents/` directory as a session-bearing payload, so a `.meta.json` sidecar\nis lowered into a `LoweredPayloadSpec` and parsed. `provider_session_id` then\nfalls back to the filename stem (`agent-\u003cid\u003e.meta`), producing a well-formed but\nmeaningless identity. The `toolu_*` and `wf_*` cases look like the same fallback\npicking up whichever id field is present in a fragment.\n\nThat should be verified in `sources/dispatch.py` and the Claude Code parser\nbefore any fix -- the shape above is inference from the data, not yet traced in\ncode.\n\n## Direction\n\nTwo candidate fixes, and the second is the one that matches\n`polylogue-aggz`'s spirit:\n\n1. Narrow: skip `*.meta.json` under `subagents/`, and attach its content to the\n subagent session it describes rather than to a session of its own.\n2. Structural: a payload may only become a session when it yields a session\n identity the PROVIDER asserted. A filename-derived or fragment-derived\n fallback identity should be a parse refusal, not a session. That kills\n `.meta`, `toolu_*` and `wf_*` in one rule, and prevents the next sidecar\n format from doing this again -- which is exactly what the hook-event fix\n failed to do.\n\nPrefer (2), with (1) only if (2) proves too broad. Under (2) this stops being a\ncategory anyone has to remember.\n\n## Acceptance criteria\n\n- No session exists whose identity was derived from a filename stem or a\n non-session fragment id.\n- The metadata carried by `*.meta.json` is still retained and attached to the\n subagent session it describes -- this must not become data loss.\n- Sampled `.meta` ids resolve to their real `%:agent-\u003cid\u003e` session, which keeps\n its content.\n- claude-code session count drops by roughly 4,945; verify against\n `.agent/scripts/corpus-fidelity-audit.py` that absences do NOT rise, i.e. that\n nothing real was removed.\n- Anti-vacuity: state the production line mutated and the resulting failure.\n\nRef polylogue-aggz\n","status":"open","priority":0,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T16:10:03Z","created_by":"Sinity","updated_at":"2026-07-30T16:10:03Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-b508","title":"21% of index sessions are metadata sidecars materialized as conversations (agent-*.meta, toolu_*, wf_*)","description":"## What the data shows\n\nClassifying every `claude-code-session` session_id in the live index by the shape\nof its native_id:\n\n 8,586 (52.6%) parent:agent-* real subagent transcripts\n 4,945 (30.3%) \u003cagent\u003e.meta SIDECAR METADATA, not a conversation\n 2,762 (16.9%) uuid real top-level sessions\n 7 wf_* workflow ids\n 3 toolu_* TOOL-USE ids\n 16,312 total\n\nContent of the suspicious classes:\n\n .meta 4,945 sessions 0 with messages 4,286 with events\n toolu_ 3 sessions 0 with messages 0 with events\n wf_ 7 sessions 0 with messages 0 with events\n\nThe `.meta` rows originate from\n`~/.claude/projects/\u003cproject\u003e/\u003csession-uuid\u003e/subagents/agent-\u003cid\u003e.meta.json` --\na per-subagent metadata sidecar. 5,053 raws come from `*.meta.json` paths.\n\n**The real subagent transcript is separately and correctly ingested.** Sampled\n300 `.meta` sessions and looked for the corresponding `%:agent-\u003cid\u003e` session:\n300 of 300 found. So these are not the only record of anything; they are\nduplicate phantom rows standing beside the real session.\n\nNet effect: 4,945 of 23,230 index sessions -- **21% of the archive's session\ncount** -- are metadata sidecars materialized as conversations.\n\n## Why this matters beyond a wrong count\n\nThis is the same pathology as the hook-event inflation already fixed once\n(83,286 -\u003e 18,391 sessions, `write_hook_event`, PR #3265): a per-session sidecar\nrecord ingested as a standalone session. A different sidecar type, the identical\nbug class, and it survived that repair because the fix was specific to hook\nevents rather than to the category.\n\nConsequences that are not merely cosmetic:\n\n- Every per-session aggregate -- counts, cost rollups, activity timelines,\n \"how many sessions did I have\" -- is inflated by 21% for claude-code.\n- 659 of them carry neither messages nor events, so they are pure empty rows.\n- Search and read surfaces can return a `.meta` session that has no content to\n show.\n- `toolu_*` sessions mean a TOOL-USE id was promoted to a session identity,\n which indicates identity derivation falling back to whatever id it found\n rather than failing loudly.\n\n## Hypothesis for the mechanism (needs confirming before fixing)\n\nProvider detection / payload lowering treats any JSON document under a\n`subagents/` directory as a session-bearing payload, so a `.meta.json` sidecar\nis lowered into a `LoweredPayloadSpec` and parsed. `provider_session_id` then\nfalls back to the filename stem (`agent-\u003cid\u003e.meta`), producing a well-formed but\nmeaningless identity. The `toolu_*` and `wf_*` cases look like the same fallback\npicking up whichever id field is present in a fragment.\n\nThat should be verified in `sources/dispatch.py` and the Claude Code parser\nbefore any fix -- the shape above is inference from the data, not yet traced in\ncode.\n\n## Direction\n\nTwo candidate fixes, and the second is the one that matches\n`polylogue-aggz`'s spirit:\n\n1. Narrow: skip `*.meta.json` under `subagents/`, and attach its content to the\n subagent session it describes rather than to a session of its own.\n2. Structural: a payload may only become a session when it yields a session\n identity the PROVIDER asserted. A filename-derived or fragment-derived\n fallback identity should be a parse refusal, not a session. That kills\n `.meta`, `toolu_*` and `wf_*` in one rule, and prevents the next sidecar\n format from doing this again -- which is exactly what the hook-event fix\n failed to do.\n\nPrefer (2), with (1) only if (2) proves too broad. Under (2) this stops being a\ncategory anyone has to remember.\n\n## Acceptance criteria\n\n- No session exists whose identity was derived from a filename stem or a\n non-session fragment id.\n- The metadata carried by `*.meta.json` is still retained and attached to the\n subagent session it describes -- this must not become data loss.\n- Sampled `.meta` ids resolve to their real `%:agent-\u003cid\u003e` session, which keeps\n its content.\n- claude-code session count drops by roughly 4,945; verify against\n `.agent/scripts/corpus-fidelity-audit.py` that absences do NOT rise, i.e. that\n nothing real was removed.\n- Anti-vacuity: state the production line mutated and the resulting failure.\n\nRef polylogue-aggz\n","status":"in_progress","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T16:10:03Z","created_by":"Sinity","updated_at":"2026-07-30T16:48:33Z","started_at":"2026-07-30T16:48:33Z","labels":["area:ingest"],"comments":[{"id":"019fb3ee-7eaf-71b8-8e5a-d1d66efffce1","issue_id":"polylogue-b508","author":"Sinity","text":"## Traced mechanism (not the original hypothesis)\n\nThe original hypothesis (\"provider detection treats any JSON under\nsubagents/ as session-bearing, provider_session_id falls back to the\nfilename stem\") was PARTLY wrong and PARTLY right, in a way that matters.\n\n**Live daemon ingest path (sources/live/batch.py + pipeline/services/\ningest_worker.py) already refuses this correctly**, and has since well\nbefore this session (classify_artifact_path's agent-*.meta.json branch\ndates to 82fc0e4ff2, 2026-03-27; the OriginSpec artifact_rule_for_path\nroute that shadows it is newer but agrees). Proved empirically: built a\nthrowaway archive and ingested 9 REAL files pulled from\n~/.claude/projects (1 top-level session, 3 real agent-*.jsonl subagent\ntranscripts, 4 real agent-*.meta.json sidecars, 1 standalone\nmeta+transcript pair) through LiveBatchProcessor (same primitives\npolylogued run wires up) -- result: exactly 5 real sessions, 0 phantom\n`.meta` rows.\n\n**The actual live bug is a second, separate parse chokepoint**:\n`sources/revision_backfill.py` (`_parse_one`/`_parse_stream`, driving\n`polylogue ops reset --index` / the offline rebuild-index path via\n`backfill_historical_revision_evidence`) calls\n`dispatch.parse_payload`/`parse_stream_payload` on every retained raw\nUNCONDITIONALLY -- no OriginSpec/artifact-taxonomy gate at all. Reproduced\nlive: rebuilding an index from the same 9-file real corpus through this\npath (bypassing the daemon) produced 9 sessions, 4 of them phantom\n`claude-code-session:agent-\u003cid\u003e.meta` rows with 0 messages/0 events --\nthe EXACT reported shape. `fallback_id = Path(source_path).stem` on\n`agent-\u003cid\u003e.meta.json` strips only the trailing `.json`, leaving\n`agent-\u003cid\u003e.meta` -- literally the observed native_id.\n\nThis means the bead's suggested remediation (\"index.db is rebuildable,\nprefer a rebuild\") would have RECREATED the defect it was meant to fix,\nnot eliminated it -- this is now fixed (see below), so the plan below is\nsafe.\n\nA structural gap also existed independent of both mechanisms:\n`dispatch.py:_generic_messages_session` (the one payload-lowering branch\nwith zero provider-specific identity handling, reached both by genuinely\nunknown providers and by the Drive-like generic fallback) fell back to\n`fallback_id` -- a filename stem the *source-discovery walk* invented --\nwhenever a payload had a `messages` list but no `id` field. Didn't\nreproduce with real `.meta.json`/`toolu_*`/`wf_*` fixtures (those are\ncovered by the OriginSpec/artifact-taxonomy path rules), but is exactly\nthe \"next sidecar format\" risk the bead is about, and closing it is what\nimplements the structural rule generically rather than per-shape.\n\n## Fixes shipped (PR, branch feature/fix/provider-asserted-session-identity)\n\n1. `polylogue/sources/dispatch.py`: `_generic_messages_session` now\n requires the payload to assert its own `id`; absent that it refuses to\n parse (returns None) instead of synthesizing an identity from\n `fallback_id`.\n2. `polylogue/sources/revision_backfill.py`: `_parse_one`/`_parse_stream`\n now consult `artifact_rule_for_path` (same OriginSpec table batch.py\n already uses) and refuse to parse (return `[]`) when the declared\n artifact's `parse_policy` isn't `\"session\"`. One rule table, enforced\n at both entry points -- a rebuild and a live ingest now agree.\n\nBoth fixes proven with:\n- Unit regression tests\n (`tests/unit/sources/test_source_laws.py::test_parse_payload_generic_messages_without_asserted_id_refuses_to_parse`,\n `tests/unit/sources/test_revision_backfill.py::test_parse_one_refuses_declared_fact_artifacts`)\n that fail before the fix and pass after (anti-vacuity verified by\n reverting each fix in isolation and re-running).\n- The real 9-file fixture-corpus rebuild: 9 sessions / 4 phantom before\n fix #2, 5 sessions / 0 phantom after, with the 3 real subagent\n transcripts' message counts (96, 120, 164, 31... unaffected across the\n run) identical in both states -- no data loss to real content.\n- `devtools test tests/unit/sources/test_source_laws.py\n tests/unit/sources/test_revision_backfill.py` -- 180 passed.\n\n## AC: metadata retention (not data loss)\n\nAlready satisfied by existing, pre-existing code, unaffected by this fix:\n`insights/claude_workflow_materializer.py` +\n`insights/claude_workflow_evidence.py` read `agent_sidecar_meta` facts\nfrom retained raw bytes (independent of whether a `sessions` row exists)\nand materialize them into the `claude-workflow:*` work-evidence graph\n(run/invocation/attempt nodes with sidecar-meta claims attached). This\nfix only removes the DUPLICATE phantom `sessions` row; the raw bytes stay\nin `raw_sessions` (admitted as \"fact\" artifacts) and the metadata content\nkeeps flowing into that graph exactly as before.\n\n## `toolu_*` / `wf_*` (10 rows total, not separately reproduced)\n\n`wf_*` (workflow_run_snapshot, `.json`) is covered by the same\nOriginSpec-declared-fact gate as `.meta.json` -- fix #2 covers it\nstructurally, same mechanism.\n\n`toolu_*` (3 rows) could not be reproduced with real fixture data: real\n`tool-results/*.txt` sidecars are excluded from the live discovery walk\nby suffix filtering (`artifact_suffixes_for_provider` only allows\n`.json`/`.jsonl`/`.ndjson` for claude-code) and are NOT declared in\nOriginSpec at all, so if a `raw_sessions` row for one of these 3 exists\nin the live archive it's very likely a relic of an older\nacquisition-scope bug already superseded by that suffix filtering. Given\nthere are only 3 (vs 4,945 `.meta`), recommend: after the rebuild below,\ncheck whether they're gone; if any survive, file a narrow follow-up bead\nwith their actual `source_path`/payload shape rather than guessing\nfurther blind.\n\n## Verified live-data remediation procedure\n\nindex.db is the rebuildable tier; source.db (raw bytes) is durable and\nuntouched by this fix. With both fixes merged and deployed:\n\n1. Stop anything writing to the live archive (already stopped per the\n session's safety rule).\n2. `polylogue ops reset --index` -- wipes only the index tier (new\n generation), source.db/user.db/ops.db untouched.\n3. `polylogued run` (or the offline `devtools`/maintenance rebuild-index\n path) -- replays EVERY `raw_sessions` row from source.db through the\n now-fixed `revision_backfill.py` path. Verified there is no\n `parsed_at_ms`-style skip: `all_index_rebuild_raw_ids` selects every\n raw unconditionally and `RebuildIndexRequest(only_missing=False)`\n forces a full non-incremental replay; content-hash idempotency only\n skips re-writing a session that ALREADY EXISTS in the target, which is\n moot against a freshly wiped, empty index.db. So no additional\n durable-tier invalidation beyond the code fix is needed -- the raws\n ARE the source of truth and will now be reparsed correctly.\n4. Verify with `.agent/scripts/corpus-fidelity-audit.py` (or equivalent\n session-count query) that: claude-code session count drops by\n approximately 4,945+7(+ up to 3), absences do NOT rise (nothing real\n removed), and the 300-sample `.meta -\u003e %:agent-\u003cid\u003e` resolution check\n from the original investigation still resolves (the real subagent\n sessions are untouched by this fix -- it only removes the duplicate).\n\nNot run against the live archive per the session's explicit\ninstruction -- this is the procedure to execute, not evidence that it was\nexecuted.\n","created_at":"2026-07-30T16:49:39Z"}],"dependency_count":0,"dependent_count":0,"comment_count":1} {"_type":"issue","id":"polylogue-aggz","title":"Collapse the failure taxonomy into three invariants that make the cases unrepresentable","description":"## The problem with the current shape\n\nOne day of investigation produced eleven separately-named defects and found four\nexisting special-case code paths. That is a taxonomy, not an architecture. Every\nnew provider quirk becomes another named category, another branch, another bead,\nand the system's correctness becomes a function of how many cases someone\nremembered. The goal is the opposite: make these failures unrepresentable, so\nthey are not known as anything at all.\n\nAlmost all of it collapses into three invariants.\n\n## Invariant 1 -- comparison identity contains only content\n\n**A conversation is a SET of messages keyed by stable provider identity, each\ncarrying only content-bearing fields. Nothing else may enter the value used to\ncompare two acquisitions of it.**\n\nCollapses, as consequences rather than cases:\n\n- polylogue-bu1i (attachment acquisition state in attachment identity) --\n acquisition state is not content.\n- polylogue-c429 (message array order) -- a set has no order.\n- polylogue-nuec (chatgpt elapsed_duration_ms) -- a measurement is not content.\n- polylogue-hith (synthetic attachment id seeded on position) -- position is not\n identity.\n- polylogue-d8al (real-id presence varies between vintages) -- identity must be\n derivable from content when the provider omits its own.\n- polylogue-oycw (positional-prefix superset test) -- set containment, not\n sequence prefix.\n- `_provider_ordered_browser_snapshots` -- exists only because DOM ordering\n differs from export ordering. Under a set, it has nothing to fix.\n- The `superseded_prefix` / `superseded_equivalent` distinction -- both are just\n \"contained or equal\".\n\nSuperset-ness becomes total and decidable, with no residual category:\n\n equal same id set, equal content per id\n contains A's id set contains B's, equal content on the intersection\n conflict content differs on the intersection\n\nOrdering remains a stored, rendered property of a session. The claim is only\nthat it is not part of the comparison value. `_direct_export_precedence` (a real\nexport outranks a browser capture) probably survives as a genuine provenance\nrule rather than a repair.\n\n## Invariant 2 -- one chokepoint may write a session\n\n**It must be structurally impossible to materialize a session without consulting\nrevision authority.**\n\npolylogue-c737, PR #3397 and PR #3398 all exist because two write paths each\ncarried their own precedence logic, and one of them forgot. #3398 then had to\ncorrect #3397's scope on one path while the other stayed wrong, which is the\nsignature of duplicated semantics rather than a missing check.\n\nThe fix is structural, not another check: one function through which every\nsession write passes, taking authority as a required argument, so a caller\ncannot forget to ask. A predicate copied into two places is a bug that has not\nhappened yet.\n\n## Invariant 3 -- derived state carries the version of the logic that derived it\n\n**Any stored conclusion records which version of which computation produced it,\nso a corrected computation invalidates its own stale outputs automatically.**\n\npolylogue-9dxn is this, and its absence is what made polylogue-bu1i inert on\nexisting data: a persisted `ambiguous` verdict has no version, so a corrected\nclassifier cannot know which verdicts it now disagrees with. The two-component\ndesign already recorded on 9dxn (separate identity and classification\nfingerprints) is the mechanism.\n\nWith this, \"stale verdict\", \"needs re-census\", and \"the fix does not apply to\nexisting rows\" all stop being categories. Correction becomes self-healing by\nconstruction.\n\n## What this does to the current bead set\n\nReframe rather than close -- the individual fixes still ship, but as instances:\n\n bu1i c429 nuec hith d8al oycw -\u003e Invariant 1\n c737 (+ the shape behind #3397/#3398) -\u003e Invariant 2\n 9dxn -\u003e Invariant 3\n ck5v -\u003e not covered; genuinely separate\n (backfill coupled to acquisition\n route -- an availability rule, not\n an identity one)\n ey3r -\u003e a measurement defect, but its cause\n is Invariant 1: it counts\n `superseded_*` as missing because\n the vocabulary has redundant\n categories that Invariant 1 removes\n\n## How to tell whether this worked\n\nNot \"the tests pass\". The observable is that the vocabulary shrinks:\n\n- The membership decision vocabulary loses `superseded_prefix` as distinct from\n `superseded_equivalent`.\n- `_provider_ordered_browser_snapshots` is deleted rather than maintained.\n- `HISTORICAL_NON_PREFIX_GOVERNANCE_DETAIL` and its legacy-detail variants stop\n needing to exist, because non-prefix growth stops being exceptional.\n- No new provider quirk requires a new branch in the classifier.\n\nIf a change adds a case instead of removing one, it is going the wrong way even\nif it makes a test pass. Per this repo's own surgical-renewal rule, the old path\nis deleted in the same change that replaces it -- these special cases must not\nsurvive as dead alternates beside the invariant.\n\n## Acceptance criteria\n\n- The comparison value for a session is constructed from an explicit\n content-only allowlist, so adding a field to a parser cannot silently enter\n identity. Adding a volatile field and observing that comparison is unaffected\n is the test.\n- Exactly one code path can write a session, and it cannot be called without\n authority.\n- Every stored verdict carries a version; changing the logic invalidates the\n affected verdicts without an operator command.\n- At least two existing special-case paths are DELETED, not merely bypassed.\n","status":"open","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T14:41:28Z","created_by":"Sinity","updated_at":"2026-07-30T14:41:28Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-oycw","title":"Coalescing rests on a positional-prefix superset test that real providers violate; 41% of the corpus depends on it","description":"## Scale first: this is the archive's normal condition, not an edge case\n\n logical identities with more than one raw 7,440\n total logical identities 18,228\n -\u003e 41% of the corpus is multi-raw\n\nCohort sizes by origin (raws in multi-member cohorts):\n\n chatgpt-export 3-member 5,817 4-member 551 +tail to 12\n claude-ai-export 4-member 3,592 3-member 258 +tail to 9\n codex-session 2-member 3,544 ... one cohort of 105\n claude-code-session 2-member 2,338 3-member 663 +tail to 25\n hermes-session 2-member 536\n aistudio-drive 2-member 302\n antigravity-session 2-member 232\n browser-capture raws 887 (786 chatgpt, 47 claude-ai, 38 unknown, 16 grok)\n\nCorrectness for nearly half the archive rests on the revision-arbitration layer.\nIt is not a rarely-exercised safety net.\n\n## Where the multiplicity comes from\n\nNot divergence. Repeated whole-account acquisition:\n\n claude-ai-data-2025-10-04 906 raws\n claude-ai-data-2026-04-23 973 raws\n claude-ai-data-2026-06-14 1,998 raws\n chatgpt-data-2025-10-20 2,072 raws\n chatgpt-data-2026-04-23 4,805 raws\n\nEvery GDPR export contains every conversation, so each conversation enters the\narchive once per export vintage. 577 of the 587 claude-ai ambiguous cohorts have\nexactly 4 members for this reason.\n\n## Layer 1 -- identity. This one is sound.\n\n`sessions.session_id` is a generated column, `origin || ':' || native_id`, where\n`native_id` is the parser's `provider_session_id` -- the provider's own\nconversation uuid. Measured: 34 of 35 sampled claude-ai cohorts have an\nIDENTICAL provider_message_id set across all members, and the conversation uuid\nis identical across all four export vintages.\n\nSession identity is stable across acquisitions. The failures found on\n2026-07-30 were narrower and are separately tracked: a dispatch bug appending a\nspurious `-0` (fixed 2026-07-20, polylogue-eqnv), and unstable synthetic\n*attachment* ids (polylogue-hith / polylogue-d8al) -- not session ids.\n\n**Identity is not the problem, and a fix aimed at identity will not help.**\n\n## Layer 2 -- coalescing. Two mechanisms that do not compose.\n\n**(a) Content-hash idempotency** (`pipeline/ids.py:session_content_hash`).\nRe-ingest with a matching hash is skipped. The hash deliberately excludes user\nmetadata, but it INCLUDES: message array order, attachment acquisition state,\nvolatile provider metadata, and synthetic ids. Across two exports of an\nunchanged conversation, at least one of those always differs.\n\nSo idempotency never fires across export vintages -- by construction, not by\naccident. Every re-export falls through to (b).\n\n**(b) Revision membership arbitration.** Decides which raw is authoritative for\na session id when hashes differ. This carries the entire load that (a) fails to\nabsorb, for 41% of the corpus.\n\n## Layer 3 -- superset determination. This is the actual defect.\n\n`_strictly_dominates` (`archive/session_revision_membership.py`) requires:\n\n older.message_hashes == newer.message_hashes[: len(older.message_hashes)]\n\na POSITIONAL PREFIX. Three assumptions are embedded there, and all three are\nviolated by real providers:\n\n1. *Messages keep a stable order across acquisitions.* Violated: 19 of 35\n sampled claude-ai cohorts differ only in array order, same ids, zero content\n differences. Claude.ai does not emit a stable sequence between exports.\n2. *A message's hash is a function of its content alone.* Violated by volatile\n provider metadata (chatgpt `elapsed_duration_ms`, polylogue-nuec) and by\n acquisition state (Drive attachment bytes, polylogue-bu1i).\n3. *Growth is append-only at the tail.* Violated whenever a provider edits or\n inserts mid-conversation, and structurally by browser-capture DOM snapshots.\n\nWhen the test fails in both directions the cohort is quarantined ambiguous and\nNOTHING is indexed -- so a conversation held complete, correct, and in four\nidentical copies is absent from the archive. That is the 1,009-1,027 absence\npopulation.\n\n## What the correct test looks like\n\nPer-message ids are stable (34/35 measured), so superset-ness is decidable on\nevidence we already hold, without ordering:\n\n equal same provider_message_id SET, equal content per id\n -\u003e semantically the same revision; `equivalent_raw_ids`,\n no arbitration needed at all\n dominates A's id set strictly contains B's, content equal on the\n intersection -\u003e A is authoritative\n fork neither contains the other, OR content differs on the\n intersection -\u003e genuinely ambiguous, and rare\n (0 of 35 sampled claude-ai; 1 plausible case archive-wide,\n in grok-export)\n\nOrdering remains a real property of a session and must still be stored and\nrendered -- the claim is only that ordering must not be the DOMINANCE key.\nA conversation is a set of identified messages plus an ordering; which evidence\nexists is a set question, and treating the sequence as identity makes every\nprovider-side reordering look like divergence.\n\nLikewise a message's identity for comparison must exclude provider-volatile\nmeasurement fields and acquisition state, for the same reason bu1i split\nattachment identity from attachment acquisition.\n\n## Browser capture\n\n887 raws, 786 of them chatgpt. A DOM snapshot legitimately carries different\nsynthetic ids and a different ordering from the same conversation's export, so\nit violates assumptions 1 and 3 by design. `_provider_ordered_browser_snapshots`\nand `_direct_export_precedence` exist to special-case it, which is evidence that\nthe general test was already known to be too strict -- the special cases are\npatches over the wrong primitive rather than genuine domain rules. Re-evaluate\nboth once the set-based test lands; `_direct_export_precedence` (a real export\noutranks a browser capture) is probably a genuine rule worth keeping, while the\nordering special-case may become unnecessary.\n\n## Acceptance criteria\n\n- Superset determination is order-independent and decided on stable per-message\n identity plus per-id content equality.\n- Equal-content cohorts resolve as `equivalent`, not `ambiguous`, and index one\n member -- no arbitration for the 34/35 case.\n- Message comparison identity excludes provider-volatile measurement fields and\n acquisition state.\n- Report how many cohorts still reach a genuine-fork verdict; it should be very\n small, and a large number means one of the above is wrong.\n- Re-run `.agent/scripts/corpus-fidelity-audit.py`: absent_documents must fall\n to approximately zero from the 1,027 baseline.\n\nRef polylogue-bu1i, polylogue-c429, polylogue-nuec, polylogue-d8al, polylogue-f1vg\n","status":"open","priority":0,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T14:35:25Z","created_by":"Sinity","updated_at":"2026-07-30T14:35:25Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-c737","title":"ArchiveStore._write_parsed_precedence_result writes a session for a raw recorded raw_session_memberships.decision='ambiguous'","description":"## What the live archive shows (coordinator measurement, confirmed independently)\n\nLive query against `/realm/db/polylogue` for `origin='aistudio-drive'`:\n34 cohorts now have BOTH members parsed (growing over the course of one\nsession). 28 of those have `raw_session_memberships.decision='ambiguous'`\non BOTH members under the SAME `logical_source_key` -- genuinely arbitrated,\ncorrectly refused a winner -- yet the cohort's session IS present in\nindex.db with zero acquired attachments: 641 attachments total across the\n28, every one `unfetched`, while the enriched sibling in the blob store\nholds the bytes.\n\n## Root cause, traced and reproduced\n\n`polylogue/storage/sqlite/archive_tiers/archive.py`'s\n`apply_raw_membership_classification` (the classify_membership_revisions\nconsumer) is innocent: for a fully-ambiguous cohort (no accepted_raw_ids)\nit explicitly clears `raw_sessions.parsed_at_ms` and never writes to\n`sessions` -- verified by reading its finalization block (`complete` check\nat the end of the function, ~line 3873-3901).\n\nThe actual writer is `_write_parsed_precedence_result` (same file), reached\nvia `write_parsed_for_retained_raw`/`write_parsed_for_retained_raw_result`\nwith `revision_authoritative=False` (the default -- used by the one-shot\nimporter, `pipeline/services/archive_ingest.py`, and by\n`_index_parsed_for_retained_raw`'s other non-membership-governed callers).\nIts ONLY revision-authority awareness before this fix was:\n\n governed = SELECT 1 FROM raw_revision_heads WHERE session_id = ?\n if governed is not None: skip\n\n`raw_revision_heads` is populated ONLY when a cohort has an ACCEPTED\nwinner. A cohort `classify_membership_revisions` genuinely refused to\narbitrate never gets an accepted head, so `governed` stays `None` and the\nfunction falls through to its own browser-capture-precedence/freshness\nlogic and writes the session unconditionally on the raw's next reparse --\nlast-writer-wins, independent of the recorded `ambiguous` verdict.\n`repair.py:1075`/`repair.py:4432-4462` (the two gates the investigation\nstarted from) are both innocent: neither is on this write path at all --\n`repair.py:4432` is a read-only reporting/accounting classifier\n(`_raw_replay_plan_outcome`), and `repair.py:1075` is a narrow inspector\nfor a different (`source-v7`/`quarantined-accepted-raw`) repair scenario.\n\nReproduced directly: a synthetic archive with a raw whose\n`raw_session_memberships` row is `decision='ambiguous'`, then calling\n`archive.write_parsed_for_retained_raw(session, raw_id=..., ...)` (no\n`revision_authoritative`) writes the session anyway pre-fix; the fix makes\nit a no-op (`content_changed=False`).\n\n## Fix landed in polylogue-af059's fast-follow PR\n\n`_write_parsed_precedence_result` now also refuses when the raw's OWN\n`raw_session_memberships.decision = 'ambiguous'`, in addition to the\nexisting `raw_revision_heads` check.\n\n## Known sibling hole, NOT fixed here (different file, different owner)\n\n`polylogue/pipeline/services/ingest_batch/_core.py` (the daemon's default\nbatch-ingest write path, used for most origins that don't go through\n`sources/live/batch.py`'s revision-authority-aware branch) has the SAME\nshape: its own precedence/freshness logic, no `raw_session_memberships`\nconsultation. The coordinator's own measurement\n(`revision_authority='quarantined'` with `parsed_at_ms` set: chatgpt-export\n7,050, codex-session 3,633, claude-code-session 2,450, claude-ai-export\n1,562 -- NOT all necessarily leaked materializations, but the same shape)\nsuggests this is where most of the non-drive volume would leak through, if\nthose origins' raws ever get genuinely `ambiguous`-recorded membership\ndecisions. Needs its own read-only census to confirm before fixing (not\ndone here -- out of file-ownership scope for this PR).\n\n## Live remediation\n\nNOT performed here (code-only fix). The 28 live aistudio-drive sessions\nwith zero-acquired attachments need their own re-materialization pass once\nthis fix (and bu1i's classifier fix) are both deployed.\n\nRef polylogue-eqnv, polylogue-bu1i, polylogue-7ilr, polylogue-9dxn","status":"closed","priority":0,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T13:09:15Z","created_by":"Sinity","updated_at":"2026-07-30T13:20:58Z","closed_at":"2026-07-30T13:20:58Z","close_reason":"Fixed in PR #3397 (feature/fix/ambiguous-membership-precedence-write-leak): ArchiveStore._write_parsed_precedence_result now also refuses to write when the raw's own raw_session_memberships.decision='ambiguous', alongside the pre-existing raw_revision_heads check. Verified with a regression test (anti-vacuity confirmed via temporary guard short-circuit + rerun). Sibling hole in pipeline/services/ingest_batch/_core.py NOT fixed here (different file, out of ownership scope) -- needs its own read-only census before fixing; tracked as residual scope in this same bead's description.","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -72,6 +72,7 @@ {"_type":"issue","id":"polylogue-tf2.1","title":"Rerun forensics on current archive; price origin_reported providers","description":"Rerun scripts/agent_forensics.py against the current archive (v23+); price origin_reported providers via the vendored LiteLLM catalog (match last path segment); all-provider headline or explicitly-labeled per-provenance figures that cannot be misread; record deltas vs 06-27; verify chart SVGs render. Cache-inclusion must be disambiguated (Codex input INCLUDES cached ~96%; see bd memories). Also blocked on logical-session token attribution — the headline must not be double-counted.","notes":"Correction to close_reason monetary values: stored/provider-priced subset was $239,453.14; catalog API-equivalent was $318,650.88; origin_reported catalog estimate was $79,197.74. The original close_reason text lost dollar-prefixed digits due shell expansion, not measurement drift.","status":"closed","priority":0,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:33Z","created_by":"Sinity","updated_at":"2026-07-03T09:59:13Z","started_at":"2026-07-03T09:28:10Z","closed_at":"2026-07-03T09:59:02Z","close_reason":"Completed with blocker caveat preserved: scripts/agent_forensics.py now prices origin_reported rows through the shared vendored LiteLLM pricing catalog while preserving stored provenance; report separates stored/provider-priced cost from catalog API-equivalent estimates and carries logical-session/cache caveats instead of claiming final billing reconciliation. Regenerated current artifact at .agent/demos/agent-forensics against /home/sinity/.local/share/polylogue schema v23: 16,498 physical sessions, 4,142,175 messages, 356.5B tokens, ,453.14 stored/provider-priced subset, ,650.88 catalog API-equivalent, and ,197.74 origin_reported catalog estimate. SVG parse check passed for 9 charts; devtools test tests/unit/scripts/test_agent_forensics.py passed; devtools verify --quick passed run 20260703T095718Z-quick-753466-96559776; devloop-review clean. Remaining final-reconciliation blocker stays open as polylogue-4ts.2.","labels":["area:usage","campaign"],"dependencies":[{"issue_id":"polylogue-tf2.1","depends_on_id":"polylogue-4ts.2","type":"blocks","created_at":"2026-07-03T06:32:45Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-tf2.1","depends_on_id":"polylogue-sru.7","type":"blocks","created_at":"2026-07-03T06:31:33Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-tf2.1","depends_on_id":"polylogue-tf2","type":"parent-child","created_at":"2026-07-03T06:31:33Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"polylogue-tf2","title":"Campaign: agent-forensics regeneration + all-provider repricing","description":"Regenerate the agent-forensics packet on the current archive with an honest all-provider headline. The 2026-06-27 report (546.6B tokens, $89,368 API-list equivalent, 216x cache amplification) is the most stranger-legible artifact on any shelf, but its numbers are pre-dedup stale and the headline prices only the priced-provenance subset (Claude Code cost_usd rows); Codex/ChatGPT/Gemini are origin_reported token counts with no dollar value (operator estimate ~$150K all-provider). Sequenced after claim-vs-evidence per operator direction 2026-07-02.","design":"Current slice design: turn the existing agent-forensics/cost headline into a product-backed all-provider repricing artifact. First inspect devtools/scripts and polylogue analyze surfaces for agent_forensics/cost code. Use active archive usage headline (detail=headline) for authoritative physical_session and logical_session_model_high_water token totals. Keep priced-provenance dollars and origin-reported token estimates separate: do not multiply every token by one blended price without a labeled lane. Add or reuse a shared pricing/projection helper so the demo artifact is regenerated from Polylogue product code, not ad hoc SQL. Acceptance for this slice: the generated agent-forensics artifact names archive root/schema, includes physical vs logical token grain, separates priced subset from origin-reported estimate lanes, gives reproduction commands, and has focused tests for any new repricing helper/surface.","acceptance_criteria":"Terminal state: regenerated forensics packet on the current archive with an honest all-provider headline (priced subset AND origin-reported estimate lanes separated), agent_forensics.py folded into polylogue analyze (tf2.2), artifact on the demo shelf with reproduction commands, cold-reader gate passed. Epic closes only when that artifact is recorded.","status":"closed","priority":0,"issue_type":"epic","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:32Z","created_by":"Sinity","updated_at":"2026-07-03T19:06:44Z","started_at":"2026-07-03T18:47:23Z","closed_at":"2026-07-03T19:06:44Z","close_reason":"Completed: provider usage headline now exposes product-backed pricing lanes in polylogue analyze usage --detail headline, separating stored/provider-priced cost from catalog API-equivalent estimates for origin_reported rows. Regenerated the current .agent/demos/agent-forensics artifact against /home/sinity/.local/share/polylogue schema v23: physical-session tokens 395,320,980,423; logical high-water tokens 288,741,229,728; stored/provider-priced USD 243,392.189328; catalog API-equivalent USD 337,565.031618; priced lane 13,889 rows / 12,331 sessions / 12,650 matched rows; origin_reported lane 2,308 rows / 2,270 sessions / 2,302 matched rows. Verification: live polylogue --plain analyze usage --detail headline --format json --limit 0 wrote /realm/tmp/polylogue-usage-headline-pricing-current.json; devtools test tests/unit/storage/test_provider_usage_report.py tests/unit/cli/test_diagnostics.py passed 23 tests; devtools verify --quick passed run 20260703T190553Z-quick-2226137-d91d4e8f; devtools workspace demo-shelf --json reported ok. Non-claim preserved: this is not final billing reconciliation and physical/logical token grains stay explicitly separated.","labels":["area:usage","campaign","size:M","spine"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-sru","title":"Campaign: claim-vs-evidence report to finding-grade","description":"Terminal state: an externally publishable finding ('how often do coding agents proceed past failed tool calls, by model/tool') with stated sample frame, calibrated markers, benign/consequential split, seeded stranger-runnable reproduction, and a passed cold-reader gate. Slice closure is NOT campaign closure; this epic stays top-of-frame until its terminal state is recorded.\\n\\nState as of 2026-07-03 after calibrated active-archive regeneration: archive root /home/sinity/.local/share/polylogue, index schema v23, 41,886 structured failures total, 5,000 origin-stratified failures inspected (3,746 claude-code-session, 1,247 codex-session, 7 claude-ai-export), 100 unpaired structured failures. Marker vocabulary was tightened to avoid broad issue/fix/block/gitignored false positives. Immediate next-turn totals: acknowledged=420, silent_proceed=1,205, ambiguous=3,375 (2,624 wordless tool continuations; 751 prose without marker). Lower-bound silent rate is 24.1%; among classified immediate next turns, silent rate is 74.2%. Next-3 sensitivity window, stopping before the next user message, finds 302 acknowledgments that appear only after the next turn; window3 silent lower bound is 37.0%. Calibration: 50 hand-labeled immediate-next-turn rows, acknowledged-marker precision=1.0, recall=0.8421052631578947, invalid rows=0. Artifact: .agent/demos/claim-vs-evidence/claim-vs-evidence.report.json.","notes":"2026-07-03 update: methodology package is now cold-read gated. .agent/demos/claim-vs-evidence contains aggregate live evidence, public-summary.json, PUBLIC_REPRODUCTION.md, COLD_READER_GATE.md, and COLD_READ_RESULT.md. Seeded reproduction is meaningful, not empty: 4 structured failures, 2 acknowledged follow-ups, 2 silent-proceed follow-ups, 0 unpaired. Cold-reader subagent PASS recovered claim/non-claim, sample frame, rates, calibration, caveats, and reproduction commands from the artifact directory only. Remaining campaign child: polylogue-sru.1 productizes action-unit outcome/followup_class capability.","status":"closed","priority":0,"issue_type":"epic","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:26Z","created_by":"Sinity","updated_at":"2026-07-03T09:28:09Z","closed_at":"2026-07-03T09:28:09Z","close_reason":"Completed: all seven campaign children are closed. The claim-vs-evidence finding now has bounded sample-frame reporting, calibrated marker precision/recall, handler-class and next-3 sensitivity splits, meaningful seeded reproduction, cold-reader PASS, and productized action-unit followup_class/followup_message_ref query capability. Current artifact lives under .agent/demos/claim-vs-evidence and was regenerated against /home/sinity/.local/share/polylogue schema v23.","labels":["area:substrate","campaign"],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-6kur","title":"Cull the repair surface: 10k lines of manual repair against 2.7k of convergence, with targets guarding schema-impossible states","description":"## Measured shape\n\n repair/maintenance surface ~10,164 lines\n storage/repair.py 7,154 (123 top-level defs, 22 public entrypoints)\n maintenance/*.py 3,010\n daemon convergence 2,665 lines\n convergence.py 637\n convergence_stages.py 2,028\n\nA 3.8:1 ratio of manual repair machinery to the automatic convergence meant to\nmake it unnecessary. Convergence registers only FIVE stages: `fts`, `embed`,\n`insights`, `claude_workflow`, `sinex_publication`. `repair.py` exposes eleven\nrepair targets.\n\nThis contradicts the project's own stated principle: *if Polylogue can maintain\na condition fully automatically it should, there is NO break-glass tier, and\nonce the automatic path maintains an invariant the redundant manual surface is\nDELETED rather than demoted.*\n\n## Per-target analysis (live archive, frozen 2026-07-30)\n\nNote first: `REPAIR_HANDLERS[target]` is a name-\u003efunction dispatch table, so\n\"no external references\" means dynamically dispatched, NOT dead. Every target\nbelow is reachable via `run_safe_repairs`/`run_archive_cleanup`.\n\n### Structurally impossible — delete (strongest case)\n\n| target | live violations | why it cannot occur |\n| --- | -- | --- |\n| `orphaned_messages` | **0** | `messages.session_id TEXT NOT NULL REFERENCES sessions(session_id) ON DELETE CASCADE` |\n| `orphaned_attachments` | **0** | `attachment_refs.session_id`/`message_id` both `NOT NULL ... ON DELETE CASCADE` |\n\n`PRAGMA foreign_keys = ON` is set in `storage/sqlite/connection_profile.py`, so\nthese are enforced, not decorative. The schema forbids the state; the repair\nscans for it anyway. Zero violations is not luck.\n\nDelete both repairs, both previews, their `SAFE_REPAIR_TARGETS`/`CLEANUP_TARGETS`\nentries, and their debt-status rows.\n\n### Spent one-shot migrations — delete once confirmed\n\n| target | live violations | note |\n| --- | -- | --- |\n| `message_type_backfill` | **0** | A backfill for a column added later. Confirm the write path always sets it (NOT NULL would settle it), then the migration is spent. |\n\nA backfill is inherently one-shot: once the historical rows are filled and the\nwriter populates the column, the repair guards nothing.\n\n### Symptom-treating — the repair is the wrong fix\n\n| target | live violations | note |\n| --- | -- | --- |\n| `session_timestamp_backfill` | **5,382, GROWING** | Was 1,117 after the hook de-inflation; now 5,382. A backfill whose backlog grows means the WRITE PATH is still producing the defect. |\n\nThis is the \"fix the automatic path\" case, and the most valuable finding here.\nDo not keep running the backfill; find why sessions are still written with\n`created_at_ms IS NULL` and stop that. The repair has been masking a live\nwriter bug, which is exactly what a break-glass tier does to you.\n\n### Cause fixed elsewhere — expect near-no-op\n\n| target | live violations | note |\n| --- | -- | --- |\n| `empty_sessions` | 5,255, of which **4,945** are `.meta` phantoms | PR #3403 fixes the cause (an ungated parse chokepoint in `revision_backfill.py`). After it lands, ~310 remain, and some of those are legitimately empty (sessions carrying only `session_events` after the v46 reclassification). Re-measure post-rebuild before deciding. |\n\n### Genuinely load-bearing — keep\n\n`raw_materialization`, `session_insights`, `orphaned_blobs`,\n`superseded_raw_snapshots`, `stale_supersession_receipts`. These were exercised\nfor real this session (raw materialization and authority blockers had to be\nunstuck manually). But note that needing them manually is itself evidence the\nautomatic path has gaps -- `session_insights` in particular overlaps the\n`insights` convergence stage and should be examined for redundancy.\n\n## Sequencing\n\nThe `archive.py` decomposition lane may relocate `repair.py`'s seam\n(`architecture-hotspots.md` note-on-#3 leaves its `storage/` vs `maintenance/`\nplacement explicitly undecided). Do the deletions after that lands, or they\ncollide.\n\n## Acceptance criteria\n\n- `orphaned_messages` and `orphaned_attachments` repair+preview+registry entries\n deleted, with the FK/CASCADE constraint cited as the replacement guarantee.\n- `message_type_backfill` deleted after confirming the writer always populates it.\n- A separate bead opened for the `created_at_ms IS NULL` WRITER defect, with the\n 1,117 -\u003e 5,382 growth as evidence; the backfill target is not deleted until\n that is fixed.\n- Line count of `storage/repair.py` reported before and after.\n- No new registry or allowlist introduced by any of this.\n","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T17:20:12Z","created_by":"Sinity","updated_at":"2026-07-30T17:20:12Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-f1vg","title":"Corpus acceptance gate: no absences and maximum fidelity, with the 2026-07-30 frozen baseline","description":"## What the operator asked for\n\n\"Ensure max fidelity as well as no absences, through the entire corpus.\" That is\na stronger bar than any existing check enforces, and nothing measured either\nhalf until now.\n\n## Why the existing checks cannot serve\n\n`verify-archive`'s `source-index-coverage` counts superseded revisions as\nmissing work (polylogue-ey3r), so it cannot reach zero on any archive that ever\ningested a conversation twice, and therefore cannot gate a rebuild. Nothing at\nall measures fidelity: an archive can report perfect coverage while every\nattachment whose bytes it holds is recorded `unfetched`, which is precisely the\nstate measured on 2026-07-30.\n\n## The gate\n\n`.agent/scripts/corpus-fidelity-audit.py` (read-only, `mode=ro` throughout,\nexits 1 on failure so it can gate a rebuild). Three measures:\n\n1. **Absences** -- logical documents (origin + provider_session_id) the archive\n holds evidence for but does not surface, bucketed by cause so a fix's effect\n is attributable rather than a single number moving for unknown reasons.\n2. **Attachment fidelity** -- acquired vs not-acquired refs, split by origin and\n upload_origin, because a Drive-hosted reference never fetched is actionable\n while a genuinely byte-less attachment kind is not.\n3. **Revision fidelity** -- documents whose indexed evidence is smaller than the\n largest revision recorded for them.\n\n## Baseline, live archive frozen 2026-07-30 (daemon stopped)\n\n ABSENCES 1,009 of 18,248 known documents\n 587 claude-ai-export/ambiguous-only\n 184 claude-code-session/ambiguous-only\n 135 chatgpt-export/ambiguous-only\n 71 aistudio-drive/settled-yet-absent\n 20 unknown-export/settled-yet-absent\n 12 gemini-cli / hermes / unknown / grok / codex\n\n ATTACHMENT FIDELITY acquired=2,118 not-acquired=7,655\n 3,684 chatgpt-export/oauth/unfetched\n 2,391 chatgpt-export/\u003cnone\u003e/unfetched\n 1,975 aistudio-drive/drive/acquired\n 1,119 aistudio-drive/drive/unfetched\n 398 claude-ai-export/oauth/unfetched\n\n REVISION FIDELITY 94 documents below best recorded evidence\n 76 hermes-session\n 16 claude-code-session\n 2 chatgpt-export\n\n VERDICT: FAIL\n\nThe `settled-yet-absent` buckets (71 drive, 20 unknown-export, 1 codex) are not\nexplained by any currently-tracked cause and want their own investigation --\nthese are documents with no ambiguous decision anywhere that are nonetheless\nmissing.\n\nThe 94 revision-fidelity documents are a residue after correcting a false\npositive, and should be treated as a prompt to investigate rather than proof of\nloss (see below).\n\n## Measurement trap this already caught\n\nThe first version compared indexed *messages* against\n`raw_session_memberships.message_count` and reported **474** shortfalls, 294 of\nthem codex-session. All false. `message_count` was recorded by whichever parser\ncensused that raw, and index v46 deliberately reclassified a large share of\nCodex/Claude Code rows from chat turns into typed `session_events`. One codex\nsession read as \"15 indexed vs 68,553 recorded\" when it actually holds 15\nmessages plus 84,612 events. Counting `messages + session_events` drops the\nfigure to 94.\n\nAnyone extending this must keep that in mind: cross-generation counts are only\napproximately comparable, so a metric built on them needs its assumption stated\nand checked against a real sample before its number is believed.\n\n## Follow-up\n\nPromote this into `devtools` as a first-class command with a `CommandSpec` (plus\n`devtools render devtools-reference`) so it is an enforced gate rather than a\nscript, and wire it into the post-rebuild acceptance path alongside\n`verify-archive`. Kept as a script for now because the fixes it measures are\nstill in flight and its thresholds will move as they land.\n\n## Acceptance criteria\n\n- Absences reach 0, or every residual is individually justified in writing.\n- Attachment refs marked not-acquired are either acquired or shown to be\n genuinely unfetchable (deleted upstream, over the size cap, byte-less kind).\n- Revision-fidelity residue is explained rather than merely small.\n","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T14:04:01Z","created_by":"Sinity","updated_at":"2026-07-30T14:04:01Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-d8al","title":"claude-ai-export: attachment real-id presence is inconsistent across export vintages, needs comparison-layer relaxation","description":"## What the data says\n\nCensus (full population, not a sample): replayed the production classifier\n(polylogue.sources.dispatch.parse_payload -\u003e session_revision_projection -\u003e\nclassify_membership_revisions) over all 566 claude-ai-export\nequal-message-count ambiguous cohorts in the live archive (read-only,\n/realm/db/polylogue), with polylogue-hith's parser-side fix (drop the\npositional-index seed for synthetic attachment ids) already applied.\n\n 566 claude-ai-export equal-message-count ambiguous cohorts (full census)\n 297 still ambiguous because message_hashes differ (polylogue-c429 /\n message-order-not-stable territory, or genuine content divergence)\n 268 still ambiguous with message_hashes EQUAL (0 content diffs) but\n attachment identity axis mismatched -- the exact shape hith\n targeted\n 0 of those 268 resolved by hith's fix\n 268 of those 268 are \"mixed real/synthetic\": one export vintage of the\n SAME conversation carries a real id (id/file_id/fileId/uuid/\n file_uuid) for an attachment; the OTHER vintage of the same\n conversation has no real id for the physically-same attachment and\n synthesizes one instead\n 0 are \"pure synthetic on both sides\" (the positional-index shape\n hith's fix targets and fully resolves when it occurs)\n\nIn other words: in the population currently persisted as ambiguous, 100% of\nthe identity-mismatch cases are this real-id-presence axis, not the\npositional-index axis. hith's fix is verified correct and regression-safe\n(250-cohort replay of already-resolved cohorts: 249/250 agree old vs new\nlogic, 1 improvement, 0 regressions) but resolves 0 of the currently-measured\n566-cohort population by itself, because no synthetic-minting scheme can ever\nmake a real UUID and a hash of (message, name, mime_type) collide.\n\n## Root cause\n\n`polylogue/sources/parsers/base_support.py:attachment_from_meta` uses the\nexport's own `id`/`file_id`/`fileId`/`uuid`/`file_uuid` field when present,\nand only falls back to synthesis when absent. Claude.ai does not consistently\nemit this field for the same attachment across export vintages of the same\nconversation -- verified directly against blob content for 6 sampled\ncohorts, all showing exactly this shape (one blob's attachment has a real\nUUID-shaped id, the other blob's attachment for the same message has no id\nfield and synthesizes `att-\u003chash\u003e`).\n\nNo id-minting scheme at the parser layer can reconcile this: a real id and a\nsynthetic hash will never be equal strings by construction, regardless of\nwhat the synthetic hash is seeded from.\n\n## Proposed fix (comparison layer, NOT parser layer)\n\nIn `polylogue/archive/session_revision_membership.py` (and/or\n`polylogue/pipeline/ids.py`'s `SessionRevisionProjection` /\n`_attachment_identity_payload`), the dominance/equivalence test should\ncompare attachments by a looser key when testing dominance -- e.g.\n`(message_provider_id, name, mime_type)` without the `id` field -- falling\nback to strict id equality only when that looser key is itself ambiguous\n(more than one attachment sharing the tuple on one side). This is the same\nclass of relaxation polylogue-bu1i introduced for acquisition state\n(`attachment_identities` vs `attachment_contents`), generalized to a third\naxis: \"same attachment referenced with and without a stable provider id\".\n\nThis bead deliberately does NOT propose an implementation in those files --\npolylogue-hith's owning lane was scoped away from\n`session_revision_membership.py`/`ids.py` because another lane owns them\nconcurrently. Whoever picks this up should re-run the census harness\ndescribed in polylogue-hith (or the updated one referenced in its closing\nnote) against the classifier change to prove the 268-cohort population above\nactually resolves, the same way polylogue-bu1i's PR proved 157/157.\n\n## Verification recipe\n\nSame read-only harness as polylogue-hith / polylogue-bu1i: parse both blobs\nof a cohort with production `parse_payload`, project with\n`session_revision_projection`, and diff the resulting\n`attachment_identities` sets. For the 268-cohort population, at least one\nattachment identity differs solely because one side has a real id string and\nthe other has a synthetic hash string for what is, by every other field\n(message anchor, name, mime_type), the same attachment.\n\nRef polylogue-hith\nRef polylogue-bu1i","notes":"Superseded by polylogue-aggz's architecture: attachment identity now unconditionally drops the provider id (content-derived: message_id+name+mime_type only), eliminating the strict/loose duality and its pairwise correlation machinery entirely rather than adding a fallback. See PR.","status":"open","priority":1,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T13:02:14Z","created_by":"Sinity","updated_at":"2026-07-30T15:15:31Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-eqnv","title":"Stale pre-fix parser identity lets a same-source_path raw pair silently split into two byte-proven singletons, downgrading fidelity","description":"## What the live archive shows\n\nFor the 5 aistudio-drive sessions Implementing-{066bb070,13ced1c8,37edfeb3,845dd573,d4d7fbab}, the index materialized the SMALLER (attachment-unfetched, \"bare\") raw and never even considered the LARGER (attachment-fetched, \"enriched\") raw. Neither raw has a `raw_session_memberships` row -- they never entered the ambiguous-membership machinery bu1i/9dxn describe. Instead both raws sit in raw_sessions with `revision_kind='full'`, `revision_authority='byte_proven'`, `baseline_raw_id=self` -- i.e. each was independently accepted as an unconditional SINGLETON byte-revision baseline under a DIFFERENT `logical_source_key`:\n\n 0064ddd16c39... (enriched, 967377B) -\u003e logical_source_key = 'gemini:Implementing-066bb070...'\n 13ae07d010bb... (bare, 252347B) -\u003e logical_source_key = 'gemini:Implementing-066bb070...-0'\n\n## Root cause, proven\n\n`raw_authority_parser_census` (source.db) records the census-time parser\nIDENTITY output for both raws:\n\n 0064ddd16c39...: fingerprint=revision-membership-v1, key=[\"gemini:Implementing-066bb070...23810576f616f90fb4254c69\"]\n 13ae07d010bb...: fingerprint=revision-membership-v1, key=[\"gemini:Implementing-066bb070...23810576f616f90fb4254c69-0\"]\n\nBoth under the SAME fingerprint string, yet different identity. Reparsing\nBOTH raw blobs from the live blob store through the CURRENT\n`polylogue/sources/dispatch.py`/`revision_backfill._parse_one` gives the\nIDENTICAL, correct, unsuffixed `provider_session_id` for both (verified with\nproduction code against the real blobs). The \"-0\" suffix is the exact\npre-#3179/z1c6 bug (`_lower_drive_like_payload`'s `_looks_like_chunked_session_list`\nbranch always appended `-{index}` regardless of list length, fixed\n2026-07-20 in b473d9256/#3179). raw_small was acquired+validated 2026-07-16,\nraw_big 2026-07-18 -- both before the fix landed 2026-07-20 -- and their\ncensus (which sets `raw_sessions.logical_source_key`) evidently ran under\nthe pre-fix parser and was never invalidated, because\n`raw_authority_parser_census`'s quiescence gate\n(`uncensused_historical_revision_raw_ids`) treats any row with the SAME\nliteral fingerprint string as \"current parser already observed this\" --\nthere is no version distinction between pre-fix and post-fix identity\noutput. `classify_raw_revision_cohort` (archive.py) then classifies each\nraw against its OWN `logical_source_key` in isolation, has no way to know\nthe two keys describe the same physical document, and unconditionally\naccepts each as a trivial one-member byte-proven chain -- the same\nstructural hole polylogue-52l2/hm2f already document for the RETIRED-SIBLING\ncase, but here the divergence is at the KEY itself, not at retirement\nstate, so the existing `raw_membership_retired_full_revision_siblings` guard\n(keyed on exact logical_source_key match) never fires.\n\n## Relationship to polylogue-9dxn\n\n9dxn's proposed fingerprint-versioning fix (permissive quiescence for any\nKNOWN fingerprint, strict-current-only for the ambiguous TERMINAL gate)\ndoes not by itself heal this case: it is designed to let previously-`ambiguous`\nverdicts be revisited without forcing a blanket re-census, but raw_small's\nstale census here was NOT ambiguous -- it was `status='complete'` with a\nWRONG identity, and 9dxn's design keeps quiescence permissive for any known\nfingerprint, so this raw would stay \"already observed\" forever even after a\nfingerprint bump. This bead's fix is a structural cross-source_path guard in\n`classify_raw_revision_cohort`, independent of fingerprint versioning, that\nalso closes the general case regardless of how two same-document raws ended\nup under different keys (stale census, race, or a future bug of the same\nshape).\n\n## Fix landed in polylogue-af059 (this branch)\n\n- `archive.py`: `classify_raw_revision_cohort` refuses unconditional\n singleton acceptance when another 'full' raw shares the same source_path\n under a different (or already-retired) logical_source_key -- forces both\n into membership governance instead of letting either become an\n unconditionally-accepted baseline.\n- `revision_backfill.py`: the retire-to-membership-governance fallback now\n buckets `membership_candidates`/`membership_keys` by the FRESHLY re-parsed\n identity (`session.provider_session_id`) instead of the stale outer-loop\n `logical_source_key`, so two same-document raws retired under different\n stale keys land in ONE membership cohort and get jointly arbitrated\n instead of each being accepted as an independent membership singleton.\n\n## Residual / follow-up\n\n- The live archive's 5 already-downgraded sessions are NOT repaired by this\n code fix (need a live remediation pass, out of scope for this PR).\n- A full census-fingerprint bump (9dxn) is still needed to catch every OTHER\n raw whose identity was assigned by pre-#3179 dispatch.py, if any exist\n beyond aistudio-drive.\n\nRef polylogue-bu1i, polylogue-7ilr, polylogue-9dxn","status":"closed","priority":1,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T12:22:33Z","created_by":"Sinity","updated_at":"2026-07-30T12:45:58Z","started_at":"2026-07-30T12:45:56Z","closed_at":"2026-07-30T12:45:58Z","close_reason":"Fixed in PR #3396 (feature/fix/ambiguous-raw-materialization-leak): ArchiveStore.classify_raw_revision_cohort gains an opt-in check_source_path_identity_split guard (used only by the offline backfill/rebuild replay loop, not the live watcher), plus revision_backfill.py's retire-to-membership-governance fallback now buckets by the freshly re-derived identity instead of the stale outer-loop key. Verified with two new regression tests (anti-vacuity confirmed both ways via direct revert+rerun). The 5 already-downgraded live sessions are NOT repaired by this fix; live remediation is a separate, explicitly out-of-scope lane.","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -453,6 +454,9 @@ {"_type":"issue","id":"polylogue-sru.2","title":"Characterize ambiguous bucket: wordless continuation vs prose-without-markers","description":"Split next-turn-is-tool-call (wordless continuation) from prose-lacking-ack-markers; state counts for both. Opus-4-7 74% ambiguous vs deepseek 17% is likely turn-structure variance, not behavior — this split disambiguates.","design":"Implementation home: the claim-vs-evidence classifier in devtools (devtools/ module behind `devtools workspace claim-vs-evidence`; tests tests/unit/devtools/test_claim_vs_evidence.py). Wordless-continuation detection: for each failure's paired next assistant message, check whether its blocks contain tool_use and no text block with \u003eN chars before the first tool_use — that is 'wordless continuation'; prose without matched ack markers stays 'ambiguous-prose'. Emit both as classification_reason variants (field already exists) and add the two counts to the report summary + by_model/by_tool cuts. Regen: `devtools workspace claim-vs-evidence --limit 5000 --out-dir .agent/demos/claim-vs-evidence --json`. Acceptance: report shows ambiguous split into wordless_continuation vs prose_no_marker with counts; per-model ambiguous variance (opus-4-7 74% vs deepseek 17%) re-examined after the split.","notes":"2026-07-03 Codex WIP: unit implementation for ambiguous split passes focused tests, but live regeneration with --limit 5000 became too slow and had to be killed twice. First attempt used correlated subqueries for next-message block shape; second used set-based CTE; third used chunked second query after sampled rows, but the full command still exceeded 90s on active archive and ignored SIGINT while inside SQLite. Do not close or commit this slice until the live regeneration path is profiled/fixed. Dirty files currently show the WIP implementation: devtools/claim_vs_evidence.py and tests/unit/devtools/test_claim_vs_evidence.py. Last passing focused proof: python -m py_compile + ruff check + devtools test tests/unit/devtools/test_claim_vs_evidence.py -\u003e 3 passed.","status":"closed","priority":1,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:28Z","created_by":"Sinity","updated_at":"2026-07-03T07:45:10Z","started_at":"2026-07-03T07:09:21Z","closed_at":"2026-07-03T07:45:10Z","close_reason":"Completed: claim-vs-evidence now splits ambiguous follow-ups into wordless tool continuations and prose-without-marker buckets, reports the counts in JSON/README summaries, and regenerates the current demo on the active archive. Focused tests pass; live regen/check completed.","labels":["area:substrate","campaign"],"dependencies":[{"issue_id":"polylogue-sru.2","depends_on_id":"polylogue-sru","type":"parent-child","created_at":"2026-07-03T06:31:27Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"polylogue-sru.3","title":"Benign-recovery vs consequential-silence split by handler kind","description":"Read failures are ~94% silent but 'tried another path' is usually benign; Bash/test failures are the consequential class. Scope the headline to consequential handler kinds or add an explicit split — credibility depends on not inflating with trivial recoveries.","design":"Handler kind is already available on the paired failure row (actions lane exposes handler/tool). Define the consequential set explicitly in code (Bash/test/build/write-class handlers) and the benign-recovery set (Read/Glob/Grep-class 'tried another path'), emit split headline rows: silent-proceed among consequential vs among all. Keep the mapping a named constant with a rationale comment so reviewers can argue with it. Report both; never let the headline mix classes silently. Same regen/tests as the other methodology children.","status":"closed","priority":1,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:28Z","created_by":"Sinity","updated_at":"2026-07-03T07:58:08Z","started_at":"2026-07-03T07:55:37Z","closed_at":"2026-07-03T07:58:08Z","close_reason":"Completed: claim-vs-evidence now reports a first-class handler-class split separating consequential shell/edit/write-class tool failures from benign read/search/path-discovery failures and other tools. The regenerated active-archive artifact shows consequential=4,177 failures with 921 silent-proceed (22.0% lower bound), benign_recovery=633 with 166 silent-proceed (26.2%), and other=190 with 92 silent-proceed (48.4%). Focused tests and demo shelf checks passed.","labels":["area:substrate","campaign"],"dependencies":[{"issue_id":"polylogue-sru.3","depends_on_id":"polylogue-sru","type":"parent-child","created_at":"2026-07-03T06:31:28Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"polylogue-sru.1","title":"Expose action-unit outcome fields + followup_class as product capability","description":"Capabilities-may-not-be-silos gate for the campaign: the facts the report needs must become composable query capability. After this, the whole report is `actions where is_error:true | group by session.origin, followup_class | count` and every future cut (model/tool/repo/time) is free.","design":"1) is_error/exit_code are normalized at parse time (sources/parsers/base_models.py:74-75) but ActionQueryRowPayload (surfaces/payloads.py:~1298) carries neither — add as filterable/groupable action-unit fields. 2) Add derived followup_class (acknowledged|silent_proceed|wordless_continuation|ambiguous) + followup_message_ref computed in the source-derived lowering (no cache tables). 3) Reduce devtools workspace claim-vs-evidence to a render preset over these query strings, or retire it. Touchpoint chain: stage parser -\u003e AST to_payload -\u003e executor -\u003e metadata.py aggregate_group_fields -\u003e shell_completion_values.py -\u003e devtools render openapi + cli-output-schemas + cli-reference. Line refs pre-07-03; re-locate.","acceptance_criteria":"Fixture session with known unacknowledged failure fires via pure query strings; report README numbers reproducible from the printed queries.","notes":"Completed: action-unit outcome follow-up classification is now shared query capability. is_error/exit_code were already wired; this slice added source-derived followup_class and followup_message_ref over existing actions/messages/blocks, exposed followup_class as filterable/groupable action metadata, added action row payload fields, routed root CLI terminal-unit aggregate expressions before session-selector compilation, and moved the report classifier from scripts into polylogue.archive.actions.followup. Reproduction/query forms are now printed in .agent/demos/claim-vs-evidence/PUBLIC_REPRODUCTION.md: actions where is_error:true | group by followup_class | count; actions where followup_class:silent_proceed. Verification: focused DSL/report/CLI tests passed; active demo packet regenerated over archive root /home/sinity/.local/share/polylogue schema v23 with 41,886 structured failures and 5,000 inspected; devtools verify --quick passed run 20260703T092510Z-quick-718233-46e8b587.","status":"closed","priority":1,"issue_type":"feature","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:27Z","created_by":"Sinity","updated_at":"2026-07-03T09:25:36Z","started_at":"2026-07-03T09:05:37Z","closed_at":"2026-07-03T09:25:36Z","close_reason":"Completed","labels":["area:query","area:substrate","campaign"],"dependencies":[{"issue_id":"polylogue-sru.1","depends_on_id":"polylogue-sru","type":"parent-child","created_at":"2026-07-03T06:31:26Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-ubwg","title":"Evaluate typed-constructor chokepoints for the remaining hash-boundary-registry sites","description":"polylogue-aggz Invariant 1 (comparison identity contains only content) is now structurally enforced in polylogue/pipeline/ids.py via fixed keyword-only constructors (message_identity_hash/attachment_identity_hash/event_base_identity_hash/event_canonical_identity_hash) instead of dict-key-list projection -- passing a non-content field is a TypeError at the call boundary, not a value a reviewer has to remember to strip. This closed the exact defect pattern behind polylogue-bu1i and polylogue-nuec.\n\ndocs/plans/hash-boundary-registry.yaml was NOT retired in that change and should stay open as tracked debt, not be treated as superseded. It governs all 198 hashlib/core.hashing call sites across polylogue/ (90 content-hash, 91 identifier, 17 other, spanning 58+ files: blob_store.py, security/excision.py, judgment/*, sinex/*, browser_capture/*, ...), the overwhelming majority of which are NOT session/message/attachment/event comparison identity -- they are content-addressed storage keys, HMAC signatures, redaction digests, and other identifier-generation sites with a different (and often already-correct) risk shape. Retiring the whole registry would have been a false claim of coverage this session did not do the work for.\n\nFollow-up: audit whether any of the 91 'identifier'-classified sites share the aggz failure shape (a mutable/acquisition-state field folded into a value used for equality/dedup comparison) and, for those that do, build the same fixed-signature-constructor pattern used in pipeline/ids.py. Sites that are pure content-hashing of raw bytes/already-hashed values (the 'content-hash'/'other' tags) don't need this -- only sites where an identifier is also treated as a stable comparison key are candidates. Only once every hash-boundary site is provably covered by a structural chokepoint (or provably out of the aggz identity-comparison class) can the registry itself be deleted.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T17:22:26Z","created_by":"Sinity","updated_at":"2026-07-30T17:22:26Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-2jga","title":"Split or delete test-closure-matrix.yaml / test-quality-coverage.yaml's unenforced narrative fields","description":"Audit (2026-07-30, meta-machinery purge) found two related but distinct\nmanifests each mixing one real enforced check with unenforced free-text\nnarrative:\n\n1. docs/plans/test-closure-matrix.yaml (381 lines): devtools/verify_closure_matrix.py\n only checks that target_files/representative_tests paths exist on disk and\n that gate:absent rows carry a known_gaps bullet — it never runs the\n representative tests or verifies they exercise the target files. Its only\n failure mode is \"a file moved/renamed and the hand-maintained matrix wasn't\n updated\" — the fossilized-diff pattern CLAUDE.md flags for deletion. Counter-\n consideration: it forces explicit known_gaps documentation per declared-\n absent domain, which has narrative value distinct from the path check, and\n git history (d068d6482, 054dfa9e1, dc6fa632a) shows only refactor/consolidation\n commits, never a caught coverage gap that wasn't already known from the\n known_gaps text itself.\n\n2. docs/plans/test-quality-coverage.yaml: check_test_quality_ci_claims verifies\n ci_gate:true dimensions actually appear in a real CI workflow step (a\n genuine, real check — keep this). But most of the file's content\n (flakiness.known_flaky, mock_depth, fuzz tool locations) is pure narrative\n with no executable check beyond generic schema/coverage-gap validation, and\n nothing re-verifies a known_flaky entry is still flaky or that\n value_percent/last_verified stay current.\n\nOperator call needed: (a) for test-closure-matrix.yaml, keep as narrative\ndocumentation with path-existence hygiene, or delete and let the real\nper-domain test suites speak for themselves; (b) for test-quality-coverage.yaml,\nsplit the ci_gate dimension (keep, real check) from the flakiness/fuzz/mock_depth\nnarrative (move to a plain doc outside docs/plans/ verification, or delete).\nNot resolved in the purge session because both are genuinely load-bearing in\npart and the split requires deciding how much narrative value survives without\nthe doc.","status":"open","priority":2,"issue_type":"chore","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T16:55:04Z","created_by":"Sinity","updated_at":"2026-07-30T16:55:04Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-ganm","title":"Reduce topology-target.yaml to a bare file inventory, drop placement-judgment columns","description":"Audit (2026-07-30, meta-machinery purge, feature/chore/purge-meta-machinery) found\ndocs/plans/topology-target.yaml's 4618 lines are almost entirely a per-file\n`target`/`reason`/`owner` placement-judgment projection that no code or doc\nreads to make a placement decision — it is written by\ndevtools/build_topology_projection.py, then only checked against itself by\ndevtools/verify_topology.py's orphan/missing/conflict checks (which need only\na bare path list) plus a narrow kernel_rule check (which needs `target`/`owner`\nonly for the ~15 files that live directly at polylogue/ root).\n\ngit log --oneline --follow on the yaml and on build_topology_projection.py /\nrender_topology_status.py (predecessor render target, already deleted this\nsession) shows only mechanical regenerate-after-adding-a-module commits,\nnever a commit that used the placement judgments to actually relocate code.\n\nReal defect class the SURVIVING checks prevent (keep these): orphan file in\ntree not declared, declared file missing from tree, duplicate declaration,\nnon-kernel file sitting at polylogue/ root. These only need a file inventory\n+ owner tag for root files, not a placement/target/reason judgment per file.\n\nProposed scope: rewrite devtools/build_topology_projection.py and\ndevtools/verify_topology.py so the generated artifact is a flat sorted list\nof declared paths (+ owner/target only for the root-level kernel_rule check),\ndropping target/reason/loc/cross_cut columns for the ~600 non-root files.\nUpdate polylogue/verification/manifests/models.py's TopologyManifest/\nTopologyEntry to match the reduced schema.\n\nNot done in the purge session because it is a generator/schema rewrite, not\na deletion — real engineering risk of breaking `render all --check` /\n`verify topology` if done without careful review, and genuinely needs an\noperator call on whether the placement-judgment metadata has narrative value\nworth keeping despite zero consumption evidence.","status":"open","priority":2,"issue_type":"chore","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T16:54:46Z","created_by":"Sinity","updated_at":"2026-07-30T16:54:46Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-cs86","title":"Full-replace DELETE cascade is ~20% of apply_s even with indexes present (small-raw probe)","description":"polylogue-9soj's blanket-index-deferral experiment found the full_replace per-session DELETE cascade (clear_projection_rows + delete_messages, ~14 tables) goes from O(log n) to O(table_size) when indexes are dropped. A follow-up single-sample probe (feature/perf/rebuild-cost-model, tests/infra/rebuild_cost_model.py) measured the SAME cascade under CURRENT production conditions (indexes present, from-empty bulk build, n=50 synthetic Codex raws ~1KB each): clear_projection_rows+delete_messages = 0.506s of apply_s=2.504s total = 20.2% of apply time; the full revision_replay.index.full_replace stage (which also includes messages/blocks insert) = 1.011s = 40.4% of apply_s. This is a SINGLE 50-raw sample, not a repeated/averaged measurement -- treat as a directional signal, not a precise number. It suggests the DELETE cascade against empty tables (a structural cost of the from-scratch bulk-build path, not merely a deferral side effect) may itself be worth investigating as a target independent of polylogue-9soj's index-deferral angle -- e.g. skipping the DELETE entirely when the session_id provably has zero existing rows (a fresh bulk-build generation, or a raw never previously ingested) rather than issuing 14 unconditional point-deletes per session. Re-measure with more samples/repetitions before treating the 20%/40% figures as load-bearing.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T15:44:02Z","created_by":"Sinity","updated_at":"2026-07-30T15:44:02Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-ey3r","title":"verify-archive source-index-coverage counts superseded revisions as missing work, so its blocking error is ~72% by-design noise","description":"## Problem\n\n`polylogue ops maintenance verify-archive` reports `source-index-coverage` as a\nblocking **error** on the live archive:\n\n 29,992 complete-census raw(s), 23,036 raw-backed session(s);\n missing_work=12,976 orphans=0\n\nThe majority of that number is a by-design state, not missing work. A raw whose\nmembership decision is `superseded_equivalent` or `superseded_prefix` is a\nrevision whose content is represented in the index through its *accepted*\nsibling; it is not supposed to own a session row. Counting it as missing work\nmakes the metric unable to reach zero on any archive that has ever ingested the\nsame conversation twice -- which is every real archive.\n\nMeasured on the live archive (complete-census raws, joined to\n`raw_session_memberships`):\n\n superseded_equivalent / superseded_prefix 9,411 \u003c- by design, no own session\n ambiguous 3,920 \u003c- genuine authority debt\n applied / \u003cnone\u003e 28,593\n\nHand-inspecting the check's own `missing_work_sample` (10 ids) splits the same\nway: 7 are `ambiguous` + unparsed + genuinely absent from the index, and 3 are\n`superseded_equivalent`/`superseded_prefix`, already parsed, and **present in\nthe index** via their accepted sibling. Those 3 are counted as missing work\nanyway.\n\n## Why it matters beyond tidiness\n\nThis is the archive's own coherence gate, the thing meant to answer \"did the\nrebuild land correctly\". A blocking error that includes states the design\nrequires trains an operator to ignore it, which is worse than not having the\ncheck: the 3,920 rows of real debt hide inside a number that is ~72% noise. It\nalso means the check cannot be used as an acceptance criterion for a rebuild or\nrestore, which is exactly what it exists for.\n\n## Proposed fix\n\nExclude raws whose membership decision is `superseded_*` from `missing_work`,\nand report them as their own evidence bucket (`superseded_count`) so coverage\nstays auditable without being conflated. Keep `ambiguous` in a distinct bucket\ntoo -- it is real debt, but it is *known, recorded* debt with an owner\n(polylogue-9dxn / polylogue-bu1i and the per-origin causes), so it should be\nreportable separately from \"we cannot account for this raw at all\", which is the\nonly thing that deserves to block.\n\nSuggested shape:\n\n missing_work_count raws with no session and no explanation\n superseded_count content represented via an accepted sibling\n ambiguous_debt_count recorded authority debt\n orphan_count (unchanged)\n\nwith `error` reserved for `missing_work_count \u003e 0` and `warning` for a nonzero\n`ambiguous_debt_count`.\n\n## Related, observed in the same run, NOT this bead\n\n`fts-parity` also errors: `messages_fts gap=36757`,\n`blocks_command_trigram gap=13235`. This one looks like genuine convergence\nbacklog rather than a measurement artifact -- every worst-offender session has\n`indexed=0` and they are all subagent sessions ingested the same day, i.e. the\nFTS repair stage had not caught up when the daemon was stopped. Re-verify after\nthe next full rebuild before filing anything; if a gap survives a rebuild, that\nis a real defect and deserves its own bead.\n\n## Acceptance criteria\n\n- `source-index-coverage` distinguishes unexplained-missing from\n superseded-by-sibling from recorded-ambiguous, with counts for each.\n- On an archive whose only residue is superseded revisions, the check does not\n report `error`.\n- A raw that is genuinely absent and unexplained still errors.\n- Anti-vacuity: state the production line mutated and the resulting failure.\n","status":"open","priority":2,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T13:14:36Z","created_by":"Sinity","updated_at":"2026-07-30T13:14:36Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-ck5v","title":"Attachment byte backfill is coupled to one acquisition route, so payloads from any other route are never backfilled","description":"## Problem\n\nResolving a Drive-hosted attachment's bytes happens in exactly one place:\n`_inject_live_drive_attachment_bytes`, called from inside\n`iter_drive_raw_data` (`polylogue/sources/drive/__init__.py:253`). Its docstring\nis explicit that this placement is deliberate -- it is the only scope where the\nlive authenticated client and the raw JSON coexist -- and that it runs on every\nread, cache hit included, so a cache file written before the feature existed\nstill gets backfilled rather than being skipped forever.\n\nThat guarantee only holds for payloads the Drive iterator enumerates. A Drive\npayload that entered the archive by any other route is structurally outside it\nand can never be backfilled, no matter how many times the daemon converges.\n\nMeasured on the live archive: 11 of 397 `aistudio-drive` raws came from a legacy\nzip backfill under `inbox/polylogue-aistudio-legacy-backfill-sha256-*.zip:members/…`\nrather than from `drive-cache/gemini/`. Those payloads carry Drive-hosted\nattachment references in the ordinary `{\"id\": \"\u003cdrive-file-id\u003e\"}` shape the\ninjector resolves successfully elsewhere, but nothing will ever visit them,\nbecause the Drive iterator enumerates the live Drive folder and a zip member is\nnot in it.\n\nUnfetched attachment counts, live archive (still moving -- convergence was\nactively writing when these were taken, so re-derive before acting):\n\n chatgpt-export 6,075\n claude-ai-export 398\n aistudio-drive 360 \u003c- of which 334 upload_origin='drive'\n grok-export 37\n\nThe 334 drive-hosted ones are fetchable in principle: a live client and a file id\nare all that is required. Some fraction will be genuinely unfetchable (deleted\nDrive files, revoked access, over the 50 MB cap) and must stay honestly\nunfetched -- that distinction is part of the work, not an inconvenience.\n\n## Why this is an invariant, not a command\n\nPer the project's automagic-invariants principle, a condition Polylogue can\nmaintain automatically belongs in daemon convergence, not in an operator\ncommand. \"Every attachment whose bytes are fetchable has been fetched\" is\nexactly such a condition, and it is currently a side effect of one acquisition\nroute instead of a maintained property of the archive.\n\nCoupling it to acquisition also has a second cost: it makes attachment fidelity\ndepend on how a payload happened to arrive. Two identical documents, one synced\nfrom Drive and one restored from a zip, end up with different evidence.\n\n## Proposed direction\n\nA convergence stage that selects attachments with `acquisition_status \u003c\u003e\n'acquired'` and a resolvable provider handle, fetches them through the owning\nsource's client in bounded windows, and records terminal failures so a\npermanently-gone Drive file is not retried forever. The existing\n`ConvergenceStage` shape fits: bounded work per pass returning `False` to push\nthe remainder into `convergence_debt` as retryable is the documented pattern for\nexactly this.\n\nNote the interaction with `polylogue-bu1i`: backfilling bytes for an\nalready-indexed session changes its content hash and therefore produces a new\nrevision to reconcile. That is now safe -- acquisition is read as a fidelity\nupgrade rather than a branch -- but it means this stage must land after bu1i,\nnot before, or it will manufacture ambiguous cohorts at scale.\n\n## Acceptance criteria\n\n- An attachment referenced by a payload that did NOT arrive through its source's\n live iterator is still backfilled. Cover the legacy-zip route specifically,\n since that is the observed miss.\n- A genuinely unfetchable attachment reaches a terminal state and stops being\n retried; nothing fabricates a hash or size for bytes never read.\n- Bounded per-pass work with the remainder in `convergence_debt`.\n- Anti-vacuity: state the production line mutated and the resulting failure.\n\nRef polylogue-bu1i\n","status":"open","priority":2,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T12:16:42Z","created_by":"Sinity","updated_at":"2026-07-30T12:16:42Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/docs/plans/hash-boundary-registry.yaml b/docs/plans/hash-boundary-registry.yaml index 21fd693f3b..469b49cde2 100644 --- a/docs/plans/hash-boundary-registry.yaml +++ b/docs/plans/hash-boundary-registry.yaml @@ -529,41 +529,41 @@ entries: call: hash_payload occurrence: 0 classification: content-hash - note: 'feeds session_content_hash / revision projection (2026-07-09 hash-boundary census (docs/audits/2026-07-09-hash-boundary-census.md), Table 1 row 1).' + note: 'per-message content hash (full, unstripped payload) -- feeds session_content_hash / revision projection (2026-07-09 hash-boundary census (docs/audits/2026-07-09-hash-boundary-census.md), Table 1 row 1).' - path: polylogue/pipeline/ids.py function: '.session_revision_projection' call: hash_payload occurrence: 1 classification: content-hash - note: 'feeds session_content_hash / revision projection (2026-07-09 hash-boundary census (docs/audits/2026-07-09-hash-boundary-census.md), Table 1 row 1).' + note: 'event_hashes -- feeds session_content_hash / revision projection, unstripped and order-preserving (polylogue-nuec).' - path: polylogue/pipeline/ids.py function: '.session_revision_projection' call: hash_payload occurrence: 2 classification: content-hash - note: 'feeds session_content_hash / revision projection (2026-07-09 hash-boundary census (docs/audits/2026-07-09-hash-boundary-census.md), Table 1 row 1).' + note: 'per-event content hash (measurement excluded via _event_content_payload''s allowlist), feeding event_contents (polylogue-aggz, polylogue-nuec).' - path: polylogue/pipeline/ids.py - function: '.session_revision_projection' + function: '.message_identity_hash' call: hash_payload - occurrence: 3 + occurrence: 0 classification: content-hash - note: 'event_hashes -- feeds session_content_hash / revision projection, unstripped and order-preserving (polylogue-nuec).' + note: 'the typed constructor for message comparison identity (polylogue-aggz) -- fixed keyword-only signature (id only) is the sole path into message identity, replacing a dict-key-list projection (polylogue-aggz constructor chokepoint).' - path: polylogue/pipeline/ids.py - function: '.session_revision_projection' + function: '.attachment_identity_hash' call: hash_payload - occurrence: 4 + occurrence: 0 classification: content-hash - note: 'event base identity (event type + anchoring message), content-derived and position-independent, feeding event_contents (polylogue-aggz, polylogue-nuec).' + note: 'the typed constructor for attachment comparison identity (polylogue-aggz, polylogue-bu1i, polylogue-d8al, polylogue-hith) -- fixed keyword-only signature (message_id/name/mime_type only) is the sole path into attachment identity.' - path: polylogue/pipeline/ids.py - function: '.session_revision_projection' + function: '.event_base_identity_hash' call: hash_payload - occurrence: 5 + occurrence: 0 classification: content-hash - note: 'per-event content hash (measurement excluded via _event_content_payload''s allowlist), feeding event_contents (polylogue-aggz, polylogue-nuec).' + note: 'the typed constructor for an event''s position-independent base identity (polylogue-aggz, polylogue-nuec) -- fixed keyword-only signature (event_type/source_message_provider_id only) is the sole path into event base identity.' - path: polylogue/pipeline/ids.py - function: '.session_revision_projection' + function: '.event_canonical_identity_hash' call: hash_payload - occurrence: 6 + occurrence: 0 classification: content-hash note: 'canonical event identity fold (base identity + content) used only when the base identity is locally ambiguous within one revision -- still content-derived, never the array index (polylogue-aggz).' - path: polylogue/pipeline/ids.py diff --git a/polylogue/archive/session_revision_membership.py b/polylogue/archive/session_revision_membership.py index d0e3c503d1..30c6ea7cd7 100644 --- a/polylogue/archive/session_revision_membership.py +++ b/polylogue/archive/session_revision_membership.py @@ -68,7 +68,7 @@ def _content_by_identity(contents: frozenset[tuple[bytes, bytes]]) -> dict[bytes Identity is not always injective: two acquired attachments on one message can share one identity (same ``message_id``/``name``/ ``mime_type``, different bytes -- the accepted limit documented on - ``_ATTACHMENT_IDENTITY_FIELDS``, not a new one). Collapsing such a + ``attachment_identity_hash``, not a new one). Collapsing such a collision to a single arbitrary content hash (a plain ``dict``, last value wins) would let a real content conflict compare as equal instead. Grouping into a set per identity means a collision always degrades to diff --git a/polylogue/pipeline/ids.py b/polylogue/pipeline/ids.py index a34fcdb421..22b0a96f73 100644 --- a/polylogue/pipeline/ids.py +++ b/polylogue/pipeline/ids.py @@ -209,21 +209,22 @@ def _message_hash_payload(message: ParsedMessage, message_id: str) -> dict[str, return payload -#: The one field of a message hash payload that answers *which message is -#: this*, as opposed to *what does it currently say*. A provider's own -#: message id is stable across re-exports even when the export's array -#: ordering is not (polylogue-c429). -_MESSAGE_IDENTITY_FIELDS = ("id",) - - -def _message_identity_payload(payload: dict[str, JSONValue]) -> dict[str, JSONValue]: - """Project the order-independent identity of one message payload. - - Reads the already-normalized value out of ``_message_hash_payload`` rather - than re-deriving it, mirroring ``_attachment_identity_payload``'s single - normalization site. +def message_identity_hash(*, id: str) -> bytes: + """The sole constructor of a message's comparison identity (polylogue-aggz). + + A provider's own message id is stable across re-exports even when the + export's array ordering is not (polylogue-c429) -- it is the only + content field that answers *which message is this*, as opposed to *what + does it currently say*. + + This is a fixed keyword-only signature, not a dict projected by a list + of field names: passing ``role``/``text``/``timestamp``/anything else is + a ``TypeError`` at the call boundary, not a value that has to be + remembered and stripped. Extending what a message's comparison identity + covers requires editing this signature -- an explicit, reviewable + decision, never a side effect of a parser gaining a new field. """ - return {field: payload[field] for field in _MESSAGE_IDENTITY_FIELDS} + return bytes.fromhex(hash_payload({"id": id})) #: Fields of an attachment hash payload that answer *which attachment is @@ -248,7 +249,19 @@ def _message_identity_payload(payload: dict[str, JSONValue]) -> dict[str, JSONVa #: genuinely distinct attachments that share one message/name/media-type and #: carry no bytes on either side of a comparison are indistinguishable by any #: signal this projection can offer. -_ATTACHMENT_IDENTITY_FIELDS = ("message_id", "name", "mime_type") + + +def attachment_identity_hash(*, message_id: JSONValue, name: JSONValue, mime_type: JSONValue) -> bytes: + """The sole constructor of an attachment's comparison identity (polylogue-aggz). + + Fixed to (anchoring message, name, media type) -- content-derived and + never the provider's own attachment id (polylogue-d8al, polylogue-hith) + or acquisition state such as ``size_bytes``/inline bytes + (polylogue-bu1i). Those fields are not parameters here; passing them + (e.g. spreading a full attachment payload dict as ``**kwargs``) is a + ``TypeError``, not a value this function has to remember to strip. + """ + return bytes.fromhex(hash_payload({"message_id": message_id, "name": name, "mime_type": mime_type})) def _attachment_hash_payload(attachment: ParsedAttachment) -> dict[str, JSONValue]: @@ -265,16 +278,6 @@ def _attachment_hash_payload(attachment: ParsedAttachment) -> dict[str, JSONValu return payload -def _attachment_identity_payload(payload: dict[str, JSONValue]) -> dict[str, JSONValue]: - """Project the acquisition-independent identity of one attachment payload. - - Reads the already-normalized values out of ``_attachment_hash_payload`` - rather than re-deriving them, so there is exactly one normalization site and - identity can never drift from the content hash it is paired with. - """ - return {field: payload[field] for field in _ATTACHMENT_IDENTITY_FIELDS} - - #: `generation_lifecycle` payload keys that are provider-reported measurement, #: not identity, when the event's own payload declares them non-durable via #: ``duration_semantics == "provider_reported_elapsed"``. ChatGPT re-derives @@ -372,12 +375,29 @@ def _event_content_payload(event: ParsedSessionEvent) -> dict[str, JSONValue]: #: depends on what else happens to be in the set. Still content-derived #: (each block's own content, including any content-intrinsic field such as #: ``block_index``, already differs), never the array position. -_EVENT_BASE_IDENTITY_FIELDS = ("event_type", "source_message_provider_id") +def event_base_identity_hash(*, event_type: JSONValue, source_message_provider_id: JSONValue) -> bytes: + """The sole constructor of an event's position-independent base identity. + + Anchoring message plus event type only -- content-derived, never the + array index and never provider-reported measurement (polylogue-nuec), + which is not a parameter here. Fixed keyword-only signature: passing a + whole event content payload as ``**kwargs`` (which also carries + ``timestamp``/``payload``) is a ``TypeError``. + """ + return bytes.fromhex( + hash_payload({"event_type": event_type, "source_message_provider_id": source_message_provider_id}) + ) -def _event_base_identity_payload(payload: dict[str, JSONValue]) -> dict[str, JSONValue]: - """Project the position-independent base correlation key of one event payload.""" - return {field: payload[field] for field in _EVENT_BASE_IDENTITY_FIELDS} +def event_canonical_identity_hash(*, base_identity: bytes, content_hash: bytes) -> bytes: + """Fold an event's base identity with its own content hash. + + Used only when a base identity (event type + anchoring message) may be + shared by more than one event within one revision (e.g. multiple + ``chatgpt_block_metadata`` events on the same message, one per block) -- + still content-derived, never the array index (polylogue-aggz). + """ + return bytes.fromhex(hash_payload({"base_identity": base_identity.hex(), "content": content_hash.hex()})) def _session_hash_payload( @@ -510,14 +530,18 @@ def session_revision_projection(convo: ParsedSession) -> SessionRevisionProjecti message_contents: set[tuple[bytes, bytes]] = set() message_hashes: list[bytes] = [] for payload in messages_payload: - identity = bytes.fromhex(hash_payload(_message_identity_payload(payload))) + message_native_id = payload["id"] + assert isinstance(message_native_id, str) # built as str above, never anything else + identity = message_identity_hash(id=message_native_id) content = bytes.fromhex(hash_payload(payload)) message_contents.add((identity, content)) message_hashes.append(content) attachment_identities: set[bytes] = set() attachment_contents: set[tuple[bytes, bytes]] = set() for payload in attachments_payload: - identity = bytes.fromhex(hash_payload(_attachment_identity_payload(payload))) + identity = attachment_identity_hash( + message_id=payload["message_id"], name=payload["name"], mime_type=payload["mime_type"] + ) inline_content_hash = payload.get("inline_content_hash") attachment_identities.add(identity) if isinstance(inline_content_hash, str): @@ -528,7 +552,12 @@ def session_revision_projection(convo: ParsedSession) -> SessionRevisionProjecti for payload, event in zip(session_events_payload, convo.session_events, strict=True): event_hashes.append(bytes.fromhex(hash_payload(payload))) content_payload = _event_content_payload(event) - event_base_identities.append(bytes.fromhex(hash_payload(_event_base_identity_payload(content_payload)))) + event_base_identities.append( + event_base_identity_hash( + event_type=content_payload["event_type"], + source_message_provider_id=content_payload["source_message_provider_id"], + ) + ) event_content_hashes.append(bytes.fromhex(hash_payload(content_payload))) event_contents: set[tuple[bytes, bytes]] = set() for base_identity, content_hash in zip(event_base_identities, event_content_hashes, strict=True): @@ -549,9 +578,7 @@ def session_revision_projection(convo: ParsedSession) -> SessionRevisionProjecti # content-intrinsic block_index), and true duplicates (same base # identity, same content, whether or not any sibling exists) # correctly collapse to one set entry either way. - canonical_identity = bytes.fromhex( - hash_payload({"base_identity": base_identity.hex(), "content": content_hash.hex()}) - ) + canonical_identity = event_canonical_identity_hash(base_identity=base_identity, content_hash=content_hash) event_contents.add((canonical_identity, content_hash)) return SessionRevisionProjection( session_hash=bytes.fromhex(session_hash_hex), diff --git a/tests/unit/pipeline/test_pipeline_ids.py b/tests/unit/pipeline/test_pipeline_ids.py index 25490bd3a2..da87f7b4dd 100644 --- a/tests/unit/pipeline/test_pipeline_ids.py +++ b/tests/unit/pipeline/test_pipeline_ids.py @@ -12,6 +12,10 @@ _message_hash_payload, _normalize_for_hash, _session_hash_payload, + attachment_identity_hash, + event_base_identity_hash, + event_canonical_identity_hash, + message_identity_hash, session_content_hash, session_id, session_revision_projection, @@ -289,3 +293,125 @@ def test_session_revision_projection_matches_independent_recomputation() -> None if "inline_content_hash" in p ) assert list(projection.event_hashes) == [bytes.fromhex(hash_payload(p)) for p in independent_event_payloads] + + +# --- polylogue-aggz: typed identity constructor ----------------------------- +# +# These tests prove the constructor property structurally, not just by +# example: acquisition state, provider-reported measurement, and any field a +# parser might add in the future cannot reach comparison identity, because +# the identity constructors are fixed keyword-only functions -- passing +# anything outside their declared parameters is a TypeError at the call +# boundary, not a value that has to be remembered and stripped. + + +def test_message_identity_hash_rejects_non_content_fields() -> None: + """The message identity constructor accepts only ``id`` -- nothing else. + + Attempting to pass ``text``/``timestamp``/``role`` (what a message says, + not which message it is) is rejected by the function signature itself, + not by a runtime filter someone has to remember to apply. + """ + with pytest.raises(TypeError): + message_identity_hash(id="m1", text="hello") # type: ignore[call-arg] + + +def test_attachment_identity_hash_rejects_acquisition_state() -> None: + """Acquisition state (polylogue-bu1i) cannot reach attachment identity. + + ``size_bytes`` and ``inline_content_hash`` describe whether an + attachment's bytes have been acquired, not which attachment it is -- + passing them is a TypeError, proving the exclusion is structural rather + than a convention encoded in a list of dict keys. + """ + with pytest.raises(TypeError): + attachment_identity_hash( # type: ignore[call-arg] + message_id="m1", name="f.txt", mime_type="text/plain", size_bytes=3 + ) + with pytest.raises(TypeError): + attachment_identity_hash( # type: ignore[call-arg] + message_id="m1", name="f.txt", mime_type="text/plain", inline_content_hash="deadbeef" + ) + + +def test_attachment_identity_hash_rejects_full_payload_spread() -> None: + """A parser adding a new field to the payload dict cannot silently enter + identity: spreading the *entire* hash-stable attachment payload (as a + real future parser change might attempt, e.g. after adding a brand new + ``upload_origin`` or ``caption`` field to what gets hashed) into the + identity constructor is rejected outright, because the payload carries + keys (``id``, ``size_bytes``, and whatever new field a parser adds) that + are simply not parameters of ``attachment_identity_hash``. + """ + attachment = ParsedAttachment( + provider_attachment_id="a1", message_provider_id="m1", name="f.txt", mime_type="text/plain", size_bytes=3 + ) + full_payload = _attachment_hash_payload(attachment) + # Simulate a parser adding a brand-new, never-seen-before field to the + # hash-stable payload -- the exact failure shape polylogue-bu1i/-nuec + # were: a NEW field silently entering identity because the extraction + # took "everything except a denylist" rather than an explicit allowlist. + full_payload["totally_new_provider_field"] = "unclassified-value" + with pytest.raises(TypeError): + attachment_identity_hash(**full_payload) + + +def test_event_base_identity_hash_rejects_measurement_fields() -> None: + """Provider-reported measurement (polylogue-nuec) cannot reach event identity.""" + with pytest.raises(TypeError): + event_base_identity_hash( # type: ignore[call-arg] + event_type="generation_lifecycle", + source_message_provider_id="m1", + payload={"elapsed_duration_ms": 13000}, + ) + + +def test_identity_constructors_ignore_new_payload_fields_when_called_correctly() -> None: + """Adding a new field to a parser fixture leaves identity unaffected. + + Two attachments differing only in a field that is not one of the three + named identity parameters -- here, acquisition state plus a synthetic + "field a future parser might add" -- still produce identical identity, + because the constructor was never given that field to begin with. + """ + acquired = ParsedAttachment( + provider_attachment_id="a1", + message_provider_id="m1", + name="f.txt", + mime_type="text/plain", + size_bytes=3, + inline_bytes=b"abc", + ) + unacquired = ParsedAttachment( + provider_attachment_id="a1-different-provider-id", + message_provider_id="m1", + name="f.txt", + mime_type="text/plain", + size_bytes=None, + ) + acquired_payload = _attachment_hash_payload(acquired) + unacquired_payload = _attachment_hash_payload(unacquired) + identity_acquired = attachment_identity_hash( + message_id=acquired_payload["message_id"], + name=acquired_payload["name"], + mime_type=acquired_payload["mime_type"], + ) + identity_unacquired = attachment_identity_hash( + message_id=unacquired_payload["message_id"], + name=unacquired_payload["name"], + mime_type=unacquired_payload["mime_type"], + ) + assert identity_acquired == identity_unacquired + + +def test_event_canonical_identity_hash_folds_base_and_content() -> None: + """The canonical fold is a pure function of the two hashes it is given.""" + base = event_base_identity_hash(event_type="chatgpt_block_metadata", source_message_provider_id="m1") + content_a = bytes.fromhex(hash_payload({"block_index": 0})) + content_b = bytes.fromhex(hash_payload({"block_index": 1})) + folded_a = event_canonical_identity_hash(base_identity=base, content_hash=content_a) + folded_b = event_canonical_identity_hash(base_identity=base, content_hash=content_b) + # Same base identity, different content -> different canonical identity + # (this is precisely what lets two distinct same-type-same-anchor events + # coexist as separate set entries -- polylogue-aggz). + assert folded_a != folded_b