diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 1d073d5fc6..1dd387648f 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,4 +1,4 @@ -{"_type":"issue","id":"polylogue-c737","title":"ArchiveStore._write_parsed_precedence_result writes a session for a raw recorded raw_session_memberships.decision='ambiguous'","description":"## What the live archive shows (coordinator measurement, confirmed independently)\n\nLive query against `/realm/db/polylogue` for `origin='aistudio-drive'`:\n34 cohorts now have BOTH members parsed (growing over the course of one\nsession). 28 of those have `raw_session_memberships.decision='ambiguous'`\non BOTH members under the SAME `logical_source_key` -- genuinely arbitrated,\ncorrectly refused a winner -- yet the cohort's session IS present in\nindex.db with zero acquired attachments: 641 attachments total across the\n28, every one `unfetched`, while the enriched sibling in the blob store\nholds the bytes.\n\n## Root cause, traced and reproduced\n\n`polylogue/storage/sqlite/archive_tiers/archive.py`'s\n`apply_raw_membership_classification` (the classify_membership_revisions\nconsumer) is innocent: for a fully-ambiguous cohort (no accepted_raw_ids)\nit explicitly clears `raw_sessions.parsed_at_ms` and never writes to\n`sessions` -- verified by reading its finalization block (`complete` check\nat the end of the function, ~line 3873-3901).\n\nThe actual writer is `_write_parsed_precedence_result` (same file), reached\nvia `write_parsed_for_retained_raw`/`write_parsed_for_retained_raw_result`\nwith `revision_authoritative=False` (the default -- used by the one-shot\nimporter, `pipeline/services/archive_ingest.py`, and by\n`_index_parsed_for_retained_raw`'s other non-membership-governed callers).\nIts ONLY revision-authority awareness before this fix was:\n\n governed = SELECT 1 FROM raw_revision_heads WHERE session_id = ?\n if governed is not None: skip\n\n`raw_revision_heads` is populated ONLY when a cohort has an ACCEPTED\nwinner. A cohort `classify_membership_revisions` genuinely refused to\narbitrate never gets an accepted head, so `governed` stays `None` and the\nfunction falls through to its own browser-capture-precedence/freshness\nlogic and writes the session unconditionally on the raw's next reparse --\nlast-writer-wins, independent of the recorded `ambiguous` verdict.\n`repair.py:1075`/`repair.py:4432-4462` (the two gates the investigation\nstarted from) are both innocent: neither is on this write path at all --\n`repair.py:4432` is a read-only reporting/accounting classifier\n(`_raw_replay_plan_outcome`), and `repair.py:1075` is a narrow inspector\nfor a different (`source-v7`/`quarantined-accepted-raw`) repair scenario.\n\nReproduced directly: a synthetic archive with a raw whose\n`raw_session_memberships` row is `decision='ambiguous'`, then calling\n`archive.write_parsed_for_retained_raw(session, raw_id=..., ...)` (no\n`revision_authoritative`) writes the session anyway pre-fix; the fix makes\nit a no-op (`content_changed=False`).\n\n## Fix landed in polylogue-af059's fast-follow PR\n\n`_write_parsed_precedence_result` now also refuses when the raw's OWN\n`raw_session_memberships.decision = 'ambiguous'`, in addition to the\nexisting `raw_revision_heads` check.\n\n## Known sibling hole, NOT fixed here (different file, different owner)\n\n`polylogue/pipeline/services/ingest_batch/_core.py` (the daemon's default\nbatch-ingest write path, used for most origins that don't go through\n`sources/live/batch.py`'s revision-authority-aware branch) has the SAME\nshape: its own precedence/freshness logic, no `raw_session_memberships`\nconsultation. The coordinator's own measurement\n(`revision_authority='quarantined'` with `parsed_at_ms` set: chatgpt-export\n7,050, codex-session 3,633, claude-code-session 2,450, claude-ai-export\n1,562 -- NOT all necessarily leaked materializations, but the same shape)\nsuggests this is where most of the non-drive volume would leak through, if\nthose origins' raws ever get genuinely `ambiguous`-recorded membership\ndecisions. Needs its own read-only census to confirm before fixing (not\ndone here -- out of file-ownership scope for this PR).\n\n## Live remediation\n\nNOT performed here (code-only fix). The 28 live aistudio-drive sessions\nwith zero-acquired attachments need their own re-materialization pass once\nthis fix (and bu1i's classifier fix) are both deployed.\n\nRef polylogue-eqnv, polylogue-bu1i, polylogue-7ilr, polylogue-9dxn","status":"open","priority":0,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T13:09:15Z","created_by":"Sinity","updated_at":"2026-07-30T13:09:15Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-c737","title":"ArchiveStore._write_parsed_precedence_result writes a session for a raw recorded raw_session_memberships.decision='ambiguous'","description":"## What the live archive shows (coordinator measurement, confirmed independently)\n\nLive query against `/realm/db/polylogue` for `origin='aistudio-drive'`:\n34 cohorts now have BOTH members parsed (growing over the course of one\nsession). 28 of those have `raw_session_memberships.decision='ambiguous'`\non BOTH members under the SAME `logical_source_key` -- genuinely arbitrated,\ncorrectly refused a winner -- yet the cohort's session IS present in\nindex.db with zero acquired attachments: 641 attachments total across the\n28, every one `unfetched`, while the enriched sibling in the blob store\nholds the bytes.\n\n## Root cause, traced and reproduced\n\n`polylogue/storage/sqlite/archive_tiers/archive.py`'s\n`apply_raw_membership_classification` (the classify_membership_revisions\nconsumer) is innocent: for a fully-ambiguous cohort (no accepted_raw_ids)\nit explicitly clears `raw_sessions.parsed_at_ms` and never writes to\n`sessions` -- verified by reading its finalization block (`complete` check\nat the end of the function, ~line 3873-3901).\n\nThe actual writer is `_write_parsed_precedence_result` (same file), reached\nvia `write_parsed_for_retained_raw`/`write_parsed_for_retained_raw_result`\nwith `revision_authoritative=False` (the default -- used by the one-shot\nimporter, `pipeline/services/archive_ingest.py`, and by\n`_index_parsed_for_retained_raw`'s other non-membership-governed callers).\nIts ONLY revision-authority awareness before this fix was:\n\n governed = SELECT 1 FROM raw_revision_heads WHERE session_id = ?\n if governed is not None: skip\n\n`raw_revision_heads` is populated ONLY when a cohort has an ACCEPTED\nwinner. A cohort `classify_membership_revisions` genuinely refused to\narbitrate never gets an accepted head, so `governed` stays `None` and the\nfunction falls through to its own browser-capture-precedence/freshness\nlogic and writes the session unconditionally on the raw's next reparse --\nlast-writer-wins, independent of the recorded `ambiguous` verdict.\n`repair.py:1075`/`repair.py:4432-4462` (the two gates the investigation\nstarted from) are both innocent: neither is on this write path at all --\n`repair.py:4432` is a read-only reporting/accounting classifier\n(`_raw_replay_plan_outcome`), and `repair.py:1075` is a narrow inspector\nfor a different (`source-v7`/`quarantined-accepted-raw`) repair scenario.\n\nReproduced directly: a synthetic archive with a raw whose\n`raw_session_memberships` row is `decision='ambiguous'`, then calling\n`archive.write_parsed_for_retained_raw(session, raw_id=..., ...)` (no\n`revision_authoritative`) writes the session anyway pre-fix; the fix makes\nit a no-op (`content_changed=False`).\n\n## Fix landed in polylogue-af059's fast-follow PR\n\n`_write_parsed_precedence_result` now also refuses when the raw's OWN\n`raw_session_memberships.decision = 'ambiguous'`, in addition to the\nexisting `raw_revision_heads` check.\n\n## Known sibling hole, NOT fixed here (different file, different owner)\n\n`polylogue/pipeline/services/ingest_batch/_core.py` (the daemon's default\nbatch-ingest write path, used for most origins that don't go through\n`sources/live/batch.py`'s revision-authority-aware branch) has the SAME\nshape: its own precedence/freshness logic, no `raw_session_memberships`\nconsultation. The coordinator's own measurement\n(`revision_authority='quarantined'` with `parsed_at_ms` set: chatgpt-export\n7,050, codex-session 3,633, claude-code-session 2,450, claude-ai-export\n1,562 -- NOT all necessarily leaked materializations, but the same shape)\nsuggests this is where most of the non-drive volume would leak through, if\nthose origins' raws ever get genuinely `ambiguous`-recorded membership\ndecisions. Needs its own read-only census to confirm before fixing (not\ndone here -- out of file-ownership scope for this PR).\n\n## Live remediation\n\nNOT performed here (code-only fix). The 28 live aistudio-drive sessions\nwith zero-acquired attachments need their own re-materialization pass once\nthis fix (and bu1i's classifier fix) are both deployed.\n\nRef polylogue-eqnv, polylogue-bu1i, polylogue-7ilr, polylogue-9dxn","status":"closed","priority":0,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T13:09:15Z","created_by":"Sinity","updated_at":"2026-07-30T13:20:58Z","closed_at":"2026-07-30T13:20:58Z","close_reason":"Fixed in PR #3397 (feature/fix/ambiguous-membership-precedence-write-leak): ArchiveStore._write_parsed_precedence_result now also refuses to write when the raw's own raw_session_memberships.decision='ambiguous', alongside the pre-existing raw_revision_heads check. Verified with a regression test (anti-vacuity confirmed via temporary guard short-circuit + rerun). Sibling hole in pipeline/services/ingest_batch/_core.py NOT fixed here (different file, out of ownership scope) -- needs its own read-only census before fixing; tracked as residual scope in this same bead's description.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-8249","title":"rebuild parse workers capped at 8 on a 24-thread host; ingest_parse_workers config is inert","design":"Found 2026-07-29 by codebase audit, while checking whether the imminent full\nrebuild honors its parse-worker configuration. TWO defects, one of which\ndirectly caps rebuild throughput.\n\n(1) THE PARSE-WORKER COUNT IS CAPPED AT 8 ON A 24-THREAD HOST\n\npolylogue/pipeline/services/process_pool.py:52\n default = max(1, min(8, (os.cpu_count() or 2) - 1))\n\nOn sinnix-prime (i7-13700K, 16 cores / 24 threads) this resolves to 8, leaving\n16 threads idle. The rebuild path reaches it directly:\n maintenance/rebuild_index.py:543 ingest_workers=None\n maintenance/replay.py:199 resolved = ... else resolve_parse_worker_count()\n\nThe daemon now runs free-threaded 3.14t (GIL disabled) and the GIL parse path\nwas deleted this session, so thread-parallel parse is the only path -- the\n`min(8, ...)` ceiling is the binding constraint on a rebuild we are trying to\nbring from 9.2h down to 1-2h. The cap predates the free-threaded deploy; on a\nGIL build 8 was a reasonable process-pool bound, but that reasoning no longer\napplies.\n\nBEFORE THE REBUILD: either raise/remove the cap, or set\nPOLYLOGUE_INGEST_PARSE_WORKERS explicitly for the rebuild run. Do NOT assume\nhigher is strictly better -- measure. Parse is decode-bound but the apply side\nis a single writer, so beyond some width the writer becomes the bottleneck and\nextra parse threads only add memory pressure. The new RebuildPassCost\ninstrumentation (replay_s / checkpoint_s / mib_per_s / parse_workers, landed\nthis session) is exactly the instrument for choosing the width from one short\nmeasured pass rather than guessing.\n\n(2) THE DOCUMENTED CONFIG KNOB IS INERT\n\nThere are two knobs for parse-worker count and only one works:\n env POLYLOGUE_INGEST_PARSE_WORKERS -- honored (process_pool.py:43,53)\n config `sources.ingest_parse_workers` -- IGNORED\n\nThe config property is defined (config.py:602), given a default\n(config.py:1876), listed in the config inventory twice (config.py:1387,1642),\nand documented (docs/configuration.md:351 \"Parallel parse workers during\ningest (default 1)\") -- but NOTHING reads it. An operator setting it in\n~/.config/polylogue/polylogue.toml gets silence.\n\nThe doc is also wrong independently of the wiring: it says \"default 1\" while\nthe resolver's actual default is min(8, cpus-1) = 8 here.\n\nFIX: make resolve_parse_worker_count read the resolved config, keeping the env\nvar as the override layer the config system already defines -- or delete the\nconfig property and document the env var as the sole knob. Per the standing\ndirective, one of the two must go; a documented knob that does nothing is\nworse than no knob. Whichever survives must be the one the rebuild reads.\n","status":"closed","priority":0,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T10:35:27Z","created_by":"Sinity","updated_at":"2026-07-29T17:16:31Z","closed_at":"2026-07-29T17:16:31Z","close_reason":"Fixed. Parse workers now scale to the interpreter: min(16, cpus-2) free-threaded, min(8, cpus-1) under the GIL. Verified under the deployed python3.14t -\u003e 16 workers, up from 8 on this 24-thread host. The module's own control-run measurement (3.9x at w=4 rising to 9.6x at w=16) is the evidence for 16 as the ceiling. Second defect also fixed: the inert sources.ingest_parse_workers config property (defined, defaulted, inventoried twice, documented as 'default 1', read by nothing) is deleted; POLYLOGUE_INGEST_PARSE_WORKERS survives as the single knob with an accurate inventory description. Commit a7945e9fe. Related: the devshell default is now the free-threaded shell (matching the daemon), so local runs no longer silently parse sequentially.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-2qx.4","title":"Field-landing decisions for the unread-wire batch: one index bump, one rebuild","description":"DECIDED. The audit established WHAT is discarded; this fixes WHERE each lands, so the parser work is mechanical and the schema changes batch into a single index-tier bump rather than one per origin.\n\n stop_reason (608,608 on wire)\n -\u003e column on messages. One value per assistant turn, low cardinality,\n feeds terminal_state directly. Replaces three columns that guess at it\n and are 85-99% 'unknown'.\n structuredPatch (105,123) + originalFile (92,313) + oldString/newString/filePath\n -\u003e new file_edits table keyed by tool_use_block_id. It is a RELATION (one\n edit per tool call), not a block attribute. This is what raises\n polylogue-cijx's file-trajectory grading from 'observed' to\n 'checkpointed' -- originalFile is the captured pre-state cijx declares\n unavailable.\n parentToolUseID (842,819 records, 185,982 distinct dispatch ids)\n -\u003e a real join-key column on session_links, plus method. It IS the\n delegation edge; it belongs where edges live. Replaces the positional\n pairing gated on count equality that resolves 12.8%.\n pr-link (20,702)\n -\u003e new session_refs table (kind, url, number, repo). Generalizes to issue\n refs and stays tracker-agnostic -- do not create a github_prs table.\n runSettings (aistudio-drive: temperature, topP, topK, maxOutputTokens,\n thinkingLevel, safetySettings, enable* flags)\n -\u003e JSON column on sessions. Genuinely per-session config; decomposing it\n into columns buys nothing and couples the schema to one provider.\n ai-title (18,422) / threads.title / slug (1,500) / agentId\n -\u003e sessions.title + title_source for the title; slug -\u003e a display_name\n column so subagent rows read 'greedy-squishing-hamming' rather than\n '5ecdb160-...:agent-af4e'.\n outcome-unknown reason\n -\u003e enum column beside blocks.tool_result_is_error. Three causes are\n collapsed into one NULL today (provider emitted nothing / parser\n deliberately distrusts it / parser does not read this provider's\n field), all knowable at parse time.\n tool-results sidecars (12,588 files, 1.34 GB, 3 ingested)\n -\u003e block content, attached to the existing tool_result block by tool_id\n (the filename IS the tool id). NEVER a session -- the hook-inflation\n incident (18,391 -\u003e 83,286 sessions) is the precedent.\n\nBATCHING: all of the above is ONE index-tier bump and ONE rebuild. Splitting by\norigin would mean four bumps and four rebuild windows against a corpus where a\nfull rebuild is the standing performance complaint (polylogue-623q). Do the\nschema change once, then the per-origin parser reads land against it\nincrementally without further bumps.\n\nSCOPE NOTE (operator, 2026-07-29): read everything SEMANTICALLY MEANINGFUL, not\neverything. Some wire fields are genuinely not worth a column -- the\nper-key classification in the OriginSpec fidelity declaration is where that\njudgement is recorded, and 'dropped, because X' is a valid outcome.","acceptance_criteria":"1. One index-tier bump covers every landing above; no second bump for a later origin. 2. Each landing is a typed column/table, not a JSON blob, except runSettings where the blob is the decision. 3. tool-results attachment leaves session count unchanged, asserted by a test. 4. Per-origin parser reads land against the new schema without further migrations. 5. The OriginSpec fidelity declaration records a per-key verdict including deliberate drops with reasons.","status":"closed","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T04:52:50Z","created_by":"Sinity","updated_at":"2026-07-29T17:16:55Z","closed_at":"2026-07-29T17:16:55Z","close_reason":"Landed. INDEX_SCHEMA_VERSION 45-\u003e46 (SEMANTIC_REPARSE), one bump for the whole batch. The version decision was settled from bootstrap.py's actual code path, not assumed: a same-version reopen only re-applies benign CREATE TABLE/INDEX IF NOT EXISTS DDL and never ALTER TABLE, so staying at v45 would have left existing v45 archives silently missing the new columns. Landed: messages.stop_reason, blocks.tool_result_outcome_unknown_reason, sessions.display_name + run_settings_json, session_links.parent_tool_use_block_id, and the file_edits and session_refs tables. Parsers now populate all of them -- measured coverage: stop_reason 44.7% of main-session messages, file_edit 7,335/44,125 tool_result blocks, display_name 65.0% of subagent sessions, session_refs 1,483 rows, outcome_unknown_reason 19,613 not_reported + 80 distrusted. parent_tool_use_provider_id deliberately left NULL: two independent samples (200 subagent transcripts; 109,853 records) found parentToolUseID appears only on the PARENT's progress records, never on a child's own, so it cannot join parent to child. Delegation resolution instead uses content identity. tool-results sidecars needed no schema change (they attach to the existing tool_result block by tool_id).","labels":["area:ingest","area:sources","delivery:K-interop-origin-export","delivery:ac-patched","horizon:frontier","lane:origin-interop-export","refactor"],"dependencies":[{"issue_id":"polylogue-2qx.4","depends_on_id":"polylogue-2qx","type":"parent-child","created_at":"2026-07-29T06:52:49Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-cijx.4","title":"Repo identity, path normalization and readable labels are ONE batch","description":"DECIDED. These were three separate items; they are one, because the label is unusable until identity is fixed and both fall out of the same normalization.\n\nTHE EVIDENCE, from eight real untitled claude-code sessions:\n repo_name = 'agent-ad682bc849a1cd0f0'\n top path = /realm/project/polylogue/.claude/worktrees/agent-ad682bc849a1cd0f0/\n polylogue/pipeline/services/ingest_batch/_core.py\nA structural label today reads 'agent-ad682bc849a1cd0f0 - 27f - 499m' -- worse\nthan the UUID it replaces. repo_name derives from cwd, the cwd is a worktree\ndirectory, so the agent id becomes the repo name.\n\nNormalize both and the same eight sessions read:\n polylogue - pipeline/services/ingest_batch/_core.py +26 - 499 msgs\n polylogue - daemon/status.py +7 - 322 msgs\n polylogue - api/archive.py +10 - 259 msgs\n polylogue - tests/unit/insights/test_delegation_work_evidence.py +5 - 163 msgs\n polylogue - storage/repair.py +1 - 91 msgs\nFor a coding session, WHICH FILES YOU TOUCHED is the topic. That beats the\nprovider echo titles, which collide 78-way.\n\nDECISION 1 -- REPOSITORY IDENTITY\n A repository is keyed on its normalized remote (all spellings of one remote\n are one repo); where no remote exists, the outermost git root. NOT the cwd.\n A worktree is a CHECKOUT OF a repository, not a repository -- every\n /realm/worktrees/polylogue-* and .claude/worktrees/agent-* is one checkout of\n polylogue. A session with no git evidence resolves to a DIRECTORY and read\n surfaces say so; do not synthesize a repository for it. Measured today:\n polylogue holds 106 distinct repo_ids, sinex 28, sinnix 31; git_branch is\n populated on 15.8% of sessions, git_repository_url on 13.2%, commit_hash on\n 15.9% -- so for ~84% the 'repo' column is really cwd.\n\nDECISION 2 -- PATHS ARE REPO-RELATIVE\n Strip the checkout root prefix (already recorded as repos.root_path) so\n action_pairs.tool_path is comparable across checkouts of one repo. Without\n this, the same file edited in two worktrees is two different paths and no\n cross-session file question works.\n\nDECISION 3 -- THE LABEL IS A PROJECTION, NEVER A COLUMN\n Form: \u003crepo\u003e - \u003cdominant repo-relative path\u003e +N - \u003csize\u003e, substituting the\n provider title for the path clause when a real one exists. Computed at read\n time in the 4p1 Projection. It must not be written to sessions.title: it\n would collide with genuine provider titles (ai-title, threads.title) and\n freeze as the session grows -- '340 msgs' is wrong the moment message 341\n lands. Measured collision rate for the structural form: 3.5% over 4,000\n sessions, max collision 10, mostly pairwise -- acceptable, and far better\n than the echo baseline's 78-way.\n\nDECISION 4 -- RESULT UNIT IS THE TOP-LEVEL SESSION\n All eight sampled sessions above are agent-* subagents; 8,614 of 18,871\n sessions (45.6%) are subagent children. A default list is unreadable because\n it is half fanout. Default unit = top-level session; children reachable\n through an explicit projection, never filling the list. Any count states its\n unit -- '18,871 sessions' unqualified is wrong when 8,614 are children.\n\nSEQUENCE: identity+paths first (write-path change, no schema bump), then the\nlabel projection. Readability cannot land before identity.","acceptance_criteria":"1. One repository per normalized remote; worktrees enumerate underneath as checkouts; polylogue/sinex/sinnix each collapse to one. 2. tool_path is repo-relative; the same file in two worktrees is one path. 3. The display label is computed per request and appears in no table; sessions.title holds only provider-supplied values. 4. Default result unit is the top-level session, proven by re-running 'polylogue find repo:polylogue' and showing named non-fanout rows. 5. Report the label collision rate against the measured 3.5% / max-10 baseline.","status":"open","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T04:52:49Z","created_by":"Sinity","updated_at":"2026-07-29T04:52:49Z","labels":["area:insights","area:interop","area:substrate","horizon:mid","lane:read-contracts","tech-tree"],"dependencies":[{"issue_id":"polylogue-cijx.4","depends_on_id":"polylogue-cijx","type":"parent-child","created_at":"2026-07-29T06:52:48Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/polylogue/storage/sqlite/archive_tiers/archive.py b/polylogue/storage/sqlite/archive_tiers/archive.py index dfab03b8c0..86ca278c76 100644 --- a/polylogue/storage/sqlite/archive_tiers/archive.py +++ b/polylogue/storage/sqlite/archive_tiers/archive.py @@ -1686,11 +1686,26 @@ def _write_parsed_precedence_result( # attachments reported unfetched despite the bytes existing in the # blob store). Refuse this raw explicitly instead of relying on an # absent head to imply "unclaimed, free to write". + # + # Scoped to the membership being written, not to the raw. One retained + # raw routinely lowers to many sessions -- a Claude Code transcript and + # its subagent sidechains, a bundle member set -- and those sessions are + # arbitrated independently. Measured on the live archive: 295 raws carry + # a mix of decisions, together holding 489 sessions whose own membership + # is NOT ambiguous, and one raw carries 106 memberships. A raw-scoped + # predicate suppresses every one of those sessions as soon as a single + # sibling membership is ambiguous, which trades a fidelity downgrade for + # outright absence -- a worse failure, and one that would have landed at + # the next full rebuild. ambiguous_membership = ( self._ensure_source_conn() .execute( - "SELECT 1 FROM raw_session_memberships WHERE raw_id = ? AND decision = 'ambiguous' LIMIT 1", - (raw_id,), + """ + SELECT 1 FROM raw_session_memberships + WHERE raw_id = ? AND provider_session_id = ? AND decision = 'ambiguous' + LIMIT 1 + """, + (raw_id, session.provider_session_id), ) .fetchone() ) diff --git a/tests/unit/storage/test_revision_replay.py b/tests/unit/storage/test_revision_replay.py index 4991d61313..a9c527ec38 100644 --- a/tests/unit/storage/test_revision_replay.py +++ b/tests/unit/storage/test_revision_replay.py @@ -683,6 +683,79 @@ def test_precedence_write_refuses_a_raw_recorded_ambiguous(tmp_path: Path) -> No assert conn.execute("SELECT COUNT(*) FROM sessions WHERE session_id = ?", (session_id,)).fetchone() == (0,) +def test_precedence_write_allows_a_non_ambiguous_sibling_membership_on_the_same_raw(tmp_path: Path) -> None: + """The ambiguity refusal is per-membership, not per-raw. + + One retained raw routinely lowers to many independently-arbitrated sessions + -- a Claude Code transcript plus its subagent sidechains, a bundle member + set. Scoping the refusal to ``raw_id`` alone suppresses every session that + raw carries the moment a single sibling membership is ambiguous, turning a + fidelity downgrade into outright absence. + + Measured on the live archive when this was caught: 295 raws carry a mix of + decisions, together holding 489 sessions whose own membership is not + ambiguous, and one raw carries 106 memberships. Their content would have + silently vanished at the next full rebuild. + """ + initialize_active_archive_root(tmp_path) + + ambiguous_session = ParsedSession( + source_name=Provider.CHATGPT, + provider_session_id="s-ambiguous", + messages=[ParsedMessage(provider_message_id="a-0", role=Role.USER, text="left")], + ) + settled_session = ParsedSession( + source_name=Provider.CHATGPT, + provider_session_id="s-settled", + messages=[ParsedMessage(provider_message_id="b-0", role=Role.USER, text="right")], + ) + + with ArchiveStore.open_existing(tmp_path, read_only=False) as archive: + raw_id = archive.write_raw_payload( + provider=Provider.CHATGPT, payload=b"two-sessions", source_path="bundle.json", acquired_at_ms=1 + ) + source_conn = archive._ensure_source_conn() + with source_conn: + # One raw, two memberships, arbitrated differently -- the live shape. + source_conn.execute( + """ + INSERT INTO raw_session_memberships ( + raw_id, logical_source_key, provider_session_id, + source_revision, normalized_content_hash, message_count, + decision, decided_at_ms + ) VALUES (?, 'chatgpt:s-ambiguous', 's-ambiguous', ?, ?, 1, 'ambiguous', 1) + """, + (raw_id, raw_id, bytes.fromhex(raw_id)), + ) + source_conn.execute( + """ + INSERT INTO raw_session_memberships ( + raw_id, logical_source_key, provider_session_id, + source_revision, normalized_content_hash, message_count, + decision, decided_at_ms + ) VALUES (?, 'chatgpt:s-settled', 's-settled', ?, ?, 1, 'applied', 1) + """, + (raw_id, raw_id + "-b", bytes.fromhex(raw_id)), + ) + + _, ambiguous_session_id = archive.write_parsed_for_retained_raw( + ambiguous_session, raw_id=raw_id, source_path="bundle.json", acquired_at_ms=2 + ) + _, settled_session_id = archive.write_parsed_for_retained_raw( + settled_session, raw_id=raw_id, source_path="bundle.json", acquired_at_ms=3 + ) + + with sqlite3.connect(tmp_path / "index.db") as conn: + # The ambiguous membership is still refused ... + assert conn.execute( + "SELECT COUNT(*) FROM sessions WHERE session_id = ?", (ambiguous_session_id,) + ).fetchone() == (0,) + # ... and its settled sibling on the same raw is not collateral damage. + assert conn.execute("SELECT COUNT(*) FROM sessions WHERE session_id = ?", (settled_session_id,)).fetchone() == ( + 1, + ) + + def test_isolated_later_raw_does_not_override_cohort_retired_under_legacy_detail_string( tmp_path: Path, ) -> None: