From e8827a25f0c7a3e4768ce9e86bdd3a8168f5ef44 Mon Sep 17 00:00:00 2001 From: Sinity Date: Mon, 20 Jul 2026 13:44:15 +0200 Subject: [PATCH] fix(sources): absorb quarantined membership head into later rebuild cohorts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v42 rebuild died (loud-context guard, post-#3204) on 'membership replay cannot retire an unrelated accepted head' for chatgpt:366f7d9f-…: cohort candidate selection is page-dependent, so a quarantined head raw decided while walking an earlier page was absent from a later page's cohort for the same logical key, and membership replay refused to retire a head it could not rank. Absorb the current quarantined head raw into the cohort candidate set so the real prefix classifier ranks it against the new members (no scalar comparisons). Chain-governed heads are deliberately not absorbed -- the #3204 yield path owns those. Verification: safe-by-construction (absorption only widens classification evidence); two attempts at a self-contained backfill fixture could not reproduce the exact census-authority state (both passed with and without the fix -- census re-derives authority on walk), so the verification artifact is the live v42 walk itself, which deterministically re-hits this key on resume; receipt to follow on the PR. --- .beads/issues.jsonl | 1 + polylogue/sources/revision_backfill.py | 12 ++++++++++++ 2 files changed, 13 insertions(+) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 8d2456c6a7..f34564477d 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -381,6 +381,7 @@ {"_type":"issue","id":"polylogue-sru.2","title":"Characterize ambiguous bucket: wordless continuation vs prose-without-markers","description":"Split next-turn-is-tool-call (wordless continuation) from prose-lacking-ack-markers; state counts for both. Opus-4-7 74% ambiguous vs deepseek 17% is likely turn-structure variance, not behavior — this split disambiguates.","design":"Implementation home: the claim-vs-evidence classifier in devtools (devtools/ module behind `devtools workspace claim-vs-evidence`; tests tests/unit/devtools/test_claim_vs_evidence.py). Wordless-continuation detection: for each failure's paired next assistant message, check whether its blocks contain tool_use and no text block with \u003eN chars before the first tool_use — that is 'wordless continuation'; prose without matched ack markers stays 'ambiguous-prose'. Emit both as classification_reason variants (field already exists) and add the two counts to the report summary + by_model/by_tool cuts. Regen: `devtools workspace claim-vs-evidence --limit 5000 --out-dir .agent/demos/claim-vs-evidence --json`. Acceptance: report shows ambiguous split into wordless_continuation vs prose_no_marker with counts; per-model ambiguous variance (opus-4-7 74% vs deepseek 17%) re-examined after the split.","notes":"2026-07-03 Codex WIP: unit implementation for ambiguous split passes focused tests, but live regeneration with --limit 5000 became too slow and had to be killed twice. First attempt used correlated subqueries for next-message block shape; second used set-based CTE; third used chunked second query after sampled rows, but the full command still exceeded 90s on active archive and ignored SIGINT while inside SQLite. Do not close or commit this slice until the live regeneration path is profiled/fixed. Dirty files currently show the WIP implementation: devtools/claim_vs_evidence.py and tests/unit/devtools/test_claim_vs_evidence.py. Last passing focused proof: python -m py_compile + ruff check + devtools test tests/unit/devtools/test_claim_vs_evidence.py -\u003e 3 passed.","status":"closed","priority":1,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:28Z","created_by":"Sinity","updated_at":"2026-07-03T07:45:10Z","started_at":"2026-07-03T07:09:21Z","closed_at":"2026-07-03T07:45:10Z","close_reason":"Completed: claim-vs-evidence now splits ambiguous follow-ups into wordless tool continuations and prose-without-marker buckets, reports the counts in JSON/README summaries, and regenerates the current demo on the active archive. Focused tests pass; live regen/check completed.","labels":["area:substrate","campaign"],"dependencies":[{"issue_id":"polylogue-sru.2","depends_on_id":"polylogue-sru","type":"parent-child","created_at":"2026-07-03T06:31:27Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"polylogue-sru.3","title":"Benign-recovery vs consequential-silence split by handler kind","description":"Read failures are ~94% silent but 'tried another path' is usually benign; Bash/test failures are the consequential class. Scope the headline to consequential handler kinds or add an explicit split — credibility depends on not inflating with trivial recoveries.","design":"Handler kind is already available on the paired failure row (actions lane exposes handler/tool). Define the consequential set explicitly in code (Bash/test/build/write-class handlers) and the benign-recovery set (Read/Glob/Grep-class 'tried another path'), emit split headline rows: silent-proceed among consequential vs among all. Keep the mapping a named constant with a rationale comment so reviewers can argue with it. Report both; never let the headline mix classes silently. Same regen/tests as the other methodology children.","status":"closed","priority":1,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:28Z","created_by":"Sinity","updated_at":"2026-07-03T07:58:08Z","started_at":"2026-07-03T07:55:37Z","closed_at":"2026-07-03T07:58:08Z","close_reason":"Completed: claim-vs-evidence now reports a first-class handler-class split separating consequential shell/edit/write-class tool failures from benign read/search/path-discovery failures and other tools. The regenerated active-archive artifact shows consequential=4,177 failures with 921 silent-proceed (22.0% lower bound), benign_recovery=633 with 166 silent-proceed (26.2%), and other=190 with 92 silent-proceed (48.4%). Focused tests and demo shelf checks passed.","labels":["area:substrate","campaign"],"dependencies":[{"issue_id":"polylogue-sru.3","depends_on_id":"polylogue-sru","type":"parent-child","created_at":"2026-07-03T06:31:28Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"polylogue-sru.1","title":"Expose action-unit outcome fields + followup_class as product capability","description":"Capabilities-may-not-be-silos gate for the campaign: the facts the report needs must become composable query capability. After this, the whole report is `actions where is_error:true | group by session.origin, followup_class | count` and every future cut (model/tool/repo/time) is free.","design":"1) is_error/exit_code are normalized at parse time (sources/parsers/base_models.py:74-75) but ActionQueryRowPayload (surfaces/payloads.py:~1298) carries neither — add as filterable/groupable action-unit fields. 2) Add derived followup_class (acknowledged|silent_proceed|wordless_continuation|ambiguous) + followup_message_ref computed in the source-derived lowering (no cache tables). 3) Reduce devtools workspace claim-vs-evidence to a render preset over these query strings, or retire it. Touchpoint chain: stage parser -\u003e AST to_payload -\u003e executor -\u003e metadata.py aggregate_group_fields -\u003e shell_completion_values.py -\u003e devtools render openapi + cli-output-schemas + cli-reference. Line refs pre-07-03; re-locate.","acceptance_criteria":"Fixture session with known unacknowledged failure fires via pure query strings; report README numbers reproducible from the printed queries.","notes":"Completed: action-unit outcome follow-up classification is now shared query capability. is_error/exit_code were already wired; this slice added source-derived followup_class and followup_message_ref over existing actions/messages/blocks, exposed followup_class as filterable/groupable action metadata, added action row payload fields, routed root CLI terminal-unit aggregate expressions before session-selector compilation, and moved the report classifier from scripts into polylogue.archive.actions.followup. Reproduction/query forms are now printed in .agent/demos/claim-vs-evidence/PUBLIC_REPRODUCTION.md: actions where is_error:true | group by followup_class | count; actions where followup_class:silent_proceed. Verification: focused DSL/report/CLI tests passed; active demo packet regenerated over archive root /home/sinity/.local/share/polylogue schema v23 with 41,886 structured failures and 5,000 inspected; devtools verify --quick passed run 20260703T092510Z-quick-718233-46e8b587.","status":"closed","priority":1,"issue_type":"feature","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:27Z","created_by":"Sinity","updated_at":"2026-07-03T09:25:36Z","started_at":"2026-07-03T09:05:37Z","closed_at":"2026-07-03T09:25:36Z","close_reason":"Completed","labels":["area:query","area:substrate","campaign"],"dependencies":[{"issue_id":"polylogue-sru.1","depends_on_id":"polylogue-sru","type":"parent-child","created_at":"2026-07-03T06:31:26Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-nfl5","title":"Prefix-dominance re-adoption of content-ahead capture evidence over chain heads","description":"PR #3204 makes chain-governed (non-quarantined) evidence win unconditionally over quarantined membership (browser-capture) heads: a scalar semantic frontier cannot prove content dominance, so the count-based content-ahead exception was removed (CodeRabbit P1s). Consequence: a capture genuinely AHEAD of a stale export (conversation continued after the export was produced) has its tail content unindexed until a newer export arrives — the capture raw stays in the source tier with SUPERSEDED/receipted decisions. This bead adds honest re-adoption: prove the chain head projection is a strict prefix (message/event/attachment hash prefix via session_revision_projection) of the capture projection, and only then let the capture take/keep the head. Needs the chain side projection at comparison time (recompute from raw or persist projections). ~228 chatgpt capture/export overlap raws + claude-ai analogues in the live corpus quantify the affected population.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-20T11:26:31Z","created_by":"Sinity","updated_at":"2026-07-20T11:26:31Z","labels":["area:ingest","area:storage","horizon:mid"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-6qjc","title":"Judgment automation actor: policy engine + trigger surface over MCP judge dispatcher","description":"From the 800m roles-to-config lane (PR #3202 design note): the judge MCP dispatcher already supports bulk policy-shaped decisions, but agent-scale judgment lacks a separate automation actor that calls judge on schedule/trigger with an explicit escalation path to human review — today every candidate defaults to human attention, which the operator has said does not scale (most agent-authored annotations will never be seen by a human). Needs: a policy engine deciding which assertion candidates are auto-judgeable, a trigger surface (daemon convergence stage or timer), and escalation semantics for the residue.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-20T10:08:22Z","created_by":"Sinity","updated_at":"2026-07-20T10:08:22Z","labels":["area:orchestration","horizon:mid"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-gt3g","title":"beads-issue provider-package marked accepted with no schema_package evidence","description":"devtools/provider_completeness.py --check fails repo-wide because Origin.BEADS_ISSUE's completeness\nmode in polylogue/sources/origin_specs.py declares maturity=\"accepted\" with schema_paths=() (no\nschemas/providers/beads catalog exists). provider_completeness._row_for_spec treats an empty\nschema_paths as status=\"missing\", which becomes a required-item blocker for any \"accepted\" row,\nso `devtools provider-completeness --check` always exits 1 on this row.\n\nDiscovered while wiring the Grok export parser (polylogue-y2hb, PR #3201): confirmed via\n`git show origin/master:polylogue/sources/origin_specs.py` that this exact pattern predates that\nPR (beads has had schema_paths=() + maturity=\"accepted\" since it was declared \"accepted\"), so it\nis not something that PR introduced -- it was pre-existing repo-wide `--check` breakage that had\ngone unnoticed because nothing runs `provider-completeness --check` as a required gate yet.\n\nFix options: (a) declare a schemas/providers/beads catalog (Beads issue-jsonl wire shape is\nsimple and stable, so a harvested catalog may be cheap to produce), or (b) downgrade Origin.\nBEADS_ISSUE's completeness-mode maturity to \"proposed\" until schema evidence exists (mirrors how\ngrok-export and unknown-export/browser-capture are declared), with an explicit caveat matching\nthe wording style used for those origins.\n\nVerification: run `python -m devtools.provider_completeness --check` before/after; should exit 0\nwith beads-issue either \"complete\" (schema catalog added) or excluded from the accepted-blockers\nlist (maturity downgraded).","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-20T10:07:48Z","created_by":"Sinity","updated_at":"2026-07-20T10:07:48Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-1ldl","title":"Stale 'archive-wide fallback is expensive' mutation assumption in action/multi-aggregate VM-step regression tests","description":"Pre-existing failures (confirmed on origin/master, unrelated to any of the z9gh.2/z9gh.3 execution-residual work in fix/query/z9gh-execution-residuals): tests/unit/storage/test_archive_tiers_archive.py::test_exact_session_action_count_bounds_pairing_before_global_ranking and tests/unit/archive/query/test_execution_control.py::test_exact_session_multi_aggregate_work_is_not_amplified_by_irrelevant_growth both monkeypatch _action_relation_for_query to force a fallback to the plain 'actions' compatibility view (simulating pre-z9gh.2 global-first behavior) and assert the resulting query costs \u003e=50000 SQLite VM steps as an anti-vacuity control. Since PR #3018 (z9gh.2) replaced the old windowed-CTE 'actions' view with one backed by the small, indexed, pre-materialized action_pairs table, that fallback is no longer expensive at these tests' data scale (measured: 0 and 400 VM steps respectively) -- the mutation no longer reproduces a meaningfully different/expensive path, so the anti-vacuity check is vacuous. Also noted in the same run: test_api_query_units_routes_through_execution_control and test_api_multi_aggregate_receipt_reports_real_work_selection_and_delivery fail identically on unmodified master with an unrelated 'query_units' vs 'api.query_units' call-log naming mismatch -- separate stale assertion, same file. Fix: either raise the mutation to something still meaningfully expensive at this data scale (e.g. force a full block_type scan directly, or scale up the noise-session count) or lower/remove the now-invalid \u003e=50000 threshold and replace with a plan-shape assertion (EQP-based, as done in the new test_bounded_action_relation_plans_session_index_not_archive_wide_tool_scan). Discovered while implementing the z9gh.2 F-006/F-007 session-alias EQP fix.","status":"open","priority":2,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-20T09:39:21Z","created_by":"Sinity","updated_at":"2026-07-20T09:39:21Z","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/polylogue/sources/revision_backfill.py b/polylogue/sources/revision_backfill.py index 58a7538dfd..212f584d5f 100644 --- a/polylogue/sources/revision_backfill.py +++ b/polylogue/sources/revision_backfill.py @@ -784,6 +784,18 @@ def commit_replay_unit() -> None: retained_bytes = 0 candidate_raw_ids = set(archive.raw_membership_rebuild_raw_ids(logical_key)) candidate_raw_ids.update(membership_candidates.get(logical_key, ())) + # Cohort absorption: candidate selection is page-dependent, so a + # head written by an EARLIER page's membership cohort for this key + # may not be in this page's candidate set -- membership replay + # would then refuse to retire an "unrelated" quarantined head and + # kill the walk. Absorb the current quarantined head raw into the + # cohort so the real prefix classifier ranks it against the new + # members instead of any scalar comparison. Chain-governed + # (non-quarantined) heads are deliberately NOT absorbed -- + # apply_raw_membership_classification yields to those. + head_raw_id = archive.raw_revision_head_raw_id(logical_key) + if head_raw_id is not None and archive._raw_revision_authority(head_raw_id) == "quarantined": + candidate_raw_ids.add(head_raw_id) for raw_id in sorted(candidate_raw_ids): sessions, payload_bytes = spill.for_raw(archive, raw_id) for session in sessions: