diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index c529612d69..45d2170d39 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1462,7 +1462,7 @@ {"_type": "issue", "id": "polylogue-yeq.2", "title": "Mine semantic contradictions and provider construct negative space", "description": "Find semantic failures without assuming which feature is broken. One bounded corpus pass should test relationships that ought to agree and enumerate common raw provider constructs that disappear, default, lose provenance, or never become queryable/renderable. This generalizes the dogfood discoveries that all Codex titles were UUIDs, nested child actions vanished, exact usage disagreed with profiles, and freshness claims ignored excluded sources.", "design": "Predeclare invariant queries such as accepted-head vs indexed hash, profile vs exact usage, titles vs authored material, failure blocks vs actions, logical vs physical lineage counts, freshness vs acquisition/materialization frontiers, and numeric zero vs absent evidence. Report denominator and contradiction classes stratified by Origin, artifact/capture route, parser/materializer version, age, and size. Separately derive a construct-flow matrix from OriginSpec/raw shape census: raw path/event -> acquired artifact -> parser field -> normalized relation -> query predicate/unit -> public projection/rendering, including unknown/opaque fields and provider-nearly-always-null normalized fields. Resolve representative rows to stable evidence refs; intentional absence needs explicit authority.", "acceptance_criteria": "1. A reproducible corpus artifact publishes every invariant, population/denominator, strata, contradiction count, representative refs, versions, and blind spots; zero contradictions is a justified confidence result, not silent omission. 2. The construct-flow matrix covers every executable OriginSpec and the top-frequency unknown/opaque raw shapes; each common construct is classified preserved, normalized, queryable, provenance-marked, rendered, intentionally unsupported, or a gap. 3. Seeded disagreement and dropped-construct mutations are caught through production readers/parsers, not a replica validator. 4. Every surviving class reconciles to an existing invariant owner or one new mechanism Bead; provider-specific symptoms do not become parallel registries. 5. Bounded execution, privacy-safe samples, exact rerun commands, and resource measurements are recorded.", "notes": "Active-set expansion 2026-07-15: admitted as independent safety, semantic, and query-law falsification lanes. They remain proof mechanisms, not substitutes for domain implementation.", "status": "open", "priority": 1, "issue_type": "task", "owner": "ezo.dev@gmail.com", "created_at": "2026-07-15T18:02:19Z", "created_by": "Sinity", "updated_at": "2026-07-31T22:35:46Z", "metadata": {"frontier": "active", "frontier_program_ref": "polylogue-88jp"}, "labels": ["area:audit", "area:sources", "area:verification", "horizon:frontier"], "dependencies": [{"issue_id": "polylogue-yeq.2", "depends_on_id": "polylogue-2qx.1", "type": "relates-to", "created_at": "2026-07-31T14:40:08Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.2", "depends_on_id": "polylogue-9e5.31", "type": "relates-to", "created_at": "2026-07-31T14:40:08Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.2", "depends_on_id": "polylogue-cuxz", "type": "relates-to", "created_at": "2026-07-31T14:40:08Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.2", "depends_on_id": "polylogue-yeq", "type": "parent-child", "created_at": "2026-07-15T20:02:19Z", "created_by": "Sinity", "metadata": "{}"}], "dependency_count": 0, "dependent_count": 1, "comment_count": 0} {"_type": "issue", "id": "polylogue-yeq.3", "title": "Prove query laws, cross-surface parity, and adversarial scale bounds", "description": "Generalize the original yeq metamorphic DSL, daemon chaos, and reference walks into one query-contract differential. A canonical selection/projection must retain identity, ordering, completeness, null/unknown/freshness semantics, continuation progress, and cancellation across CLI, Python, HTTP, and MCP, including p50/p95/max live shapes. This is broader than the incident-specific terminal replay but reuses its query transaction and receipts.", "design": "Generate bounded query plans from executable declarations. Laws include declared predicate commutativity, page-concatenate equals unpaged logical membership, LIMIT monotonicity, grouped counts sum to the matching-grain population, equivalent structured/DSL plans, exact-ref canonicalization, and ref list-to-detail closure. Execute semantic differentials across surfaces and compare selections, stable order, pages/totals, evidence/world refs, types, errors, and refinements. Mine live size/selectivity/family/tool-id/result-lag distributions for p50/p95/max/pathological fixtures. Emit 1xc.14 WorkloadReceipts for rows visited, wall/CPU, process/cgroup RSS/PSS anon/cache/swap, temp/I/O/response bytes, cancellation/progress, and cleanup; compare expensive routes with the cheapest correct primitive. Use one bounded reader over a reflink snapshot.", "acceptance_criteria": "1. Generated metamorphic laws and cross-surface differentials cover every declared query unit/stage/read projection and every list-emitted ref family, with explicit exemptions and semantic\u2014not byte-format\u2014comparison. 2. Page concatenation enumerates each logical member exactly once; continuation is progressing/replayable; cancellation halts server work; unknown/error/coverage facts agree across surfaces. 3. Fixtures derive from recorded live distributions and include duplicate/missing/late tool results, wide/deep lineage, active growth, large payloads, low/high selectivity, and the 2026-07-15 mandate incident. A serialized workload census on one reflink archive copy captures EQP scans/temp B-trees, rows visited, wall/CPU, RSS/swap/temp I/O, and response bytes for every declared query family, including coordinator-scoped actions/delegations and tool:Workflow. 4. A deliberately broken predicate pushdown, continuation state, public type, and ref route each fail the production harness. 5. Budgets from SLO owners are enforced with exact resource receipts; every unexpected full scan/materialization is classified or linked to an invariant owner, and expensive routes are compared with the cheapest correct primitive. 6. The census uses one bounded reader and never parallel dbstat/EQP walks or a mutable live database.", "notes": "2026-07-15 portfolio convergence: absorbs the executable workload/EQP half of polylogue-20d.7. The former one-shot sweep becomes a permanent query differential fixture, including the known coordinator/delegation/tool:Workflow incident and the one-reader reflink safety constraint.\nActive-set expansion 2026-07-15: admitted as independent safety, semantic, and query-law falsification lanes. They remain proof mechanisms, not substitutes for domain implementation.\nGPT Pro testdiet-01/r02 admission (2026-07-17): merged as PR #3019 / d42cc1497ee91fded8c46313a46e18733f9084ee. Added a test-owned native Codex wire manifest that crosses production ingest, DSL parse/lower, canonical action relation, repository/terminal execution, root CLI read, and public delete preview/apply. It proves a five-action duplicate/missing/orphan population, exact is_error partition 2/2/1, stable pages, selected-session deletion, output-only decoy survival, and rejection of the historical naive same-session/tool-id join (7 rows). Verification: focused query compatibility set 211 passed; Ruff, strict Mypy, and devtools verify --quick passed. This is a strong query-law seed, not closure of the broad cross-surface/cancellation/receipt census AC.\n2026-07-17 testdiet-01/r01 reconciliation: current master still dropped compiled boolean_predicate only on the final selector-only root CLI list_summaries route. The package production hunk applied cleanly; a minimal repair now forwards the existing typed filter_kwargs map and adds a real native-Codex corpus CLI law. The law returns exactly the two selected sessions and their total; removing the map returns every session. Focused 86-test and quick-gate evidence will be recorded with the PR.\n2026-07-17 testdiet-01/r01 admitted and merged: PR #3022 / d1c08af640a07b27c3fb04185e34f4fda2f814ec. The final selector-only root list route now forwards the established filter_kwargs map, preserving boolean_predicate. Regression uses native Codex provider-wire facts through ingest and public Click root execution: selected membership and total are exact; deleting the forwarding broadens to every session. Verification: devtools test tests/unit/cli/test_query_composition_laws.py tests/unit/cli/test_query_exec_laws.py (86 passed); devtools verify --quick (16/16). This closes only this r01 candidate; yeq.3 remains open for its declared cross-surface/cancellation/receipt census.\n2026-07-17 paired raw-package audit: testdiet-01 r01 contained the root CLI boolean_predicate forwarding repair now merged by PR #3022 / d1c08af; r02 supplied the native-Codex cardinality survivor merged by PR #3019 / d42cc149. No additional r01 implementation is to be replayed. Their different contributions are retained in the campaign receipts/index; broad cross-surface/cancellation scope remains open.\n2026-07-17: Test Diet 07 public session-profile fact parity was reconciled and merged in PR #3044 / 1d3145afa. Repository, fa\u00e7ade, CLI, and daemon now share provenance under a real-route survivor. Broad cross-surface parity remains open.\nWarroom sweep It.17: claiming session closed; partial landed via testdiet-01 admission (#3019 action cardinality composition test, #3022 boolean predicate fix, #3023 admission record). Cross-surface parity + adversarial scale bounds remain. Reset to open.\nVERDICT: PARTIAL \u2014 Substantial real progress landed (testdiet-01 boolean_predicate fix #3022, action-cardinality composition test #3019, session-profile parity #3044) but the bead's own last note (Warroom It.17) explicitly resets status to open: 'Cross-surface parity + adversarial scale bounds remain.' The core AC (generated metamorphic laws + CLI/Python/HTTP/MCP semantic differential + p50/p95/max workload receipts + adversarial scale bounds) is not built as one coherent harness \u2014 only isolated point-fixes/tests exist so far. \u2014 evidence: bd show polylogue-yeq.3 --json (status=open, last note dated 2026-07-17 explicitly says remaining scope; dependency 1xc.14 (WorkloadReceipts) also still open per its own notes).", "status": "open", "priority": 1, "issue_type": "task", "assignee": "Sinity", "owner": "ezo.dev@gmail.com", "created_at": "2026-07-15T18:02:20Z", "created_by": "Sinity", "updated_at": "2026-07-31T22:35:46Z", "started_at": "2026-07-17T12:28:28Z", "metadata": {"frontier": "active", "frontier_program_ref": "polylogue-88jp"}, "labels": ["area:mcp", "area:query", "area:test", "area:verification", "horizon:frontier"], "dependencies": [{"issue_id": "polylogue-yeq.3", "depends_on_id": "polylogue-1xc.14", "type": "blocks", "created_at": "2026-07-15T20:45:46Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.3", "depends_on_id": "polylogue-20d.7", "type": "supersedes", "created_at": "2026-07-15T20:28:29Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.3", "depends_on_id": "polylogue-t67b", "type": "relates-to", "created_at": "2026-07-31T14:40:08Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.3", "depends_on_id": "polylogue-t8t", "type": "relates-to", "created_at": "2026-07-31T14:40:08Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.3", "depends_on_id": "polylogue-yeq", "type": "parent-child", "created_at": "2026-07-15T20:02:20Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.3", "depends_on_id": "polylogue-z9gh.1", "type": "relates-to", "created_at": "2026-07-31T14:40:08Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.3", "depends_on_id": "polylogue-z9gh.7", "type": "relates-to", "created_at": "2026-07-31T14:40:08Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.3", "depends_on_id": "polylogue-z9gh.9.1", "type": "relates-to", "created_at": "2026-07-31T14:40:08Z", "created_by": "Sinity", "metadata": "{}"}], "dependency_count": 1, "dependent_count": 1, "comment_count": 0} {"_type": "issue", "id": "polylogue-yeq.4", "title": "Evaluate cold comprehension, accessibility, and comparative operator value", "description": "After evidence truth and query operability are proven, test whether a cold operator/model can discover and correctly use Polylogue, understand uncertainty, recover from errors, and outperform raw history tools. Expert dogfood and visual design are insufficient because they hide learned vocabulary/workarounds. Accessibility and calibrated comprehension are product correctness for a continuity archive.", "design": "Use a fixed known-answer task set: find work from a phrase; determine whether a file changed; reconstruct failure and demonstrated repair; resume unresolved work; explain a usage number; diagnose stale/partial evidence. Run CLI, MCP/model, and web independently with no command hints. Record time-to-first-correct-answer, wrong turns, payload read, help/refinements, abandonment, correctness, confidence calibration, and evidence traceability. Exercise keyboard, focus, landmarks/names/live regions, screen reader, zoom/reflow, contrast/non-color, reduced motion, long content, tables/trees, and loading/empty/error/stale/partial states. Compare with raw rg/SQLite/provider history; ablate lineage, structured outcomes, freshness, semantic titles, and compact views to measure actual value. The first unaided external adoption receipt from hg8n.1 may seed recruitment, friction hypotheses, or one compatible observation, but it cannot satisfy the fixed comparative/accessibility protocol by itself.", "acceptance_criteria": "1. Cold task protocols, known answers, participant/model context, transcripts/recordings, metrics, privacy terms, and limitations are reproducible; CLI, MCP, and web results are not averaged into one score. 2. Each task reports correctness, evidence traceability, confidence calibration, discovery/recovery friction, and accessibility blockers across healthy and degraded states. 3. Keyboard plus screen-reader/manual checks complement automated browser checks; every critical state is perceivable without color alone and recoverable without pointer use. 4. Comparative and ablation results identify which Polylogue mechanisms materially improve outcomes over raw/provider tools; unsupported value claims are withdrawn or narrowed. 5. Findings map to existing query/legibility/accessibility owners or one distinct mechanism, with no aesthetic-only substitute for measured comprehension.", "notes": "Priority calibration 2026-07-15: P2 to P3. This remains part of the full project ambition, but it is a sequenced demo, experiment, governed analytic extension, or evaluation layer rather than a present failure of archive truth, bounded queryability, durability, or source fidelity. Priority is urgency, not deletion or scope reduction; horizon is unchanged.", "status": "open", "priority": 3, "issue_type": "task", "owner": "ezo.dev@gmail.com", "created_at": "2026-07-15T18:02:21Z", "created_by": "Sinity", "updated_at": "2026-07-31T22:35:46Z", "labels": ["area:legibility", "area:test", "area:web", "horizon:mid"], "dependencies": [{"issue_id": "polylogue-yeq.4", "depends_on_id": "polylogue-hg8n.1", "type": "relates-to", "created_at": "2026-07-15T20:43:44Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.4", "depends_on_id": "polylogue-yeq", "type": "parent-child", "created_at": "2026-07-15T20:02:21Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.4", "depends_on_id": "polylogue-yeq.1", "type": "blocks", "created_at": "2026-07-15T20:03:37Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.4", "depends_on_id": "polylogue-yeq.2", "type": "blocks", "created_at": "2026-07-15T20:03:38Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.4", "depends_on_id": "polylogue-yeq.3", "type": "blocks", "created_at": "2026-07-15T20:03:38Z", "created_by": "Sinity", "metadata": "{}"}, {"issue_id": "polylogue-yeq.4", "depends_on_id": "polylogue-z9gh.7", "type": "blocks", "created_at": "2026-07-15T20:03:39Z", "created_by": "Sinity", "metadata": "{}"}], "dependency_count": 4, "dependent_count": 0, "comment_count": 0} -{"_type": "issue", "id": "polylogue-ygfwa", "title": "Determine if doctor --repair --target session_insights mutate path is redundant with daemon convergence", "description": "Found while auditing ops maintenance/doctor manual repair surfaces against\nthe automagic-invariants doctrine (polylogue-cfvvt).\n\nEvidence found: session_insights staleness is already fully covered by two\nindependent automatic daemon mechanisms, not just one:\n\n1. Per-ingest ConvergenceStage (polylogue/daemon/convergence_stages.py\n ~line 478-670): check/execute + check_many/execute_many +\n check_sessions/execute_sessions variants that call\n rebuild_session_insights_sync directly, including hot-file quiet-window\n deferral for still-appending sessions.\n2. A dedicated periodic convergence_debt retry loop\n (polylogue/daemon/cli.py:_periodic_convergence_check ->\n _retry_convergence_debt_once -> _drain_convergence_debt_once), which\n runs on its own interval and re-drives make_default_convergence_stages\n (including the insights stage) against any session_id/source_path debt\n the per-ingest path deferred.\n\nThis is structurally the same shape PR #3286 used to justify removing\nembedding-orphan-reconcile's --yes apply path and adding\nperiodic_blob_gc_check for blob-gc (also removed in the cfvvt pass,\npolylogue-cfvvt). session_insights looks like a strong redundancy\ncandidate by the same doctrine.\n\nWhy this was NOT deleted directly in the cfvvt pass instead of filing this\nfollow-up: unlike blob-gc (one CLI file, no other surface coupling),\nsession_insights repair is threaded through more surface area that needs\ncareful handling before a safe removal:\n\n- storage/repair.py: repair_session_insights/preview_session_insights are\n wired into REPAIR_HANDLERS, the doctor --repair umbrella (SAFE_REPAIR_TARGETS),\n the maintenance planner/replay executor, and archive-debt status reporting.\n- MaintenanceTargetSpec(session_insights) has include_preview_when_ready=True,\n meaning even a \"ready\" archive still surfaces this target's preview --\n need to confirm read-only preview survives untouched (as with\n embedding-orphan-reconcile) while only the mutate/apply path is removed.\n- Need to confirm no daemon-less workflow (tests, devtools scenarios,\n cloud lane, single-shot `polylogue import` without `polylogued run`)\n depends on doctor's mutate path as its only route to fresh insights --\n demo/seed.py calls rebuild_session_insights_sync directly (bypassing\n doctor), which is a good sign but not a full survey of every caller.\n- `doctor --repair` (no --target) iterates every REPAIR-mode target\n (SAFE_REPAIR_TARGETS = session_insights + message_type_backfill); if\n session_insights becomes read-only-only, the umbrella's remaining\n effective mutate scope shrinks to message_type_backfill alone, which the\n CLI help text and MaintenanceTargetMode modeling should reflect honestly\n rather than silently.\n\nmessage_type_backfill (the other REPAIR-mode target) was investigated in\nthe same pass and is NOT a redundancy candidate: it's finite one-time\nlegacy-row remediation with no recurring daemon condition (new-ingest rows\nare classified correctly since PR #836/#944) -- see the docstring added to\npolylogue/storage/message_type_backfill.py in the cfvvt PR.\n\nAcceptance criteria:\n- Decide DELETE (demote mutate path to read-only, mirroring\n embedding-orphan-reconcile) or KEEP-WITH-REASON for session_insights doctor\n repair, backed by a full-repo survey of callers/tests/scenarios that rely\n on its mutate behavior.\n- If DELETE: remove the --yes-equivalent mutate path from\n repair_session_insights while preserving preview_session_insights,\n update REPAIR_HANDLERS/SAFE_REPAIR_TARGETS wiring and doctor CLI help\n text/tests accordingly, run devtools verify.\n- If KEEP-WITH-REASON: document the specific structural reason (e.g. a gap\n in the periodic retry loop's coverage, or a genuine daemon-less use case)\n inline in repair_session_insights's docstring.", "status": "open", "priority": 2, "issue_type": "task", "owner": "ezo.dev@gmail.com", "created_at": "2026-08-02T16:33:22Z", "created_by": "Sinity", "updated_at": "2026-08-02T16:33:22Z", "dependency_count": 0, "dependent_count": 0, "comment_count": 0} +{"_type": "issue", "id": "polylogue-ygfwa", "title": "Determine if doctor --repair --target session_insights mutate path is redundant with daemon convergence", "description": "Found while auditing ops maintenance/doctor manual repair surfaces against\nthe automagic-invariants doctrine (polylogue-cfvvt).\n\nEvidence found: session_insights staleness is already fully covered by two\nindependent automatic daemon mechanisms, not just one:\n\n1. Per-ingest ConvergenceStage (polylogue/daemon/convergence_stages.py\n ~line 478-670): check/execute + check_many/execute_many +\n check_sessions/execute_sessions variants that call\n rebuild_session_insights_sync directly, including hot-file quiet-window\n deferral for still-appending sessions.\n2. A dedicated periodic convergence_debt retry loop\n (polylogue/daemon/cli.py:_periodic_convergence_check ->\n _retry_convergence_debt_once -> _drain_convergence_debt_once), which\n runs on its own interval and re-drives make_default_convergence_stages\n (including the insights stage) against any session_id/source_path debt\n the per-ingest path deferred.\n\nThis is structurally the same shape PR #3286 used to justify removing\nembedding-orphan-reconcile's --yes apply path and adding\nperiodic_blob_gc_check for blob-gc (also removed in the cfvvt pass,\npolylogue-cfvvt). session_insights looks like a strong redundancy\ncandidate by the same doctrine.\n\nWhy this was NOT deleted directly in the cfvvt pass instead of filing this\nfollow-up: unlike blob-gc (one CLI file, no other surface coupling),\nsession_insights repair is threaded through more surface area that needs\ncareful handling before a safe removal:\n\n- storage/repair.py: repair_session_insights/preview_session_insights are\n wired into REPAIR_HANDLERS, the doctor --repair umbrella (SAFE_REPAIR_TARGETS),\n the maintenance planner/replay executor, and archive-debt status reporting.\n- MaintenanceTargetSpec(session_insights) has include_preview_when_ready=True,\n meaning even a \"ready\" archive still surfaces this target's preview --\n need to confirm read-only preview survives untouched (as with\n embedding-orphan-reconcile) while only the mutate/apply path is removed.\n- Need to confirm no daemon-less workflow (tests, devtools scenarios,\n cloud lane, single-shot `polylogue import` without `polylogued run`)\n depends on doctor's mutate path as its only route to fresh insights --\n demo/seed.py calls rebuild_session_insights_sync directly (bypassing\n doctor), which is a good sign but not a full survey of every caller.\n- `doctor --repair` (no --target) iterates every REPAIR-mode target\n (SAFE_REPAIR_TARGETS = session_insights + message_type_backfill); if\n session_insights becomes read-only-only, the umbrella's remaining\n effective mutate scope shrinks to message_type_backfill alone, which the\n CLI help text and MaintenanceTargetMode modeling should reflect honestly\n rather than silently.\n\nmessage_type_backfill (the other REPAIR-mode target) was investigated in\nthe same pass and is NOT a redundancy candidate: it's finite one-time\nlegacy-row remediation with no recurring daemon condition (new-ingest rows\nare classified correctly since PR #836/#944) -- see the docstring added to\npolylogue/storage/message_type_backfill.py in the cfvvt PR.\n\nAcceptance criteria:\n- Decide DELETE (demote mutate path to read-only, mirroring\n embedding-orphan-reconcile) or KEEP-WITH-REASON for session_insights doctor\n repair, backed by a full-repo survey of callers/tests/scenarios that rely\n on its mutate behavior.\n- If DELETE: remove the --yes-equivalent mutate path from\n repair_session_insights while preserving preview_session_insights,\n update REPAIR_HANDLERS/SAFE_REPAIR_TARGETS wiring and doctor CLI help\n text/tests accordingly, run devtools verify.\n- If KEEP-WITH-REASON: document the specific structural reason (e.g. a gap\n in the periodic retry loop's coverage, or a genuine daemon-less use case)\n inline in repair_session_insights's docstring.", "notes": "DISPOSITION: KEEP-WITH-REASON.\n\nIndependently verified (not just trusting cfvvt's audit summary). Traced both automatic mechanisms cfvvt found, plus the underlying repair function's full call graph:\n\n1. make_insights_stage (polylogue/daemon/convergence_stages.py:475) -- both the plain-index-db branch and the multi-tier _archive_insights_check/_archive_insights_execute[_many/_sessions] branch -- only ever call rebuild_session_insights_sync (per-session profile/work_events/phases rebuild).\n2. The periodic convergence_debt retry loop (_periodic_convergence_check -> _retry_convergence_debt_once -> _drain_convergence_debt_once, daemon/cli.py) only re-drives make_default_convergence_stages(); no separate aggregate-refresh call.\n\nNeither automatic mechanism ever calls refresh_session_insight_aggregates_sync (polylogue/storage/insights/session/rebuild.py:1619), the archive-wide (non-per-session) refresh of thread materialization (threads/thread_sessions), tag rollups (session_tag_rollups), and provider-day aggregates. Grepped the whole tree: polylogue/storage/repair.py's repair_session_insights is the ONLY caller of refresh_session_insight_aggregates_sync anywhere in the codebase. It calls that refresh whenever _session_insight_aggregate_debt_count(status) > 0, i.e. missing_thread_materialization_count / stale_thread_count / orphan_thread_count / stale_tag_rollup_count / stale_day_summary_count are nonzero -- real, live-checked fields feeding threads_ready/tag_rollups_ready in storage/insights/session/status.py, not dead code.\n\nReal-world trigger: a SESSION_INSIGHT_MATERIALIZER_VERSION bump (or anything else that stales thread/tag-rollup aggregates archive-wide) leaves that debt permanently stuck -- the daemon has no automatic route to clear it. Only the manual doctor --repair --target session_insights path (or an equivalent direct call to repair_session_insights) drains it.\n\nSecond, independent reason repair_session_insights can't be deleted even for the narrower \"just the doctor CLI's active-archive invocation\" framing: the SAME function (with archive_root_override + owned_inactive_generation) is called directly by maintenance/rebuild_index.py:953 as the terminal materialization stage when building a brand-new INACTIVE generation before promotion (also used by sharded_rebuild.py/revision_backfill.py through ArchiveStore.open_owned_inactive_generation). The daemon's convergence mechanisms only ever touch the live/active generation's index.db -- they cannot reach an inactive generation under construction. This mirrors the polylogue-rpuqn (rebuild-index) finding: the daemon path is not actually equivalent once you look past fixture-scale/happy-path usage.\n\nmessage_type_backfill (the other SAFE_REPAIR_TARGETS member) was already correctly ruled NOT a redundancy candidate in the cfvvt pass (finite one-time legacy backfill, no recurring daemon condition) -- unchanged by this investigation.\n\nAction taken: documented this exact gap inline in repair_session_insights's docstring (polylogue/storage/repair.py) rather than leaving it to be re-derived on the next redundancy audit. No functional/behavioral change -- REPAIR_HANDLERS/SAFE_REPAIR_TARGETS/planner/replay/archive-debt-status wiring is untouched because the manual mutate path is genuinely load-bearing.\n\nPR: https://github.com/Sinity/polylogue/pull/3579 (docs-only, devtools verify --quick exit 0, all 19 checks pass)\n\nFollow-up NOT filed: giving thread/tag-rollup/day-summary aggregate staleness its own automatic daemon convergence stage would close this gap for good, but that's a separate feature-scoped change with its own design tradeoffs (batching/cost of an archive-wide DELETE+rebuild on every daemon cycle vs. current on-demand-only refresh) -- out of scope for this audit-follow-up bead. Not filing a bead for it now per no-completeness-check-theater guidance; revisit if the manual repair path is ever observed going stale in practice.", "status": "closed", "priority": 2, "issue_type": "task", "owner": "ezo.dev@gmail.com", "created_at": "2026-08-02T16:33:22Z", "created_by": "Sinity", "updated_at": "2026-08-02T19:31:55Z", "closed_at": "2026-08-02T19:31:55Z", "close_reason": "Investigated independently: KEEP-WITH-REASON. Daemon automatic mechanisms (per-ingest ConvergenceStage + periodic convergence_debt retry) only ever call rebuild_session_insights_sync (per-session profile/work_events/phases); neither ever calls refresh_session_insight_aggregates_sync (archive-wide thread/tag-rollup/day-summary aggregate refresh), which repair_session_insights is the sole caller of. Also reused directly by rebuild_index.py for inactive-generation materialization, unreachable by the daemon. Documented in repair_session_insights docstring, PR #3579 (docs-only).", "dependency_count": 0, "dependent_count": 0, "comment_count": 0} {"_type": "issue", "id": "polylogue-yhgc", "title": "Wire reported_cost_usd through archive/query/ session hydration (query-pipeline gap)", "description": "polylogue-gt1z added sessions.reported_cost_usd (v49) and wired it through the primary session-read path (storage/sqlite/archive_tiers/write.py -> api/archive.py:_archive_session_to_session -> Session.reported_cost_usd -> pricing.py:_session_level_estimate). archive/query/archive_execution.py:_session_to_session (the query-DSL 'sessions where ... | ...' pipeline read path) builds Session objects from a *different* envelope (ArchiveSessionEnvelope via a separate query path) and was deliberately left out of polylogue-gt1z's scope (explicit AVOID: archive/query/ belongs to another lane). AC: thread reported_cost_usd through that hydration path too so query-pipeline session reads carry the same exact-cost evidence as the primary read path.", "status": "open", "priority": 3, "issue_type": "task", "owner": "ezo.dev@gmail.com", "created_at": "2026-07-31T10:54:04Z", "created_by": "Sinity", "updated_at": "2026-07-31T10:54:04Z", "dependency_count": 0, "dependent_count": 0, "comment_count": 0} {"_type": "issue", "id": "polylogue-ykhy", "title": "Extract a shared read-only SQLite open helper for devtools/", "description": "At least 18 devtools/ files hand-roll their own sqlite3.connect() call for opening the archive read-only, with no shared helper anywhere in devtools/ (confirmed: no open_archive/_open_readonly/ArchiveHandle/DevtoolsContext symbol exists). Sampled 4 concrete variants that disagree with each other: deployment_smoke.py and cost_reconciliation_probe.py use file:{path}?mode=ro; index_fast_forward.py adds timeout=30.0; archive_schema_fast_forward.py adds immutable=1. The divergence is a real risk, not just duplication - a devtools command missing immutable=1 can behave differently under concurrent daemon writes than one that has it. Surfaced during a 2026-07-16 refactoring-opportunity survey.", "design": "Add one devtools/_sqlite.py (or similar) with open_readonly(path) and open_immutable_readonly(path) helpers encoding the correct, single URI construction (decide the right default including immutable=1 and timeout by consulting the storage/ tier docs on WAL/locking behavior for read-only access during live daemon writes). Migrate the 18 call sites to use it. Do not change behavior beyond making the URI construction consistent unless a site is found to be using a genuinely wrong mode for its use case, in which case fix that specific bug separately and note it in the bead.", "acceptance_criteria": "A single shared open helper exists and is used by all 18 identified call sites; devtools test on affected files passes; any behavior change found necessary during migration (e.g. a site missing immutable=1 that needed it) is called out explicitly, not silently folded in.", "notes": "PR #3316 opened: https://github.com/Sinity/polylogue/pull/3316 (branch feature/refactor/devtools-sqlite-open-helper). Migrated all 18 identified mode=ro call sites across 14 devtools files (degraded_archive_proof.py, index_v37_fast_forward.py x2, index_fast_forward.py x3, archive_schema_fast_forward.py, cost_reconciliation_probe.py, scale_regression_probe.py, deployment_smoke.py, self_verify.py, render_demo_corpus_datasheet.py, read_package.py, dev_loop.py x2, schema_generate.py, failure_context.py, test_economics_report.py) onto the pre-existing polylogue.storage.sqlite.connection_profile.open_readonly_connection helper (already used correctly by 6 other devtools files before this change). Extended that helper with immutable: bool=False, kept True only at the two sites (index_v37_fast_forward.py, archive_schema_fast_forward.py) that already prove zero WAL/SHM/journal sidecars before opening -- verified genuinely load-bearing, not drift. Preserved index_fast_forward.py's 30s/120s timeout overrides (live-archive lock contention); dropped their now-redundant manual PRAGMA query_only=ON since the helper sets it. Unified read_package.py's timeout=5.0 (was byte-identical to the canonical default). Flagged one real behavior change in the PR body: archive_schema_fast_forward.py previously built its immutable URI via Path.as_uri() (percent-encoded); the canonical helper uses the same plain f-string URI construction as its other 30+ existing callers, so that one site loses percent-encoding for paths with reserved URI chars -- an existing risk shared by all other callers, not newly introduced. Did not touch: row_factory assignments (query ergonomics, not connection semantics), the many read-write sqlite3.connect() calls in the same files, turso_probe.py (different library), or ATTACH DATABASE statements in pipeline_probe/result.py and daemon_workload_probe.py (attach to an already-open connection, not a new connect()). Verification: ruff+mypy strict clean, devtools render all --check exit 0, devtools test green on all touched files' test modules (149+360 passed); pre-existing unrelated failures (14 in test_index_v37_fast_forward.py/test_index_fast_forward_lifecycle.py, 1 in test_status.py) confirmed identical on origin/master via git stash A/B. Not closing -- leaving for operator review/merge.", "status": "closed", "priority": 3, "issue_type": "task", "owner": "ezo.dev@gmail.com", "created_at": "2026-07-16T10:43:09Z", "created_by": "Sinity", "updated_at": "2026-07-27T10:12:02Z", "closed_at": "2026-07-27T10:12:02Z", "close_reason": "Fixed and merged via PR #3316. Unified 18 divergent read-only sqlite3.connect() call sites across 14 devtools/ files onto the already-existing canonical helper open_readonly_connection (polylogue/storage/sqlite/connection_profile.py), which 6 other devtools files already used correctly. Investigated each sampled divergence honestly: both immutable=1 sites (index_v37_fast_forward.py, archive_schema_fast_forward.py) genuinely check for zero WAL/SHM/journal sidecars first - load-bearing, so extended the helper with an immutable: bool=False parameter rather than erasing the distinction. index_fast_forward.py's 30s/120s timeout overrides read a potentially-live archive under daemon lock contention - kept as deliberate per-caller overrides; dropped now-redundant manual PRAGMA query_only=ON since the helper already sets it. read_package.py's timeout=5.0 was byte-identical to the canonical default - collapsed as accidental drift. One honest behavior note flagged in the PR: archive_schema_fast_forward.py's Path.as_uri() percent-encoding is replaced by the helper's plain f-string URI construction (same as 30+ other existing callers already do) - an existing risk shared repo-wide, not newly introduced by this change. mypy --strict clean (1250 files), ruff clean, devtools render all --check clean, devtools test green on all touched files (149+360 passed), pre-existing unrelated failures confirmed identical via git stash A/B against master. Personally reviewed the full diff (CodeRabbit rate-limited) before merging.", "dependency_count": 0, "dependent_count": 0, "comment_count": 0} {"_type": "issue", "id": "polylogue-yky4", "title": "Backfill checkpoint mirror: quota re-check on overwrite growth + orphan GC", "description": "Reviewer of PR #2871 (polylogue-06zm) found two non-blocking design gaps in the receiver-side backfill-checkpoint mirror (polylogue/browser_capture/receiver.py write_backfill_checkpoint):\n\n1. Quota not re-checked on same-instance overwrite growth: write_backfill_checkpoint only runs _check_spool_quota when the target file does not yet exist (a NEW instance id). An existing instance can grow its checkpoint file arbitrarily large on repeated overwrites (last-write-wins) without ever being quota-checked again, since BACKFILL_CHECKPOINT_MAX_BYTES is enforced only at new-file creation time, not on total-bytes-on-disk after an overwrite.\n2. No GC for orphaned per-instance checkpoints: if a browser-profile reseed mints a new extension_instance_id (because chrome.storage.local, which stores the instance id itself, is wiped along with IndexedDB), the previous instance's mirrored checkpoint file on the receiver becomes permanently orphaned -- there is no expiry, TTL sweep, or manual reconciliation path to reclaim that spool space or let an operator adopt the orphaned checkpoint under a new instance id.", "notes": "Filed 2026-07-14 during the polylogue-06zm fix round for PR #2871 (branch feature/browser-ext/checkpoint-mirror-and-message-layer), addressing an independent reviewer's two minor/non-blocking findings that were correctly left out of that PR's scope (the PR's own major finding -- silent-coercion checkpoint validation hole -- was fixed in the same round). Not fixed here because both require an actual design decision (bytes-on-overwrite quota re-check semantics vs total remains 200MiB soft cap; GC/TTL/adoption policy for orphaned per-instance checkpoints) rather than a mechanical one-line fix.", "status": "closed", "priority": 3, "issue_type": "task", "owner": "ezo.dev@gmail.com", "created_at": "2026-07-14T01:27:04Z", "created_by": "Sinity", "updated_at": "2026-07-14T23:12:44Z", "closed_at": "2026-07-14T23:12:44Z", "dependencies": [{"issue_id": "polylogue-yky4", "depends_on_id": "polylogue-06zm", "type": "supersedes", "created_at": "2026-07-15T01:12:43Z", "created_by": "Sinity", "metadata": "{}"}], "dependency_count": 0, "dependent_count": 0, "comment_count": 0} diff --git a/polylogue/storage/repair.py b/polylogue/storage/repair.py index 40810b5b67..b6281ac183 100644 --- a/polylogue/storage/repair.py +++ b/polylogue/storage/repair.py @@ -6063,6 +6063,35 @@ def repair_session_insights( When ``session_ids`` is given, the rebuild is narrowed to that set instead of touching the full archive — used by the maintenance planner to honor :class:`MaintenanceScopeFilter.session_ids`. + + KEEP-WITH-REASON (polylogue-ygfwa): this mutate path is *not* + fully redundant with the daemon's automatic convergence mechanisms + (the per-ingest ``make_insights_stage`` ``ConvergenceStage`` and the + periodic ``convergence_debt`` retry loop in ``daemon/cli.py``). Both + automatic mechanisms only ever call ``rebuild_session_insights_sync`` + (per-session profile/work_events/phases). Neither one ever calls + ``refresh_session_insight_aggregates_sync`` — the archive-wide, + non-per-session-scoped refresh of thread materialization + (``threads``/``thread_sessions``), tag rollups + (``session_tag_rollups``), and provider-day aggregates. This + function is the *only* caller of + ``refresh_session_insight_aggregates_sync`` in the codebase (verified + by grep across ``daemon/`` and the rest of the tree, 2026-08-02): it + runs that refresh whenever ``_session_insight_aggregate_debt_count`` + (``missing_thread_materialization_count``, ``stale_thread_count``, + ``orphan_thread_count``, ``stale_tag_rollup_count``, + ``stale_day_summary_count``) is nonzero. So a bump to + ``SESSION_INSIGHT_MATERIALIZER_VERSION`` (or any other event that + stales thread/tag-rollup aggregates archive-wide) leaves those + aggregates stale forever unless something calls this manual repair + path — the daemon has no automatic route to clear that debt. This + function is also reused directly (not via the doctor CLI) by + ``maintenance/rebuild_index.py``'s terminal stage to materialize + insights for a freshly built *inactive* generation before promotion, + a scenario the daemon (which only ever touches the live/active + generation) cannot reach at all. Do not remove the mutate path + without first giving thread/tag-rollup/day-summary aggregate + staleness its own automatic convergence mechanism. """ from polylogue.api.archive import _rebuild_archive_session_insights from polylogue.paths import archive_root as _resolve_archive_root