daemon: in-memory TEMP page corruption suspect and ~1 GB unlinked SQLite temp files

Follow-up from the #4094 root-cause work (fix: delete the projection writer's TEMP output-state cache, PR pending on fix/4094-projection-temp-cache).

1. Unidentified memory corruption. The #4094 corruption ("database disk image is malformed" in observation_projection_output_state) lived only in process memory: bundled rusqlite builds SQLite with SQLITE_TEMP_STORE=1, so TEMP tables sit in unlinked /var/tmp/etilqs_* files, and a copy of the live daemon's TEMP file failed PRAGMA integrity_check (Tree 2 page 8004: btreeInitPage() returns error code 11) on pages that filefrag shows were never written. Dirty TEMP pages are only ever in the page cache, so something overwrote page-cache memory. The writer is single-threaded, holds no open cursors across rollbacks, and a 26k-transaction probe of the production upsert pattern on SQLite 3.53.2 stayed clean, so SQLite usage is ruled out. Leading unproven suspect: #2773 (2026-09-30), which routes SQLite allocations to mimalloc v3 in crates/tracedecay-cli/src/process_allocator.rs and installs owner heaps through mimalloc internals. Removing the TEMP cache removes the visible victim, not the cause; a clean main-database page is reloaded from disk, so corruption elsewhere may be silent.

2. Large unlinked temp files. The live beta.88 daemon (PID 1277835) holds SQLite temp files in /var/tmp that are deleted but open, so no storage census or du of ~/.tracedecay sees them: on 2026-10-16 ~19:53 UTC, fd 89 was 974,143,785 bytes, fd 78 34 MB, fd 92 24 MB (an earlier look saw a 1.9 GB one). Identify which statements spill (large sorts, GROUP BY/ORDER BY without an index, rebuild-time TEMP tables in observation_projection/rebuild.rs), bound or remove them, and include temp spill in the #4099 footprint accounting.

Part B status. #4173 removes mimalloc owner heaps. In mimalloc 3.3.2, mi_heap_delete frees the building thread's part of the heap while that thread's pages still point at it. The builder's next free that empties such a page then segfaults in _mi_arenas_collect with a NULL tld. A test using only the public mimalloc API crashes 5 of 5, and the regression test crashes 9 of 10 on master. The graph runtime deletes the engine heap while state the engine built is still alive, so the daemon can reach this. SQLite and tree-sitter stay routed to mimalloc. Every stress arm that routed SQLite to mimalloc without owner-heap churn was clean.

The corruption question is still open. A use-after-free in the allocator's page bookkeeping is a plausible way for a page-cache buffer to be overwritten. A retired page kept pointing at a deleted heap whose page counters had gone negative, which means pages were being tracked inconsistently. But no run has shown it. Every stress run that hit the bug segfaulted first, and none reported a failed integrity_check or a corrupted block. Settling it needs a run that observes a SQLite page or block overwritten while owner heaps are deleted, for example the same stress harness with the crash path avoided. With owner heaps gone, a recurrence of the #4094 symptom on a build containing #4173 would rule this cause out.

Merge pull request #4173 from ScriptedAlchemy/fix/4152-no-owner-heaps

fix(storage): deduplicate SQLite TEMP spill descriptors

fix(storage): exclude non-file SQLite TEMP fd targets

fix(projection): avoid connection-lifetime TEMP ownership sets

Focused partial fix staged (NOT on master): 04a4aa8 on held DRAFT #4776, integration/application-repairs-20261105-next. Refs #4152, not Fixes; issue remains OPEN.

The projection rebuild's clear_active_projection created two connection-local TEMP ownership sets (observation_projection_rebuild_retained_outputs and observation_projection_rebuild_cleared_outputs). It reused the SQLite writer connection and only deleted rows on later activations; these tables could remain in an unlinked etilqs_* temp DB. A disposable Linux SQLite temp_store=FILE probe established that even DROP TABLE can leave such files open and sized until the connection closes. The focused change removes these two TEMP tables, using existing indexed provenance and staged-output ownership predicates directly while preserving foreign projector and staged output protections. A new #[asupersync::test] regression checks the same writer connection's TEMP schema; the pre-fix implementation would leave the two tables visible.

Independent Python SQLite before/after execution passed five assertions across six ownership cases (sole created, shared owner, noncreated, staged, foreign-only, multiple provenance rows), showing identical surviving message/provenance state and no new TEMP tables from this function. This is NOT a Rust/Bazel, native macOS, or production verification. No Bazel workspace/toolchain is available in the current execution environment; exact-head hosted CI is not yet established.

Still unresolved: the distinct prepare_rebuild_output_activation TEMP materialization, any large query-sort spills and complete /proc/self/fd accounting, and the original in-memory page corruption cause (merged #4173 eliminated a proven mimalloc owner-heap use-after-free crash but did not establish page-corruption causality). Need measured production-statement footprint, nonzero Bazel test/CI and native macOS coverage, and recurrence/allocator evidence before #4152 can close. No merge/auto-merge performed; mixed Kev/decision #4776 stays DRAFT and held.

fix(projection): validate rebuild outputs without TEMP cache

test(projection): check durable SQLite integrity on preflight

Additional focused partial fix published as DRAFT #4802, commit 61eefab. Refs #4152 (not Fixes). Removes the other per-rebuild SQLite TEMP table (observation_projection_rebuild_preexisting_outputs) in collision preflight, replacing it with indexed direct ownership probes before any provenance is cleared. A new real registered-global-DB Rust behavioral regression checks conflict refusal, compatible foreign output acceptance, absent/uncreated output refusal, TEMP-schema absence, durable body preservation and PRAGMA integrity_check. Independent disposable SQLite 3 differential probe passed 32 ownership/body cases with indexed plan/no TEMP B-tree. Bazel Rust test / exact-head CI remain unproven; hosted CI on the new PR was queued at publication, and no Bazel workspace is mounted here. #4776 remains DRAFT/untouched (its two separate retained/cleared TEMP ownership caches remain off master). Memory page corruption root cause, other spills, native/macOS checks and verified safe reclamation remain open. No merges, no release/infra changes; #4152 stays OPEN.

test(projection): prove persisted output and TEMP pages survive rollback

test(projection): compare TEMP page allocation to preflight baseline

test(projection): stage normalized stored messages in preflight regre…

fix(projection): import normalized message helper in regression

test(storage): exercise SQLite TEMP integrity and spill release

Refs #4152 — published a further focused SQLite TEMP spill fix on existing PR #4853, now at exact verified head 2eed865, following nonforce expected-old-head commits f5358c and 2eed865. Independent review is COMMENTED, not approval.

New proven source defect: The existing two-pass INTEGER PRIMARY KEY remap in increment.rs::shift_rowid_keys, used for surviving row_blocks and clone_occurrences, can open an unlinked statement-local etilqs_* file while restoring negative keys. Stock SQLite 3.46.1 on disk, temp_store=FILE, small caches and 16,000 keys measured 1,529,300 bytes of transient unlinked allocation on the second old bulk UPDATE, even though PRAGMA temp.page_count=0 afterward. This is distinct from both the explicit clone TEMP copy and the bulk WITHOUT ROWID PK spill already repaired in the preceding #4853 commits.

Published repair: The shared rowid helper now uses bounded, key-ordered primary-key windows and exact-key updates with cancellation/deadline checkpoints and one-row guards. The original two-phase negative staging, transaction rollback and durable schemas remain. New nonvacuous Rust tests exercise both real table shapes, positive-key collision pressure, shrink/expand cases, interruption with partial writes followed by rollback, on-disk reopen/commit, independent table survival, main/TEMP integrity, and Linux live-unlinked-FD observation with the actual two-pass bulk positive control. Independent disposable SQLite differential probes passed on the source-equivalent algorithm with zero observed ephemeral spill.

Review correction: An exploratory row_chunks non-key UPDATE rewrite in the first new commit was reverted in 2eed865 after independent tests of the actual production positive-ID transform on up to 120,000 rows produced no observed spill. No unproven row_chunks "fix" remains in the proposed diff. Other issue-owner branches (#4794/#4805 etc.) were not modified.

Verification hold: Exact-head hosted Bazel CI #42723821065 is queued; neither the new Rust tests, rustfmt nor Clippy have reported a current-head pass. Local Bazel is unavailable and Cargo was not used. Do not merge or close #4152 until required checks and all original memory-corruption, sorter-bound, and projection ownership acceptance are proven on master. Keep mixed #4776 and Kev/decision-model PRs DRAFT and unmerged; release #4763 read-only. No production/infra/tags/credentials, CI weakening, delegation or schedule changes.

fix(sqlite): repair clone TEMP regression and Bazel formatting

Refs #4152 — published CI/test repair on the existing focused PR #4853. Commit 21b2fd47 advances the branch by exact expected-old-head, nonforce fast-forward from 2eed865; no other issue owners were modified.

The prior exact-head hosted CI #42723821065 failed the mandatory Bazel Rustfmt gate (ten hunks in the bounded SQLite remap source/tests). The new commit applies the precise reported formatting. Independent inspection also found a real Rust compile/test defect that the earlier formatter failure masked: the physical-spill positive control still called shift_rowid_keys with its obsolete 4-argument API. Since that helper is now bounded, calling it would also make the old bulk spill comparison vacuous. The test now runs the original two full-table composite-key UPDATE statements directly as its positive control, without changing production remap behavior.

An independent on-disk SQLite 3.46.1 execution of that exact restored SQL with 48,000 postings measured 4,739,004 live-unlinked bytes while the statement was executing, with zero temp.page_count after it returned, and preserved 48,000 shifted rows. This is SQL mechanism evidence, not an executed Rust/Bazel test pass.

Current exact-head CI #42730056362 is queued; no merge is justified. #4152 remains OPEN for original in-connection corruption causality, residual sorter bounds and held ownership-TEMP tables. Mixed #4776 and Kev/decision-model PRs remain DRAFT/unmerged; #4763 read only; no Cargo, CI weakening, production/infra/tag/credential, delegation or schedule changes.

fix(sqlite): bound cancellable row chunk carry updates

test(sqlite): independently verify chunk remap boundaries

Refs #4152 — published and reviewed a focused correction to existing PR #4853. New exact head 407e488, following source commit 45d908d. Both commits were appended with exact expected-old-head nonforce fast-forwards, without replacing concurrent code. Source/behavior review is COMMENTED, not approval.

Root cause of the new cancellation blocker: the row_chunks carry kept one bulk UPDATE of every surviving non-key document_id without any internal cancellation/deadline checkpoint. Unlike the separate composite- and INTEGER-PRIMARY-KEY spills already addressed by earlier #4853 commits, this statement has not been shown to spill FILE TEMP (the earlier 120k-row exact-statement probe observed zero unlinked bytes). Its demonstrated defect was unbounded interruption latency while the builder transaction stayed occupied.

Published repair: bounded 4,096-key chunk_id PRIMARY KEY seeks, checked mapping/one-row UPDATEs, execution-control checkpoints, and skipping unchanged values. The new on-disk SQLite Rust regression exercises 12,000 real rows and both cancellation and deadline during a partially completed remap, rollback and restart, successful mixed shrinking/expanding ranges with an independent expected-value mapping, preservation of unrelated durable data, and main/TEMP integrity checks. No schema or writer-policy changes.

Disposable independent SQL verification: the same SQL keyset plan uses SEARCH row_chunks USING PRIMARY KEY (chunk_id>?). A 4,100-row partial remap rolled back to all 12,000 originals; a successful run committed and reopened 11,987 survivor rows with integrity_check=ok. This is not a Rust/Bazel test pass. Exact-head CI #42736892284 is queued. Local Bazel is unavailable; no Cargo or weakened test gate.

Issue remains OPEN. PR #4853 is unmerged pending nonzero Bazel/Clippy/rustfmt/current-head validation. Original dirty TEMP page corruption causality and current-master active-sorter bounds remain unproven. Other projection ownership TEMP tables remain exclusively in held mixed DRAFT #4776 (must not merge, auto-merge, copy or bypass); other Kev/decision drafts, independent #4794/#4805 owner branches, and all schedules remain untouched. Release #4763 read-only; no production, infra, credentials, tags, delegation, or CI changes.

fix(sqlite): preserve literal chunk keys during bounded carry

Refs #4152 — published a current-head correctness fix to the existing focused PR #4853: 8631a6331, appended to 407e488 by exact-expected-old-head, nonforce fast-forward. Verified GitHub's PR head and branch contents at the new SHA. Review is COMMENTED, not approval to merge.

New blocker found and repaired: the newly bounded carry_row_chunks reader decoded row_chunks.chunk_id exclusively as Vec<u8> and always bound its keyset cursor as SQLite BLOB. Production stored_chunk_key stores canonical SHA-256 keys as BLOB but other valid literal chunk IDs as TEXT, so incremental carry of a shifted literal ID would fail before completing. The source now carries the exact SQLite Value (TEXT/BLOB) through ordered scans, seek bindings and checked updates, preserving stable key identity and cancellation/rollback semantics. No schema or allocator-policy change.

Nonvacuous regression updated: existing 12,000-row on-disk SQLite test now writes and asserts 6,000 TEXT and 6,000 BLOB keys in one WITHOUT ROWID index before invoking actual carry_row_chunks for mixed shrinking/expanding shifts; it still checks partial cancellation and deadline interruption, full rollback, independent expected keys, commit/reopen durability, surviving unrelated rows, and main/temp.integrity_check. The previous Vec<u8> code fails on the TEXT fixture. Fixed two minor review findings (module-level atomic imports and timeless physical-spill positive-control comment); all three inline threads received fixes/replies and are marked resolved.

Independent disposable SQLite 3.46.1 check (not Rust/Bazel): a 12,000-key disk fixture retained 6,000 of each SQLite storage class; the mixed keyset scan visited three bounded windows, performed 11,980 checked remaps after skipping replaced-range rows in the standalone probe, matched independently computed values, and rolled back to the complete original 12,000 keys with integrity_check=ok. This only supports the algorithm and storage-class mechanism; it is not a compiled or executed Rust result.

New exact-head CI #42742985758 is queued, so Bazel Rust tests, rustfmt, Clippy, Linux/Windows and native macOS for this commit are not proven; Bazel isn't available in this execution environment, and Cargo was not used. PR #4853 remains unmerged; #4152 stays OPEN. Original live in-connection page-overwrite causality and residual current-master sorter/ownership-TEMP acceptance remain unresolved. Owner-held #4794 untouched; mixed Kev/decision-model DRAFT #4776 remains held and was not copied/merged, and release #4763 remains read-only. No production/infra/tags/credentials, CI weakening, delegation or schedule change.

style(sqlite): format mixed-key carry rollback assertions

fix(sqlite): bound clone payload ordinal restoration

Refs #4152 — published a further focused SQLite TEMP spill fix to existing PR #4853: commit f835a9e2. The ref advanced from 96bc6551 with exact expected_sha, force=false; no peer branches were overwritten. Exact-head review is COMMENTED, not merge approval.

New root-cause evidence: even after prior clone-posting and rowid carry repairs, renumber_clone_payloads still bulk-updated the clone_body_payloads INTEGER PRIMARY KEY to restore negative ordinals. Stock SQLite 3.46.1 on disk with FILE TEMP, a small cache, and 16,000 reverse-rekeyed 256-byte payloads observed 5,088,100 bytes of transient open-deleted etilqs_* storage during that bulk statement. The corresponding 4,096-key scan and exact point-updates observed zero; both variants had 16,000 identical payload rows after commit/reopen and integrity_check=ok. This confirms another actual statement-local spill, not a permanent TEMP pager allocation; a post-statement PRAGMA temp.page_count=0 alone misses it.

Published source fix and regression: bound the existing negative-ordinal restoration with ordered primary-key windows, one-row update guards and cancellation/deadline checkpoints, also checking interruption during the preceding per-row staging. Added nonvacuous on-disk Rust coverage for a true 16,000-row key reversal, constraint-collision positive control, partial-restoration interruption/rollback, occurrence repointing, durable reopen, byte-exact payload survival, independent-row preservation and main/TEMP integrity. Linux subprocess regression measures live unlinked descriptors during both old bulk and new bounded SQL, requiring a positive old spill and no observed new spill. These Rust regressions are committed but not yet run successfully; no local Bazel is available and no Cargo was used.

Current-head Bazel CI #42751477632 is queued, not passed. Do not merge #4853 or close #4152 on this partial result. The original 2026-10 in-connection corruption cause and residual active-sorter/ownership TEMP acceptance remain open; related #4794 is separately owned. Kev/decision-model PRs and mixed #4776 remain DRAFT and must never merge/auto-merge or be bypassed. Release #4763 remains read-only. No CI weakening, production, infrastructure, tags, credentials, delegation, other owner task or schedule changes.

fix(sqlite): satisfy Bazel formatting and strict Clippy

style(sqlite): align progress-handler error chains with Rustfmt

Refs #4152 — published targeted Bazel CI repairs to the existing issue-specific PR #4853, now exact head 18cc1a415202df9f4870318918a68f3e5f9df3fc.

The previous head f835a9e2 exposed new, exact-source blockers in CI 42751477632: Rustfmt rejected nine hunks in the new clone payload spill regression; strict Clippy rejected the obsolete ShiftV1::sql_image method plus six ignored Connection::progress_handler Results. The latter could silently skip live FD observations if a handler failed to install. f91b68a removes the dead API, applies the exact formatter output and checks every install/clear Result; 18cc1a4 aligns chain layout with existing Rustfmt conventions. Both are single-parent expected-old-head nonforce fast-forwards preserving concurrent issue fixes.

Source/diff reviewed in commented review #5837264806. Exact-head Bazel CI 42762915407 is queued, not green. Local Bazel is absent; no Rust test pass or merge is claimed. No Cargo, weakened gates or test assertions.

Current master 2f9d6592 still has merged TEMP-only FULL vacuum policy and partial FD-census safety, but TableSizes GROUP BY and the two retained/cleared ownership TEMP tables are present; owner-held #4794 and mixed DRAFT #4776 are untouched. The original October in-connection memory page-corruption cause is unproven. #4152 stays open (Refs only), PR #4853 stays unmerged. Kev/decision drafts, read-only release #4763, production/infra and all schedules were untouched.

Refs #4152 — new exact-head CI blocker found during review on focused PR #4853, unchanged at 18cc1a415202df9f4870318918a68f3e5f9df3fc.

Hosted Bazel CI #42762915407 now proves the changed crate's //crates/tracedecay-query:unit_test PASSED (7.8s). Rustfmt/repository gates, strict Clippy, Feature gates, Dashboard, change scope and benchmark tests passed. However the Linux fleet finished 182/183 Bazel targets because //crates/tracedecay-global-db:unit_test TIMEOUT (300.9s). The 500-test log flags background_upgrade_is_atomic_at_every_cancellation_point as running over 60s; it does not prove that one test is the sole cause of the timeout. Linux fallback remained queued and Windows was in progress when checked; ordinary macOS skipped. Exact-head full green is not established, so do not merge yet.

tracedecay-global-db has no dependency on tracedecay-query, and #4853 changes only its own four query source/test paths. No source regression attributable to #4152 was shown by the failure; I have not altered the separately owned suite or weakened its deadline. Exact-head review is COMMENTED, not approval. The accepted master TEMP reclamation and indexed session lookup remain present, but original live in-connection corruption causality and residual workload sorter bounds are unproven. Keep #4152 OPEN / Refs, not Fixes. Protected DRAFT #4776, Kev/decision PRs, release #4763, peer branches, and schedules untouched; no Cargo or production/infra actions.

Navigation Menu

Repository navigation

daemon: in-memory TEMP page corruption suspect and ~1 GB unlinked SQLite temp files #4152

25 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

daemon: in-memory TEMP page corruption suspect and ~1 GB unlinked SQLite temp files #4152

Description

Activity

ScriptedAlchemy commented on Oct 17, 2026

ScriptedAlchemy commented on Nov 8, 2026

ScriptedAlchemy commented on Nov 8, 2026

25 remaining items

ScriptedAlchemy commented on Nov 9, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

ScriptedAlchemy commented on Nov 10, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions