Repository navigation
fix(sqlite): reclaim dropped TEMP pages without deleting live state - #4839
Conversation
Refs #4152 Configure FULL auto-vacuum on writer and maintenance TEMP databases before schema creation; preserve live TEMP and durable main objects. Add a real FILE-backed page reclamation, survival and integrity regression.
|
Refs #4152 Assert the normal writer connection, not only maintenance setup, enables FULL TEMP auto-vacuum before the first TEMP schema object.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Refs #4152 A failed TEMP table removal must restore all committed rows and page count, then a committed removal must shrink pages while an unrelated table lives.
ScriptedAlchemy
left a comment
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Refs #4152 — independent source/behavior review at exact head 9ca0cf1bb3461ca9fbabd07d7b2d75f1c4c5ffba.
Checked the canonical connection policy and the FILE-backed regression against merged master be2e1349. temp.auto_vacuum=FULL is applied before TEMP objects on Writer and Maintenance modes; neither main-database auto-vacuum nor temp_store is changed, avoiding the destructive TEMP reset. The test allocates 1,024 real 8 KiB TEMP rows, verifies that a rolled-back DROP restores them and the page count, then checks a committed DROP materially shrinks TEMP pages while an unrelated live TEMP row and main durable row survive; both schema integrity checks are explicit. The normal Writer open is separately asserted to install the policy. A disposable SQLite 3.46.1 FILE probe reproduced 2,182 -> 3 TEMP pages, 12,288 final unlinked-file bytes, live survivor/main data, and intact integrity; under NONE, dropping the TEMP relation retained the large allocation.
Do not treat this as Rust/Bazel acceptance or original corruption-causality proof. This policy only reclaims pages after a TEMP relation is dropped; active sorter spills, writer TEMP ownership tables held in mixed DRAFT #4776, and the 2026-10 in-memory page-overwrite cause remain independent gaps. A new head 9ca0cf1 needs an uncached nonzero //crates/tracedecay-rusqlite-runtime:unit_test run, full applicable CI, and platform verification before merging. The connection/mod.rs overlap with held #4776 is at disjoint existing file-identity code; its branch and protected merge hold are untouched. No Cargo or CI weakening.
ScriptedAlchemy
left a comment
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Refs #4152 — independent follow-up review at exact PR head fdfd93c1b030a8fb4c693b18c8bc8c7b0fad48dd.
Re-read the current root AGENTS.md (recursive master tree has no nested AGENTS.md), the full issue thread, the writer/maintenance policy, the three-file PR diff, and current master/related held draft. The latest revision now limits TEMP FULL auto-vacuum to the canonical Writer connection; Maintenance does not set it, preserving main/attached databases' independent auto_vacuum modes when an ordinary full VACUUM executes. The writer's exact-SQL regression rejects full VACUUM. The FILE-backed regression checks actual VFS TEMP bytes, rollback of a DROP, committed shrink with a separate live TEMP row, main-row durability, and both integrity checks; this exercises behavior beyond a PRAGMA value assertion.
No new source defect found in these three files. The posted native macOS/Linux counts are author-reported evidence, not tests I executed in this environment. The current hosted checks (runs 42587729252 and 42587736163) are queued at this head; neither is a verified exact-head CI pass. Do not merge until the required gates are green. The branch is currently GitHub-mergeable but blocked by checks, not source conflicts.
The fix is partial: it reclaims a dropped TEMP b-tree, not active sorter spills, the two other persistent projection ownership TEMP tables exclusive to held DRAFT #4776, or the 2026-10 memory-page overwrite mechanism. Keep Refs #4152; do not close #4152 or merge held #4776/Kev. No Cargo, delegated work, CI weakening, release or infrastructure actions.
ScriptedAlchemy
left a comment
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Refs #4152 — independent exact-head review at 6db9d115b0a3bf41e001e95d0956387dc5c721fb.
Compared this head against current master 8d6fe6791a1b9983a066f9ada990d70cb6eb451a: exactly three SQLite-runtime files remain changed. The writer-only temp.auto_vacuum=FULL policy is installed before TEMP schema creation; maintenance retains its separate durable VACUUM modes; existing tests assert live TEMP row survival, rollback, actual VFS TEMP-file truncation, main durability, and integrity. No new source defect found in that focused implementation; the current exact-head hosted job is still queued, so I make no claim of green CI or independently executed Rust tests.
New independent mechanism measurement posted on #4152: with SQLite 3.46.1, a live 40,000-row FILE-backed external sort held two unlinked file descriptors totaling 169,796,611 bytes at first row and after 100 more rows, for both temp.auto_vacuum=NONE and FULL; exhausting the cursor released them. This does not invalidate #4839's dropped-btree reclamation; it establishes that active sorter files remain a separate #4152 acceptance gap. The synthetic SQL is not claimed to occur in production. Keep Refs #4152, do not close the issue or bypass held DRAFT #4776/Kev PRs.
Current Linux and macOS verification
The earlier fleet job failed downloading both pinned x86-64 launchers from their release URLs. The branch now includes the dependency cache and downloader already merged to master. At
fdfd93c1b0, the feature diff still has three SQLite files. On that commit the complete native SQLite suite passed 435 Linux tests as an ordinary user on ext4 and 429 macOS tests. Both suites measured TEMP file reclamation, preserved both durable vacuum modes, and exercised writer exclusion. Hosted Linux and Windows verification. The independent review hold remains.Safe SQLite TEMP file reclamation (partial) — Refs #4152
Root cause and measured behavior
On a long-lived SQLite connection with
temp_store=FILE, dropping a large TEMP table releases the logical rows but by default does not truncate the already-unlinkedetilqs_*file. A 3.46.1 disposable test with 1,024 × 8 KiB rows and a second live TEMP table measuredtemp.page_countat 2,182 before DROP and still large underauto_vacuum=NONE. WithPRAGMA temp.auto_vacuum=FULLconfigured before TEMP schema creation, it shrank to 3 pages (12,288 bytes) after DROP; the separate live TEMP row remained readable, the durable main row remained intact, and bothPRAGMA main.integrity_checkandPRAGMA temp.integrity_checkreturnedok. Live/proc/self/fdconfirmed a retained unlinked TEMP file reduced to 12,288 bytes.PRAGMA temp_storereset was deliberately not used because it deletes live TEMP objects.Writer policy and maintenance compatibility
The canonical repository writer sets and verifies
temp.auto_vacuum=FULLbefore TEMP schema creation. A committed TEMP DROP can then return file space while other TEMP objects remain live. The runtime keeps its existing main-database policy and TEMP storage mode.SQLite remembers the most recent auto-vacuum assignment for any later full VACUUM on the same connection, even when the assignment named TEMP. In a disposable database, applying this setting to unrestricted maintenance changed both a NONE and an INCREMENTAL main database to FULL on a later VACUUM; an attached NONE database also became FULL. The writer's exact-SQL authority rejects full VACUUM. Unrestricted maintenance keeps SQLite's original behavior, so a full VACUUM of main or an attached store preserves each store's own mode.
Verification and merge hold
ok.The owner’s merge hold remains until independent review and exact-head CI succeed.
Partial only: this improves reclaim after a TEMP relation is actually dropped. It does not remove the two existing persistent TEMP ownership tables, bound active sorter spills, prove the 2026-10 in-memory page overwrite cause, or make unsafe connection resets acceptable. The ownership-table change remains exclusively on mixed DRAFT #4776, which must stay draft and unmerged; this PR does not copy any of its work. That draft also touches
connection/mod.rsat disjoint file-identity logic; preserve that owner's changes and do not force-update or modify the draft. #4152 remains open.No Kev/decision-model or release #4763 changes; no production, infra, tags, credentials, agent delegation, Kody, Copilot, or CI weakening.
On
3a3ecf40eb, native Linux ARM64 Bazel also executed all 434 rusqlite-runtime tests as an ordinary user on ext4. All passed with zero ignored, including the real SQLite VFS TEMP-file reclamation and the retention of durable vacuum modes. The suite finished in 8.42 seconds and Bazel exited successfully.