fix(persistence): harden chat and memory storage

[dovvnloading]

Problem

Chat and permanent-memory persistence had eight small but real weaknesses, all of them about user data:

• Two groups made at the same moment could get the same position, and a group deleted while a chat was being moved into it left the chat filed under a group that no longer exists (hidden from the sidebar until the next launch).
• On Windows a scanner or indexer holding a file for a few milliseconds made a memory save or a database backup fail outright, and the memory backup copy was never flushed to disk.
• The legacy JSON chat_history folder survived every migration, so every launch logged a warning that legacy history had been found.
• Deleting chats never returned disk space.
• The chat revision was the number of messages, so regenerating a reply did not move it and two regenerations from the same state both passed the compare-and-swap.
• The in-memory chat repository that every API test runs against had drifted from the SQLite one users run (rename reordered the sidebar, a fork could overwrite another chat and rewrote every message time, group positions could repeat, empty fields read back differently).
• The downgrade and newer-settings gates and a full disk on write had no tests, and a full disk was reported as a generic 500.

Root cause

• Group creation read the highest position and inserted afterwards, and set_chat_group checked the group and then updated without holding the write lock.
• os.replace was called unguarded at the publish points, and _atomic_copy_memos copied with shutil.copy2 and no fsync.
• migrate_from_json_if_needed warned whenever the directory existed, and nothing ever removed it.
• SQLite only shrinks a file by VACUUM or an incremental vacuum, and nothing set auto_vacuum or ran either.
• _check_chat_revision compared COUNT(*), and replace_message does not change the count.
• No test compared the two repositories.
• _raise_repository_error mapped every failure to 500.

Change

One commit per concern (ten on the branch as first opened; after merging main, the four follow-up commits at the end of this section):

1. BE-54 create_group is a single INSERT ... SELECT and set_chat_group takes BEGIN IMMEDIATE before it looks at the group.
2. BE-57 replace_with_retry (in sqlite_backup.py, which both stores already share) retries only PermissionError, four tries in all, waiting 50, 100, 200 ms. It now publishes the chat backup and its rotation, the settings backup, the memory file, its backup copy and the set-aside of a corrupt memory file. The memory backup copy is fsynced before it replaces the backup.
3. BE-58 A pass lists the .json files first and returns quietly when there are none. After a pass the directory is renamed to chat_history.retired (.retired-2, .retired-3, ... if that name is taken) only when it is verified empty (an empty quarantine/ counts as empty). Nothing is deleted: no rmdir remains, so a wrong emptiness check, or a file that lands in the folder between the check and the rename, loses nothing (it is still there under the new name), and the retired folder is the user's to remove. The retired name is not one the migration scans. Quarantined files, un-archivable files and stray files keep the directory where it is; quarantine content is logged once at info level.
4. BE-52 / BE-59 tests/test_chat_repository_contract.py runs 31 behaviours against both repositories (62 tests). Drift found and fixed in the in-memory double: rename no longer bumps recency (SQLite never did), fork keeps message timestamps, starts ungrouped and refuses an id that is taken, new groups go after the highest position, empty sources/attachments/stats read back as absent. On the SQLite side, renaming an unknown chat, adding a message to an unknown chat without a title, and replacing a missing or non-assistant message now raise ChatNotFound / MessageNotFound like the double (previously silent, a foreign-key error, and a wrapped error).
5. BE-64 Schema v6 (main's fix(generation): generation-path cleanups, prompt quality, Ollama load status, translated history, request ids #315 already took v5 for messages.original_content; this step is _migrate_to_v6, ordered after it, SCHEMA_VERSION = 6) adds threads.revision INTEGER NOT NULL DEFAULT 0, raised by add_message, replace_message and delete_last_assistant_message, set to the message count for a fork or a legacy import, backfilled for existing chats from their message count. Rename and moving into a group do not touch it (so neither can turn a generating answer into a conflict). load_chat reads the thread row and messages in one read transaction. chat_revision() returns the persisted counter (falling back to the count for a chat that carries none), the regeneration route hands admission the revision it read instead of len(transcript), and a new turn compares the revision. The API field and its name are unchanged. The in-memory repository has the same counter.
6. BE-60 New chat databases are created with auto_vacuum = INCREMENTAL (before WAL is enabled, the only moment that works). At startup, after that launch's backup succeeded, sqlite_reclaim.reclaim_free_space acts only when more than 25% of the file (and at least 64 pages) is free: incremental files get their free pages back in small transactional steps; older files are rewritten once with VACUUM, which also converts them. Plus a follow-up commit so a step that frees nothing ends the pass instead of spinning. A VACUUM that fails or exceeds the 20 s limit is rolled back, and used to be retried at every launch (up to 20 s added to each start, for good, for a non-incremental database under the 1 GiB ceiling); it now leaves <db>.reclaim-backoff (only a timestamp) and no rewrite is tried for seven days. See "Follow-up" below.
7. BE-55 A full disk (SQLite database or disk is full, ENOSPC, the Windows disk-full codes, found through the cause chain) is answered 507 with "Could not because the disk is full. Free some disk space and try again." on every route that reports repository failures through _raise_repository_error (chat and group create, rename, move, delete, fork); everything else there is still 500 with the same text plus main's request id. The send path (POST /generations) is not among them, see Limits. New tests inject the full-disk error at the write and check the chat comes back exactly as it was, and add the newer-schema gates for the chat file, the settings file and the settings row.
8. README: two bullets in the upgrade and recovery section.

Follow-up after merging main and an independent review (four commits on top of the merge):

• Merge with main. Schema numbering (below), _raise_repository_error (main's request-id logging plus the 507 mapping), and the in-memory repository (empty-means-absent plus main's original_content).
• Tests moved off the removed route. POST /chats/{id}/messages went away in fix(api): small API defects - drop the raw message write, kind-checked /generations, can_cancel, client route check, one-query task list, 1 MiB body ceiling #316, so the two HTTP full-disk tests (507 with the chat unchanged and no half-written fork; a non-disk-full failure still 500) now use POST /chats/{id}/forks, and the newer-settings test expects main's (Request ID: ...) suffix. The revision tests already used /generations.
• Two untested hunks now covered. A test lands a real add_message between load_chat's two reads (through a trace callback on the reading connection) and requires the revision and the messages to come from the same moment; another checks a legacy-imported chat's revision equals its message count and that a stale revision is refused. Each was verified to fail when the hunk is reverted (BEGIN removed; import revision set to 0).
• chat_history.retired instead of rmdir (BE-58, above).
• Back-off after a failed whole-file rewrite (BE-60, above): a lock held by another program is not a failed rewrite and leaves no marker, an unreadable or future-dated marker is ignored, the incremental path is never delayed, and deleting the marker lifts the wait.

Compatibility and rollback

Schema change (v5 to v6, on top of main's v5): additive only (one column, plus a backfill of derived values). The existing ladder runs it. A store written by current main (v5) gets <db>.pre-v5.bak, written and verified first (the upgrade refuses to run without it); a store still at v4 gets <db>.pre-v4.bak and climbs through main's v5 step and then this one in the same launch (one snapshot, of where the upgrade began). Each step is one transaction with its version bump, and this one is idempotent (MAX(revision, count) never lowers a revision). Nothing is dropped or rewritten, original_content is untouched, and test_a_v4_database_upgrades_through_5_and_6... and test_a_v5_database_upgrades_to_v6... check data, originals, snapshots and revisions for both starting points.

To go back to the previous release (which reads up to schema 5): close Cortex, move cortex_db.sqlite and any -wal and -shm files beside it aside, copy cortex_db.sqlite.pre-v5.bak (or .pre-v4.bak for a store that came straight from v4; the previous release upgrades that itself) to cortex_db.sqlite, start the previous release. Anything written after the upgrade is only in the files moved aside. The previous release refuses a v6 file without touching it and names the newest snapshot it can read, exactly as the ladder already refuses any newer store. test_the_previous_release_refuses_the_upgraded_database_and_the_named_snapshot_restores_it walks that procedure for a previous release that reads 5 and for one that reads 4 (refusal naming the snapshot, move aside, restore, previous release opens it with its data and originals, upgrading again works). Reverting only this PR's code, without restoring the snapshot, leaves a v6 file that the previous release refuses rather than misreads.

The other on-disk changes: <db>.reclaim-backoff (a timestamp, only after a failed whole-file rewrite; safe to delete) and chat_history.retired (an emptied folder, safe to delete). Neither is read by any earlier release.

Behaviour changes to know about: revision in a chat response now also moves on a regenerate (it used to move only on appends); a client holding an older revision after someone regenerated gets the existing 409 "reload". The frontend needed no change: it takes the revision from the chat it reloads when a generation completes, and its optimistic +1 for the user message the server just accepted is still what the server does. The backend contract check (generate_contracts.py --check) is unchanged. If that reload fails, the client keeps a stale revision and its next send gets the 409, as it would after any turn.

The in-memory repository changes (rename no longer reorders, fork keeps timestamps) change what the demo and API tests see, deliberately, to match SQLite.

Checks

Run from the worktree root, on the merge of this branch with main at eb55570 plus the follow-ups.

• python -m ruff check backend tests tools main.py app_factory.py scripts: all checks passed.
• python -m mypy: no issues in 98 source files.
• python tools/generate_contracts.py --check: exit 0.
• python tools/artifact_boundary_review.py --json --strict: exit 0, 12 of 12 cases passed.
• python -m pytest -q on Python 3.14: 3071 passed, 25 subtests passed.
• Pre-push hook (check.ps1 quick tier), first attempt, Python 3.14: 12 of 12 checks passed in 334.6 s, including backend tests (3071 passed), artifact-boundary review, contract check, frontend types, eslint and vitest (914 passed in 72 files).
• Python 3.12 (py -3.12, gguf supplied from a copy of the 3.14 package on PYTHONPATH): 3070 passed, 1 failed, 25 subtests passed. The failure is tests/test_repository_conventions.py::test_collection_works_without_cwd_on_sys_path, which runs a child pytest with PYTHONPATH removed on purpose, so that child cannot import gguf on this machine; it does not touch this PR's files.
• Each new test in this round was checked against a mutation of the code it covers and fails there: back-off predicate disabled (2 tests fail), load_chat without its BEGIN (the same-moment test fails: the messages include the write that landed after the revision was read), legacy import revision set to 0 (fails: 0 against 3), 507 mapping disabled (500 instead of 507).
• Earlier evidence still stands: each behaviour test from the first version of this PR was run against the unfixed source and failed there (group positions, orphaned-chat interleaving, sharing violations, legacy directory, the contract suite, the revision tests, startup reclaim, the 507 mapping, the no-progress guard).
• Nothing under frontend/ changed; the hook's frontend checks are the only frontend run.
• Flaky tests: none seen in the runs above.

Security / data-loss / concurrency

• No user data is deleted or overwritten, and no folder is deleted: the emptied legacy folder is renamed. The only file this change removes is its own .reclaim-backoff marker, after a rewrite succeeds. Space reclaim is all-or-nothing (VACUUM rolls back on interruption, failure or a full disk; incremental steps are separate transactions) and never runs without a verified backup of the same content from the same launch. Tests interrupt a VACUUM with a zero time limit, hold the write lock from another connection, remove the free disk space and shrink the size ceiling, and each leaves every row in place with a passing integrity check.
• Reclaim runs in the DatabaseManager constructor before the app serves a request, so it never competes with a write; another connection holding the write lock only makes it give up.
• Group creation and chat moves are atomic; the tests run eight threads behind a barrier and land a second writer exactly between the check and the update.
• The revision compare-and-swap keeps its semantics under BEGIN IMMEDIATE; a test races two regenerations from the same revision against both repositories and exactly one wins.
• Logs carry only exception type names and counts, never paths, titles or content. The 507 detail has no path or user text (asserted). Fixtures are synthetic.

Limits

• The 1 GiB / 20 s / 25% thresholds are choices, not measurements; fixtures are a few megabytes. A database above the ceiling is never compacted at startup.
• Retry on sharing violations is not applied to recovery's quarantine moves or the sidecar moves; those fail closed on purpose and their tests depend on it.
• Creating a duplicate chat, group or fork still raises a different error type in each store (the contract accepts either); the routes never do this (ids are generated).
• DatabaseManager.set_chat_group is still translated to ChatGroupNotFound by matching the message text in the adapter (unchanged).
• The 507 mapping is in _raise_repository_error, so it covers the routes that use it. The send path (POST /generations) writes the user message while the job is prepared, outside that helper, and a full disk there reaches main's outermost handler as 500 Internal server error. (Request ID: ...) (measured with the same injected fault). It is a chat turn that is refused and nothing is half saved, but the message does not say why. Not changed here: it means touching the job-start error path or main's request-failure handler, which belong to a separate change.
• The whole-file rewrite back-off is a fixed seven days from the failure, keyed on the wall clock; a clock set far back is ignored rather than honoured. A database whose rewrite keeps failing therefore still costs up to 20 s once a week, not at every launch.
• Retiring the legacy folder now leaves chat_history.retired (an empty folder, or one holding only an empty quarantine/) beside the data instead of no folder; that is the price of never deleting one.
• Outside this bundle, noticed and not touched: SQLiteSettingsRepository.save upserts schema_version from the current release, so a payload written by a newer build would be overwritten if something saved without loading first; the routes load first, so it is not reachable today.
• BE-55's downgrade-refusal test for the chat file already existed in test_chat_groups.py; this adds the stricter directory-wide variant rather than a duplicate. The plan's BE-59 was already fixed for SQLite (only the double drifted), and BE-57's "shared helper from BE-56" does not exist yet, so replace_with_retry lives in sqlite_backup.py.

Plan item: BE-52, BE-54, BE-55, BE-57, BE-58, BE-59, BE-60, BE-64

🤖 Generated with Claude Code

[dovvnloading]

Pushed c90828c: main (eb55570) merged in, and three review findings fixed. The description above is updated (schema v6, rollback, checks).

Merge with main (by merge, no rebase, no force-push). Conflicts were in routes.py, repositories/chats.py and repositories/storage.py; main's changes to requirements*.lock.txt and everything else were kept as they came. git diff origin/main --stat lists only this PR's 20 files.

Renumbered: schema v5 to v6. Main's #315 already took v5 (messages.original_content). This branch's threads.revision step is now _migrate_to_v6, SCHEMA_VERSION = 6, ordered after main's untouched v5. The pre-upgrade snapshot logic is generic (<db>.pre-v<version>.bak of the version the upgrade began at), so a v4 store gets pre-v4.bak and climbs 5 then 6 in one launch, and a v5 store written by current main gets pre-v5.bak. Tests: a v4 store and a v5 store are upgraded to v6 and checked for data, original_content kept, snapshots, and revisions equal to message counts; the rollback test runs for a previous release that reads 5 and one that reads 4 (refusal names the snapshot, restore works, upgrading again works); the newer-database refusal test now follows SCHEMA_VERSION. The _raise_repository_error conflict keeps main's request-id logging and this branch's 507 for a full disk.

Moved: tests off the removed route. POST /chats/{id}/messages is gone (#316). The two HTTP full-disk tests now use POST /chats/{id}/forks (507 with the message, the chat unchanged, no half-written fork, and the same request succeeding once there is room; a non-disk-full failure is still 500), and the newer-settings API test expects main's (Request ID: ...) suffix. The revision tests in test_chat_revision_cas.py already used /generations.

Fixed (a): the two untested hunks. New tests: one lands a real add_message between load_chat's two reads and requires the revision and the messages to be from the same moment; one checks a legacy-imported chat's revision equals its message count and that a stale revision is refused. Each fails when its hunk is reverted (verified: BEGIN removed, import revision set to 0).

Fixed (b): no folder is deleted any more. The emptied legacy folder (verified empty first, an empty quarantine/ counting as empty) is renamed to chat_history.retired, or .retired-2, .retired-3 if the name is taken; os.rmdir is gone from the path. Reason: a wrong emptiness check or a file landing in the folder between check and rename now loses nothing. Tests: renamed, never deleted (rmdir/rmtree forbidden in the test), a taken name is not reused, a folder that cannot be renamed is left and retired next launch, stray/quarantined files still keep the folder. README updated.

Fixed (c): back-off for the whole-file rewrite. A non-incremental database under the 1 GiB ceiling whose VACUUM exceeded 20 s was interrupted, rolled back and retried at every launch (up to 20 s added to each start). A failed or timed-out rewrite now writes <db>.reclaim-backoff (only a timestamp) and no rewrite is tried for seven days; success removes it. A lock held by another program is not a failure and leaves no marker; an unreadable or future-dated marker is ignored; the incremental path is never delayed; deleting the marker lifts the wait. Tests at function level (boundaries at exactly seven days, garbage markers, lock, unwritable marker, incremental unaffected) and at startup level (three launches after a failure do not retry, then one does once the marker is old). README updated.

Not fixed, stated in Limits: POST /generations (the send path) writes the user message outside _raise_repository_error, so a full disk there is still main's generic 500 ... (Request ID: ...), not 507 (measured). Fixing it means touching the job-start error path or main's request-failure handler, which is a separate change.

Checks. ruff, mypy (98 files), contract check, artifact-boundary review (12 of 12): clean. python -m pytest -q on Python 3.14: 3071 passed. Pre-push hook, first attempt: 12 of 12 checks passed (backend 3071 passed; vitest 914 passed in 72 files). Python 3.12 with gguf from a copy of the 3.14 package: 3070 passed, 1 failed, the failure being test_collection_works_without_cwd_on_sys_path, whose child pytest removes PYTHONPATH on purpose and so cannot import gguf.