[RyeMutt]

Adds aldb, the SQLite layer the viewer's settings will move onto, and later the cache's index and the chat logs. Nothing links it yet. Also adds LL::ThreadPoolBase::abandon(), which aldb's writer needs. Built on develop after #596, whose pool scheduler it keeps. Draft until macOS and Linux pass this push's CI.

Commits

1. A thread pool can be let go of while its threads run (llcommon). Until now a pool was closed by joining its threads, so a pool whose thread is stuck held whoever closed it. abandon() takes the pool from its owner instead of destroying it:
   • It closes the queue and stops listening for the viewer's shutdown, on the caller's thread.
   • If every thread has finished, it joins them and destroys the pool. Otherwise it lets them go, and the last thread to finish destroys the pool.
   • The count of running threads, and the pool abandon() left them, live in a state the threads share and keep once the pool is gone, since a pool's own run() may destroy it (the Win32 window's thread does). A thread touches nothing of the pool after run() returns.
   • The pool's destructor now stops listening only if the pool listened, so a pool can be destroyed on its own thread without touching the event pumps.
2. Add aldb (indra/aldb, the sqlite3 port, wiring, an entry in doc/ARCHITECTURE.md).

What aldb does

• One writer thread per file, an LL::ThreadPool of one: named for the OS and Tracy, not closed by the viewer's quitting.

  • Jobs queued together commit in one BEGIN IMMEDIATE transaction, each in a savepoint of its own, so a job that fails or throws is dropped alone.
  • A failure that may pass (busy, locked, disk full, I/O) is retried with backoff, from 100 ms up to 5 s, and reported once until a write next succeeds. Where SQLite undoes the whole transaction for it, the jobs that ran before it are tried again with it rather than dropped, and none runs outside a transaction.
  • A job may carry a completion, told on the writer thread whether it committed. flush() waits up to closeWait and says whether everything was written.
  • Close writes what is queued and checkpoints the WAL. After closeWait it abandons the pool to a writer stuck on a hung disk rather than hang an exit.

• Coroutine-safe: the queue, the reader pool and the waits use the coroutines' own locks and promises (LLCoros::Mutex, ConditionVariable, Promise). A coroutine waiting in postAndWait, flush or a backup gives its thread's other coroutines their turn, and a write from a coroutine never trips LLMutex's main-coroutine assert.

• A pool of read-only connections for reads.

• Profiles:

  • Durable syncs every commit (fullfsync on macOS).
  • Journal survives a crash.
  • Regenerable is unsynced and memory-mapped.

  All are WAL, falling back to a DELETE journal where there is no shared memory.

• Migrations by user_version run in one transaction, with a VACUUM INTO copy kept before each (the last two are kept). The version is read again under the transaction's lock, so two viewers opening a new file at once make its tables once. A file written by a newer viewer is opened and left alone, and a file whose application_id names another kind of file is refused.

• Read-only files: a file this process may only read (marked read-only, owned by another user, on read-only media) is opened for reading, and ALDbOpenResult::readOnly says so. Its reads run; its writes are refused, reported once.

• Recovery: after a crash the file is checked.

  • The check waits for a file another viewer is writing, as every read does, and takes only what reports damage as damage, not a check that could not run.
  • A damaged file is moved aside with its WAL and salvaged into a new file with SQLite's recover API; a Regenerable one is rebuilt instead.
  • A file that can't be used at all leaves the database in memory for the session.
  • Damage found while writing is reported once for as long as the file is open.

• ALDbConnection:

  • It caches statements by their SQL. A single statement exec() runs is cached too, so BEGIN, COMMIT and each job's savepoint are prepared once. A script still goes through sqlite3_exec.
  • A second borrower of a statement in use gets one of its own.
  • It keeps the first failure, and stores an LLUUID as 16 bytes.
  • Its Foreign access is for imported backups and presets. It opens them read-only in defensive mode, with triggers and views disabled, untrusted schemas, no memory mapping, limits on value size and query depth, and an integrity check before anything is read.

Since the first push

• The review's fixes, folded into the two commits:
  • A pool thread called back into its pool after run() returned, when run() may have destroyed it. The shared state above fixes it.
  • A batch SQLite rolled back whole lost the jobs that ran before the failure. They are now tried again.
  • A read-only file opened as though writable and dropped every write silently. It is now opened for reading and reported.
  • user_version was read outside the migration's lock, so two viewers could both migrate a new file.
  • A check that could not run (the file busy) quarantined a healthy file.
  • exec() of a single statement is now cached.
  • The Foreign defences are read back from SQLite in their test, and the tests' temporary directory says why it could not be made.
• CI fixes: macOS's libc++ refuses a discarded std::future::get(), which one test had. The thread pool's wake check is no longer timed under AddressSanitizer or ThreadSanitizer, where a busy runner took 3 ms.
• Rebased onto develop after Pool workers sleep until woken, and UUIDs are random v4 without a lock #596. Pool workers sleep until woken, and UUIDs are random v4 without a lock #596 replaced the Windows pool scheduler that slept a timer tick at a time, which this branch had fixed too. Its scheduler is kept and this branch's is gone. The two threadpool_test.cpp files are merged.

Dependencies

• sqlite3 3.54.1 comes from alchemy-registry. Its port builds the recover API into the library and sets the build options in sqlite3-vcpkg-config.h:

  • multi-thread mode;
  • WAL synchronous NORMAL;
  • no double-quoted strings;
  • no memory statistics;
  • no deprecated or shared-cache API.

  The registry baseline moves to f516d94, which adds sqlite3 alone.

• attribution.json gives sqlite3 a holder line, since its copyright file only says it is public domain.

Testing

• Windows RelWithDebInfo, at this push: all 75 llcommon and aldb ctest entries pass, including llapp_test, which shuts the viewer's pools down.

• aldb_tests: 38 cases. They include:

  • a fault-injection VFS: full disk, hung disk, busy file, no shared memory;
  • two processes writing one file, and two viewers migrating one file at once;
  • a batch SQLite rolls back whole, a read-only file, a check that finds the file busy;
  • salvage of a damaged index and of rows only in the WAL;
  • transaction and savepoint statements prepared once;
  • a foreign file's trigger, view and oversized value.

  Each case written for a fault was seen to fail before its fix.

• threadpool_test: 9 cases, Pool workers sleep until woken, and UUIDs are random v4 without a lock #596's three and this branch's six:

  • three stuck threads let go and destroyed by the last of them;
  • a pool never started;
  • a pool whose queued work still runs after it is let go;
  • a listening pool that stops;
  • a pool its own run() destroys, whose memory nothing writes afterwards;
  • a thread woken by a ping in under half a millisecond at the median.

  The first three fail with the last thread's deletion taken out, the fourth with the listener left, and the fifth crashed before the shared state.

• GCC 15.3: every source this PR changes compiles with -Werror and the project's warning flags.

• CI at the previous push (before the rebase): every job passed but ASan, whose only failure was the wake timing above.

• Owed: this push's CI.

Left for later

Before the cache index or the chat logs use it, aldb grows what a second and third consumer need and settings does not:

• one policy table per profile, read by the writer and the readers alike;
• capped batches and queues, and a keyed post;
• posting from any thread, with results and bounded waits;
• an authorizer over jobs, and a statement cache with a limit;
• WAL upkeep and damage marked for the next open;
• per-file fault tests.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}