fix(db): WAL, because watching a fetch was killing it

[eaitbrahim]

Refs #437, #435. Found while verifying D4's acceptance end to end — by running it, not by testing it.

The bug

A first market-data fetch, started from the browser and watched on the page that says "this page will show its progress", died at 45 seconds:

market_data · failed · OperationalError: disk I/O error
                       (31,709 candles written)

Not disk space — 362 GiB free. Not corruption — PRAGMA integrity_check → ok.

The setup page auto-refreshes every 5 seconds and opens the database to render the checklist. The background job was writing to it. In SQLite's default rollback journal a writer takes an EXCLUSIVE lock and readers take SHARED ones, so the two cannot coexist.

Measured, same fetch, same machine, real venue

polling	journal	outcome
every 5 s (the page)	rollback	FAILED at 45 s, 31,709 candles
none	rollback	ran 150 s, 108,202 candles
every 0.2 s (25× harder)	WAL	ran 150 s, 108,501 candles, 694 clean reads

The middle row is what makes this worth taking seriously: the fetch was fine, and observing it was the bug. Worse, observing it is the encouraged behaviour — the action's own message invites it.

The fix

In WAL, readers never block the writer and the writer never blocks readers.

The hazard was never specific to the job runner, either: an agent writing a cycle while a dashboard refreshes is the same shape. The web UI is simply what made it reachable.

A busy timeout comes with it — SQLite's default is zero, it raises immediately, which is the wrong default for a process that now reads and writes one file at the same time.

Journal mode is a property of the file, not the connection, so an existing deployment converts on its next connection and needs nothing from an operator. :memory: is excluded: no file to journal, SQLite refuses WAL there, and a shared in-memory database is single-connection anyway.

Two things checked rather than assumed

• keel update's backups are unaffected. It uses SQLite's own online-backup API — precisely because "a plain file copy of a database with a live rollback journal is not a snapshot" — and that API reads committed WAL content too.
• The -wal/-shm sidecars must not be mistaken for databases. Both the backup set and is_deployment_root glob keel*.db, which does not match them. Now pinned by test, because a -wal counted as a database would be backed up as one and, worse, would make any folder holding one look like a deployment.

Also checked: a read-only URI open (file:…?mode=ro, how keel setup inspects a deployment) still works against a WAL database with no writer present.

Verified end to end after the fix

The identical scenario, polled once per second — five times harder than the page does:

market_data · running · 102s elapsed · 74,714 candles · no error

Verification

4092 passed, 3 skipped (5 new). ruff check clean repo-wide, mypy clean.

The operator runbook now explains why a deployment folder has three files per database, and says not to hand-copy the .db while keel is running.

🤖 Generated with Claude Code