Deadlines on every wait for the database (#172)

[Devski]

Closes #172.

Every wait on the database had no end to it. src/db/client.ts built the pool
with a connection string and nothing else, so pg's defaults stood — and the
default for connectionTimeoutMillis is 0, which means a caller waiting for a
free connection waits for ever. Neither statement_timeout nor lock_timeout
was set on either side. Contention therefore queued where nothing could be
read: no error, no log line, a page that simply never came back.

What this changes

• The pool is spelled out — max, connectionTimeoutMillis (5 s),
  idleTimeoutMillis, with the reasoning next to each number.
• The server cuts off what runs too long — statement_timeout 10 s,
  lock_timeout 3 s, and idle_in_transaction_session_timeout at the same
  bound as a statement, because a transaction gone quiet still holds its locks.
• Every number is per environment — DB_POOL_MAX,
  DB_CONNECT_TIMEOUT_MS, DB_STATEMENT_TIMEOUT_MS, DB_LOCK_TIMEOUT_MS,
  documented in .env.example. The built-in values are sized for dev's one
  core, shared by dev and every open preview (PR preview deployments on the dev instance #31, PR previews: a database cloned from dev per preview, not the shared one #113); prod's managed
  database (Production environment: instance, managed Postgres, prod bucket, CDN, G10 drill #24) is another machine with another cap and will set its own.
• A batch job gets no web deadlines — runAsBatchJob() in the three
  scripts (seed, TERYT import, file-key backfill). The migration runner needs
  no opt-out at all, which is the point of the next item.
• The bounds travel in the connection, not in DATABASE_URL. The address
  is shared by deploy/migrate.mjs, psql and the scripts; a migration cut off
  half-applied by a bound meant for a page is worse than a slow migration. The
  migration runner does gain the opposite bound — 30 s to reach the database at
  all, so an unreachable one fails the deploy instead of holding it open.
• A failure now has a name. databaseStall() reads the bound out of the
  error (57014, 55P03, 25P03, and pg-pool's own connect-timeout message),
  looking through the wrapper drizzle puts around driver errors, and
  onRequestError writes it to the log. A stall used to look exactly like a
  slow request.
• application_name says who is holding the connection —
  platform-lite/pr-170/web. Dev and every preview share one database, so
  "something is queueing" is only actionable if pg_stat_activity says which.
• The pool has an error listener. Without one, an idle connection the
  server drops arrives as an unhandled 'error' event — a crash, not a log.

Decisions taken

• A refused connection stays a plain error, not a 503 with Retry-After.
  The issue left it open. Nothing on the client retries, and inviting a retry
  into a database that is already saturated is the wrong advice; what was
  missing was an answer and a name in the log, and both are here.
• idle_in_transaction_session_timeout equals the statement bound. No
  transaction in this codebase does I/O — the S3 calls are all outside — so a
  gap inside one longer than a whole statement may run means the caller is gone.

Verification

pnpm check green, pnpm build green. The issue's verification list is
automated in src/db/client.test.ts against a real PostgreSQL (the CI service
container; the tunnel locally — PGlite cannot stand in, every bound here is
about the second connection):

• a statement waiting on a lock somebody else holds is cut off — and it is the
  advisory lock lockUser takes on every write, measured rather than
  assumed, because the PostgreSQL documentation's "table, index, row, or other
  database object" does not say it in so many words. It is 55P03 after the
  lock bound, not the statement one.
• a statement that runs too long (pg_sleep) is cut off;
• a caller that cannot get a connection is answered instead of queued.

🤖 Generated with Claude Code

---

What the two review lanes changed

One crash, and it was mine to introduce. The new
idle_in_transaction_session_timeout ends a connection while a caller holds
it, and pg-pool removes its own error listener from a client for exactly as
long as that client is checked out. An 'error' event with no listener is a
process exit in Node. Probed against dev's PostgreSQL: without a listener of
ours the process dies (exit 9, terminating connection due to idle-in-transaction timeout), with it the query rejects and the request fails on its own. The new
test fails when the listener is removed — checked, not assumed.

Also fixed, all from the reviews:

• A batch job no longer reads the web variables. They belong to an
  environment's web process, and .env is loaded by the scripts before the
  pool opens — so DB_STATEMENT_TIMEOUT_MS=10000 on the instance would have
  rolled back a TERYT import at ten seconds. Batch takes the built-in numbers
  and nothing else, and a server is refused runAsBatchJob() outright.
• The R360 collector gets its own pool. It runs inside the server, so it
  could not opt out; under the web bound its sweep over every files row
  would not fail loudly, it would return "nothing found" as the table grew —
  which reads exactly like a clean bucket.
• DB_IDLE_TX_TIMEOUT_MS is its own variable, so lifting the statement
  bound for one slow report no longer also lets a transaction sit on its locks.
• The reader refuses what looks permissive and is not: 0 for the pool
  (pg silently substitutes 10) or for the connect wait (that is the bug this
  closes), hexadecimal (0x10 parses as 16), and anything past setTimeout's
  ceiling, where a number meant as "effectively never" fires after 1 ms.
• The cause walk is capped at 16 links — it runs on the failure path of
  every request, and a self-referential cause would have spun the core.
• The logged path drops its query string, where a verification token lives.
• A session read that fails on a bound now says so. sessionUserId()
  catches and returns null, so a stalled database used to hang and now
  silently signs everyone out — with nothing in the log, because the request
  succeeds and Next's error path never runs.
• deploy/migrate.mjs gains a five-second lock_timeout. That one guards
  the site rather than the migration: the old container is still serving, and
  a migration queueing for ACCESS EXCLUSIVE queues every reader of that
  table behind itself.
• 0 on a server-side bound means "the server's setting stands", not "off"
  — corrected in .env.example and in the code, because a managed database
  (Production environment: instance, managed Postgres, prod bucket, CDN, G10 drill #24) may well ship a role-level statement_timeout.

Two findings were filed rather than folded in: #179 (anonymous renders
have no cache and no per-address limit, and the proxy's 1.5 s lookup keeps a
5 s place in the pool queue) and #180 (three queries that scan a whole
table, now under a deadline).