Skip to content

fix(tests): stop exhausting test-db connections and migrate once - #98

Merged
KenTaniguchi-R merged 1 commit into
mainfrom
fix/test-db-connection-exhaustion
Aug 29, 2026
Merged

KenTaniguchi-R merged 1 commit into
mainfrom
fix/test-db-connection-exhaustion

Conversation

@KenTaniguchi-R

Copy link
Copy Markdown
Owner

The problem

pnpm test at vitest's default concurrency reported 137 failures that were not real:

Test Files  56 failed | 56 passed (112)
     Tests  137 failed | 479 passed | 123 skipped (739)
  Duration  339.05s

Errors were 57P01 ProcessInterrupts — Postgres terminating connections — attributed to whichever tests were in flight, not to any actual defect.

Root cause

Each integration file opens its own pg.Pool against the single shared testcontainer. pg pools default to 10 connections. On a 14-core machine vitest runs ~14 workers, demanding up to 140 connections against a server whose max_connections is 100 (verified against postgres:17-alpine).

Postgres starts dropping connections, and the failures land wherever they land. That also explains the flakiness: three runs on identical code gave 5 → 1 → 0 failures.

The fix

Connections. max_connections=300 on the container; each pool capped at 4. The suite needs a couple of concurrent queries per file, not ten.

Migrations — the dominant cost. All 52 integration files replayed the full 9-migration chain: 468 migration runs per suite. Now migrated once into a template database in global setup, with each file cloning it via CREATE DATABASE ... TEMPLATE. Concurrent clones briefly conflict (55006), so that specific error retries with backoff. When no template is present the old path still replays migrations, so a bare DATABASE_URL keeps working.

Durability. fsync, synchronous_commit, and full_page_writes off — the server is destroyed at teardown.

Results

pnpm test, default concurrency, no worker cap:

Before After
Result 137 failed / 479 passed 727 passed / 727
Duration 339s 21s

16x faster, and green. Five consecutive integration runs all passed 301/301, where the suite previously varied run to run.

This should also help the CI backlog: there is one self-hosted runner (macmini-ledgr) and every branch's jobs queue behind it serially, so shortening each job compounds.

Also: ESLint

.stryker-tmp/** added to globalIgnores. Stryker leaves sandbox copies of the entire tree behind, and linting them produced 981 of 1024 errors, failing pnpm lint on duplicated copies of our own code. Lint is now 0 errors (1 pre-existing warning, _accountType in src/lib/money.ts).

Verification

Run in an isolated worktree off main:

  • pnpm install --frozen-lockfile — clean
  • pnpm typecheck — clean
  • pnpm lint — 0 errors (verified with a sandbox present that previously errored)
  • pnpm test — 727 passed / 112 files
  • 5x consecutive integration runs — 301/301 each

🤖 Generated with Claude Code

`pnpm test` at vitest's default concurrency reported ~137 failures that
were not real. Each integration file opens its own pg.Pool against the
single shared container, and pg pools default to 10 connections. At 14
workers that demands ~140 against a server whose max_connections is 100,
so Postgres terminates connections mid-run (57P01) and the failures land
on whichever tests happened to be in flight. That also explains the
suite's flakiness: consecutive runs on identical code gave 5, then 1,
then 0 failures.

Raise max_connections to 300 and cap each pool at 4.

While here, cut the dominant cost: every one of the 52 integration files
replayed the full migration chain. Migrate once into a template database
in global setup and have each file clone it with CREATE DATABASE ...
TEMPLATE. Concurrent clones briefly conflict (55006), so retry. Falls
back to replaying migrations when no template is present, keeping a bare
DATABASE_URL working. Durability settings are also disabled — the server
is destroyed at teardown.

    pnpm test, default concurrency
    before: 339s, 137 failed / 479 passed
    after:   21s, 727 passed

Five consecutive integration runs are now green where the suite
previously varied run to run.

Also ignore .stryker-tmp/** in ESLint. Stryker leaves sandbox copies of
the tree behind, and linting them reported 981 of 1024 errors, failing
`pnpm lint` on code that is not ours. Lint is now 0 errors.
@KenTaniguchi-R
KenTaniguchi-R merged commit c02da36 into main Aug 29, 2026
3 of 5 checks passed
@KenTaniguchi-R
KenTaniguchi-R deleted the fix/test-db-connection-exhaustion branch August 29, 2026 21:58
KenTaniguchi-R added a commit that referenced this pull request Aug 29, 2026
…ions (#101)

#98 replaced per-file migrations with a template database that each test
file cloned via CREATE DATABASE ... TEMPLATE. That broke CI: every
integration file now times out in beforeAll at createTestDb().

    Test Files  57 failed | 56 passed (113)
    Error: Hook timed out in 10000ms.
      185|   beforeAll(async () => {
      186|     ({ db, close } = await createTestDb());

CREATE DATABASE ... TEMPLATE takes a lock on the source database, so
concurrent clones serialize rather than run in parallel. On the CI runner
they block past the 10s hook timeout.

The change was measured only on a 14-core dev machine, where the suite
went 339s -> 21s. That speedup was real but local: the problem it solved
— pool connections (14 workers x pg's default 10) overrunning
max_connections=100 — never occurred on CI, which has fewer cores and ran
the same suite in 27s beforehand.

Restore tests/global-setup.ts and tests/integration/setup.ts to their
pre-#98 state. The .stryker-tmp ESLint ignore from #98 is unrelated and
stays; it takes `pnpm lint` from 1024 errors to 0.

The local exhaustion is better addressed with a maxWorkers cap, which
costs CI nothing. Not included here — this commit only restores green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant