CI: two flake sources produce reds that look like code defects (pnpm setup race, post-teardown 57P01)

[KenTaniguchi-R]

Two distinct flake sources hit four separate runs today. Neither is a code defect, and both have narrow fixes. Filing with evidence so the next red is triaged in seconds rather than by bisecting a diff.

1. pnpm/action-setup races on a shared install dir

Error: ENOTEMPTY: directory not empty, rmdir
  '/Users/taniguchiryusei/setup-pnpm/node_modules/.bin/store/v11/files/03'

The action installs into ~/setup-pnpm, which is shared across every job on the self-hosted runner. Two jobs from different refs starting close together race on it, and the loser fails before a single test runs.

The workflow's concurrency group is keyed on github.ref, so it correctly serialises runs for one branch but does nothing across branches — which is exactly the case that collides.

Seen on: run 33297952151 (PR #124). A re-run with no code change passed.

Fix options, cheapest first:

• standalone: true on pnpm/action-setup — installs a self-contained binary, skipping the node_modules layout that races.
• or dest: ~/setup-pnpm-${{ github.run_id }} — unique per run, at the cost of leaving directories behind.

2. A post-teardown 57P01 fails a run where every test passed

Run 33297710143 (main @ 97a3c23):

Test Files  120 passed (120)
Tests       865 passed (865)
Errors      1 error
→ exit code 1

The error is a Postgres FATAL 57P01 ("terminating connection due to administrator command") surfacing after returns zeros on Plaid API error (non-fatal) completed, in tests/integration/recurring-sync.test.ts. A pooled connection is still open when the testcontainer is torn down, so the reset lands as an unhandled error outside any test and Vitest exits non-zero.

Note this is not the hook-timeout problem — that one is fixed (hookTimeout: 30_000, PR #122) and no hook timed out in this run. Every assertion passed. Only the teardown ordering is wrong.

Likely fix: ensure the pool is drained before the container stops — await pool.end() in the afterEach/global teardown path rather than letting the container shutdown race an open client.

Why it is worth fixing rather than re-running

Both produce a red X that looks like a broken diff. Today that cost a full triage cycle on main and on two PRs, and the natural reaction — reverting the most recent change — is exactly wrong. See #103 for the related mutation-gate noise, and the pattern already recorded there of a revert being opened for a red the commit did not cause.

[KenTaniguchi-R]

Two more occurrences of source 1, with different symptoms — worth recording, because the shared-dir race does not always surface as ENOTEMPTY, and the other faces are harder to recognise.

PR #119 — reflink from a store path that no longer exists:

[ERR_PNPM_ENOENT] ... the source path is not an existing regular file, reflink
'/Users/taniguchiryusei/setup-pnpm/node_modules/.bin/store/v11/files/29/844c...'
  -> '.../node_modules/.pnpm/next@16.3.3_.../node_modules/next_tmp_76812_2/dist/...'

PR #123 — pnpm's own package incomplete in the store:

[MODULE_NOT_FOUND] Cannot find module
'/Users/taniguchiryusei/setup-pnpm/node_modules/.bin/store/v11/links/@/pnpm/11.5.2/.../node_modules/pnpm/dist/worker.js'

Both re-ran green with no code change, and both were on actions-runner-ledgr-2 while other jobs were active — same shared ~/setup-pnpm path, same cross-ref race you identified.

So the race has produced at least three distinct faces: ENOTEMPTY on rmdir, ENOENT on reflink, and a half-written package that fails at require. Only the first is obviously an install problem; the third in particular reads as a broken pnpm installation and invites reinstalling the toolchain.

One detail that makes all three easy to misread: the failing step is Run pnpm install --frozen-lockfile, and --frozen-lockfile suggests a stale lockfile — while the line immediately above says Lockfile is up to date, resolution step is skipped.

standalone: true looks right, and better than the per-runner store dir I had been assuming: it removes the node_modules layout the race needs, rather than reducing the odds of colliding on it.

A fast triage command, since the symptom varies but the step does not:

gh api repos/KenTaniguchi-R/ledgr/actions/runs/<id>/jobs \
  --jq '.jobs[] | {name, conclusion, failed: [.steps[]|select(.conclusion=="failure")|.name]}'

If the failed step is the install, it is this, whatever the message says.

[KenTaniguchi-R]

Correcting the diagnosis above — the damage is worse than "the installer directory is dirty", and #130 alone does not clear it.

pnpm/action-setup sets PNPM_HOME=~/setup-pnpm/node_modules/.bin, and the pnpm content-addressable store lives underneath that, at ~/setup-pnpm/node_modules/.bin/store/v11/. So the original error:

ENOTEMPTY: directory not empty, rmdir '~/setup-pnpm/node_modules/.bin/store/v11/files/03'

was a half-completed rmdir inside the package store, not just the installer. The consequence shows up later as:

[ERR_PNPM_ENOENT] [importPackage .../node_modules/@vitest/runner]
  the source path is not an existing regular file, reflink
  '~/setup-pnpm/node_modules/.bin/store/v11/files/03/13e240...' -> ...

pnpm install reflinks from the store; those store files are gone, so it fails. That is why runs are now failing at pnpm install --frozen-lockfile rather than at setup.

Two consequences worth recording:

Both runners are affected. macmini-ledgr and macmini-ledgr-2 are separate runner processes on one machine under the same user, so they have separate _work directories but a shared ~/setup-pnpm — which is both why they race and why the corruption hits both. Jobs still pass intermittently only when they happen not to need a deleted file.

#130 prevents recurrence but does not repair. standalone: true stops the installer racing on that directory, which is the right long-term fix, but PNPM_HOME still resolves under ~/setup-pnpm and the existing store damage persists.

The repair is one command on the runner host:

rm -rf ~/setup-pnpm

That drops the broken installer and the corrupt store together; the next run re-downloads both. Worth doing before merging #130 so its own green check reflects a clean store rather than a lucky one.

Longer term it is worth moving the store off that path entirely — e.g. a pnpm config set store-dir ~/.pnpm-store on the host, so a future installer race can never take the package cache with it.