Skip to content

fix(tooling): cap local lint concurrency at 4 threads - #423

Open
PeronGH wants to merge 1 commit into
masterfrom
fix/lint-concurrency
Open

fix(tooling): cap local lint concurrency at 4 threads#423
PeronGH wants to merge 1 commit into
masterfrom
fix/lint-concurrency

Conversation

@PeronGH

@PeronGH PeronGH commented Aug 5, 2026

Copy link
Copy Markdown
Member

pnpm lint freezes a 32-core workstation. --concurrency=auto resolves to 16 worker threads there, each of which builds its own full TypeScript program set, and the sum blows past the machine's memory before anything reports an error. This pins local lint to 4 threads.

lint:ci is untouched — CI stays at --concurrency=off with the single 6144 MB heap that CODE-468 landed.

Why auto is wrong on a workstation

calculateWorkerCount (eslint/lib/eslint/eslint.js:410) derives the cap straight from the core count:

case "auto": {
  const maxWorkers = availableParallelism() >> 1;   // 32 cores -> 16
  return calculateAutoWorkerCount(eslint, filePaths, maxWorkers);
}

getWorkerCountFor is ceil(files / 50) clamped to that cap. With ~1333 lintable files it's ceil(1333/50) = 27 -> 16: maximum fan-out, every run.

Each worker calls configLoader.loadConfigArrayForFile itself (worker.js:127), so each spins up its own typescript-eslint project service and its own TS programs. Nothing is shared between threads, and it all lands in one process's RSS. This repo's own CI comment (ci.yml:31) puts one such program set at over 4 GB — hence the 6144 MB heap there. Sixteen of them is not survivable on a 32 GB box.

Two details make it worse than it looks:

  • A warm cache doesn't help. calculateAutoWorkerCount sets countAllMatched = !lintResultCache || cacheStrategy === "content" (eslint.js:355). Under --cache-strategy content every matched file counts toward the worker calculation even when its cached result is valid, so a fully warm run still starts 16 workers and 16 project services.
  • auto scales the wrong variable. It tracks core count, which on CI runners is small and on workstations is exactly where memory pressure hurts most. The failure gets worse the better your machine is.

Why 4

The worker path is either/or, not additive: eslint.js:1048 selects lintFilesWithMultithreading or lintFilesWithoutMultithreading, so the main thread only coordinates and never holds a program of its own. Four workers means four program sets.

Numeric concurrency also skips calculateAutoWorkerCount entirely (eslint.js:424, just Math.min(4, filePaths.length)), which sidesteps the cache-strategy content behaviour above. And 4 clears ESLint's own workerCount <= 2 threshold, so it doesn't emit the "just disable concurrency" advisory.

Measurements (32-core / 32 GB WSL2 VM, 4 GB swap)

Run Wall clock Peak RAM Swap
Warm cache 9.9 s ~3.7 GB untouched
Cold cache, default heap 89 s 19.8 GB untouched
Cold cache, --max-old-space-size=5120 ~90 s 20.3 GB untouched

Idle baseline is ~1.5 GB. Both cold runs exited 0 with 0 errors (375 pre-existing warnings).

Warm is the common case and 10 seconds is not a bottleneck. Cold peaks at 19.8 GB against a 32 GB ceiling — roughly 12 GB of headroom, and swap is never touched, which is what separates "slow" from "the desktop stops repainting".

No heap flag needed

Verified against the stock 4192 MB default with NODE_OPTIONS unset: cold run completes, no OOM. Adding a heap flag would actively hurt, because runWorkers passes no resourceLimits (eslint.js:470-477), so any heap setting is per worker and multiplies by 4 — 6144 would raise the worst case from ~17 GB to 24 GB, undoing the fix.

Leaving the default also keeps a useful failure mode. If the repo grows past what one worker can hold, you get a clean JavaScript heap out of memory naming the thread instead of a frozen machine. The right answer that day is --concurrency=3, not a bigger heap.

Trade-off

4 is a fixed number, so a low-core machine now oversubscribes slightly where auto would have backed off. That's the cheap direction to be wrong in: 4 threads on a 4-core box is a scheduling inefficiency, whereas 16 threads on a 32-core box is an unrecoverable freeze. The memory ceiling is a property of the repo's TS program size, not of the host's core count, so a constant models it better than auto does.

Verification

  • pnpm check:ci — format, lint, typecheck all pass.
  • pnpm test — 2521 passed, 1 failed: packages/host/assets/tests/integration/registry-client.test.ts fails identically on unmodified master (a loopback-timeout issue on the test machine), unrelated to this change.

Prior art: CODE-468 / #312 fixed the same class of OOM on the CI side and introduced the lint:ci split; this is the local-workstation half that was left on auto.

Copilot AI lite review requested due to automatic review settings August 5, 2026 12:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants