Skip to content

fix(node-lts): repair the bundled npm's shadowed minipass — every npm install is broken - #670

Closed
bryan-minimal wants to merge 1 commit into
mainfrom
fix/node-lts-npm-minipass
Closed

bryan-minimal wants to merge 1 commit into
mainfrom
fix/node-lts-npm-minipass

Conversation

@bryan-minimal

Copy link
Copy Markdown
Member

node-lts 24.19.0 ships an npm whose own bundled dependency tree is
self-inconsistent, which breaks every npm install. That has been failing
next since 2026-08-27, and is currently blocking #665, #667, #669 and #659 —
all four fail the same minimal build check on the same step.

Root cause

Upstream's node-v24.19.0 tarball bundles npm 11.17.0 containing:

npm/node_modules/minipass-flush/index.js
    const { Minipass } = require('minipass')        # minipass 7.x NAMED export
npm/node_modules/minipass-flush/package.json
    "minipass": "^7.1.3"
npm/node_modules/minipass-flush/node_modules/minipass
    version 3.3.6                                   # default export, no .Minipass

The nested minipass@3.3.6 shadows the correct hoisted minipass@7.1.3, so
Minipass resolves to undefined and class Flush extends undefined throws
Class extends value undefined is not a constructor or null. minipass-pipeline
is unaffected only because it still uses the old default-import style that
matches its own nested 3.x.

Nothing in pkgs causes this — both node/build.sh and node-lts/build.sh are
plain ./configure && make && make install and never touch node_modules. We
inherit it from the upstream tarball.

Why it was so hard to see: with --prefix on the command line npm dies
during config resolution, before it can print an error. The build's entire
stderr is:

npm error A complete log of this run can be found in: /state/home/.npm/_logs/…-debug-0.log

and that debug log contains no error line either — it stops after verbose cwd
and jumps to verbose exit 1. It reads exactly like a network fault. It is not:
it reproduces offline with a two-line dependency-free package.json.

Evidence

Run against the real store tree on res-server-amd64, using node-lts's own
binaries. Removing one directory is the only variable:

npm --version                     -> 11.17.0, exit 0
npm config get prefix             -> /tmp/mh/.local, exit 0
npm install --global --prefix …   -> exit 1, no error output at all
npm install  (plain, local)       -> exit 1, Class extends value undefined …

rm -rf minipass-flush/node_modules/minipass
npm install --global --prefix …   -> added 1 package in 162ms, exit 0

Note the third and fourth lines: a plain local install with no --global
and no --prefix fails identically, which is what proves this has nothing to do
with prefixes, next, or the network.

The fix

Delete the shadowing nested copy so minipass-flush resolves the hoisted 7.1.3
it actually asks for.

Guarded: if upstream changes minipass-flush off the named-import shape, the
build fails loudly rather than silently carrying a workaround that no longer
applies. We shipped five obsolete carried workarounds in one emacs bump recently;
this one announces itself when it stops being needed.

The test matters more than the fix

This shipped completely silently, and would have again, because every existing
test only ran node
— and both npm --version and npm config get prefix
succeed on a totally broken npm. Nothing exercised the module graph.

npm_bundled_deps_load requires minipass-flush and cacache (the module that
actually threw). Added to both node and node-lts.

Confirmed falsifiable rather than assumed — same tree, one directory apart:

BROKEN cacache rc=1      <- test fails
FIXED  cacache rc=0      <- test passes

I first wrote this as an npm install test; it returned exit 254 with no output
because the standalone harness doesn't persist /tmp between cmds. The
module-load form needs no network, no cache, no writable $HOME, and no
cross-command state.

Verification

Local, in the Linux sandbox, from a clean origin/main worktree:

  • min package build --rebuild node-lts → exit 0
  • min check --packages node-lts → 15/15 Pass, including standalone tests
  • min package build --rebuild node + min check --packages node → see below

Not fixed here

next declares node but builds against node-lts. Both are in its
closure (pnpm carries node-lts in runtime_deps for its #!/usr/bin/env node
shebang), both ship /usr/bin/node, and the sandbox hardlinks first-writer-wins
in content-hash order — so the declaration is never consulted and the winner can
flip on any rebuild. res-server-amd64 logged 4,010 such silent
Not linking … already exists collisions in one day.

This PR fixes the npm that next happens to land on. It does not fix a package
building against a toolchain it did not declare. That deserves its own issue
against minimal.

🤖 Generated with Claude Code

…very install

Upstream node-v24.19.0 bundles npm 11.17.0 with a self-inconsistent dependency
tree: minipass-flush imports minipass by its 7.x NAMED export but ships a
nested minipass@3.3.6 (default export only) that shadows the hoisted 7.1.3.
`class Flush extends undefined` throws, so EVERY `npm install` fails — global
or local, prefix or not.

This has failed `next` since 2026-08-27 and currently blocks #665, #667, #669
and #659. With `--prefix` on the command line npm dies during config
resolution before it can print an error, so the build's whole stderr is a
pointer to a debug log that is itself empty of errors — it reads like a network
fault, but reproduces offline with a dependency-free package.json.

Remove the shadowing copy, guarded so the build fails loudly if upstream
changes shape rather than silently carrying a dead workaround.

Add `npm_bundled_deps_load` to node AND node-lts. This shipped silently because
every existing test only ran `node`, and both `npm --version` and
`npm config get prefix` succeed on a completely broken npm. Confirmed
falsifiable: same tree one directory apart, cacache require rc=1 vs rc=0.

Verified locally in the sandbox: node-lts and node each build --rebuild clean
and pass min check 15/15, including standalone tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bryan-minimal

Copy link
Copy Markdown
Member Author

Closing — the diagnosis behind this PR was wrong, and the fix is a no-op.

What I got wrong

I claimed upstream node-lts 24.19.0 bundles an npm with a self-inconsistent
dependency tree. It does not. Measured against the real store artifacts on
res-server-arm64:

tree minipass-flush import style nested minipass
node-lts npm 11.17.0 1.0.6 const { Minipass } = require(…) none — resolves hoisted 7.1.3
node 25.8.2 npm 11.11.1 1.0.5 const Minipass = require(…) 3.3.6 — correct for 1.0.5
the failing next rootfs — const { Minipass } = 3.3.6

Both packages are internally consistent and work standalone. The merge is
what breaks: node and node-lts both ship /usr/lib/node_modules/npm/**, and
the sandbox hardlinks closure members first-writer-wins per file. So the
rootfs got index.js from npm 11.17.0 and the nested node_modules/minipass
from npm 11.11.1 — a tree that has never existed upstream and cannot work.

Why this PR cannot help

The removal is guarded by if [ -d "$NPM_NM/minipass-flush/node_modules/minipass" ]
evaluated against node-lts's own $OUTPUT_DIR, which has no such directory.
The branch never fires. That is exactly why next still failed on arm64 with
this branch checked out.

The amd64 leg going green on this run was not this fix either — no next
sandbox was created there, and the hash-ordered link race simply landed
differently on that arch.

My error was doing the A/B on a merged sandbox rootfs and attributing the
result to node-lts. Removing that directory did fix npm in the merged tree,
which is true and misleading: it repairs the blend, not the package.

The npm_bundled_deps_load test has the same flaw — a standalone test sandbox
contains only node-lts, where npm is genuinely fine, so it passes honestly and
proves nothing about the real defect.

The actual bug

Two packages own the same paths, and nothing objects. res-server-amd64 logged
4,010 Not linking … already exists warnings in a single day; this is one
of them turning into a two-day outage.

Superseded by a fix at the composition level — keeping a second npm out of
next's closure, and (separately, in minimal) making a same-path collision
between two different packages an error rather than a warn!.

cvince pushed a commit to capysc/pkgs that referenced this pull request Oct 2, 2026
…ak each other (gominimal#671)

* fix(pnpm): depend on node, not node-lts — two npms in one sandbox breaks both

pnpm only needs *a* node for its `#!/usr/bin/env node` shebang, but whichever
one it names enters the closure of every package built with pnpm. `next` and
`agent-browser` both declare `node` themselves, so each sandbox held TWO npm
installs at the same paths.

The sandbox hardlinks closure members first-writer-wins PER FILE, so
/usr/lib/node_modules/npm came out a blend: minipass-flush/index.js from npm
11.17.0 (named `{ Minipass }` import) beside a nested minipass@3.3.6 from npm
11.11.1 (default export only). That tree never existed upstream and cannot
load — `class Flush extends undefined` throws and every `npm install` dies.
It broke `next` for two days and blocked gominimal#665, gominimal#667, gominimal#669 and gominimal#659.

Verified: closure of next 75->74 and agent-browser 114->113 with node-lts gone,
capy (the only other node-lts consumer) untouched; next, pnpm and agent-browser
each build --rebuild clean and pass min check 15/15.

Supersedes gominimal#670, whose fix was a no-op against node-lts's own output.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refine: take only the node BINARY, not the whole package

Review point from Bryan: pnpm should not be dictating a node version to its
consumers at all. The spec language cannot express that today — RuntimeDep is
BuildSpec | Subset | UpstreamPkg, all concrete, with no "consumer provides it"
form — but `subsetOf` gets most of the way there.

pnpm needs a node to interpret `#!/usr/bin/env node` and nothing else. Taking
`subsetOf node ["node"]` instead of the whole package means pnpm can never
contribute an npm tree to any closure, so the two-npms-merged-per-file failure
is structurally impossible rather than merely absent from today's fleet.

Residual, and worth stating: pnpm still names a version, so a future node-lts
consumer adding pnpm would collide on the `node` binary. That is a much smaller
failure than an unloadable module graph, and the real fix is a virtual/provided
dependency in the spec language.

Verified with the subset: next, pnpm and agent-browser all build --rebuild
clean and pass min check, including pnpm's `missing runtime_deps` — which
independently confirms the bare binary satisfies the shebang.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant