feat(dependency-resolver): opt-in support for pnpm's global virtual store

[zkochan]

Opt-in support for pnpm's global virtual store: dependency directories are created once in a store shared by every workspace and capsule on the machine, instead of being re-created inside each node_modules/.pnpm.

Draft because of one remaining engine dependency: pnpm/pnpm#13669. The settings surface (pnpm/pnpm#13648) and the link: slot fix (pnpm/pnpm#13658) are released in @pnpm/napi 12.0.0-rc.0, which this branch now pins — but rc.0 still carries a race in slot materialization that #13669 fixes: peer variants of an injected component whose dependency hashes are equal share one slot by design, and the link pass raced one forced import per variant against the same directory (failed to remove existing directory ... prior to swap). The bit repo hits it on every install. Until a release carries it, scripts/link-local-pnpm-engine.sh installs a locally built engine over the published one.

Effect

After a fresh install the project-local virtual store holds nothing but lock.yaml and the hoisted node_modules — every dependency directory, including injected workspace components, lives in the shared store:

workspace	node_modules/.pnpm entries, no GVS	with GVS
a Bit Cloud workspace	20832	0
the bit repo itself	6441	0
bitdev-envs	4017	0

Bit uses pnpm's own <storeDir>/links, so slots are reused across bit versions and shared with every other pnpm project on the machine, upgrades stay incremental, and pnpm store prune accounts for them.

Migration note: switching an existing workspace over does not remove what the previous project-local store already contains — pnpm does not prune it once the effective virtual store points elsewhere — so the old directories linger until node_modules/.pnpm is deleted. New workspaces never see them.

Turning it on

Off by default. Per workspace under teambit.dependencies/dependency-resolver, machine-wide with bit config set enable_global_virtual_store true, or per run with BIT_ENABLE_GLOBAL_VIRTUAL_STORE=true — which is how the e2e suite runs the whole matrix under it.

What it took to behave identically

A shared store breaks one assumption that runs through Bit: that a package can walk up out of node_modules/.pnpm into the workspace. Consequences:

• Core aspects (CJS). Bit deliberately does not declare @teambit/* core aspects as dependencies of the envs and aspects that use them — they have to be the single copy from the running installation, and declaring them would install duplicates. Project-locally they resolved by walking up into the workspace root; nothing in a shared store can. This branch uses pnpm's documented mechanism for hoisted dependencies in this layout: the privately hoisted directory stays project-local at node_modules/.pnpm/node_modules even under the global virtual store, the core aspects are linked there, and bootstrap puts that directory on NODE_PATH before any aspect loads. Nothing is written inside the store itself, which is what allows the shared <storeDir>/links to be used at all.
• Core aspects (ESM). Node ignores NODE_PATH for ESM, and bit supports ESM components, so the same hoisted-directory fallback is provided to the ESM resolver via a module.register() loader (adapted from pnpm's @pnpm/plugin-esm-node-path, MIT), installed at bootstrap and propagated to child processes through NODE_OPTIONS.
• Phantom tsconfig extends in published envs. TypeScript resolves extends with its own resolver — neither NODE_PATH nor the ESM loader applies — and several published envs extend a package they never declare (@teambit/react/typescript/tsconfig.json from the node mocha envs, base-react-env, rspack.envs.react-env and react-env 1.x; rspack.envs.react-env/config/tsconfig.json from cloud-react). Project-locally the walk up out of .pnpm passed the privately hoisted directory and caught these by accident. Fixed with pnpm's documented remedy: built-in packageExtensions declaring the missing dependency for the known envs, merged under the workspace's own packageExtensions so users can cover third-party envs the same way. Extensions only add what a manifest lacks, an unmatched entry is not an error (react-env's is scoped to @1 — 2.x inlined the config), and none of these envs import their extends-target at runtime, so the added copy cannot shadow the host-provided singletons. (A patchedDependencies-based fix for react-env 1.x moved to fix(dependency-resolver): patch react-env's tsconfig extends for the global virtual store #10572; it stays blocked until @pnpm/napi exposes allowUnusedPatches.)
• The bit repo itself. A workspace that authors the core aspects bridges them into the hoisted store only until they compile, then removes the bridge. Right project-locally, fatal from a shared store. The bridge now persists, repointed at the compiled workspace copy.
• Injected dirs. getInjectedDirs reports an injected copy absolute when it cannot be made relative to the lockfile dir — under a shared store, always. compiler.task.ts joined it onto the capsule root regardless, hard-linking build artifacts into a path that does not exist. (node-modules-linker and workspace-compiler already guarded this; only that one site did not.)
• Capsules stay project-local. A capsule is a self-contained build sandbox and TypeScript's declaration emit depends on that: it names an inferred type through the package that declares it, which it can only do while that package sits inside the capsule. Moving env/aspect packages out made bit build of any custom env fail with TS2742. Workspace installs — the ones users wait on repeatedly — still get the shared store.

Self-hosting: a workspace whose node_modules provides the running bit

Both layouts are safe in steady state: repeat installs preserve the top-level package directories the running process resolves from (verified by sampling them through live installs), and the end-of-install compile refreshes the per-variant copies. A layout switch is different — it relocates every injected component package, rebuilding the top-level directories from source, so their compiled dist disappears mid-install and a bit running from them dies before reaching the compile that would restore it, leaving node_modules unusable. The installer now detects the combination (current layout read off .modules.yaml, self-hosting by realpath of the running module) and refuses up front with instructions to run the one-time transition from an external bit installation.

With that and the engine fix, the bit repo itself runs under the global virtual store: full install + compile of all 332 components, repeat self-hosted installs in ~50s, npm run lint green. The repo's own tsc needed the type-layer counterpart of the runtime bridges — tsconfig.json paths routing react typings and @teambit/* through the workspace's node_modules, since a published package's .d.ts cannot reach undeclared @types/core aspects from a store slot either.

Phantom dependencies, generally

A shared store makes an under-declared package fail loudly: it lives outside the project, so nothing the project hoisted is on its resolution path any more. That is a real constraint, not a bug to work around, and it applies to pnpm generally rather than to Bit. Every failure this branch hit was a genuine phantom dependency — an env's undeclared tsconfig base, a fixture env with detection switched off (@bit-no-check removed so the dependency gets declared), @teambit/webpack.webpack-bundler requiring events/ without declaring it (covered by packageExtensions, documented in the aspect docs), and the repo's own undeclared @types/graceful-fs (now declared).

Testing

• The bit repo (332 components): full global-virtual-store install, repeat self-hosted installs, transition guard verified in both directions (clean refusal, workspace intact), lint green. Runs on the released rc.0 engine plus the #13669 fix.
• A Bit Cloud workspace, 105 components: wiped-node_modules install + compile, slot reuse verified (a repeat install creates 0 of 30766 slots), core-aspect resolution through NODE_PATH verified positively and negatively, bit start working.
• bitdev-envs (609 components, 20 apps): bit app list byte-identical to the non-GVS baseline, apps running — against an earlier revision of the core-aspect mechanism.
• The full e2e suite under BIT_ENABLE_GLOBAL_VIRTUAL_STORE=true was last run against an earlier revision (net result then: no GVS-specific failures; the validate command failures reproduce with the store off). CI re-runs it against the current branch.

The first commit (fix(doctor)) is unrelated to the rest and can be split out — it surfaced because the e2e temp dir contains a dot, which is why doctor-infra.e2e.ts fails locally but passes on CI.

[zkochan]

Updated for the pnpm side dropping the directory-resolution change (pnpm/pnpm#13648).

The pnpm PR originally moved injected/file: packages out of the shared store so they could still reach what the project hoisted. That was the wrong fix — reaching a package you never declared is a phantom dependency, and the answer is to declare it, not to move the package. So that commit is gone, injected components stay in the shared store as upstream intends, and this branch had to stand on its own.

It does. Re-verified with an engine carrying only the two remaining pnpm commits:

workspace	result
bitdev-envs (609 components, 20 apps)	installs + compiles, bit app list unchanged
a Bit Cloud workspace (105 components)	installs + compiles
the bit repo itself (332 components)	installs + compiles

The single case that did fail was a test fixture, and it was a genuine phantom dependency of our own making: my-react-env.bit-env.ts opened with // @bit-no-check, which switches Bit's dependency detection off for the file, so @teambit/react.react-env never reached the component's manifest. It only ever resolved by reaching a hoisted copy. Removing that line — @ts-nocheck, which is what the tsconfig scenario actually needs, stays — makes Bit detect the import, declare it, and the env loads under both layouts. install-and-compile.e2e.ts is 6/6 with the store on and off.

Worth stating plainly: none of the real env components in those workspaces carry @bit-no-check, and Bit's dependency detection was never at fault.

The core-aspect mirroring in this branch is a different thing and stays. Bit deliberately does not declare @teambit/* core aspects as dependencies — they have to be the single copy from the running installation, and declaring them would install duplicates. They are provided by the host, like a peer, and the workspace root node_modules is how that is done today. Mirroring the same links where the shared store can reach them is the direct analogue, not a workaround.