diff --git a/docs/remote-bridge/README.md b/docs/remote-bridge/README.md index 464e4763..a37745a2 100644 --- a/docs/remote-bridge/README.md +++ b/docs/remote-bridge/README.md @@ -307,7 +307,10 @@ execution. - Code API negotiates a bounded number of active workspace assignments per worker. The lower API or worker slot ceiling wins, and assignments sharing the same workspace isolation key remain serialized while independent - conversation worktrees may run concurrently. + conversation worktrees may run concurrently. A linked-worktree lane + (`worktree: `) nests beneath its checkout's key: sibling lanes run + concurrently, while a lane and its checkout exclude each other, and a lane + cannot start while its checkout is quarantined. - Workspace tool admission waits for capacity up to 30 seconds without a `X-LibreChat-Workspace-Queue-Wait-Ms` header. A caller can advertise a longer per-request allowance, bounded by five minutes and any server queue ceiling. diff --git a/docs/remote-bridge/projects.md b/docs/remote-bridge/projects.md index 987ce112..26887ee9 100644 --- a/docs/remote-bridge/projects.md +++ b/docs/remote-bridge/projects.md @@ -41,7 +41,11 @@ workspace registration remains available for independent project directories. filesystem boundary and coordination for the common Git directory. - A worktree beneath its parent checkout overlaps that checkout. Either use disjoint execution roots under a discovery grant or explicitly exclude and - coordinate descendant worktrees before relaxing root exclusion. + coordinate descendant worktrees before relaxing root exclusion. Linked + worktree lanes (`--linked-worktree-lanes`) take the second approach for + `.worktrees/`: hierarchical admission keeps a lane and its checkout + exclusive, and each lane's sandbox can write only its worktree, shared + object and ref storage, and its own worktree metadata. - Setup and dependency links must remain within the execution policy. Sharing writable dependency directories between supposedly isolated worktrees reintroduces overlap and requires an explicit operator decision. diff --git a/packages/code/README.md b/packages/code/README.md index 95d86485..e3051930 100644 --- a/packages/code/README.md +++ b/packages/code/README.md @@ -791,6 +791,57 @@ Legacy requests without a conversation identity continue to use the selected source root. Older Code API deployments do not negotiate the capability, so the worker omits it until every request path understands the isolation boundary. +#### Linked worktree lanes + +Agents that keep one checkout and give each task its own linked worktree +(`git worktree add .worktrees/`) can run those tasks concurrently: + +```sh +librechat-code run \ + --worker-dir /projects/LibreChat \ + --workspace-lease-slots 4 \ + --linked-worktree-lanes \ + --allow-workspace-writes \ + --allow-workspace-commands +``` + +`LIBRECHAT_CODE_LINKED_WORKTREE_LANES=true` is the environment equivalent. The +worker then advertises `workspaceScopes: ['git_linked_worktree']` for each +registered root, and a request that names `worktree: ` runs in its own +lane at `/.worktrees/`, with `cwd` and file paths relative to that +worktree. Sibling lanes run concurrently up to the negotiated slot count. A +lane and its checkout never run at the same time: requests without a +`worktree`, including `git worktree add` or `remove` run at the root, wait for +every lane beneath the checkout, and a waiting root request holds back newer +lanes so it cannot be starved. + +Before admission the worker verifies, without running Git, that the directory +is a real linked worktree of that checkout: no symlinks on the path, a `.git` +file pointing at `/.git/worktrees/`, and metadata whose `commondir` +and `gitdir` point back. Shared Git storage paths granted for writes must be +real directories, not symlinks into sibling metadata; optional log and LFS +paths may be absent. A lane's sandbox can write only its worktree, the +shared object and ref storage (`.git/objects`, `.git/refs`, `.git/logs/refs`, +`.git/lfs`) and its own `.git/worktrees/` metadata. Everything else in +`.git` stays read-only: configuration, hooks and `info`, the checkout's own +`HEAD`, index and merge or rebase state, and sibling metadata. Automatic `gc` +and maintenance are disabled, and `git gc` itself cannot run in a lane (it +needs to write `.git/gc.pid` and `packed-refs`); run storage maintenance from +the checkout. Explicit `git prune --expire now`, `git repack -ad`, or LFS +pruning can still delete objects another lane is writing; the sandbox does +**not** prevent this race. Do not enable lanes for agents that run destructive +maintenance until those commands are blocked or serialized. Deleting a branch +or tag also needs the checkout, because Git locks `packed-refs` for every ref +deletion. Each lane has its own durable quarantine guard. A lane +cannot start while its checkout is quarantined, and a checkout cannot start +while any lane beneath it is. + +Lanes require native-srt commands and at least two lease slots, and cannot yet +be combined with conversation worktrees. Code API must advertise +`supportedWorkspaceScopes`; older deployments do not, and the worker omits the +scope for them. Deploy consumers that read worker status (such as LibreChat) +with support for `workspaceScopes` before enabling lanes on a worker. + On an updated Code API, admission waits up to 30 seconds without the `X-LibreChat-Workspace-Queue-Wait-Ms` request header. A caller may advertise a positive integer millisecond allowance up to five minutes, capped by any server @@ -850,6 +901,11 @@ To recover a quarantined native root: 3. Run the normal worker command with all its root/slot options plus `--reset-workspace-quarantine second`. This verifies the local guard is cleared, resets the server fence, then exits. 4. Restart the normal worker command without the reset option. +A linked-worktree lane keeps its own guard and fence. Inspect or restore +`/.worktrees/`, clear its guard with +`--worker-dir /.worktrees/`, then add +`--reset-workspace-worktree ` to the reset command in step 3. + The workspace selector in LibreChat must preserve these registered IDs. Adding roots here does not grant a principal access or change an agent's selected root. # Named project environments diff --git a/packages/code/src/cli.ts b/packages/code/src/cli.ts index 6e3a0062..135b9f8e 100644 --- a/packages/code/src/cli.ts +++ b/packages/code/src/cli.ts @@ -1,8 +1,8 @@ #!/usr/bin/env node import { createHash, createHmac, randomBytes } from 'node:crypto'; import { readFileSync } from 'node:fs'; -import { realpath, stat } from 'node:fs/promises'; -import { basename, resolve, relative, isAbsolute, sep } from 'node:path'; +import { readdir, realpath, stat } from 'node:fs/promises'; +import { basename, join, resolve, relative, isAbsolute, sep } from 'node:path'; import { pairBridgeWorker } from './pairing.js'; import { discoverProjects } from './projects.js'; @@ -37,6 +37,7 @@ import { RuntimeWorkspaceCommandSandbox } from './workspace-runtime.js'; import { NativeProcessWorkspaceCommandSandbox } from './native-process.js'; import { NativeWorkspaceCommandPool } from './native-pool.js'; import { GitWorktreeWorkspaceTools, internalWorkspaceId } from './workspace-instances.js'; +import { LINKED_WORKTREE_DIRECTORY, LinkedWorktreeWorkspaceTools } from './linked-worktrees.js'; import { GitWorktreeManager } from './worktrees.js'; import { captureWorkspaceRootIdentity } from './root-identity.js'; import { @@ -66,6 +67,7 @@ import { BridgeProtocolError, isValidBridgeWorkerCapabilities, isValidBridgeWorkerId, + isValidLinkedWorktreeName, workspaceIsolationKey, } from './protocol.js'; @@ -514,6 +516,11 @@ async function run( (args.includes('--allow-workspace-commands') || process.env.LIBRECHAT_CODE_ALLOW_WORKSPACE_COMMANDS?.trim().toLowerCase() === 'true'); + const linkedWorktreeLanes = + runtimeSessionId == null && + (args.includes('--linked-worktree-lanes') || + process.env.LIBRECHAT_CODE_LINKED_WORKTREE_LANES?.trim().toLowerCase() === + 'true'); const commandSandboxMode = option(args, '--command-sandbox') ?? process.env.LIBRECHAT_CODE_COMMAND_SANDBOX?.trim().toLowerCase() ?? @@ -783,6 +790,21 @@ async function run( 'Conversation worktrees require native-srt commands and at least two workspace lease slots', ); } + if ( + linkedWorktreeLanes && + (!allowWorkspaceCommands || + commandSandboxMode !== 'native-srt' || + workspaceLeaseSlots < 2) + ) { + throw new Error( + 'Linked worktree lanes require native-srt commands and at least two workspace lease slots', + ); + } + if (linkedWorktreeLanes && conversationWorktreeRoot) { + throw new Error( + 'Linked worktree lanes cannot be combined with conversation worktrees', + ); + } if ( roots.length > 1 && process.env.LIBRECHAT_CODE_WORKSPACE_QUARANTINE_FILE?.trim() @@ -1073,7 +1095,7 @@ async function run( }; const nativeCommandSandbox = allowWorkspaceCommands && commandSandboxMode === 'native-srt' - ? roots.length > 1 || workspaceLeaseSlots > 1 || conversationWorktreeRoot + ? roots.length > 1 || workspaceLeaseSlots > 1 || conversationWorktreeRoot || linkedWorktreeLanes ? new NativeWorkspaceCommandPool( new Map( roots.map(root => [ @@ -1199,6 +1221,41 @@ async function run( ), }); workspaceTools = conversationWorkspaceTools; + } + let linkedWorktreeTools: LinkedWorktreeWorkspaceTools | undefined; + if (linkedWorktreeLanes && workspaceTools) { + if (!(nativeCommandSandbox instanceof NativeWorkspaceCommandPool)) { + throw new Error('Linked worktree lanes require a native command pool'); + } + linkedWorktreeTools = new LinkedWorktreeWorkspaceTools({ + commandPool: nativeCommandSandbox, + delegate: workspaceTools, + programmaticDelegate: conversationWorkspaceTools ?? nativeCommandSandbox, + onResolve(workspaceId, root) { + if (admittedGitHubRepositories) { + admittedGitHubRepositories.set( + root, + repositoriesByWorkspace?.get(workspaceId), + ); + } + }, + onRelease(root) { + admittedGitHubRepositories?.delete(root); + }, + sources: new Map( + roots.map((root) => [ + root.id, + { + root: root.root, + identity: root.identity, + command: nativeOptions, + repositoryInstructions: args.includes('--repository-instructions'), + writable: root.writable ?? false, + }, + ]), + ), + }); + workspaceTools = linkedWorktreeTools; } if (workspaceTools && environments.length) { workspaceTools = new EnvironmentWorkspaceTools( @@ -1304,7 +1361,7 @@ async function run( ...(nativeProgrammaticEnabled && nativeCommandSandbox ? { workspaceProgrammatic: - conversationWorkspaceTools ?? nativeCommandSandbox, + linkedWorktreeTools ?? conversationWorkspaceTools ?? nativeCommandSandbox, } : {}), ...(conversationWorktrees @@ -1331,6 +1388,40 @@ async function run( ), } : {}), + ...(linkedWorktreeTools + ? { + linkedWorktreeQuarantineResolver: ( + selectedWorkspaceId: string, + worktree: string, + ) => { + const source = roots.find((root) => root.id === selectedWorkspaceId); + if (!source) { + throw new BridgeProtocolError('Linked worktree source is not registered'); + } + return workspaceMutationGuard( + defaultWorkspaceQuarantinePath({ + codeApiUrl, + workerId, + workspaceRoot: join(source.root, LINKED_WORKTREE_DIRECTORY, worktree), + }), + workerId, + workspaceIsolationKey(selectedWorkspaceId, undefined, worktree), + incarnationId, + ); + }, + linkedWorktreeNames: async (selectedWorkspaceId: string) => { + const source = roots.find((root) => root.id === selectedWorkspaceId); + if (!source) return []; + try { + const entries = await readdir(join(source.root, LINKED_WORKTREE_DIRECTORY)); + return entries.filter(isValidLinkedWorktreeName); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return []; + throw error; + } + }, + } + : {}), ...(workspaceLeaseSlots > 1 || roots.length > 1 ? { workspaceQuarantines: new Map( @@ -1445,15 +1536,20 @@ async function run( args, '--reset-workspace-instance', ); + const resetWorkspaceWorktree = option( + args, + '--reset-workspace-worktree', + ); await worker.refreshCredential(controller.signal); await worker.registerForMaintenance(controller.signal); await worker.resetNativeWorkspace( resetNativeRoot, controller.signal, resetWorkspaceInstance, + resetWorkspaceWorktree, ); process.stdout.write( - `librechat-code: reset acknowledged for native workspace ${resetNativeRoot}${resetWorkspaceInstance ? ` instance ${resetWorkspaceInstance}` : ''}\n`, + `librechat-code: reset acknowledged for native workspace ${resetNativeRoot}${resetWorkspaceInstance ? ` instance ${resetWorkspaceInstance}` : ''}${resetWorkspaceWorktree ? ` worktree ${resetWorkspaceWorktree}` : ''}\n`, ); return; } diff --git a/packages/code/src/linked-worktrees.test.ts b/packages/code/src/linked-worktrees.test.ts new file mode 100644 index 00000000..53ec156e --- /dev/null +++ b/packages/code/src/linked-worktrees.test.ts @@ -0,0 +1,294 @@ +import assert from 'node:assert/strict'; +import { execFile } from 'node:child_process'; +import { + mkdir, + mkdtemp, + readFile, + realpath, + rm, + stat, + symlink, + writeFile, +} from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { promisify } from 'node:util'; +import test from 'node:test'; + +import { + LinkedWorktreeWorkspaceTools, + linkedWorktreeWorkspaceId, + verifyLinkedWorktree, +} from './linked-worktrees.js'; +import { LocalWorkspaceTools, WorkspaceToolError } from './workspace.js'; + +import type { NativeWorkspaceCommandPool } from './native-pool.js'; +import type { NativeProcessSandboxOptions } from './native-process.js'; +import type { WorkspaceToolRequest } from './protocol.js'; + +const execFileAsync = promisify(execFile); + +async function git(cwd: string, ...args: string[]): Promise { + await execFileAsync( + 'git', + ['-c', 'user.name=test', '-c', 'user.email=test@example.com', '-c', 'commit.gpgsign=false', ...args], + { cwd }, + ); +} + +async function checkout(): Promise<{ parent: string; root: string }> { + const parent = await realpath(await mkdtemp(join(tmpdir(), 'linked-worktree-'))); + const root = join(parent, 'repo'); + await mkdir(root); + await git(root, 'init', '-q', '-b', 'main'); + await writeFile(join(root, 'README.md'), 'root\n'); + await git(root, 'add', '.'); + await git(root, 'commit', '-q', '-m', 'init'); + await mkdir(join(root, '.worktrees')); + await git(root, 'worktree', 'add', '-q', '-b', 'task-a', '.worktrees/task-a'); + return { parent, root }; +} + +async function rejects(promise: Promise, code = 'INVALID_REQUEST'): Promise { + await assert.rejects(promise, (error: unknown) => error instanceof WorkspaceToolError && error.code === code); +} + +test('verifies a linked worktree of the checkout and lists the only Git paths it may write', async (t) => { + const { parent, root } = await checkout(); + t.after(() => rm(parent, { recursive: true, force: true })); + await git(root, 'worktree', 'add', '-q', '-b', 'task-b', '.worktrees/task-b'); + + const lane = await verifyLinkedWorktree(root, 'task-a'); + + assert.equal(lane.root, join(root, '.worktrees', 'task-a')); + assert.equal(lane.checkoutRoot, root); + assert.equal(lane.commonGitDir, join(root, '.git')); + assert.deepEqual(lane.writableGitPaths, [ + join(root, '.git', 'objects'), + join(root, '.git', 'refs'), + join(root, '.git', 'logs', 'refs'), + join(root, '.git', 'lfs'), + join(root, '.git', 'worktrees', 'task-a'), + ]); +}); + + +test('rejects directories that are not linked worktrees of this checkout', async (t) => { + const { parent, root } = await checkout(); + t.after(() => rm(parent, { recursive: true, force: true })); + + await mkdir(join(root, '.worktrees', 'plain')); + await rejects(verifyLinkedWorktree(root, 'plain')); + + await mkdir(join(root, '.worktrees', 'borrowed')); + await writeFile( + join(root, '.worktrees', 'borrowed', '.git'), + `gitdir: ${join(root, '.git', 'worktrees', 'task-a')}\n`, + ); + await rejects(verifyLinkedWorktree(root, 'borrowed')); + + await symlink(join(root, '.worktrees', 'task-a'), join(root, '.worktrees', 'alias')); + await rejects(verifyLinkedWorktree(root, 'alias')); + + const other = join(parent, 'other'); + await mkdir(other); + await git(other, 'init', '-q', '-b', 'main'); + await writeFile(join(other, 'file'), 'x\n'); + await git(other, 'add', '.'); + await git(other, 'commit', '-q', '-m', 'other'); + await git(other, 'worktree', 'add', '-q', '-b', 'foreign', join(root, '.worktrees', 'foreign')); + await rejects(verifyLinkedWorktree(root, 'foreign')); + + for (const name of ['..', '.git', 'a/b', 'task.lock']) { + await rejects(verifyLinkedWorktree(root, name)); + } + await rejects(verifyLinkedWorktree(root, 'missing')); +}); + +test('rejects a checkout whose .worktrees directory is a symlink', async (t) => { + const { parent, root } = await checkout(); + t.after(() => rm(parent, { recursive: true, force: true })); + const elsewhere = join(parent, 'elsewhere'); + await git(root, 'worktree', 'move', '.worktrees/task-a', elsewhere); + await rm(join(root, '.worktrees'), { recursive: true }); + await symlink(parent, join(root, '.worktrees')); + await rejects(verifyLinkedWorktree(root, 'elsewhere')); +}); + +test('rejects shared Git storage symlinks into sibling worktree metadata', async (t) => { + const { parent, root } = await checkout(); + t.after(() => rm(parent, { recursive: true, force: true })); + await git(root, 'worktree', 'add', '-q', '-b', 'task-b', '.worktrees/task-b'); + const commonGitDir = join(root, '.git'); + const siblingMetadata = join(commonGitDir, 'worktrees', 'task-b'); + await rm(join(commonGitDir, 'objects'), { recursive: true }); + await symlink(siblingMetadata, join(commonGitDir, 'objects')); + + await rejects(verifyLinkedWorktree(root, 'task-a')); +}); + +function recordingPool(): { + pool: NativeWorkspaceCommandPool; + calls: Array<{ action: string; id: string; options?: NativeProcessSandboxOptions }>; +} { + const calls: Array<{ action: string; id: string; options?: NativeProcessSandboxOptions }> = []; + const pool = { + async registerRoot(id: string, options: NativeProcessSandboxOptions) { + calls.push({ action: 'register', id, options }); + }, + async unregisterRoot(id: string) { + calls.push({ action: 'unregister', id }); + }, + async execute(request: WorkspaceToolRequest) { + calls.push({ action: 'execute', id: request.workspaceId }); + return { + protocolVersion: 1, + operation: 'execute_command', + workspaceId: request.workspaceId, + exitCode: 0, + stdout: '', + stderr: '', + truncated: false, + timedOut: false, + }; + }, + async executeProgrammatic(id: string) { + calls.push({ action: 'programmatic', id }); + return {}; + }, + } as unknown as NativeWorkspaceCommandPool; + return { pool, calls }; +} + +async function laneTools( + root: string, + pool?: NativeWorkspaceCommandPool, + onRelease?: (root: string) => void, +) { + const delegate = await LocalWorkspaceTools.create({ + repositoryInstructions: false, + workspaces: [{ id: 'repo', root, writable: true }], + }); + return new LinkedWorktreeWorkspaceTools({ + commandPool: pool, + delegate, + onRelease, + sources: new Map([ + [ + 'repo', + { + root, + command: { workspaceRoot: root } as NativeProcessSandboxOptions, + repositoryInstructions: false, + writable: true, + }, + ], + ]), + }); +} + +test('lane file tools are confined to the worktree and report the public workspace', async (t) => { + const { parent, root } = await checkout(); + t.after(() => rm(parent, { recursive: true, force: true })); + const tools = await laneTools(root); + + assert.deepEqual(tools.capabilities.workspaces[0]?.workspaceScopes, ['git_linked_worktree']); + const written = await tools.execute({ + protocolVersion: 1, + operation: 'write_file', + workspaceId: 'repo', + worktree: 'task-a', + path: 'lane.txt', + content: 'from the lane\n', + }); + assert.equal(written.workspaceId, 'repo'); + assert.equal(await readFile(join(root, '.worktrees', 'task-a', 'lane.txt'), 'utf8'), 'from the lane\n'); + await assert.rejects(stat(join(root, 'lane.txt'))); + + const read = await tools.execute({ + protocolVersion: 1, + operation: 'read_file', + workspaceId: 'repo', + path: 'README.md', + }); + assert.equal(read.operation === 'read_file' && read.content.trimEnd(), 'root'); + + await rejects( + tools.execute({ + protocolVersion: 1, + operation: 'read_file', + workspaceId: 'repo', + worktree: 'task-a', + workspaceInstanceId: 'a'.repeat(64), + path: 'README.md', + }), + ); +}); + +test('lane commands register a confined root once, whatever siblings come and go', async (t) => { + const { parent, root } = await checkout(); + t.after(() => rm(parent, { recursive: true, force: true })); + const { pool, calls } = recordingPool(); + const tools = await laneTools(root, pool); + const command = { + protocolVersion: 1 as const, + operation: 'execute_command' as const, + workspaceId: 'repo', + worktree: 'task-a', + command: 'git status', + }; + const id = linkedWorktreeWorkspaceId('repo', 'task-a'); + + const result = await tools.execute(command); + await tools.execute(command); + assert.equal(result.workspaceId, 'repo'); + assert.deepEqual( + calls.map((call) => call.action), + ['register', 'execute', 'execute'], + ); + const registered = calls[0]!.options!; + assert.equal(calls[0]!.id, id); + assert.equal(registered.workspaceRoot, join(root, '.worktrees', 'task-a')); + assert.equal(registered.linkedWorktree?.commonGitDir, join(root, '.git')); + assert.equal(registered.linkedWorktree?.checkoutRoot, root); + + assert.ok(!registered.linkedWorktree?.writableGitPaths.includes(join(root, '.git'))); + + await rejects(tools.execute({ + ...command, + environmentAction: { name: 'unresolved', fingerprint: 'a'.repeat(64) }, + })); + assert.equal(calls.length, 3, 'an unresolved action must never reach the command pool'); + + await git(root, 'worktree', 'add', '-q', '-b', 'task-b', '.worktrees/task-b'); + await tools.execute(command); + assert.deepEqual( + calls.slice(3).map((call) => call.action), + ['execute'], + ); +}); + +test('a removed lane releases its command root and credential route', async (t) => { + const { parent, root } = await checkout(); + t.after(() => rm(parent, { recursive: true, force: true })); + const { pool, calls } = recordingPool(); + const released: string[] = []; + const tools = await laneTools(root, pool, (lane) => released.push(lane)); + const command = { + protocolVersion: 1 as const, + operation: 'execute_command' as const, + workspaceId: 'repo', + worktree: 'task-a', + command: 'git status', + }; + await tools.execute(command); + + await git(root, 'worktree', 'remove', '.worktrees/task-a'); + await rejects(tools.execute(command)); + + assert.deepEqual( + calls.map((call) => call.action), + ['register', 'execute', 'unregister'], + ); + assert.deepEqual(released, [join(root, '.worktrees', 'task-a')]); +}); diff --git a/packages/code/src/linked-worktrees.ts b/packages/code/src/linked-worktrees.ts new file mode 100644 index 00000000..e049aa6e --- /dev/null +++ b/packages/code/src/linked-worktrees.ts @@ -0,0 +1,386 @@ +import { createHash } from 'node:crypto'; +import { lstat, open, realpath } from 'node:fs/promises'; +import { isAbsolute, join, resolve } from 'node:path'; + +import { isValidLinkedWorktreeName } from './protocol.js'; +import { + captureWorkspaceRootIdentity, + matchesWorkspaceRoot, +} from './root-identity.js'; +import { LocalWorkspaceTools, WorkspaceToolError } from './workspace.js'; + +import type { NativeWorkspaceCommandPool } from './native-pool.js'; +import type { NativeProcessSandboxOptions } from './native-process.js'; +import type { WorkspaceRootIdentity } from './root-identity.js'; +import type { + BridgeWorkspaceProgrammaticRequest, + WorkspaceExecuteCommandRequest, + WorkspaceToolRequest, + WorkspaceToolResult, +} from './protocol.js'; +import type { WorkspaceToolExecutor } from './workspace.js'; + +/** Linked worktrees are only admitted from this directory beneath a checkout. */ +export const LINKED_WORKTREE_DIRECTORY = '.worktrees'; +/** + * Shared Git storage a lane may write beneath the common Git directory. Every + * other path there (config, hooks, the checkout's own HEAD, index and + * operation state, sibling metadata) stays read-only. + */ +const LINKED_WORKTREE_SHARED_GIT_PATHS = ['objects', 'refs', join('logs', 'refs'), 'lfs']; +/** Lane registrations kept per worker; the least recently used idle lanes are released first. */ +const LINKED_WORKTREE_LANE_LIMIT = 32; +/** Git pointer files are a single line; anything larger is not one. */ +const GIT_POINTER_MAX_BYTES = 4096; + +export interface LinkedWorktreeSource { + root: string; + identity?: WorkspaceRootIdentity; + command?: NativeProcessSandboxOptions; + repositoryInstructions: boolean; + writable: boolean; +} + +export interface VerifiedLinkedWorktree { + root: string; + identity: WorkspaceRootIdentity; + checkoutRoot: string; + commonGitDir: string; + /** Shared object and ref storage plus the lane's own metadata; nothing else in the common Git directory. */ + writableGitPaths: string[]; +} + +export interface LinkedWorktreeWorkspaceToolsOptions { + commandPool?: NativeWorkspaceCommandPool; + delegate: WorkspaceToolExecutor; + /** Called with each verified lane root, e.g. to route credentials for its repository. */ + onResolve?: (workspaceId: string, root: string) => void; + /** Called when a lane root is released, e.g. to drop its credential route. */ + onRelease?: (root: string) => void; + programmaticDelegate?: { + executeProgrammatic( + workspaceId: string, + request: BridgeWorkspaceProgrammaticRequest, + signal?: AbortSignal, + ): Promise; + }; + sources: ReadonlyMap; +} + +export function linkedWorktreeWorkspaceId(workspaceId: string, worktree: string): string { + return `lane-${createHash('sha256').update(`${workspaceId}\0${worktree}`).digest('hex')}`; +} + +function rejected(message: string): WorkspaceToolError { + return new WorkspaceToolError(message, 'INVALID_REQUEST'); +} + +async function realDirectory(path: string): Promise { + try { + const status = await lstat(path); + return status.isDirectory() && !status.isSymbolicLink(); + } catch { + return false; + } +} + +/** Optional Git paths may be absent, but cannot redirect a write grant into sibling metadata. */ +async function safeSharedGitStorage(commonGitDir: string): Promise { + for (const path of ['objects', 'refs', 'logs', join('logs', 'refs'), 'lfs']) { + try { + const status = await lstat(join(commonGitDir, path)); + if (!status.isDirectory() || status.isSymbolicLink()) return false; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT' || + path === 'objects' || path === 'refs') return false; + } + } + return true; +} + +async function readPointer(path: string): Promise { + let handle; + try { + const status = await lstat(path); + if (!status.isFile() || status.isSymbolicLink() || status.size > GIT_POINTER_MAX_BYTES) { + return undefined; + } + handle = await open(path, 'r'); + const buffer = Buffer.alloc(GIT_POINTER_MAX_BYTES + 1); + const { bytesRead } = await handle.read(buffer, 0, buffer.length, 0); + if (bytesRead > GIT_POINTER_MAX_BYTES) return undefined; + return buffer.subarray(0, bytesRead).toString('utf8'); + } catch { + return undefined; + } finally { + await handle?.close().catch(() => undefined); + } +} + +async function canonicalOrUndefined(path: string): Promise { + try { + return await realpath(path); + } catch { + return undefined; + } +} + +function singleLine(value: string | undefined): string | undefined { + const line = value?.replace(/\r?\n$/, ''); + return line == null || line.length === 0 || /[\r\n\0]/.test(line) ? undefined : line; +} + +/** + * Verify, without running Git, that `/.worktrees/` is a linked + * worktree of that checkout: a real directory whose `.git` file points at + * `/.git/worktrees/`, whose metadata points back at it, and + * whose common directory is the checkout's own `.git`. Neither the lane path + * nor a writable shared Git storage directory may be a symlink, so a forged + * worktree cannot borrow a lane or redirect its Git write grants. + */ +export async function verifyLinkedWorktree( + checkoutRoot: string, + name: string, + checkoutIdentity?: WorkspaceRootIdentity, +): Promise { + if (!isValidLinkedWorktreeName(name)) { + throw rejected('Invalid linked worktree name'); + } + const checkout = await canonicalOrUndefined(checkoutRoot); + if (checkout == null || !(await realDirectory(checkout))) { + throw new WorkspaceToolError('Selected project is unavailable', 'REGISTRATION_INVALID'); + } + if (checkoutIdentity != null && !(await matchesWorkspaceRoot(checkout, checkoutIdentity))) { + throw new WorkspaceToolError('Selected project changed before lane admission', 'REGISTRATION_INVALID'); + } + const commonGitDir = join(checkout, '.git'); + const worktreesDirectory = join(checkout, LINKED_WORKTREE_DIRECTORY); + const root = join(worktreesDirectory, name); + const metadata = join(commonGitDir, 'worktrees', name); + if ( + !(await realDirectory(commonGitDir)) || + !(await realDirectory(worktreesDirectory)) || + !(await realDirectory(root)) || + (await canonicalOrUndefined(root)) !== root || + !(await realDirectory(metadata)) || + (await canonicalOrUndefined(metadata)) !== metadata || + !(await safeSharedGitStorage(commonGitDir)) + ) { + throw rejected(`No linked worktree named ${name} in ${LINKED_WORKTREE_DIRECTORY}`); + } + const dotGit = join(root, '.git'); + const pointer = singleLine(await readPointer(dotGit))?.match(/^gitdir: (.+)$/)?.[1]; + const gitDir = pointer == null ? undefined : isAbsolute(pointer) ? pointer : resolve(root, pointer); + const commonPointer = singleLine(await readPointer(join(metadata, 'commondir'))); + const backPointer = singleLine(await readPointer(join(metadata, 'gitdir'))); + if ( + gitDir == null || + (await canonicalOrUndefined(gitDir)) !== metadata || + commonPointer == null || + (await canonicalOrUndefined(resolve(metadata, commonPointer))) !== commonGitDir || + backPointer == null || + (await canonicalOrUndefined(resolve(metadata, backPointer))) !== dotGit + ) { + throw rejected(`${LINKED_WORKTREE_DIRECTORY}/${name} is not a linked worktree of this project`); + } + let identity: WorkspaceRootIdentity; + try { + identity = await captureWorkspaceRootIdentity(root); + } catch { + throw rejected(`${LINKED_WORKTREE_DIRECTORY}/${name} is unavailable`); + } + const writableGitPaths = [ + ...LINKED_WORKTREE_SHARED_GIT_PATHS.map((path) => join(commonGitDir, path)), + metadata, + ]; + return { root, identity, checkoutRoot: checkout, commonGitDir, writableGitPaths }; +} + +function publicResult(result: WorkspaceToolResult, workspaceId: string): WorkspaceToolResult { + return { ...result, workspaceId }; +} + +/** + * Route requests that name a linked worktree into their own isolated executor. + * Code API schedules each `.worktrees/` as its own lane beneath the + * checkout, so file tools and commands here run confined to that worktree while + * sibling lanes run concurrently. Requests without a worktree pass through. + */ +export class LinkedWorktreeWorkspaceTools implements WorkspaceToolExecutor { + readonly mutationFailuresAreAtomic?: true; + readonly capabilities: WorkspaceToolExecutor['capabilities']; + private readonly executors = new Map< + string, + { fingerprint: string; value: Promise } + >(); + private readonly commandRoots = new Map(); + /** Verified lane roots by internal ID, least recently used first. */ + private readonly lanes = new Map(); + + constructor(private readonly options: LinkedWorktreeWorkspaceToolsOptions) { + this.mutationFailuresAreAtomic = options.delegate.mutationFailuresAreAtomic; + this.capabilities = { + ...options.delegate.capabilities, + workspaces: options.delegate.capabilities.workspaces.map((workspace) => ({ + ...workspace, + ...(options.sources.has(workspace.id) + ? { workspaceScopes: ['git_linked_worktree' as const] } + : {}), + })), + }; + } + + private async resolveLane( + workspaceId: string, + worktree: string, + workspaceInstanceId: string | undefined, + ): Promise<{ lane: VerifiedLinkedWorktree; source: LinkedWorktreeSource; internalId: string }> { + if (workspaceInstanceId != null) { + throw rejected('Linked worktree lanes are not available inside conversation worktrees'); + } + const source = this.options.sources.get(workspaceId); + if (!source) { + throw rejected('Workspace does not allow linked worktree lanes'); + } + const internalId = linkedWorktreeWorkspaceId(workspaceId, worktree); + let lane: VerifiedLinkedWorktree; + try { + lane = await verifyLinkedWorktree(source.root, worktree, source.identity); + } catch (error) { + await this.release(internalId); + throw error; + } + this.lanes.delete(internalId); + this.lanes.set(internalId, lane.root); + this.options.onResolve?.(workspaceId, lane.root); + await this.releaseIdleLanes(internalId); + return { lane, source, internalId }; + } + + /** Forget a lane's executors and routes; a lane still running a command is kept. */ + private async release(internalId: string): Promise { + const root = this.lanes.get(internalId); + if (root == null) return; + if (this.commandRoots.has(internalId)) { + try { + await this.options.commandPool?.unregisterRoot(internalId); + } catch { + return; + } + this.commandRoots.delete(internalId); + } + this.executors.delete(internalId); + this.lanes.delete(internalId); + this.options.onRelease?.(root); + } + + private async releaseIdleLanes(current: string): Promise { + for (const internalId of [...this.lanes.keys()]) { + if (this.lanes.size <= LINKED_WORKTREE_LANE_LIMIT) return; + if (internalId !== current) await this.release(internalId); + } + } + + private async fileExecutor( + internalId: string, + lane: VerifiedLinkedWorktree, + source: LinkedWorktreeSource, + ): Promise { + const fingerprint = `${lane.identity.path}\0${lane.identity.dev}\0${lane.identity.ino}`; + let cached = this.executors.get(internalId); + if (cached == null || cached.fingerprint !== fingerprint) { + cached = { + fingerprint, + value: LocalWorkspaceTools.create({ + repositoryInstructions: source.repositoryInstructions, + workspaces: [ + { id: internalId, identity: lane.identity, root: lane.root, writable: source.writable }, + ], + }), + }; + this.executors.set(internalId, cached); + } + return await cached.value; + } + + /** Register the lane's command root, replacing it when its identity or writable Git paths changed. */ + private async registerCommandRoot( + internalId: string, + lane: VerifiedLinkedWorktree, + source: LinkedWorktreeSource, + ): Promise { + const pool = this.options.commandPool; + if (!source.command || !pool) { + throw new WorkspaceToolError('Linked worktree commands are unavailable', 'COMMAND_DISABLED'); + } + const fingerprint = JSON.stringify([ + lane.identity.path, + lane.identity.dev, + lane.identity.ino, + lane.writableGitPaths, + ]); + const registered = this.commandRoots.get(internalId); + if (registered !== fingerprint) { + if (registered != null) await pool.unregisterRoot(internalId); + await pool.registerRoot(internalId, { + ...source.command, + workspaceIdentity: lane.identity, + workspaceRoot: lane.root, + linkedWorktree: { + checkoutRoot: lane.checkoutRoot, + commonGitDir: lane.commonGitDir, + writableGitPaths: lane.writableGitPaths, + }, + }); + this.commandRoots.set(internalId, fingerprint); + } + return pool; + } + + async execute(request: WorkspaceToolRequest, signal?: AbortSignal): Promise { + if (request.worktree == null) { + return await this.options.delegate.execute(request, signal); + } + if (request.operation === 'execute_command' && request.environmentAction) { + throw rejected('Environment action was not resolved by this worker'); + } + const { worktree, ...baseRequest } = request; + const { lane, source, internalId } = await this.resolveLane( + request.workspaceId, + worktree, + request.workspaceInstanceId, + ); + const laneRequest = { ...baseRequest, workspaceId: internalId } as WorkspaceToolRequest; + if (request.operation === 'execute_command') { + const pool = await this.registerCommandRoot(internalId, lane, source); + return publicResult( + await pool.execute(laneRequest as WorkspaceExecuteCommandRequest, signal), + request.workspaceId, + ); + } + const executor = await this.fileExecutor(internalId, lane, source); + return publicResult(await executor.execute(laneRequest, signal), request.workspaceId); + } + + async executeProgrammatic( + workspaceId: string, + request: BridgeWorkspaceProgrammaticRequest, + signal?: AbortSignal, + ): Promise { + const worktree = request.body.workspace_worktree; + if (worktree == null) { + if (!this.options.programmaticDelegate) { + throw new WorkspaceToolError('Workspace programmatic execution is unavailable', 'COMMAND_DISABLED'); + } + return await this.options.programmaticDelegate.executeProgrammatic(workspaceId, request, signal); + } + const { lane, source, internalId } = await this.resolveLane( + workspaceId, + worktree, + request.body.workspace_instance_id, + ); + const pool = await this.registerCommandRoot(internalId, lane, source); + const { workspace_worktree: _worktree, ...body } = request.body; + return await pool.executeProgrammatic(internalId, { ...request, body }, signal); + } +} diff --git a/packages/code/src/native-process.test.ts b/packages/code/src/native-process.test.ts index 5f7eedbb..8abf263c 100644 --- a/packages/code/src/native-process.test.ts +++ b/packages/code/src/native-process.test.ts @@ -150,6 +150,22 @@ test('executor bootstrap excludes bridge credentials and Node injection variable await sandbox.close(); }); +test('executor forwards the linked worktree policy to the sandbox process', async () => { + const fake = fixture(); + const linkedWorktree = { + checkoutRoot: '/checkout', + commonGitDir: '/checkout/.git', + writableGitPaths: ['/checkout/.git/objects', '/checkout/.git/refs'], + }; + const sandbox = new NativeProcessWorkspaceCommandSandbox( + { workspaceRoot: '/checkout/.worktrees/task-a', linkedWorktree }, + fake.fork, + ); + await sandbox.prepare(); + assert.deepEqual(fake.messages[0].options.linkedWorktree, linkedWorktree); + await sandbox.close(); +}); + test('executor forwards the resolved command policy without worker credentials', async () => { const fake = fixture(); const sandbox = new NativeProcessWorkspaceCommandSandbox( diff --git a/packages/code/src/native-process.ts b/packages/code/src/native-process.ts index ba411042..77fe8269 100644 --- a/packages/code/src/native-process.ts +++ b/packages/code/src/native-process.ts @@ -343,6 +343,7 @@ export class NativeProcessWorkspaceCommandSandbox implements WorkspaceCommandSan homeDirectory, shellPath, programmaticFileUpstream, + linkedWorktree, } = this.options; await this.rpc( 'prepare', @@ -356,6 +357,7 @@ export class NativeProcessWorkspaceCommandSandbox implements WorkspaceCommandSan homeDirectory, shellPath, programmaticFileUpstream, + linkedWorktree, variables: this.options.maskedEnvironment?.variables, }, }, diff --git a/packages/code/src/native-sandbox.test.ts b/packages/code/src/native-sandbox.test.ts index eab72966..7ecb06f3 100644 --- a/packages/code/src/native-sandbox.test.ts +++ b/packages/code/src/native-sandbox.test.ts @@ -1598,3 +1598,64 @@ test('cleans allocated command state exactly once on every execution exit', asyn } } }); + +test('a linked worktree lane may write only shared Git storage and its own metadata', async t => { + const checkoutRoot = await realpath(await mkdtemp(join(tmpdir(), 'librechat-code-lane-'))); + t.after(() => rm(checkoutRoot, { recursive: true, force: true })); + const commonGitDir = join(checkoutRoot, '.git'); + const lane = join(checkoutRoot, '.worktrees', 'task-a'); + const writableGitPaths = [ + join(commonGitDir, 'objects'), + join(commonGitDir, 'refs'), + join(commonGitDir, 'worktrees', 'task-a'), + ]; + await Promise.all( + [lane, ...writableGitPaths].map(path => mkdir(path, { recursive: true })), + ); + const prepare = async (paths: string[]) => { + const fake = fakeManager(); + const sandbox = new NativeSrtWorkspaceCommandSandbox({ + workspaceRoot: lane, + linkedWorktree: { checkoutRoot, commonGitDir, writableGitPaths: paths }, + environment: { PATH: '/usr/bin' }, + manager: fake.manager, + }); + t.after(() => sandbox.close()); + await sandbox.prepare(); + return fake.config!; + }; + + const config = await prepare(writableGitPaths); + assert.deepEqual(config.filesystem.allowWrite.slice(0, 4), [lane, ...writableGitPaths]); + assert.ok(!config.filesystem.allowWrite.includes(commonGitDir)); + assert.ok(config.filesystem.allowRead?.includes(commonGitDir)); + + const probed = fakeManager(); + const prober = new NativeSrtWorkspaceCommandSandbox({ + workspaceRoot: lane, + linkedWorktree: { checkoutRoot, commonGitDir, writableGitPaths }, + environment: { PATH: '/usr/bin' }, + manager: probed.manager, + }); + t.after(() => prober.close()); + const dataDirectory = await prober.createExecutionDirectory(); + await prober.executeProgrammatic(request, dataDirectory, undefined, { probe: true }); + assert.ok(probed.customConfigSeenDuringWrap?.filesystem?.allowRead?.includes(commonGitDir)); + assert.ok(!probed.customConfigSeenDuringWrap?.filesystem?.allowWrite?.includes(commonGitDir)); + + await assert.rejects( + prepare([commonGitDir]), + (error: unknown) => + error instanceof WorkspaceToolError && error.code === 'REGISTRATION_INVALID', + ); + + const siblingMetadata = join(commonGitDir, 'worktrees', 'task-b'); + await mkdir(siblingMetadata, { recursive: true }); + await rm(join(commonGitDir, 'objects'), { recursive: true }); + await symlink(siblingMetadata, join(commonGitDir, 'objects')); + await assert.rejects( + prepare(writableGitPaths), + (error: unknown) => + error instanceof WorkspaceToolError && error.code === 'REGISTRATION_INVALID', + ); +}); diff --git a/packages/code/src/native-sandbox.ts b/packages/code/src/native-sandbox.ts index 5bf61c50..6ac51acc 100644 --- a/packages/code/src/native-sandbox.ts +++ b/packages/code/src/native-sandbox.ts @@ -106,10 +106,15 @@ const TRUSTED_GIT_ENVIRONMENT = { GIT_CONFIG_KEY_3: 'filter.lfs.required', GIT_CONFIG_VALUE_3: 'true', } as const; -const { - GIT_CONFIG_COUNT: TRUSTED_GIT_CONFIG_COUNT, - ...TRUSTED_GIT_CONFIG_ENTRIES -} = TRUSTED_GIT_ENVIRONMENT; +/** Sibling lanes share object storage, so a lane never starts automatic gc or maintenance. */ +const LINKED_WORKTREE_GIT_ENVIRONMENT = { + ...TRUSTED_GIT_ENVIRONMENT, + GIT_CONFIG_COUNT: '6', + GIT_CONFIG_KEY_4: 'gc.auto', + GIT_CONFIG_VALUE_4: '0', + GIT_CONFIG_KEY_5: 'maintenance.auto', + GIT_CONFIG_VALUE_5: 'false', +} as const; const NATIVE_SANDBOX_SCRATCH_PREFIX = 'librechat-code-srt-'; // SRT grants these shared compatibility paths by default. A worker-specific @@ -162,6 +167,15 @@ type SpawnCommand = ( export interface NativeSrtWorkspaceCommandSandboxOptions { workspaceIdentity?: WorkspaceRootIdentity; workspaceRoot: string; + /** Present when `workspaceRoot` is a verified linked worktree lane of a checkout. */ + linkedWorktree?: { + /** The checkout that owns the worktree; trusted as a Git safe directory. */ + checkoutRoot: string; + /** `/.git`: readable, but writable only at `writableGitPaths`. */ + commonGitDir: string; + /** Shared objects and refs plus the lane's own metadata beneath `commonGitDir`. */ + writableGitPaths: string[]; + }; commandPolicy?: NativeSrtCommandPolicy; /** Trusted worker files that must never become workspace-readable or writable. */ protectedPaths?: string[]; @@ -295,9 +309,18 @@ export class NativeSrtWorkspaceCommandSandbox implements WorkspaceCommandSandbox private readonly platform: NodeJS.Platform; private initialized?: Promise; private canonicalRoot?: string; + /** A lane's shared Git directory; replay probes of the lane must read it too. */ + private canonicalCommonGitDir?: string; private runtimeConfig?: SandboxRuntimeConfig; private denyReadPaths: string[] = []; private denyWritePaths: string[] = []; + private get gitEnvironment(): + | typeof TRUSTED_GIT_ENVIRONMENT + | typeof LINKED_WORKTREE_GIT_ENVIRONMENT { + return this.options.linkedWorktree + ? LINKED_WORKTREE_GIT_ENVIRONMENT + : TRUSTED_GIT_ENVIRONMENT; + } private scratchDirectory?: string; private scratchHandle?: FileHandle; private execution?: Promise; @@ -424,6 +447,33 @@ export class NativeSrtWorkspaceCommandSandbox implements WorkspaceCommandSandbox ); } } + const lane = this.options.linkedWorktree; + const commonGitDir = lane ? await canonicalPath(lane.commonGitDir) : undefined; + const checkoutRoot = lane ? await canonicalPath(lane.checkoutRoot) : undefined; + const writableGitPaths = lane + ? await Promise.all(lane.writableGitPaths.map(canonicalPath)) + : []; + if ( + lane && + (commonGitDir == null || + checkoutRoot == null || + !isWithin(checkoutRoot, commonGitDir) || + !isWithin(checkoutRoot, root) || + isWithin(commonGitDir, home) || + protectedPaths.some(path => isWithin(commonGitDir, path)) || + writableGitPaths.some( + (path, index) => + path !== resolve(lane.writableGitPaths[index]!) || + path === commonGitDir || + !isWithin(commonGitDir, path), + )) + ) { + throw new WorkspaceToolError( + 'Linked worktree Git storage is outside its checkout', + 'REGISTRATION_INVALID', + ); + } + const laneGitPaths = commonGitDir ? [commonGitDir] : []; const canonicalScratchDirectory = await this.createScratchDirectory(sharedScratchPaths); if ( @@ -459,17 +509,22 @@ export class NativeSrtWorkspaceCommandSandbox implements WorkspaceCommandSandbox ], allowRead: [ root, + ...laneGitPaths, ...(canonicalScratchDirectory ? [canonicalScratchDirectory] : []), ], allowWrite: [ root, + ...writableGitPaths, ...(canonicalScratchDirectory ? [canonicalScratchDirectory] : []), ], - denyWrite: [...protectedPaths, ...deniedInheritedWritablePaths], + denyWrite: [ + ...protectedPaths, + ...deniedInheritedWritablePaths, + ], allowGitConfig: false, }, credentials: { @@ -520,13 +575,14 @@ export class NativeSrtWorkspaceCommandSandbox implements WorkspaceCommandSandbox allowAppleEvents: false, enableWeakerNestedSandbox: false, enableWeakerNetworkIsolation: false, - git: { safeDirectories: [root] }, + git: { safeDirectories: checkoutRoot ? [root, checkoutRoot] : [root] }, }; await this.manager.initialize( config, unrestrictedNetwork ? async () => true : undefined, ); this.canonicalRoot = root; + this.canonicalCommonGitDir = commonGitDir; this.runtimeConfig = config; this.denyReadPaths = [ home, @@ -758,6 +814,9 @@ export class NativeSrtWorkspaceCommandSandbox implements WorkspaceCommandSandbox allowRead: [ canonicalWorkspaceRoot ?? this.canonicalRoot!, canonicalDataDirectory, + ...(this.canonicalCommonGitDir + ? [this.canonicalCommonGitDir] + : []), ], allowWrite: [ ...(canonicalWorkspaceRoot != null @@ -894,7 +953,7 @@ export class NativeSrtWorkspaceCommandSandbox implements WorkspaceCommandSandbox : request.command; wrapped = await this.withTemporaryHostEnvironment( { - ...TRUSTED_GIT_ENVIRONMENT, + ...this.gitEnvironment, ...(credentialEnvironment ?? {}), ...this.scratchSelectorEnvironment(sandboxScratchDirectory), }, @@ -999,10 +1058,10 @@ export class NativeSrtWorkspaceCommandSandbox implements WorkspaceCommandSandbox ...wrapped.env, ...this.scratchEnvironment(), ...trustedEnvironment, - ...TRUSTED_GIT_CONFIG_ENTRIES, + ...this.gitEnvironment, GIT_CONFIG_COUNT: wrapped.env.GIT_CONFIG_COUNT ?? - TRUSTED_GIT_CONFIG_COUNT, + this.gitEnvironment.GIT_CONFIG_COUNT, GIT_CONFIG_GLOBAL: this.platform === 'win32' ? 'NUL' diff --git a/packages/code/src/protocol.test.ts b/packages/code/src/protocol.test.ts index 5c26c462..729b20e2 100644 --- a/packages/code/src/protocol.test.ts +++ b/packages/code/src/protocol.test.ts @@ -11,6 +11,8 @@ import { isWorkspaceToolRequest, isWorkspaceToolResult, workspaceIsolationKey, + workspaceIsolationKeysConflict, + workspaceIsolationParent, } from './protocol.js'; import type { WorkspaceEditFileRequest, @@ -816,3 +818,62 @@ test('bridge artifact policy and media types match the hardened gateway contract assert.equal(bridgeArtifactMediaType('reports/result.json'), 'application/json'); assert.equal(bridgeArtifactMediaType('Dockerfile'), 'application/octet-stream'); }); + +test('linked-worktree lanes nest beneath their checkout key and conflict only with it', () => { + const instanceId = 'a'.repeat(64); + const lane = workspaceIsolationKey('repo', undefined, 'task-a'); + const sibling = workspaceIsolationKey('repo', undefined, 'task-b'); + const nested = workspaceIsolationKey('repo', instanceId, 'task-a'); + assert.notEqual(lane, workspaceIsolationKey('repo')); + assert.notEqual(lane, workspaceIsolationKey('repo\0task-a')); + assert.notEqual(nested, lane); + assert.equal(workspaceIsolationParent(lane), 'repo'); + assert.equal(workspaceIsolationParent(nested), workspaceIsolationKey('repo', instanceId)); + assert.equal(workspaceIsolationParent('repo'), undefined); + assert.equal(workspaceIsolationParent(workspaceIsolationKey('repo', instanceId)), undefined); + assert.equal(workspaceIsolationKeysConflict(lane, 'repo'), true); + assert.equal(workspaceIsolationKeysConflict('repo', lane), true); + assert.equal(workspaceIsolationKeysConflict(lane, lane), true); + assert.equal(workspaceIsolationKeysConflict(lane, sibling), false); + assert.equal(workspaceIsolationKeysConflict(nested, 'repo'), false); + assert.equal(workspaceIsolationKeysConflict(nested, workspaceIsolationKey('repo', instanceId)), true); +}); + +test('workspace requests accept only a single safe worktree name', () => { + const base = { protocolVersion: 1, operation: 'read_file', workspaceId: 'repo', path: 'README.md' }; + for (const worktree of ['task-a', 'fix_16464', 'v2.0']) { + assert.equal(isWorkspaceToolRequest({ ...base, worktree }), true, worktree); + } + for (const worktree of ['', '.hidden', '..', 'a/b', 'a\\b', 'task.lock', 'x'.repeat(129), 7, null]) { + assert.equal(isWorkspaceToolRequest({ ...base, worktree }), false, String(worktree)); + } + const programmatic = (workspace_worktree: string) => ({ + headers: {}, + body: { + language: 'bash', + version: '5.2.0', + session_id: 'session', + files: [{ name: 'main.sh', content: 'true' }], + workspace_worktree, + }, + }); + assert.equal(isBridgeWorkspaceProgrammaticRequest(programmatic('task-a')), true); + assert.equal(isBridgeWorkspaceProgrammaticRequest(programmatic('../x')), false); +}); + +test('workspace capabilities advertise linked-worktree scopes exactly', () => { + const capabilities = (workspaceScopes: unknown) => ({ + statefulWorkspace: false, + sandboxProfile: 'anthropic-srt', + runtimes: [], + workspaceTools: { + protocolVersion: 1, + operations: ['read_file'], + workspaces: [{ id: 'repo', workspaceScopes }], + }, + }); + assert.equal(isValidBridgeWorkerCapabilities(capabilities(['git_linked_worktree'])), true); + assert.equal(isValidBridgeWorkerCapabilities(capabilities(['git_worktree'])), false); + assert.equal(isValidBridgeWorkerCapabilities(capabilities([])), false); + assert.equal(isValidBridgeWorkerCapabilities(capabilities('git_linked_worktree')), false); +}); diff --git a/packages/code/src/protocol.ts b/packages/code/src/protocol.ts index 7657fc9c..b23ad327 100644 --- a/packages/code/src/protocol.ts +++ b/packages/code/src/protocol.ts @@ -259,14 +259,51 @@ export function bridgeArtifactMediaType(name: string): string { export type BridgeProtocolVersion = typeof BRIDGE_PROTOCOL_VERSION; -/** Collision-free identity shared by scheduling and worker quarantine state. */ +/** One path segment naming a linked worktree at `/.worktrees/`. */ +export const BRIDGE_LINKED_WORKTREE_NAME_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/; + +export function isValidLinkedWorktreeName(value: unknown): value is string { + return ( + typeof value === 'string' && + BRIDGE_LINKED_WORKTREE_NAME_PATTERN.test(value) && + !value.endsWith('.lock') + ); +} + +const LINKED_WORKTREE_KEY_PREFIX = '\0linked-worktree\0'; + +/** Collision-free identity shared by scheduling and worker quarantine state. + * A linked worktree lane nests beneath the key of the checkout that owns it. */ export function workspaceIsolationKey( workspaceId: string, instanceId?: string, + worktree?: string, ): string { - return instanceId === undefined + const base = instanceId === undefined ? workspaceId : `\0git-worktree\0${workspaceId}\0${instanceId}`; + return worktree === undefined + ? base + : `${LINKED_WORKTREE_KEY_PREFIX}${base}\0${worktree}`; +} + +/** The checkout key a linked-worktree lane nests beneath, or undefined for a root key. + * Scheduling treats a lane and its parent as conflicting; sibling lanes do not. */ +export function workspaceIsolationParent(key: string): string | undefined { + if (!key.startsWith(LINKED_WORKTREE_KEY_PREFIX)) return undefined; + const separator = key.lastIndexOf('\0'); + return separator <= LINKED_WORKTREE_KEY_PREFIX.length + ? undefined + : key.slice(LINKED_WORKTREE_KEY_PREFIX.length, separator); +} + +/** Two isolation keys may not execute concurrently when either nests the other. */ +export function workspaceIsolationKeysConflict(left: string, right: string): boolean { + return ( + left === right || + workspaceIsolationParent(left) === right || + workspaceIsolationParent(right) === left + ); } export type BridgeWorkspaceToolOperation = @@ -292,6 +329,9 @@ export interface BridgeWorkspaceDescriptor { operations?: BridgeWorkspaceToolOperation[]; /** Worker-owned isolation schemes available beneath this selected root. */ workspaceInstances?: ['git_worktree']; + /** Scheduling scopes available beneath this root. `git_linked_worktree` gives each + * verified `.worktrees/` linked worktree its own lane. */ + workspaceScopes?: ['git_linked_worktree']; environment?: { fingerprint: string; repo?: string; @@ -321,6 +361,8 @@ export interface WorkspaceReadFileRequest { operation: 'read_file'; workspaceId: string; workspaceInstanceId?: string; + /** Linked worktree lane at `.worktrees/`; paths and cwd are relative to it. */ + worktree?: string; path: string; startLine?: number; maxLines?: number; @@ -364,6 +406,8 @@ export interface WorkspaceSearchTextRequest { operation: 'search_text'; workspaceId: string; workspaceInstanceId?: string; + /** Linked worktree lane at `.worktrees/`; paths and cwd are relative to it. */ + worktree?: string; query: string; path?: string; maxResults?: number; @@ -389,6 +433,8 @@ export interface WorkspaceListFilesRequest { operation: 'list_files'; workspaceId: string; workspaceInstanceId?: string; + /** Linked worktree lane at `.worktrees/`; paths and cwd are relative to it. */ + worktree?: string; path?: string; maxResults?: number; /** Continue strictly after this canonical path from a previous page. */ @@ -410,6 +456,8 @@ export interface WorkspaceWriteFileRequest { operation: 'write_file'; workspaceId: string; workspaceInstanceId?: string; + /** Linked worktree lane at `.worktrees/`; paths and cwd are relative to it. */ + worktree?: string; path: string; content: string; /** False requires an atomic create and refuses to replace an existing file. */ @@ -430,6 +478,8 @@ interface WorkspaceEditFileRequestBase { operation: 'edit_file'; workspaceId: string; workspaceInstanceId?: string; + /** Linked worktree lane at `.worktrees/`; paths and cwd are relative to it. */ + worktree?: string; path: string; /** Refuses the mutation unless current file bytes match this preview revision. */ expectedBaseSha256?: string; @@ -475,6 +525,8 @@ interface WorkspacePreviewEditRequestBase { operation: 'preview_edit'; workspaceId: string; workspaceInstanceId?: string; + /** Linked worktree lane at `.worktrees/`; paths and cwd are relative to it. */ + worktree?: string; path: string; } @@ -513,6 +565,8 @@ export interface WorkspaceExecuteCommandRequest { operation: 'execute_command'; workspaceId: string; workspaceInstanceId?: string; + /** Linked worktree lane at `.worktrees/`; paths and cwd are relative to it. */ + worktree?: string; /** Shell source evaluated only inside the selected sandbox runtime. */ command: string; /** Portable path relative to the workspace root; defaults to '.'. */ @@ -558,6 +612,7 @@ const WORKSPACE_READ_REQUEST_KEYS = new Set([ 'operation', 'workspaceId', 'workspaceInstanceId', + 'worktree', 'path', 'startLine', 'maxLines', @@ -567,6 +622,7 @@ const WORKSPACE_SEARCH_REQUEST_KEYS = new Set([ 'operation', 'workspaceId', 'workspaceInstanceId', + 'worktree', 'query', 'path', 'maxResults', @@ -576,6 +632,7 @@ const WORKSPACE_LIST_REQUEST_KEYS = new Set([ 'operation', 'workspaceId', 'workspaceInstanceId', + 'worktree', 'path', 'maxResults', 'afterPath', @@ -585,6 +642,7 @@ const WORKSPACE_WRITE_REQUEST_KEYS = new Set([ 'operation', 'workspaceId', 'workspaceInstanceId', + 'worktree', 'path', 'content', 'overwrite', @@ -594,6 +652,7 @@ const WORKSPACE_EDIT_REQUEST_KEYS = new Set([ 'operation', 'workspaceId', 'workspaceInstanceId', + 'worktree', 'path', 'oldText', 'newText', @@ -605,6 +664,7 @@ const WORKSPACE_PREVIEW_EDIT_REQUEST_KEYS = new Set([ 'operation', 'workspaceId', 'workspaceInstanceId', + 'worktree', 'path', 'oldText', 'newText', @@ -617,6 +677,7 @@ const WORKSPACE_COMMAND_REQUEST_KEYS = new Set([ 'operation', 'workspaceId', 'workspaceInstanceId', + 'worktree', 'command', 'cwd', 'timeoutMs', @@ -730,6 +791,8 @@ export interface BridgeWorkerRegistrationResponse { supportedWorkspaceProgrammaticLanguages?: WorkspaceProgrammaticLanguage[]; /** Workspace isolation schemes this Code API understands and can route. */ supportedWorkspaceInstanceTypes?: ['git_worktree']; + /** Scheduling scopes this Code API can admit as independent lanes. */ + supportedWorkspaceScopes?: ['git_linked_worktree']; } /** Administrator-visible liveness for a configured worker. Credentials, @@ -803,6 +866,8 @@ export interface BridgeWorkspaceProgrammaticBody { language: 'bash'; version: string; workspace_instance_id?: string; + /** Linked worktree lane at `.worktrees/` beneath the selected checkout. */ + workspace_worktree?: string; /** Stable identity shared by every replay iteration of one execution. */ execution_id?: string; /** Declared replay tools; zero allows the worker to skip the probe pass. */ @@ -970,6 +1035,8 @@ export function isBridgeWorkspaceProgrammaticRequest( (body.workspace_instance_id !== undefined && (typeof body.workspace_instance_id !== 'string' || !/^[a-f0-9]{64}$/.test(body.workspace_instance_id))) || + (body.workspace_worktree !== undefined && + !isValidLinkedWorktreeName(body.workspace_worktree)) || (body.execution_id !== undefined && (typeof body.execution_id !== 'string' || !/^[A-Za-z0-9_-]{1,128}$/.test(body.execution_id))) || @@ -1222,7 +1289,8 @@ export function isWorkspaceToolRequest( !isValidBridgeWorkerId(request.workspaceId) || (request.workspaceInstanceId !== undefined && (typeof request.workspaceInstanceId !== 'string' || - !/^[a-f0-9]{64}$/.test(request.workspaceInstanceId))) + !/^[a-f0-9]{64}$/.test(request.workspaceInstanceId))) || + (request.worktree !== undefined && !isValidLinkedWorktreeName(request.worktree)) ) { return false; } @@ -1676,6 +1744,7 @@ export function isValidBridgeWorkspaceToolCapabilities( key !== 'name' && key !== 'operations' && key !== 'workspaceInstances' && + key !== 'workspaceScopes' && key !== 'instructions' && key !== 'environment', ) || @@ -1686,6 +1755,10 @@ export function isValidBridgeWorkspaceToolCapabilities( (!Array.isArray(descriptor.workspaceInstances) || descriptor.workspaceInstances.length !== 1 || descriptor.workspaceInstances[0] !== 'git_worktree')) || + (descriptor.workspaceScopes !== undefined && + (!Array.isArray(descriptor.workspaceScopes) || + descriptor.workspaceScopes.length !== 1 || + descriptor.workspaceScopes[0] !== 'git_linked_worktree')) || (descriptor.instructions !== undefined && (!Array.isArray(descriptor.instructions) || descriptor.instructions.length > 1 || !descriptor.instructions.every(isRepositoryInstructionDescriptor))) || (descriptor.environment !== undefined && !isValidCodeEnvironmentDescriptor(descriptor.environment)) || diff --git a/packages/code/src/worker-slots.test.ts b/packages/code/src/worker-slots.test.ts index fda6c659..72052974 100644 --- a/packages/code/src/worker-slots.test.ts +++ b/packages/code/src/worker-slots.test.ts @@ -377,3 +377,204 @@ test('programmatic work on an independent workspace bypasses another root cleanu 'previous', ); }); + +test('sibling linked-worktree lanes run past each other, but a lane and its checkout wait for one another', async () => { + const worker = new BridgeWorker({ + codeApiUrl: 'http://localhost:1', + token: 'fixture', + workerId: 'worker', + sandboxEndpoint: 'http://localhost:2', + capabilities: { + statefulWorkspace: false, + sandboxProfile: 'fixture', + runtimes: [], + }, + }); + const internals = worker as unknown as { + activeWorkspaceAssignments: Map }>; + executeOwned: (assignment: BridgeAssignment) => Promise; + }; + const executed: string[] = []; + internals.executeOwned = async (assignment) => { + executed.push(assignment.assignmentId); + }; + const assignment = (assignmentId: string, worktree?: string) => + ({ + assignmentId, + executionKind: 'workspace_tool', + remainingMs: 1_000, + request: { + protocolVersion: 1, + workspaceId: 'repo', + operation: 'read_file', + path: 'README.md', + ...(worktree ? { worktree } : {}), + }, + }) as BridgeAssignment; + let releaseLane!: () => void; + internals.activeWorkspaceAssignments.set(workspaceIsolationKey('repo', undefined, 'task-a'), { + id: 'lane-a', + done: new Promise((resolve) => { + releaseLane = resolve; + }), + }); + + await worker.executeAndSettle(assignment('lane-b', 'task-b')); + assert.deepEqual(executed, ['lane-b']); + + const checkout = worker.executeAndSettle(assignment('checkout')); + await new Promise((resolve) => setTimeout(resolve, 5)); + assert.deepEqual(executed, ['lane-b']); + internals.activeWorkspaceAssignments.delete(workspaceIsolationKey('repo', undefined, 'task-a')); + releaseLane(); + await checkout; + assert.deepEqual(executed, ['lane-b', 'checkout']); + + let releaseCheckout!: () => void; + internals.activeWorkspaceAssignments.set(workspaceIsolationKey('repo'), { + id: 'root', + done: new Promise((resolve) => { + releaseCheckout = resolve; + }), + }); + const lane = worker.executeAndSettle(assignment('lane-c', 'task-c')); + await new Promise((resolve) => setTimeout(resolve, 5)); + assert.deepEqual(executed, ['lane-b', 'checkout']); + internals.activeWorkspaceAssignments.delete(workspaceIsolationKey('repo')); + releaseCheckout(); + await lane; + assert.deepEqual(executed, ['lane-b', 'checkout', 'lane-c']); +}); + +test('a linked worktree lane fence is reset through its own guard and isolation key', async () => { + const bodies: Record[] = []; + const guarded: string[] = []; + const worker = new BridgeWorker({ + codeApiUrl: 'http://localhost:1', + token: 'fixture', + workerId: 'worker', + incarnationId: 'incarnation-reset', + sandboxEndpoint: 'http://localhost:2', + capabilities: { + statefulWorkspace: false, + sandboxProfile: 'native-srt', + runtimes: [], + workspaceLeaseSlots: 2, + requiresReadyConfirmation: true, + workspaceTools: { + protocolVersion: 1, + operations: ['read_file'], + workspaces: [{ id: 'repo', workspaceScopes: ['git_linked_worktree'] }, { id: 'plain' }], + }, + }, + workspaceQuarantines: new Map( + ['repo', 'plain'].map((id) => [ + id, + { + assertAvailable: async () => undefined, + arm: async () => undefined, + clear: async () => undefined, + quarantine: async () => undefined, + }, + ]), + ), + workspaceTools: { + capabilities: { + protocolVersion: 1, + operations: ['read_file'], + workspaces: [{ id: 'repo', workspaceScopes: ['git_linked_worktree'] }, { id: 'plain' }], + }, + async execute() { + throw new Error('must not execute'); + }, + }, + linkedWorktreeQuarantineResolver: (workspaceId, worktree) => { + guarded.push(`${workspaceId}/${worktree}`); + return { + assertAvailable: async () => undefined, + arm: async () => undefined, + clear: async () => undefined, + quarantine: async () => undefined, + }; + }, + fetchImpl: async (url, init) => { + assert.ok(new URL(String(url)).pathname.endsWith('/workspaces/reset')); + bodies.push(JSON.parse(String(init?.body))); + return Response.json({ protocolVersion: 1, reset: true }); + }, + }); + + await worker.resetNativeWorkspace('repo', undefined, undefined, 'task-a'); + + assert.deepEqual(guarded, ['repo/task-a']); + assert.equal( + bodies[0]?.runtimeSessionId, + `native-workspace:${workspaceIsolationKey('repo', undefined, 'task-a')}`, + ); + await assert.rejects( + worker.resetNativeWorkspace('plain', undefined, undefined, 'task-a'), + /worktree lanes/, + ); + await assert.rejects( + worker.resetNativeWorkspace('repo', undefined, 'a'.repeat(64), 'task-a'), + /worktree lanes/, + ); + await assert.rejects( + worker.resetNativeWorkspace('repo', undefined, undefined, '../task-a'), + /worktree lanes/, + ); + assert.equal(bodies.length, 1); +}); + +test('a checkout waits on quarantined linked worktrees beneath it', async () => { + const quarantinedLanes = new Set(['task-b']); + const worker = new BridgeWorker({ + codeApiUrl: 'http://localhost:1', + token: 'fixture', + workerId: 'worker', + sandboxEndpoint: 'http://localhost:2', + capabilities: { + statefulWorkspace: false, + sandboxProfile: 'fixture', + runtimes: [], + }, + linkedWorktreeQuarantineResolver: (_workspaceId, worktree) => ({ + assertAvailable: async () => { + if (quarantinedLanes.has(worktree)) throw new Error(`lane ${worktree} is quarantined`); + }, + arm: async () => undefined, + clear: async () => undefined, + quarantine: async () => undefined, + }), + linkedWorktreeNames: async () => ['task-a', 'task-b'], + }); + const internals = worker as unknown as { + quarantinedWorkspaces: Set; + assertLaneFamilyAvailable: (key: string, assignment: BridgeAssignment) => Promise; + }; + const assignment = (worktree?: string) => + ({ + assignmentId: 'probe', + executionKind: 'workspace_tool', + request: { + protocolVersion: 1, + workspaceId: 'repo', + operation: 'read_file', + path: 'README.md', + ...(worktree ? { worktree } : {}), + }, + }) as BridgeAssignment; + const check = (worktree?: string) => + internals.assertLaneFamilyAvailable( + workspaceIsolationKey('repo', undefined, worktree), + assignment(worktree), + ); + + await assert.rejects(check(), /task-b is quarantined/); + await check('task-a'); + + quarantinedLanes.clear(); + await check(); + internals.quarantinedWorkspaces.add(workspaceIsolationKey('repo', undefined, 'task-c')); + await assert.rejects(check(), /linked worktree/); +}); diff --git a/packages/code/src/worker.ts b/packages/code/src/worker.ts index 2a5896fd..37223b34 100644 --- a/packages/code/src/worker.ts +++ b/packages/code/src/worker.ts @@ -6,8 +6,11 @@ import { BridgeProtocolError, bridgeWorkerPath, isBridgeWorkspaceProgrammaticRequest, + isValidLinkedWorktreeName, isWorkspaceToolResult, workspaceIsolationKey, + workspaceIsolationKeysConflict, + workspaceIsolationParent, } from './protocol.js'; import { EndpointRuntimeSupervisor } from './runtime.js'; import { signBridgeRequest } from './identity.js'; @@ -62,6 +65,15 @@ export interface BridgeWorkerOptions { ) => | WorkspaceMutationQuarantine | Promise; + /** Resolve a durable guard for a linked-worktree lane beneath a registered root. */ + linkedWorktreeQuarantineResolver?: ( + workspaceId: string, + worktree: string, + ) => + | WorkspaceMutationQuarantine + | Promise; + /** Names of the linked worktrees currently beneath a registered root. */ + linkedWorktreeNames?: (workspaceId: string) => Promise; leaseWaitMs?: number; leaseTransportGraceMs?: number; registrationTransportTimeoutMs?: number; @@ -223,7 +235,13 @@ function workspaceCapabilitiesMatch( (instanceType, instanceIndex) => instanceType === executor.workspaces[index]?.workspaceInstances?.[instanceIndex], - ) ?? executor.workspaces[index]?.workspaceInstances == null), + ) ?? executor.workspaces[index]?.workspaceInstances == null) && + workspace.workspaceScopes?.length === + executor.workspaces[index]?.workspaceScopes?.length && + (workspace.workspaceScopes?.every( + (scope, scopeIndex) => + scope === executor.workspaces[index]?.workspaceScopes?.[scopeIndex], + ) ?? executor.workspaces[index]?.workspaceScopes == null), ) ); } @@ -239,7 +257,9 @@ function registrationCompatibleCapabilities( ) && workspaceTools.workspaces.every( (workspace) => - workspace.operations == null && workspace.workspaceInstances == null, + workspace.operations == null && + workspace.workspaceInstances == null && + workspace.workspaceScopes == null, )) ) { return capabilities; @@ -263,6 +283,7 @@ function registrationCompatibleCapabilities( const { operations: _operations, workspaceInstances: _workspaceInstances, + workspaceScopes: _workspaceScopes, ...compatibleWorkspace } = workspace; return [{ ...compatibleWorkspace, ...(workspace.environment ? { @@ -355,6 +376,10 @@ function supportedWorkspaceCapabilities( ) ? { workspaceInstances: workspace.workspaceInstances } : { workspaceInstances: undefined }), + ...(workspace.workspaceScopes != null && + registration.supportedWorkspaceScopes?.includes('git_linked_worktree') + ? { workspaceScopes: workspace.workspaceScopes } + : { workspaceScopes: undefined }), ...(workspace.operations ? { operations: workspaceOperations } : {}), ...(workspace.environment && !workspaceOperations.includes('execute_command') ? { environment: { ...workspace.environment, actions: [] }, @@ -506,7 +531,8 @@ export class BridgeWorker { ) === true && options.workspaceMutationQuarantine == null && options.workspaceQuarantines == null && - options.workspaceQuarantineResolver == null + options.workspaceQuarantineResolver == null && + options.linkedWorktreeQuarantineResolver == null ) { throw new BridgeProtocolError( 'Workspace mutation capabilities require durable quarantine storage', @@ -522,6 +548,17 @@ export class BridgeWorker { 'Workspace instance capabilities require a durable quarantine resolver', ); } + if ( + options.capabilities.workspaceTools?.workspaces.some( + (root) => (root.workspaceScopes?.length ?? 0) > 0, + ) && + (options.linkedWorktreeQuarantineResolver == null || + (options.capabilities.workspaceLeaseSlots ?? 1) < 2) + ) { + throw new BridgeProtocolError( + 'Linked worktree lanes require a durable quarantine resolver and at least two lease slots', + ); + } if ((options.capabilities.workspaceLeaseSlots ?? 1) > 1) { if ( options.capabilities.requiresReadyConfirmation !== true || @@ -823,18 +860,33 @@ export class BridgeWorker { workspaceId: string, signal?: AbortSignal, workspaceInstanceId?: string, + worktree?: string, ): Promise { const workspace = this.options.capabilities.workspaceTools?.workspaces.find( (root) => root.id === workspaceId, ); - const key = workspaceIsolationKey(workspaceId, workspaceInstanceId); - const guard = - workspaceInstanceId == null - ? this.options.workspaceQuarantines?.get(workspaceId) - : await this.options.workspaceQuarantineResolver?.( - workspaceId, - workspaceInstanceId, - ); + if ( + worktree != null && + (workspaceInstanceId != null || + !isValidLinkedWorktreeName(worktree) || + workspace?.workspaceScopes?.includes('git_linked_worktree') !== true) + ) { + throw new BridgeProtocolError( + 'Linked worktree reset requires a registered checkout with worktree lanes', + ); + } + const key = workspaceIsolationKey(workspaceId, workspaceInstanceId, worktree); + let guard: WorkspaceMutationQuarantine | undefined; + if (worktree != null) { + guard = await this.options.linkedWorktreeQuarantineResolver?.(workspaceId, worktree); + } else if (workspaceInstanceId != null) { + guard = await this.options.workspaceQuarantineResolver?.( + workspaceId, + workspaceInstanceId, + ); + } else { + guard = this.options.workspaceQuarantines?.get(workspaceId); + } if ( !guard || this.activeWorkspaceAssignments.size > 0 || @@ -1313,8 +1365,11 @@ export class BridgeWorker { ): Promise { const root = this.assignmentWorkspaceId(assignment); const waitingAt = Date.now(); - while (root != null && this.activeWorkspaceAssignments.has(root)) { - const active = this.activeWorkspaceAssignments.get(root)!; + for ( + let active = root == null ? undefined : this.conflictingActiveAssignment(root); + root != null && active != null; + active = this.conflictingActiveAssignment(root) + ) { if (active.id === assignment.assignmentId) throw new BridgeProtocolError( 'Code API replayed an active workspace assignment', @@ -1386,6 +1441,50 @@ export class BridgeWorker { } } + /** An active assignment on the same key, the key's parent checkout, or a lane beneath it. */ + private conflictingActiveAssignment( + key: string, + ): { id: string; done: Promise } | undefined { + for (const [activeKey, active] of this.activeWorkspaceAssignments) { + if (workspaceIsolationKeysConflict(activeKey, key)) return active; + } + return undefined; + } + + /** A lane may not run while the checkout it belongs to is quarantined. */ + /** + * A lane may not start while its checkout is quarantined, and a checkout may + * not start while any linked worktree beneath it is: root commands can reach + * `.worktrees/*`, including `git worktree remove`. + */ + private async assertLaneFamilyAvailable( + workspaceKey: string, + assignment: BridgeAssignment, + ): Promise { + const workspaceId = this.assignmentBaseWorkspaceId(assignment); + const parent = workspaceIsolationParent(workspaceKey); + if (parent != null) { + if (this.quarantinedWorkspaces.has(parent)) { + throw new Error('Parent workspace requires an explicit quarantine reset'); + } + if (workspaceId != null && parent === workspaceId) { + await this.options.workspaceQuarantines?.get(workspaceId)?.assertAvailable(); + } + return; + } + const resolveLaneGuard = this.options.linkedWorktreeQuarantineResolver; + if (workspaceId == null || workspaceKey !== workspaceId || resolveLaneGuard == null) return; + for (const key of this.quarantinedWorkspaces) { + if (workspaceIsolationParent(key) === workspaceKey) { + throw new Error('A linked worktree in this workspace requires an explicit quarantine reset'); + } + } + const names = (await this.options.linkedWorktreeNames?.(workspaceId)) ?? []; + await Promise.all( + names.map(async (name) => (await resolveLaneGuard(workspaceId, name)).assertAvailable()), + ); + } + private workspaceGuard( assignment: BridgeAssignment, ): @@ -1394,6 +1493,10 @@ export class BridgeWorker { | undefined { const workspaceId = this.assignmentBaseWorkspaceId(assignment); const instanceId = this.assignmentWorkspaceInstanceId(assignment); + const worktree = this.assignmentWorktree(assignment); + if (workspaceId != null && worktree != null) { + return this.options.linkedWorktreeQuarantineResolver?.(workspaceId, worktree); + } if (workspaceId != null && instanceId != null) { return this.options.workspaceQuarantineResolver?.( workspaceId, @@ -1412,7 +1515,27 @@ export class BridgeWorker { const workspaceId = this.assignmentBaseWorkspaceId(assignment); if (workspaceId == null) return undefined; const instanceId = this.assignmentWorkspaceInstanceId(assignment); - return workspaceIsolationKey(workspaceId, instanceId); + return workspaceIsolationKey( + workspaceId, + instanceId, + this.assignmentWorktree(assignment), + ); + } + + private assignmentWorktree(assignment: BridgeAssignment): string | undefined { + if ( + assignment.executionKind === 'workspace_tool' && + isWorkspaceToolRequest(assignment.request) + ) { + return assignment.request.worktree; + } + if ( + assignment.executionKind === 'workspace_programmatic' && + isBridgeWorkspaceProgrammaticRequest(assignment.request) + ) { + return assignment.request.body.workspace_worktree; + } + return undefined; } private assignmentBaseWorkspaceId( @@ -1580,9 +1703,11 @@ export class BridgeWorker { } if ( this.options.workspaceQuarantines != null || - this.options.workspaceQuarantineResolver != null + this.options.workspaceQuarantineResolver != null || + this.options.linkedWorktreeQuarantineResolver != null ) { await guard?.assertAvailable(); + await this.assertLaneFamilyAvailable(workspaceKey, assignment); } } catch (error) { throw new BridgeWorkspaceQuarantinedError( @@ -1615,6 +1740,14 @@ export class BridgeWorker { 'Workspace instance type is not advertised', ); } + if ( + workspaceRequest.worktree != null && + workspace.workspaceScopes?.includes('git_linked_worktree') !== true + ) { + throw new BridgeProtocolError( + 'Linked worktree lanes are not advertised for workspace', + ); + } if ( workspace.operations != null && !workspace.operations.includes(workspaceRequest.operation) @@ -1739,9 +1872,11 @@ export class BridgeWorker { } if ( this.options.workspaceQuarantines != null || - this.options.workspaceQuarantineResolver != null + this.options.workspaceQuarantineResolver != null || + this.options.linkedWorktreeQuarantineResolver != null ) { await guard?.assertAvailable(); + await this.assertLaneFamilyAvailable(workspaceKey, assignment); } } catch (error) { throw new BridgeWorkspaceQuarantinedError( @@ -1772,6 +1907,14 @@ export class BridgeWorker { 'Workspace instance type is not advertised', ); } + if ( + assignment.request.body.workspace_worktree != null && + workspace.workspaceScopes?.includes('git_linked_worktree') !== true + ) { + throw new BridgeProtocolError( + 'Linked worktree lanes are not advertised for workspace', + ); + } this.mutationGuardArmed = true; try { this.armedWorkspaces.add(workspaceKey); diff --git a/packages/code/src/workspace-cli.test.ts b/packages/code/src/workspace-cli.test.ts index 1de68818..52743bae 100644 --- a/packages/code/src/workspace-cli.test.ts +++ b/packages/code/src/workspace-cli.test.ts @@ -147,6 +147,47 @@ test('CLI requires concurrent native slots for conversation worktrees', async (t assert.match(result.stderr, /at least two workspace lease slots/i); }); +test('CLI requires concurrent native slots and no conversation worktrees for linked worktree lanes', async (t) => { + const root = await mkdtemp(join(tmpdir(), 'librechat-code-lanes-')); + const workspaceRoot = join(root, 'workspace'); + await mkdir(workspaceRoot); + t.after(() => rm(root, { recursive: true, force: true })); + const run = (...extra: string[]) => + spawnSync( + process.execPath, + [ + fileURLToPath(new URL('./cli.js', import.meta.url)), + 'run', + '--worker-dir', + workspaceRoot, + '--allow-workspace-writes', + '--allow-workspace-commands', + '--linked-worktree-lanes', + ...extra, + ], + { + encoding: 'utf8', + env: { + ...process.env, + LIBRECHAT_CODE_URL: 'http://127.0.0.1:1/v1', + LIBRECHAT_CODE_WORKER_TOKEN: 'worker-secret', + LIBRECHAT_CODE_WORKER_ID: 'engineering-vm', + }, + }, + ); + const serial = run(); + assert.notEqual(serial.status, 0); + assert.match(serial.stderr, /Linked worktree lanes require .*at least two workspace lease slots/i); + const combined = run( + '--workspace-lease-slots', + '2', + '--conversation-worktree-root', + join(root, 'conversations'), + ); + assert.notEqual(combined.status, 0); + assert.match(combined.stderr, /cannot be combined with conversation worktrees/i); +}); + test('CLI advertises explicitly enabled writes without exposing the workspace root', async (t) => { const root = await mkdtemp(join(tmpdir(), 'librechat-code-cli-')); const workspaceRoot = join(root, ' '); diff --git a/service/src/bridge/linked-worktree.test.ts b/service/src/bridge/linked-worktree.test.ts new file mode 100644 index 00000000..5a286726 --- /dev/null +++ b/service/src/bridge/linked-worktree.test.ts @@ -0,0 +1,197 @@ +import { createHash, randomUUID } from 'node:crypto'; +import { expect, test } from 'bun:test'; +import Redis from 'ioredis'; +import { RedisBridgeStore } from './store'; +import type { CodeBridgeAssignment } from './store'; +import type { WorkspaceToolRequest } from '../../../packages/code/src/protocol'; + +const redisUrl = process.env.BRIDGE_TEST_REDIS_URL; +const incarnationId = 'lane-incarnation'; + +function fenceKey(workerId: string, fence: string): string { + const hash = createHash('sha256').update(fence).digest('hex'); + return `codeapi:bridge:v1:worker:${encodeURIComponent(workerId)}:workspace:${hash}:quarantined`; +} + +async function withLaneWorker( + run: (context: { + store: RedisBridgeStore; + redis: Redis; + workerId: string; + dispatch: ( + request: Partial, + budgetMs?: number, + ) => Promise; + lease: (slot: number) => Promise; + settle: (assignment: CodeBridgeAssignment) => Promise; + }) => Promise, + scopes = true, +): Promise { + const redis = new Redis(redisUrl!); + const store = new RedisBridgeStore(redis, 60, 1000, 3); + const workerId = `lanes-${randomUUID()}`; + const controller = new AbortController(); + const pending: Promise[] = []; + try { + const generation = await store.register({ + protocolVersion: 1, + workerId, + incarnationId, + capabilities: { + statefulWorkspace: false, + sandboxProfile: 'native-srt', + runtimes: [], + workspaceLeaseSlots: 3, + requiresReadyConfirmation: true, + workspaceTools: { + protocolVersion: 1, + operations: ['read_file'], + workspaces: [ + { + id: 'repo', + ...(scopes ? { workspaceScopes: ['git_linked_worktree' as const] } : {}), + }, + ], + }, + }, + }); + await store.confirmReady(workerId, incarnationId, generation); + await run({ + store, + redis, + workerId, + dispatch(request, budgetMs = 3000) { + const result = store.dispatchWorkspaceTool({ + workerId, + signal: controller.signal, + deadlineAtMs: Date.now() + budgetMs, + executionTimeoutMs: 5000, + request: { + protocolVersion: 1, + operation: 'read_file', + workspaceId: 'repo', + path: 'probe', + ...request, + } as WorkspaceToolRequest, + }); + void result.catch(() => undefined); + pending.push(result); + return result; + }, + lease: (slot) => store.lease(workerId, incarnationId, 1000, undefined, undefined, slot), + async settle(assignment) { + await store.acknowledgeLease( + workerId, + incarnationId, + assignment.assignmentId, + assignment.generation, + assignment.leaseToken, + ); + await store.settle(workerId, assignment.assignmentId, { + protocolVersion: 1, + incarnationId, + generation: assignment.generation, + leaseToken: assignment.leaseToken, + status: 'rejected', + error: 'probe complete', + }); + await store.confirmWorkspaceCleanup(workerId, assignment.assignmentId, { + protocolVersion: 1, + incarnationId, + generation: assignment.generation, + leaseToken: assignment.leaseToken, + status: 'rejected', + error: 'local cleanup confirmed', + }); + }, + }); + } finally { + controller.abort(); + await Promise.allSettled(pending); + await redis.quit(); + } +} + +test.skipIf(!redisUrl)( + 'sibling linked-worktree lanes run together while their checkout waits for both', + async () => { + await withLaneWorker(async ({ dispatch, lease, settle }) => { + const laneA = dispatch({ worktree: 'task-a' }); + const laneB = dispatch({ worktree: 'task-b' }); + const first = await lease(0); + const second = await lease(1); + expect( + [first?.request, second?.request].map( + (request) => (request as WorkspaceToolRequest).worktree, + ).sort(), + ).toEqual(['task-a', 'task-b']); + + const checkout = dispatch({}, 400); + expect(await lease(2)).toBeUndefined(); + await expect(checkout).rejects.toMatchObject({ code: 'WORKSPACE_QUEUE_TIMEOUT' }); + + await settle(first!); + await settle(second!); + await Promise.allSettled([laneA, laneB]); + }); + }, +); + +test.skipIf(!redisUrl)('a busy checkout holds every lane beneath it', async () => { + await withLaneWorker(async ({ dispatch, lease, settle }) => { + const checkout = dispatch({}); + const held = await lease(0); + expect((held?.request as WorkspaceToolRequest).worktree).toBeUndefined(); + + const lane = dispatch({ worktree: 'task-a' }); + expect(await lease(1)).toBeUndefined(); + + await settle(held!); + await Promise.allSettled([checkout]); + const admitted = await lease(0); + expect((admitted?.request as WorkspaceToolRequest).worktree).toBe('task-a'); + await settle(admitted!); + await Promise.allSettled([lane]); + }); +}); + +test.skipIf(!redisUrl)('a lane may not start while its checkout is quarantined', async () => { + await withLaneWorker(async ({ redis, workerId, dispatch }) => { + await redis.set(fenceKey(workerId, 'native-workspace:repo'), 'quarantined:earlier'); + await expect(dispatch({ worktree: 'task-a' })).rejects.toMatchObject({ + code: 'WORKSPACE_QUARANTINED', + }); + expect(await redis.exists(fenceKey(workerId, 'native-workspace:\0linked-worktree\0repo\0task-a'))).toBe(0); + }); +}); + +test.skipIf(!redisUrl)('a checkout is refused while a lane beneath it keeps a stuck fence', async () => { + await withLaneWorker(async ({ redis, workerId, dispatch, lease, settle }) => { + const settleNext = async (request: Partial) => { + const pending = dispatch(request); + const assignment = await lease(0); + expect((assignment?.request as WorkspaceToolRequest).worktree).toBe(request.worktree); + await settle(assignment!); + await Promise.allSettled([pending]); + }; + await settleNext({ worktree: 'task-a' }); + await settleNext({}); + + // A lane whose cleanup was never acknowledged keeps its fence after its slot is released. + await settleNext({ worktree: 'task-a' }); + await redis.set( + fenceKey(workerId, 'native-workspace:\0linked-worktree\0repo\0task-a'), + 'quarantined:cleanup-unacknowledged', + ); + + await expect(dispatch({})).rejects.toMatchObject({ code: 'WORKSPACE_QUARANTINED' }); + }); +}); + +test.skipIf(!redisUrl)('a lane is refused unless the worker advertises linked-worktree scopes', async () => { + await withLaneWorker(async ({ dispatch }) => { + await expect(dispatch({ worktree: 'task-a' })).rejects.toMatchObject({ + code: 'WORKER_MISMATCH', + }); + }, false); +}); diff --git a/service/src/bridge/router.ts b/service/src/bridge/router.ts index b3d549b1..962c6242 100644 --- a/service/src/bridge/router.ts +++ b/service/src/bridge/router.ts @@ -539,6 +539,7 @@ router.post( supportedWorkspaceListFileFeatures: ['after_path'], supportedWorkspaceProgrammaticLanguages: ['bash'], supportedWorkspaceInstanceTypes: ['git_worktree'], + supportedWorkspaceScopes: ['git_linked_worktree'], }); } catch (error) { if (error instanceof BridgeStoreError) { diff --git a/service/src/bridge/slots.test.ts b/service/src/bridge/slots.test.ts index dfe89495..1efe3b97 100644 --- a/service/src/bridge/slots.test.ts +++ b/service/src/bridge/slots.test.ts @@ -3,6 +3,7 @@ import RedisMock from 'ioredis-mock'; import type Redis from 'ioredis'; import { BridgeAdmissionQueue } from './admission'; import { BridgeWorkspaceSlots } from './slots'; +import { workspaceIsolationKey } from '../../../packages/code/src/protocol'; const redis = new RedisMock() as unknown as Redis; const admission = new BridgeAdmissionQueue(redis); @@ -13,20 +14,27 @@ const prefix = `codeapi:bridge:v1:worker:${workerId}`; afterEach(async () => { await redis.flushall(); }); -async function enqueue(assignmentId: string, workspaceId: string) { +async function enqueue(assignmentId: string, workspaceId: string, capacity = 2) { await redis.set(`${prefix}:incarnation`, incarnationId); - await redis.set(`${prefix}:workspace-slot-capacity`, '2'); + await redis.set(`${prefix}:workspace-slot-capacity`, String(capacity)); await admission.enter(workerId, assignmentId, Date.now() + 5000, workspaceId); return { workerId, incarnationId, assignmentId, workspaceId, - capacity: 2, + capacity, expiresAtMs: Date.now() + 10000, }; } +async function finish(assignmentId: string) { + await slots.release(workerId, incarnationId, assignmentId); + await admission.leave(workerId, assignmentId); +} + +const lane = (name: string, parent = 'repo') => workspaceIsolationKey(parent, undefined, name); + test('slots admit independent workspaces, skip a busy root, and bound capacity', async () => { const a = await enqueue('a', 'root-a'); const a2 = await enqueue('a2', 'root-a'); @@ -91,3 +99,51 @@ test('releasing a long slot shortens the aggregate expiry to remaining work', as await slots.release(workerId, incarnationId, 'a'); expect(await redis.pttl(`${prefix}:lock`)).toBeLessThanOrEqual(3000); }); + +test('sibling linked-worktree lanes share their checkout, which waits for both', async () => { + const a = await enqueue('a', lane('task-a'), 3); + const b = await enqueue('b', lane('task-b'), 3); + const root = await enqueue('root', 'repo', 3); + expect(await slots.reserve(a)).toBe(0); + expect(await slots.reserve(b)).toBe(1); + expect(await slots.reserve(root)).toBeUndefined(); + await finish('a'); + expect(await slots.reserve(root)).toBeUndefined(); + await finish('b'); + expect(await slots.reserve(root)).toBe(0); +}); + +test('a busy checkout holds every lane beneath it but not other checkouts', async () => { + const root = await enqueue('root', 'repo', 3); + const a = await enqueue('a', lane('task-a'), 3); + const other = await enqueue('other', lane('task-a', 'other-repo'), 3); + expect(await slots.reserve(root)).toBe(0); + expect(await slots.reserve(a)).toBeUndefined(); + expect(await slots.reserve(other)).toBe(1); + await finish('root'); + expect(await slots.reserve(a)).toBe(0); +}); + +test('a waiting checkout holds back newer lanes so root work cannot starve', async () => { + const a = await enqueue('a', lane('task-a'), 3); + expect(await slots.reserve(a)).toBe(0); + const root = await enqueue('root', 'repo', 3); + const b = await enqueue('b', lane('task-b'), 3); + const other = await enqueue('other', 'other-repo', 3); + expect(await slots.reserve(root)).toBeUndefined(); + expect(await slots.reserve(b)).toBeUndefined(); + expect(await slots.reserve(other)).toBe(1); + await finish('a'); + expect(await slots.reserve(b)).toBeUndefined(); + expect(await slots.reserve(root)).toBe(0); +}); + +test('lanes nest beneath a conversation checkout, not the source root', async () => { + const instance = workspaceIsolationKey('repo', 'c'.repeat(64)); + const conversation = await enqueue('conversation', instance, 3); + const nested = await enqueue('nested', workspaceIsolationKey('repo', 'c'.repeat(64), 'task-a'), 3); + const source = await enqueue('source', lane('task-a'), 3); + expect(await slots.reserve(conversation)).toBe(0); + expect(await slots.reserve(nested)).toBeUndefined(); + expect(await slots.reserve(source)).toBe(1); +}); diff --git a/service/src/bridge/slots.ts b/service/src/bridge/slots.ts index 9c9e7d3b..81f60ebe 100644 --- a/service/src/bridge/slots.ts +++ b/service/src/bridge/slots.ts @@ -6,6 +6,12 @@ export const MAX_WORKSPACE_LEASE_SLOTS = 8; /** Reservations share the legacy admission queue and aggregate worker lock. * Thus a serial dispatcher or replacement incarnation cannot race active slots. * Workspace mutation uncertainty is fenced separately by the assignment store. + * + * Keys are hierarchical: a linked-worktree lane (`\0linked-worktree\0\0`) + * conflicts with itself and with its parent checkout, and a checkout conflicts + * with every lane beneath it. Sibling lanes run concurrently. A waiting checkout + * request also holds back newer lanes beneath it, so root operations such as + * `git worktree add` cannot be starved by a steady stream of lane work. */ export class BridgeWorkspaceSlots { constructor(private readonly redis: Redis) {} @@ -47,10 +53,26 @@ export class BridgeWorkspaceSlots { [ "if redis.call('GET', KEYS[4]) ~= ARGV[1] then return -2 end", "if (redis.call('GET', KEYS[8]) or '1') ~= ARGV[4] then return -2 end", + // Parent of a linked-worktree lane key; nil for a checkout key. + 'local lanePrefix = "\\0linked-worktree\\0"', + 'local function parentOf(key)', + ' if string.sub(key, 1, #lanePrefix) ~= lanePrefix then return nil end', + ' local last = nil', + ' local cursor = #lanePrefix + 1', + ' while true do', + ' local found = string.find(key, "\\0", cursor, true)', + ' if not found then break end', + ' last = found', + ' cursor = found + 1', + ' end', + ' if last == nil or last <= #lanePrefix + 1 then return nil end', + ' return string.sub(key, #lanePrefix + 1, last - 1)', + 'end', "local lock = redis.call('GET', KEYS[2])", "local owner = 'workspace-slots:' .. ARGV[1]", 'if lock and lock ~= owner then return -1 end', 'local busy = {}', + 'local busyChildren = {}', 'local free = nil', 'local latest = tonumber(ARGV[5])', `for slot = 0, ${MAX_WORKSPACE_LEASE_SLOTS - 1} do`, @@ -63,12 +85,20 @@ export class BridgeWorkspaceSlots { ' else', ' if entry[1] == ARGV[2] then return slot end', ' busy[entry[3]] = true', + ' local busyParent = parentOf(entry[3])', + ' if busyParent then busyChildren[busyParent] = true end', ' latest = math.max(latest, tonumber(entry[4]))', ' end', ' end', ' if not occupied and free == nil and slot < tonumber(ARGV[4]) then free = slot end', 'end', - 'if free == nil or busy[ARGV[3]] then return -1 end', + 'local function conflicts(key)', + ' if busy[key] then return true end', + ' local parent = parentOf(key)', + ' if parent then return busy[parent] == true end', + ' return busyChildren[key] == true', + 'end', + 'if free == nil or conflicts(ARGV[3]) then return -1 end', // Expiry removes queue metadata, never a workspace uncertainty fence. "local expired = redis.call('ZRANGEBYSCORE', KEYS[6], '-inf', ARGV[6])", 'for _, id in ipairs(expired) do', @@ -78,11 +108,15 @@ export class BridgeWorkspaceSlots { 'end', "local pending = redis.call('ZRANGE', KEYS[5], 0, 31)", 'local selected = nil', + 'local waitingCheckouts = {}', 'for _, id in ipairs(pending) do', " local workspace = redis.call('HGET', KEYS[7], id)", // An older serial request remains a barrier until its dispatcher finishes. ' if not workspace then return -1 end', - ' if not busy[workspace] then selected = id; break end', + ' local parent = parentOf(workspace)', + ' local held = conflicts(workspace) or (parent ~= nil and waitingCheckouts[parent] == true)', + ' if not held then selected = id; break end', + ' if parent == nil then waitingCheckouts[workspace] = true end', 'end', 'if selected ~= ARGV[2] then return -1 end', "redis.call('HSET', KEYS[1], 'a:' .. free, ARGV[2], 'i:' .. free, ARGV[1], 'w:' .. free, ARGV[3], 'e:' .. free, ARGV[5])", diff --git a/service/src/bridge/store.ts b/service/src/bridge/store.ts index 0ca75566..123e7ee2 100644 --- a/service/src/bridge/store.ts +++ b/service/src/bridge/store.ts @@ -18,6 +18,7 @@ import { isWorkspaceToolRequest, isWorkspaceToolResult, workspaceIsolationKey, + workspaceIsolationParent, } from '../../../packages/code/src/protocol'; import type { BridgeWorkerBinding } from './pairing'; import { BridgeAdmissionQueue } from './admission'; @@ -105,6 +106,25 @@ type AssignmentOwnership = Pick< | 'runtimeSessionId' >; +const NATIVE_WORKSPACE_FENCE_PREFIX = 'native-workspace:'; + +/** Fence keys of the linked-worktree lanes enqueued beneath a checkout fence. */ +function workspaceLaneFenceIndexKey(checkoutFenceKey: string): string { + return `${checkoutFenceKey}:lanes`; +} + +/** The fence of the checkout a linked-worktree lane nests beneath. While that + * checkout is quarantined, its lanes may not start either. */ +function workspaceFenceParent(fence: string): string | undefined { + if (!fence.startsWith(NATIVE_WORKSPACE_FENCE_PREFIX)) return undefined; + const parent = workspaceIsolationParent( + fence.slice(NATIVE_WORKSPACE_FENCE_PREFIX.length), + ); + return parent === undefined + ? undefined + : `${NATIVE_WORKSPACE_FENCE_PREFIX}${parent}`; +} + function workspaceFenceReceiptKey(assignmentId: string): string { return `${assignmentKey(assignmentId)}:workspace-fence-owner`; } @@ -151,6 +171,12 @@ function supportsWorkspaceTool( ) { return false; } + if ( + request.worktree !== undefined && + workspace.workspaceScopes?.includes('git_linked_worktree') !== true + ) { + return false; + } if (request.operation === 'list_files' && request.afterPath !== undefined) { return capabilities?.listFileFeatures?.includes('after_path') === true; } @@ -184,6 +210,7 @@ function supportsWorkspaceProgrammatic( workspaceId: string, language: string, workspaceInstanceId?: string, + worktree?: string, ): boolean { const capabilities = registration.capabilities.workspaceTools; const workspace = capabilities?.workspaces.find( @@ -193,6 +220,8 @@ function supportsWorkspaceProgrammatic( workspace != null && (workspaceInstanceId === undefined || workspace.workspaceInstances?.includes('git_worktree') === true) && + (worktree === undefined || + workspace.workspaceScopes?.includes('git_linked_worktree') === true) && capabilities?.operations.includes('execute_command') === true && (workspace.operations == null || workspace.operations.includes('execute_command')) && @@ -214,11 +243,24 @@ function workspaceInstanceId(body: t.PayloadBody): string | undefined { return undefined; } +function workspaceWorktree(body: t.PayloadBody): string | undefined { + if ( + typeof body === 'object' && + body != null && + 'workspace_worktree' in body && + typeof body.workspace_worktree === 'string' + ) { + return body.workspace_worktree; + } + return undefined; +} + export function workspaceAdmissionId( workspaceId: string, instanceId?: string, + worktree?: string, ): string { - return workspaceIsolationKey(workspaceId, instanceId); + return workspaceIsolationKey(workspaceId, instanceId, worktree); } function workerKey(workerId: string): string { @@ -862,6 +904,7 @@ export class RedisBridgeStore { args.workspaceId, args.body.language, workspaceInstanceId(args.body), + workspaceWorktree(args.body), ) ) { throw new BridgeStoreError( @@ -899,12 +942,15 @@ export class RedisBridgeStore { args.workspaceRequest?.workspaceId ?? args.workspaceId; const selectedWorkspaceInstanceId = args.workspaceRequest?.workspaceInstanceId ?? workspaceInstanceId(args.body); + const selectedWorktree = + args.workspaceRequest?.worktree ?? workspaceWorktree(args.body); const selectedWorkspaceAdmissionId = selectedWorkspaceId == null ? undefined : workspaceAdmissionId( selectedWorkspaceId, selectedWorkspaceInstanceId, + selectedWorktree, ); const workspaceSlots = selectedWorkspaceId != null && @@ -1033,6 +1079,7 @@ export class RedisBridgeStore { args.workspaceId, args.body.language, workspaceInstanceId(args.body), + workspaceWorktree(args.body), )) ) { throw new BridgeStoreError( @@ -1059,13 +1106,13 @@ export class RedisBridgeStore { leaseToken, leaseTokenHash: tokenHash(leaseToken), ...(selectedWorkspaceAdmissionId == null ? {} : { - workspaceFence: `native-workspace:${selectedWorkspaceAdmissionId}`, + workspaceFence: `${NATIVE_WORKSPACE_FENCE_PREFIX}${selectedWorkspaceAdmissionId}`, }), ...(workspaceLeaseSlot === undefined ? {} : { workspaceLeaseSlot, - workspaceFence: `native-workspace:${selectedWorkspaceAdmissionId!}`, + workspaceFence: `${NATIVE_WORKSPACE_FENCE_PREFIX}${selectedWorkspaceAdmissionId!}`, }), ...(registration.identityId != null ? { workerIdentityId: registration.identityId } @@ -1155,6 +1202,7 @@ export class RedisBridgeStore { args.workspaceId, args.body.language, workspaceInstanceId(args.body), + workspaceWorktree(args.body), ) ) { throw new BridgeStoreError( @@ -2185,6 +2233,13 @@ export class RedisBridgeStore { "if redis.call('GET', KEYS[1]) ~= ARGV[1] then return 0 end", 'if ARGV[7] ~= "" and redis.call(\'GET\', KEYS[6]) ~= ARGV[7] then return 0 end', "if #KEYS >= 7 and redis.call('EXISTS', KEYS[7]) == 1 then return -1 end", + // A lane refuses a quarantined checkout. A checkout reaches enqueue only once no + // lane beneath it holds a slot, so any lane fence still indexed here is stuck. + "if ARGV[9] == 'lane' and redis.call('EXISTS', KEYS[10]) == 1 then return -1 end", + "if ARGV[9] == 'checkout' then", + " for i = 11, #KEYS do if redis.call('EXISTS', KEYS[i]) == 1 then return -1 end end", + " for i = 11, #KEYS do redis.call('SREM', KEYS[10], KEYS[i]) end", + 'end', 'redis.call(\'SET\', KEYS[2], ARGV[2], \"EX\", ARGV[3])', "redis.call('RPUSH', KEYS[3], ARGV[4])", "redis.call('EXPIRE', KEYS[3], ARGV[3])", @@ -2193,7 +2248,8 @@ export class RedisBridgeStore { : []), 'redis.call(\'SET\', KEYS[5], "1", \"PXAT\", ARGV[6])', "if #KEYS >= 7 then redis.call('SET', KEYS[7], ARGV[4]) end", - 'if #KEYS == 9 then', + "if ARGV[9] == 'lane' then redis.call('SADD', KEYS[11], KEYS[7]) end", + 'if #KEYS >= 9 then', " local epoch = redis.call('GET', KEYS[9])", " if type(epoch) ~= 'string' then epoch = '0'; redis.call('SET', KEYS[9], epoch, 'EX', ARGV[3]) end", " if redis.call('PTTL', KEYS[9]) < tonumber(ARGV[3]) * 1000 then redis.call('EXPIRE', KEYS[9], ARGV[3]) end", @@ -2233,11 +2289,29 @@ export class RedisBridgeStore { workerIdentityId: assignment.workerIdentityId, expiresAt: assignment.expiresAt, }; + let fenceScope: '' | 'lane' | 'checkout' = ''; if (assignment.workspaceLeaseSlot !== undefined) { keys.push( workspaceFenceReceiptKey(assignment.assignmentId), `${workspaceQuarantineKey(assignment.workerId, assignment.workspaceFence!)}:epoch`, ); + const parent = workspaceFenceParent(assignment.workspaceFence!); + if (parent !== undefined) { + const parentFence = workspaceQuarantineKey(assignment.workerId, parent); + keys.push(parentFence, workspaceLaneFenceIndexKey(parentFence)); + fenceScope = 'lane'; + } else { + const index = workspaceLaneFenceIndexKey( + workspaceQuarantineKey(assignment.workerId, assignment.workspaceFence!), + ); + const laneFences = await boundedCommand( + this.redis.smembers(index), + this.redisCommandTimeoutMs, + 'Bridge linked worktree fence index read', + ); + keys.push(index, ...laneFences); + fenceScope = 'checkout'; + } } const result = await this.redis.eval( script, @@ -2251,6 +2325,7 @@ export class RedisBridgeStore { String(Date.parse(assignment.expiresAt)), readyToken ?? '', JSON.stringify(receipt), + fenceScope, ); if (Number(result) === -1) { throw new BridgeStoreError( diff --git a/service/src/service/programmatic-router.ts b/service/src/service/programmatic-router.ts index a646b9ad..1e2a367c 100644 --- a/service/src/service/programmatic-router.ts +++ b/service/src/service/programmatic-router.ts @@ -92,7 +92,11 @@ import { CODEAPI_BRIDGE_WORKSPACE_HEADER, resolveBridgeWorkerSelection, } from '../bridge/selection'; -import { isValidBridgeWorkerId, BRIDGE_WORKSPACE_PROGRAMMATIC_MAX_INPUT_FILES } from '../../../packages/code/src/protocol'; +import { + isValidBridgeWorkerId, + isValidLinkedWorktreeName, + BRIDGE_WORKSPACE_PROGRAMMATIC_MAX_INPUT_FILES, +} from '../../../packages/code/src/protocol'; import logger from '../logger'; import { type ExecutionState, @@ -505,6 +509,7 @@ async function handleReplayInitial( bridgeWorkerId?: string; workspaceId?: string; workspaceInstanceId?: string; + workspaceWorktree?: string; }, cancellation: ReplayRequestCancellation, ): Promise { @@ -514,6 +519,7 @@ async function handleReplayInitial( bridgeWorkerId, workspaceId, workspaceInstanceId, + workspaceWorktree, } = params; const { code, tools, user_id, files } = req.body as t.ProgrammaticRequestBody; @@ -671,6 +677,7 @@ async function handleReplayInitial( bridgeWorkerId, workspaceId, workspaceInstanceId, + workspaceWorktree, executionProfile: env.EXECUTION_PROFILE, executionProfileSource: env.EXECUTION_PROFILE_SOURCE, sandboxBackend: resolveReplayStateSandboxBackend({ @@ -1291,6 +1298,7 @@ router.post( let bridgeWorkerId: string | undefined; let workspaceId: string | undefined; let workspaceInstanceId: string | undefined; + let workspaceWorktree: string | undefined; if (continuation_token == null || continuation_token === '') { try { const bridgeSelection = resolveBridgeWorkerSelection({ @@ -1341,6 +1349,15 @@ router.post( principalId: principal.userId, }); } + const requestedWorktree = rawBody.workspace_worktree; + if (requestedWorktree !== undefined) { + if (workspaceId == null || !isValidLinkedWorktreeName(requestedWorktree)) { + return res.status(400).json({ + error: 'Invalid code workspace worktree', + }); + } + workspaceWorktree = requestedWorktree; + } } catch (error) { if (error instanceof BridgeWorkerSelectionError) { return res @@ -1458,6 +1475,7 @@ router.post( bridgeWorkerId, workspaceId, workspaceInstanceId, + workspaceWorktree, }, cancellation); } if (workspaceId != null) { diff --git a/service/src/service/programmatic-state.ts b/service/src/service/programmatic-state.ts index 38c72486..056fcbb5 100644 --- a/service/src/service/programmatic-state.ts +++ b/service/src/service/programmatic-state.ts @@ -40,6 +40,7 @@ export interface BuildReplayExecutionStateParams { bridgeWorkerId?: string; workspaceId?: string; workspaceInstanceId?: string; + workspaceWorktree?: string; sandboxBackend?: SandboxBackendName; executionProfile: ExecutionProfile; executionProfileSource: ExecutionProfileSource; @@ -70,6 +71,7 @@ export function buildReplayExecutionState( bridgeWorkerId: params.bridgeWorkerId, workspaceId: params.workspaceId, workspaceInstanceId: params.workspaceInstanceId, + workspaceWorktree: params.workspaceWorktree, sandboxBackend: params.sandboxBackend, executionProfile: params.executionProfile, executionProfileSource: params.executionProfileSource, @@ -86,12 +88,21 @@ export function buildReplayExecutionState( }; } -/** Bind the authenticated conversation checkout to every replay iteration. */ +/** Bind the authenticated conversation checkout and linked-worktree lane to every replay iteration. */ export function bindReplayWorkspaceInstance( payload: t.PayloadBody, - state: Pick, + state: Pick, ): t.PayloadBody { - return state.workspaceInstanceId == null - ? payload - : { ...payload, workspace_instance_id: state.workspaceInstanceId }; + if (state.workspaceInstanceId == null && state.workspaceWorktree == null) { + return payload; + } + return { + ...payload, + ...(state.workspaceInstanceId == null + ? {} + : { workspace_instance_id: state.workspaceInstanceId }), + ...(state.workspaceWorktree == null + ? {} + : { workspace_worktree: state.workspaceWorktree }), + }; } diff --git a/service/src/service/replay-state.ts b/service/src/service/replay-state.ts index fd4e90e3..003ce6b5 100644 --- a/service/src/service/replay-state.ts +++ b/service/src/service/replay-state.ts @@ -120,6 +120,8 @@ export interface ExecutionState { workspaceId?: string; /** Selected conversation checkout retained across every replay iteration. */ workspaceInstanceId?: string; + /** Selected linked-worktree lane retained across every replay iteration. */ + workspaceWorktree?: string; /** Original queue/backend target retained across replay continuations. */ sandboxBackend?: SandboxBackendName; /** Original producer profile retained so continuations use the same queue. */ diff --git a/service/src/types/service.ts b/service/src/types/service.ts index 91c3ed89..b0375b38 100644 --- a/service/src/types/service.ts +++ b/service/src/types/service.ts @@ -206,6 +206,8 @@ export interface PayloadBody { version: string; /** Opaque conversation checkout selected and authenticated by the API. */ workspace_instance_id?: string; + /** Linked worktree lane at `.worktrees/` beneath the selected checkout. */ + workspace_worktree?: string; /** Stable identity shared by all replay iterations of one execution. */ execution_id?: string; replay_tool_count?: number; @@ -397,6 +399,8 @@ export interface ProgrammaticRequestBody { lang?: 'python' | 'bash'; /** Opaque conversation checkout binding for a selected native workspace. */ workspace_instance_id?: string; + /** Linked worktree lane at `.worktrees/` beneath the selected checkout. */ + workspace_worktree?: string; } export interface ProgrammaticToolCall {