Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/remote-bridge/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,10 @@ execution.
- Code API negotiates a bounded number of active workspace assignments per
worker. The lower API or worker slot ceiling wins, and assignments sharing
the same workspace isolation key remain serialized while independent
conversation worktrees may run concurrently.
conversation worktrees may run concurrently. A linked-worktree lane
(`worktree: <name>`) nests beneath its checkout's key: sibling lanes run
concurrently, while a lane and its checkout exclude each other, and a lane
cannot start while its checkout is quarantined.
- Workspace tool admission waits for capacity up to 30 seconds without a
`X-LibreChat-Workspace-Queue-Wait-Ms` header. A caller can advertise a longer
per-request allowance, bounded by five minutes and any server queue ceiling.
Expand Down
6 changes: 5 additions & 1 deletion docs/remote-bridge/projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,11 @@ workspace registration remains available for independent project directories.
filesystem boundary and coordination for the common Git directory.
- A worktree beneath its parent checkout overlaps that checkout. Either use
disjoint execution roots under a discovery grant or explicitly exclude and
coordinate descendant worktrees before relaxing root exclusion.
coordinate descendant worktrees before relaxing root exclusion. Linked
worktree lanes (`--linked-worktree-lanes`) take the second approach for
`.worktrees/<name>`: hierarchical admission keeps a lane and its checkout
exclusive, and each lane's sandbox can write only its worktree, shared
object and ref storage, and its own worktree metadata.
- Setup and dependency links must remain within the execution policy. Sharing
writable dependency directories between supposedly isolated worktrees
reintroduces overlap and requires an explicit operator decision.
Expand Down
56 changes: 56 additions & 0 deletions packages/code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -791,6 +791,57 @@ Legacy requests without a conversation identity continue to use the selected
source root. Older Code API deployments do not negotiate the capability, so the
worker omits it until every request path understands the isolation boundary.

#### Linked worktree lanes

Agents that keep one checkout and give each task its own linked worktree
(`git worktree add .worktrees/<task>`) can run those tasks concurrently:

```sh
librechat-code run \
--worker-dir /projects/LibreChat \
--workspace-lease-slots 4 \
--linked-worktree-lanes \
--allow-workspace-writes \
--allow-workspace-commands
```

`LIBRECHAT_CODE_LINKED_WORKTREE_LANES=true` is the environment equivalent. The
worker then advertises `workspaceScopes: ['git_linked_worktree']` for each
registered root, and a request that names `worktree: <name>` runs in its own
lane at `<root>/.worktrees/<name>`, with `cwd` and file paths relative to that
worktree. Sibling lanes run concurrently up to the negotiated slot count. A
lane and its checkout never run at the same time: requests without a
`worktree`, including `git worktree add` or `remove` run at the root, wait for
every lane beneath the checkout, and a waiting root request holds back newer
lanes so it cannot be starved.

Before admission the worker verifies, without running Git, that the directory
is a real linked worktree of that checkout: no symlinks on the path, a `.git`
file pointing at `<root>/.git/worktrees/<name>`, and metadata whose `commondir`
and `gitdir` point back. Shared Git storage paths granted for writes must be
real directories, not symlinks into sibling metadata; optional log and LFS
paths may be absent. A lane's sandbox can write only its worktree, the
shared object and ref storage (`.git/objects`, `.git/refs`, `.git/logs/refs`,
`.git/lfs`) and its own `.git/worktrees/<name>` metadata. Everything else in
`.git` stays read-only: configuration, hooks and `info`, the checkout's own
`HEAD`, index and merge or rebase state, and sibling metadata. Automatic `gc`
and maintenance are disabled, and `git gc` itself cannot run in a lane (it
needs to write `.git/gc.pid` and `packed-refs`); run storage maintenance from
the checkout. Explicit `git prune --expire now`, `git repack -ad`, or LFS
pruning can still delete objects another lane is writing; the sandbox does
**not** prevent this race. Do not enable lanes for agents that run destructive
maintenance until those commands are blocked or serialized. Deleting a branch
or tag also needs the checkout, because Git locks `packed-refs` for every ref
deletion. Each lane has its own durable quarantine guard. A lane
cannot start while its checkout is quarantined, and a checkout cannot start
while any lane beneath it is.

Lanes require native-srt commands and at least two lease slots, and cannot yet
be combined with conversation worktrees. Code API must advertise
`supportedWorkspaceScopes`; older deployments do not, and the worker omits the
scope for them. Deploy consumers that read worker status (such as LibreChat)
with support for `workspaceScopes` before enabling lanes on a worker.

On an updated Code API, admission waits up to 30 seconds without the
`X-LibreChat-Workspace-Queue-Wait-Ms` request header. A caller may advertise a
positive integer millisecond allowance up to five minutes, capped by any server
Expand Down Expand Up @@ -850,6 +901,11 @@ To recover a quarantined native root:
3. Run the normal worker command with all its root/slot options plus `--reset-workspace-quarantine second`. This verifies the local guard is cleared, resets the server fence, then exits.
4. Restart the normal worker command without the reset option.

A linked-worktree lane keeps its own guard and fence. Inspect or restore
`<root>/.worktrees/<name>`, clear its guard with
`--worker-dir <root>/.worktrees/<name>`, then add
`--reset-workspace-worktree <name>` to the reset command in step 3.

The workspace selector in LibreChat must preserve these registered IDs. Adding
roots here does not grant a principal access or change an agent's selected root.
# Named project environments
Expand Down
106 changes: 101 additions & 5 deletions packages/code/src/cli.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
#!/usr/bin/env node
import { createHash, createHmac, randomBytes } from 'node:crypto';
import { readFileSync } from 'node:fs';
import { realpath, stat } from 'node:fs/promises';
import { basename, resolve, relative, isAbsolute, sep } from 'node:path';
import { readdir, realpath, stat } from 'node:fs/promises';
import { basename, join, resolve, relative, isAbsolute, sep } from 'node:path';

import { pairBridgeWorker } from './pairing.js';
import { discoverProjects } from './projects.js';
Expand Down Expand Up @@ -37,6 +37,7 @@ import { RuntimeWorkspaceCommandSandbox } from './workspace-runtime.js';
import { NativeProcessWorkspaceCommandSandbox } from './native-process.js';
import { NativeWorkspaceCommandPool } from './native-pool.js';
import { GitWorktreeWorkspaceTools, internalWorkspaceId } from './workspace-instances.js';
import { LINKED_WORKTREE_DIRECTORY, LinkedWorktreeWorkspaceTools } from './linked-worktrees.js';
import { GitWorktreeManager } from './worktrees.js';
import { captureWorkspaceRootIdentity } from './root-identity.js';
import {
Expand Down Expand Up @@ -66,6 +67,7 @@ import {
BridgeProtocolError,
isValidBridgeWorkerCapabilities,
isValidBridgeWorkerId,
isValidLinkedWorktreeName,
workspaceIsolationKey,
} from './protocol.js';

Expand Down Expand Up @@ -514,6 +516,11 @@ async function run(
(args.includes('--allow-workspace-commands') ||
process.env.LIBRECHAT_CODE_ALLOW_WORKSPACE_COMMANDS?.trim().toLowerCase() ===
'true');
const linkedWorktreeLanes =
runtimeSessionId == null &&
(args.includes('--linked-worktree-lanes') ||
process.env.LIBRECHAT_CODE_LINKED_WORKTREE_LANES?.trim().toLowerCase() ===
'true');
const commandSandboxMode =
option(args, '--command-sandbox') ??
process.env.LIBRECHAT_CODE_COMMAND_SANDBOX?.trim().toLowerCase() ??
Expand Down Expand Up @@ -783,6 +790,21 @@ async function run(
'Conversation worktrees require native-srt commands and at least two workspace lease slots',
);
}
if (
linkedWorktreeLanes &&
(!allowWorkspaceCommands ||
commandSandboxMode !== 'native-srt' ||
workspaceLeaseSlots < 2)
) {
throw new Error(
'Linked worktree lanes require native-srt commands and at least two workspace lease slots',
);
}
if (linkedWorktreeLanes && conversationWorktreeRoot) {
throw new Error(
'Linked worktree lanes cannot be combined with conversation worktrees',
);
}
if (
roots.length > 1 &&
process.env.LIBRECHAT_CODE_WORKSPACE_QUARANTINE_FILE?.trim()
Expand Down Expand Up @@ -1073,7 +1095,7 @@ async function run(
};
const nativeCommandSandbox =
allowWorkspaceCommands && commandSandboxMode === 'native-srt'
? roots.length > 1 || workspaceLeaseSlots > 1 || conversationWorktreeRoot
? roots.length > 1 || workspaceLeaseSlots > 1 || conversationWorktreeRoot || linkedWorktreeLanes
? new NativeWorkspaceCommandPool(
new Map(
roots.map(root => [
Expand Down Expand Up @@ -1199,6 +1221,41 @@ async function run(
),
});
workspaceTools = conversationWorkspaceTools;
}
let linkedWorktreeTools: LinkedWorktreeWorkspaceTools | undefined;
if (linkedWorktreeLanes && workspaceTools) {
if (!(nativeCommandSandbox instanceof NativeWorkspaceCommandPool)) {
throw new Error('Linked worktree lanes require a native command pool');
}
linkedWorktreeTools = new LinkedWorktreeWorkspaceTools({
commandPool: nativeCommandSandbox,
delegate: workspaceTools,
programmaticDelegate: conversationWorkspaceTools ?? nativeCommandSandbox,
onResolve(workspaceId, root) {
if (admittedGitHubRepositories) {
admittedGitHubRepositories.set(
root,
repositoriesByWorkspace?.get(workspaceId),
);
}
},
onRelease(root) {
admittedGitHubRepositories?.delete(root);
},
sources: new Map(
roots.map((root) => [
root.id,
{
root: root.root,
identity: root.identity,
command: nativeOptions,
repositoryInstructions: args.includes('--repository-instructions'),
writable: root.writable ?? false,
},
]),
),
});
workspaceTools = linkedWorktreeTools;
}
if (workspaceTools && environments.length) {
workspaceTools = new EnvironmentWorkspaceTools(
Expand Down Expand Up @@ -1304,7 +1361,7 @@ async function run(
...(nativeProgrammaticEnabled && nativeCommandSandbox
? {
workspaceProgrammatic:
conversationWorkspaceTools ?? nativeCommandSandbox,
linkedWorktreeTools ?? conversationWorkspaceTools ?? nativeCommandSandbox,
}
: {}),
...(conversationWorktrees
Expand All @@ -1331,6 +1388,40 @@ async function run(
),
}
: {}),
...(linkedWorktreeTools
? {
linkedWorktreeQuarantineResolver: (
selectedWorkspaceId: string,
worktree: string,
) => {
const source = roots.find((root) => root.id === selectedWorkspaceId);
if (!source) {
throw new BridgeProtocolError('Linked worktree source is not registered');
}
return workspaceMutationGuard(
defaultWorkspaceQuarantinePath({
codeApiUrl,
workerId,
workspaceRoot: join(source.root, LINKED_WORKTREE_DIRECTORY, worktree),
}),
workerId,
workspaceIsolationKey(selectedWorkspaceId, undefined, worktree),
incarnationId,
Comment on lines +1405 to +1409

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add a reset path for linked-worktree quarantines

A failed or ambiguous lane execution now creates a durable guard keyed by workspaceIsolationKey(..., worktree), but the only operator recovery path, BridgeWorker.resetNativeWorkspace, accepts only a root or workspaceInstanceId and resolves only workspaceQuarantines/workspaceQuarantineResolver; the CLI likewise exposes only --reset-workspace-instance. After clearing the local file, the server-side lane fence therefore cannot be acknowledged and removed through the supported tooling, leaving that worktree name permanently unusable unless Redis is edited or the lane is renamed. Extend the reset protocol and CLI to select the linked-worktree guard and isolation key.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in abf5349: resetNativeWorkspace accepts a worktree, resolves the lane guard, and resets native-workspace:<lane key>. The CLI gains --reset-workspace-worktree <name>, and the README recovery steps cover lanes.

);
},
linkedWorktreeNames: async (selectedWorkspaceId: string) => {
const source = roots.find((root) => root.id === selectedWorkspaceId);
if (!source) return [];
try {
const entries = await readdir(join(source.root, LINKED_WORKTREE_DIRECTORY));
return entries.filter(isValidLinkedWorktreeName);
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return [];
throw error;
}
},
}
: {}),
...(workspaceLeaseSlots > 1 || roots.length > 1
? {
workspaceQuarantines: new Map(
Expand Down Expand Up @@ -1445,15 +1536,20 @@ async function run(
args,
'--reset-workspace-instance',
);
const resetWorkspaceWorktree = option(
args,
'--reset-workspace-worktree',
);
await worker.refreshCredential(controller.signal);
await worker.registerForMaintenance(controller.signal);
await worker.resetNativeWorkspace(
resetNativeRoot,
controller.signal,
resetWorkspaceInstance,
resetWorkspaceWorktree,
);
process.stdout.write(
`librechat-code: reset acknowledged for native workspace ${resetNativeRoot}${resetWorkspaceInstance ? ` instance ${resetWorkspaceInstance}` : ''}\n`,
`librechat-code: reset acknowledged for native workspace ${resetNativeRoot}${resetWorkspaceInstance ? ` instance ${resetWorkspaceInstance}` : ''}${resetWorkspaceWorktree ? ` worktree ${resetWorkspaceWorktree}` : ''}\n`,
);
return;
}
Expand Down
Loading
Loading