Repository navigation
Description
With 5 or more agents committing at once on the native machine, some commits stall for 1 to 3.8 s (up to 4.3 s in one run with 8 agents) while the other agents' commits keep going through, up to 58 of them during one wait. I saw it on #570 (6be2f6c) and on the base d430474 (stalls of 2.0 to 3.8 s), on my M3 Max, macOS 26.6.2, release build. Each agent is its own node process with its own Sandbox and Branch: fork, sandbox.create, mount, then rounds of writing 20 small files from the host and workspace.commit, with all agents starting at the same moment.
I ran sample on all five daemons and agents during a run where two commits stalled (3.1 and 3.7 s). The two stuck daemons had their control thread (layerfs-control-0) in sync in every sample, called from flush_export (engine/crates/layerfs-daemon/src/control/export.rs:584, the same at f2896f7), and their node processes were waiting in recvfrom. The other three daemons had it in sync for about 80% of the samples too. With LAYERFS_NFS_FLUSH unset, every commit calls nix::unistd::sync(), which is sync(2) for every mounted volume on the Mac, the other agents' NFS mounts included. I didn't find out why one call blocks for seconds while the other agents' calls keep returning
To check that the flush only needs this Workspace's mount, I set LAYERFS_NFS_FLUSH to a small C helper that calls sync_volume_np(dir, SYNC_VOLUME_WAIT) (macOS 10.8+) on {dir}. The daemon starts with a cleared environment, so the variable goes in a wrapper script passed to machine.update("native", {daemon}). I ran the same 5 and 10 agent runs again, the helper ran on all 400 commits, each time on that Workspace's mount, and the stalls were gone:
sync(2), as now sync_volume_np on the mount
5 agents x 20 commits, p50 85-162 ms 11-12 ms
5 agents x 20 commits, max 3.7 s 25 ms
10 agents x 10 commits, p50 114-312 ms 13-15 ms
10 agents x 10 commits, max 2.4 s 34 ms
commits over 1 s 20 of 400 0 of 400
one agent, 200 files a commit 66-80 ms 23-25 ms
(3 runs of 5 agents and 1 of 10 in each column, p50 is per agent.)
A commit still gets what the host wrote. Each round of my check writes 200 small files and closes them, writes 128 KB to one file and a short line to another and leaves both open, commits, and reads them back from the Branch with file.read and file.stat. With the helper set to /usr/bin/true the short line was missing after 4 of 5 commits. With sync(2) and with the per-volume helper it was there 5 of 5, and every time the 128 KB file had its full size and the small files I read back (40 of 200) matched.
So I think on macOS flush_export could call sync_volume_np on the Workspace's mount instead of sync(2). With SYNC_VOLUME_WAIT it waits for that mount's write-back, which sync(2) doesn't promise (man 2 sync: "may return before the buffers are completely flushed"). libc 0.2.189 doesn't bind it and layerfs-daemon has #![forbid(unsafe_code)], so it needs one extern "C" somewhere. Should I send a PR for that, and if so, where would you want the extern? I only ran this on macOS with my scripts, not with real coding agents.
Stalled commits, scripts and the helper
Commits over 1 s in the 5 agent runs: run, agent, commit index, ms, commits by the other agents that started and finished during it, other agents that had already done all their commits. b5* and bs5* are the base (bs5 with starts spread over 3 s; in b5b one agent stopped at the start with Store busy, so 4 committed), x5* and sm are #570, sm is the sampled run.
b5a b5a2 0 3725 57 3/4
b5a b5a4 0 3787 58 3/4
b5b b5b3 0 3738 58 3/4
bs5a bs5a2 0 3015 49 3/4
bs5b bs5b5 11 2000 32 2/4
sm sm2 3 3069 48 1/4
sm sm5 3 3729 58 3/4
x5a x5a4 0 3266 57 3/4
x5a x5a5 0 3320 57 3/4
x5b x5b2 0 3677 57 3/4
x5b x5b5 0 3733 58 3/4
x5c x5c1 0 1769 29 0/4
x5c x5c3 10 1778 27 1/4
x5c x5c4 10 3638 57 4/4
x5c x5c5 0 1708 28 0/4
One agent (conc-570.mjs), on a Store vps with a project vps/vps:
// One "agent": own process, own Sandbox, own Branch; the first Store call opens the Store at openAt, then commits in a loop.
// usage: LAYERFS_HOME=<home> node conc-570.mjs <repo> <name> <rounds> <openAtEpochMs>
import { writeFileSync } from "node:fs";
import { join } from "node:path";
const [repo, name, rounds = "20", openAt = "0"] = process.argv.slice(2);
const lfs = await import(join(repo, "api/node/src/index.js"));
const P = "vps/vps";
const out = { name, ok: 0, errors: {} };
const fail = (stage, e) => { out.errors[`${stage}:${e.code}`] = (out.errors[`${stage}:${e.code}`] ?? 0) + 1; out.first ??= `${stage}: ${String(e.message).slice(0, 300)}`; };
const times = []; const spans = [];
await new Promise((r) => setTimeout(r, Math.max(0, +openAt - Date.now())));
const t0 = performance.now();
let ws, sb;
try {
try { await lfs.branch.fork(P, name, "main"); out.openMs = Math.round(performance.now() - t0); }
catch (e) { fail("open", e); throw e; }
sb = `sb-${name}`;
await lfs.sandbox.create(P, sb, "native");
ws = (await lfs.workspace.mount(sb, name)).ws;
const path = (await lfs.sandbox.get(sb)).workspaces.find((w) => w.ws === ws).path;
out.readyAt = Date.now();
for (let i = 0; i < +rounds; i++) {
for (let j = 0; j < 20; j++) writeFileSync(join(path, `${name}-${i}-${j}.txt`), `${i} ${j}\n`);
const c0 = performance.now();
try { await lfs.workspace.commit(ws, { message: `${name} ${i}` }); times.push(performance.now() - c0); spans.push([Math.round(performance.timeOrigin + c0), Math.round(performance.timeOrigin + performance.now())]); out.ok++; }
catch (e) { fail("commit", e); }
}
} catch (e) { if (!out.first) fail("setup", e); }
finally {
out.teardownAt = Date.now();
try { if (ws) await lfs.workspace.unmount(ws, { discard: true }); if (sb) await lfs.sandbox.close(sb); }
catch (e) { fail("teardown", e); }
lfs.close();
times.sort((a, b) => a - b);
out.p50 = Math.round(times[times.length >> 1] ?? 0); out.max = Math.round(times.at(-1) ?? 0);
out.totalMs = Math.round(performance.now() - t0); out.endAt = Date.now();
if (process.env.SPANS) out.spans = spans;
console.log(JSON.stringify(out));
}Start N agents at the same moment (conc-run-570.sh <home> <label> <agents> <rounds> [stagger_ms], run with SPANS=1):
#!/bin/sh
# usage: conc-run-570.sh <home> <label> <agents> <rounds> [stagger_ms]
# starts <agents> processes whose first Store call (the open) lands at the same instant (+ random stagger)
H=$1 L=$2 N=$3 R=$4 S=${5:-0}
D=$(cd "$(dirname "$0")" && pwd)
at=$(( $(date +%s) * 1000 + 3000 ))
for i in $(seq 1 "$N"); do
off=0; [ "$S" -gt 0 ] && off=$(( $(od -An -N2 -tu2 /dev/urandom) % S ))
LAYERFS_HOME=$H perl -e 'alarm 300; exec @ARGV' node "$D/conc-570.mjs" "$D/${REPO:-pr570}" "$L$i" "$R" $((at + off)) > "$D/c570/$L$i.log" 2>&1 &
done
wait
for i in $(seq 1 "$N"); do tail -c 600 "$D/c570/$L$i.log"; doneThe helper and the wrapper set with machine.update("native", { daemon: ".../d-volsync.sh" }):
// LAYERFS_NFS_FLUSH helper: write back one volume (the Workspace's NFS mount) instead of sync(2).
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char **argv) {
if (argc != 2) return 2;
const char *log = getenv("VOLSYNC_LOG");
if (log) { FILE *f = fopen(log, "a"); if (f) { fprintf(f, "%s\n", argv[1]); fclose(f); } }
if (sync_volume_np(argv[1], SYNC_VOLUME_WAIT) != 0) { perror("sync_volume_np"); return 1; }
return 0;
}#!/bin/sh
export LAYERFS_NFS_FLUSH="<tools>/volsync {dir}" VOLSYNC_LOG=<tools>/volsync.log
exec <repo>/target/release/layerfs-daemon "$@"The flush check (flushcheck.mjs <repo> <daemon> <home> <src> [rounds]):
// Does a commit see what the host just wrote? Small files written and closed, plus one file still open with unflushed writes.
// usage: node flushcheck.mjs <repo> <daemon> <home> <src> [rounds]
import { closeSync, openSync, writeFileSync, writeSync } from "node:fs";
import { randomBytes } from "node:crypto";
import { join } from "node:path";
const [repo, daemon, home, src, rounds = "5"] = process.argv.slice(2);
const sdk = await import(join(repo, "api/node/src/index.js"));
const lfs = sdk.client({ home });
const P = "fc/x";
await lfs.machine.update("native", { daemon });
await lfs.store.create("fc");
await lfs.project.init(P, src);
await lfs.branch.fork(P, "b", "main");
await lfs.sandbox.create(P, "fcs", "native");
const { ws, path } = await lfs.workspace.mount("fcs", "b");
const out = [];
try {
for (let r = 0; r < +rounds; r++) {
const want = {};
for (let i = 0; i < 200; i++) { const t = randomBytes(32).toString("hex") + "\n"; want[`/r${r}-${i}.txt`] = t; writeFileSync(join(path, `r${r}-${i}.txt`), t); }
const fd = openSync(join(path, `open${r}.txt`), "w");
const big = randomBytes(64 * 1024).toString("hex");
writeSync(fd, big); // left open across the commit
const fd2 = openSync(join(path, `tiny${r}.txt`), "w");
const tiny = `tiny ${r} ${randomBytes(8).toString("hex")}\n`;
writeSync(fd2, tiny); // a small write, still open and buffered
const t0 = performance.now();
await lfs.workspace.commit(ws, { message: `r${r}` });
const ms = Math.round(performance.now() - t0);
let missing = 0, wrong = 0;
for (const [p, t] of Object.entries(want).slice(0, 40)) {
try { if ((await lfs.file.read(P, "b", p)).text !== t) wrong++; } catch { missing++; }
}
let openSize;
try { openSize = (await lfs.file.stat(P, "b", `/open${r}.txt`)).size; } catch (e) { openSize = e.code; }
let tinyOk;
try { tinyOk = (await lfs.file.read(P, "b", `/tiny${r}.txt`)).text === tiny; } catch (e) { tinyOk = e.code; }
closeSync(fd); closeSync(fd2);
out.push({ r, commitMs: ms, small: `${40 - missing - wrong}/40 ok`, missing, wrong, openFile: `${openSize}/${big.length}`, tinyOpen: tinyOk });
}
} finally {
await lfs.workspace.unmount(ws, { discard: true });
await lfs.sandbox.close("fcs");
sdk.close();
}
console.log(JSON.stringify(out));
Activity
jiangkoumo commentedon Nov 10, 2026
@ImmuneFOMO 感谢你的 sample 分析和 sync_volume_np 试验,按这个思路开了 #647,commit 里加了你的 Co-authored-by。
改法:macOS 上 Commit 前只刷 export 挂载点所在卷(sync_volume_np + SYNC_VOLUME_WAIT),失败时回退 sync(),不跳过 flush。
我们在 M2 Air 上测得:3 会话并发 commit p50 约 130 ms → 12 ms。方便的话请在你 5+ agent 的场景下复测一下,看卡顿是否消失。
ImmuneFOMO commentedon Nov 10, 2026
Retested #647 (1c0a6fd) against its base bdeb9e9 on my M3 Max, macOS 26.6.2, with the scripts from #578 (each round writes 20 files and commits, all agents start at once). The stalls are gone:
flushcheck passes 5 of 5 rounds on #647 (40/40 small files, the open 128 KiB file and the short line all read back).
The commit counts differ because many commits on #647 failed: 96 of 240 in the 5-agent runs and 50 of 100 in the 10-agent run. The first failures are
Store busy before effectorService(Busy), and on some Sandboxes the later ones areadmitted no connection for 60s. On bdeb9e9 with my volsync helper from #578 it's the same, and on an older #570 head (6be2f6c, on an older base) with the helper it was 0 of 400. So I think it's the #534 Busy, which now shows up because commits no longer wait in sync(2), and not #647 itself. 7 Sandboxes kept their daemon and NFS mount after the agent exited. Closing them later, 3 gotBusyafter 60 s (all 8 control slots taken, looks like #540), 2 gotMountsActiveand 2 closed. Maybe #570 should go in before #647 or together with itjiangkoumo commentedon Nov 11, 2026
@ImmuneFOMO 感谢你昨天的复测,并发 commit 的 Store busy 已经修了,都合在 #647(头 7d2bfbd),#570 已关闭并入。
改动:
• 把 #570 的 macOS 写入准入(目录 flock,非阻塞)移植进来,拿不到锁返回确定的 Busy,不再落到 SQLITE_BUSY
• digest 拒绝归为确定 Busy,不再判成「结果未知」去占控制槽,残留 daemon/挂载的问题一并消失
我们用你的脚本在 macOS 27.2(M2)上跑了 5×20×3 和 10×10×2,共 500 次:0 失败、0 残留,flushcheck 5/5。Linux 全量无新增失败,并发 commit Busy 0/8700。
方便的话请在 #647 新头上再跑一遍你那套(尤其 M3 Max 和 10 个以上 agent,我们没测到),有问题直接在 #647 说。
Mola-maker commentedon Nov 11, 2026
molamolamiao
ImmuneFOMO commentedon Nov 14, 2026
Fixed by #647. On layerfs_v2_0_0 at 8cfb6e2 on my M3 Max, macOS 26.6.2, with 5 agents x 20 commits and then 10 x 10 all starting at once, 200 of 200 commits went through with no Busy and a max of 34 ms, and flushcheck passes 5 of 5. Closing