Skip to content

native macOS: with 5+ agents a commit can stall for seconds in sync(2) in flush_export #578

Closed

Description

@ImmuneFOMO

With 5 or more agents committing at once on the native machine, some commits stall for 1 to 3.8 s (up to 4.3 s in one run with 8 agents) while the other agents' commits keep going through, up to 58 of them during one wait. I saw it on #570 (6be2f6c) and on the base d430474 (stalls of 2.0 to 3.8 s), on my M3 Max, macOS 26.6.2, release build. Each agent is its own node process with its own Sandbox and Branch: fork, sandbox.create, mount, then rounds of writing 20 small files from the host and workspace.commit, with all agents starting at the same moment.

I ran sample on all five daemons and agents during a run where two commits stalled (3.1 and 3.7 s). The two stuck daemons had their control thread (layerfs-control-0) in sync in every sample, called from flush_export (engine/crates/layerfs-daemon/src/control/export.rs:584, the same at f2896f7), and their node processes were waiting in recvfrom. The other three daemons had it in sync for about 80% of the samples too. With LAYERFS_NFS_FLUSH unset, every commit calls nix::unistd::sync(), which is sync(2) for every mounted volume on the Mac, the other agents' NFS mounts included. I didn't find out why one call blocks for seconds while the other agents' calls keep returning

To check that the flush only needs this Workspace's mount, I set LAYERFS_NFS_FLUSH to a small C helper that calls sync_volume_np(dir, SYNC_VOLUME_WAIT) (macOS 10.8+) on {dir}. The daemon starts with a cleared environment, so the variable goes in a wrapper script passed to machine.update("native", {daemon}). I ran the same 5 and 10 agent runs again, the helper ran on all 400 commits, each time on that Workspace's mount, and the stalls were gone:

                               sync(2), as now    sync_volume_np on the mount
5 agents x 20 commits, p50     85-162 ms          11-12 ms
5 agents x 20 commits, max     3.7 s              25 ms
10 agents x 10 commits, p50    114-312 ms         13-15 ms
10 agents x 10 commits, max    2.4 s              34 ms
commits over 1 s               20 of 400          0 of 400
one agent, 200 files a commit  66-80 ms           23-25 ms

(3 runs of 5 agents and 1 of 10 in each column, p50 is per agent.)

A commit still gets what the host wrote. Each round of my check writes 200 small files and closes them, writes 128 KB to one file and a short line to another and leaves both open, commits, and reads them back from the Branch with file.read and file.stat. With the helper set to /usr/bin/true the short line was missing after 4 of 5 commits. With sync(2) and with the per-volume helper it was there 5 of 5, and every time the 128 KB file had its full size and the small files I read back (40 of 200) matched.

So I think on macOS flush_export could call sync_volume_np on the Workspace's mount instead of sync(2). With SYNC_VOLUME_WAIT it waits for that mount's write-back, which sync(2) doesn't promise (man 2 sync: "may return before the buffers are completely flushed"). libc 0.2.189 doesn't bind it and layerfs-daemon has #![forbid(unsafe_code)], so it needs one extern "C" somewhere. Should I send a PR for that, and if so, where would you want the extern? I only ran this on macOS with my scripts, not with real coding agents.

Stalled commits, scripts and the helper

Commits over 1 s in the 5 agent runs: run, agent, commit index, ms, commits by the other agents that started and finished during it, other agents that had already done all their commits. b5* and bs5* are the base (bs5 with starts spread over 3 s; in b5b one agent stopped at the start with Store busy, so 4 committed), x5* and sm are #570, sm is the sampled run.

b5a b5a2 0 3725 57 3/4
b5a b5a4 0 3787 58 3/4
b5b b5b3 0 3738 58 3/4
bs5a bs5a2 0 3015 49 3/4
bs5b bs5b5 11 2000 32 2/4
sm sm2 3 3069 48 1/4
sm sm5 3 3729 58 3/4
x5a x5a4 0 3266 57 3/4
x5a x5a5 0 3320 57 3/4
x5b x5b2 0 3677 57 3/4
x5b x5b5 0 3733 58 3/4
x5c x5c1 0 1769 29 0/4
x5c x5c3 10 1778 27 1/4
x5c x5c4 10 3638 57 4/4
x5c x5c5 0 1708 28 0/4

One agent (conc-570.mjs), on a Store vps with a project vps/vps:

// One "agent": own process, own Sandbox, own Branch; the first Store call opens the Store at openAt, then commits in a loop.
// usage: LAYERFS_HOME=<home> node conc-570.mjs <repo> <name> <rounds> <openAtEpochMs>
import { writeFileSync } from "node:fs";
import { join } from "node:path";
const [repo, name, rounds = "20", openAt = "0"] = process.argv.slice(2);
const lfs = await import(join(repo, "api/node/src/index.js"));
const P = "vps/vps";
const out = { name, ok: 0, errors: {} };
const fail = (stage, e) => { out.errors[`${stage}:${e.code}`] = (out.errors[`${stage}:${e.code}`] ?? 0) + 1; out.first ??= `${stage}: ${String(e.message).slice(0, 300)}`; };
const times = []; const spans = [];
await new Promise((r) => setTimeout(r, Math.max(0, +openAt - Date.now())));
const t0 = performance.now();
let ws, sb;
try {
  try { await lfs.branch.fork(P, name, "main"); out.openMs = Math.round(performance.now() - t0); }
  catch (e) { fail("open", e); throw e; }
  sb = `sb-${name}`;
  await lfs.sandbox.create(P, sb, "native");
  ws = (await lfs.workspace.mount(sb, name)).ws;
  const path = (await lfs.sandbox.get(sb)).workspaces.find((w) => w.ws === ws).path;
  out.readyAt = Date.now();
  for (let i = 0; i < +rounds; i++) {
    for (let j = 0; j < 20; j++) writeFileSync(join(path, `${name}-${i}-${j}.txt`), `${i} ${j}\n`);
    const c0 = performance.now();
    try { await lfs.workspace.commit(ws, { message: `${name} ${i}` }); times.push(performance.now() - c0); spans.push([Math.round(performance.timeOrigin + c0), Math.round(performance.timeOrigin + performance.now())]); out.ok++; }
    catch (e) { fail("commit", e); }
  }
} catch (e) { if (!out.first) fail("setup", e); }
finally {
  out.teardownAt = Date.now();
  try { if (ws) await lfs.workspace.unmount(ws, { discard: true }); if (sb) await lfs.sandbox.close(sb); }
  catch (e) { fail("teardown", e); }
  lfs.close();
  times.sort((a, b) => a - b);
  out.p50 = Math.round(times[times.length >> 1] ?? 0); out.max = Math.round(times.at(-1) ?? 0);
  out.totalMs = Math.round(performance.now() - t0); out.endAt = Date.now();
  if (process.env.SPANS) out.spans = spans;
  console.log(JSON.stringify(out));
}

Start N agents at the same moment (conc-run-570.sh <home> <label> <agents> <rounds> [stagger_ms], run with SPANS=1):

#!/bin/sh
# usage: conc-run-570.sh <home> <label> <agents> <rounds> [stagger_ms]
# starts <agents> processes whose first Store call (the open) lands at the same instant (+ random stagger)
H=$1 L=$2 N=$3 R=$4 S=${5:-0}
D=$(cd "$(dirname "$0")" && pwd)
at=$(( $(date +%s) * 1000 + 3000 ))
for i in $(seq 1 "$N"); do
  off=0; [ "$S" -gt 0 ] && off=$(( $(od -An -N2 -tu2 /dev/urandom) % S ))
  LAYERFS_HOME=$H perl -e 'alarm 300; exec @ARGV' node "$D/conc-570.mjs" "$D/${REPO:-pr570}" "$L$i" "$R" $((at + off)) > "$D/c570/$L$i.log" 2>&1 &
done
wait
for i in $(seq 1 "$N"); do tail -c 600 "$D/c570/$L$i.log"; done

The helper and the wrapper set with machine.update("native", { daemon: ".../d-volsync.sh" }):

// LAYERFS_NFS_FLUSH helper: write back one volume (the Workspace's NFS mount) instead of sync(2).
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char **argv) {
  if (argc != 2) return 2;
  const char *log = getenv("VOLSYNC_LOG");
  if (log) { FILE *f = fopen(log, "a"); if (f) { fprintf(f, "%s\n", argv[1]); fclose(f); } }
  if (sync_volume_np(argv[1], SYNC_VOLUME_WAIT) != 0) { perror("sync_volume_np"); return 1; }
  return 0;
}
#!/bin/sh
export LAYERFS_NFS_FLUSH="<tools>/volsync {dir}" VOLSYNC_LOG=<tools>/volsync.log
exec <repo>/target/release/layerfs-daemon "$@"

The flush check (flushcheck.mjs <repo> <daemon> <home> <src> [rounds]):

// Does a commit see what the host just wrote? Small files written and closed, plus one file still open with unflushed writes.
// usage: node flushcheck.mjs <repo> <daemon> <home> <src> [rounds]
import { closeSync, openSync, writeFileSync, writeSync } from "node:fs";
import { randomBytes } from "node:crypto";
import { join } from "node:path";
const [repo, daemon, home, src, rounds = "5"] = process.argv.slice(2);
const sdk = await import(join(repo, "api/node/src/index.js"));
const lfs = sdk.client({ home });
const P = "fc/x";
await lfs.machine.update("native", { daemon });
await lfs.store.create("fc");
await lfs.project.init(P, src);
await lfs.branch.fork(P, "b", "main");
await lfs.sandbox.create(P, "fcs", "native");
const { ws, path } = await lfs.workspace.mount("fcs", "b");
const out = [];
try {
  for (let r = 0; r < +rounds; r++) {
    const want = {};
    for (let i = 0; i < 200; i++) { const t = randomBytes(32).toString("hex") + "\n"; want[`/r${r}-${i}.txt`] = t; writeFileSync(join(path, `r${r}-${i}.txt`), t); }
    const fd = openSync(join(path, `open${r}.txt`), "w");
    const big = randomBytes(64 * 1024).toString("hex");
    writeSync(fd, big); // left open across the commit
    const fd2 = openSync(join(path, `tiny${r}.txt`), "w");
    const tiny = `tiny ${r} ${randomBytes(8).toString("hex")}\n`;
    writeSync(fd2, tiny); // a small write, still open and buffered
    const t0 = performance.now();
    await lfs.workspace.commit(ws, { message: `r${r}` });
    const ms = Math.round(performance.now() - t0);
    let missing = 0, wrong = 0;
    for (const [p, t] of Object.entries(want).slice(0, 40)) {
      try { if ((await lfs.file.read(P, "b", p)).text !== t) wrong++; } catch { missing++; }
    }
    let openSize;
    try { openSize = (await lfs.file.stat(P, "b", `/open${r}.txt`)).size; } catch (e) { openSize = e.code; }
    let tinyOk;
    try { tinyOk = (await lfs.file.read(P, "b", `/tiny${r}.txt`)).text === tiny; } catch (e) { tinyOk = e.code; }
    closeSync(fd); closeSync(fd2);
    out.push({ r, commitMs: ms, small: `${40 - missing - wrong}/40 ok`, missing, wrong, openFile: `${openSize}/${big.length}`, tinyOpen: tinyOk });
  }
} finally {
  await lfs.workspace.unmount(ws, { discard: true });
  await lfs.sandbox.close("fcs");
  sdk.close();
}
console.log(JSON.stringify(out));

Activity

  1. added
    bugSomething isn't working
    integrationIntegrations with external tools and agent hosts
    on Nov 6, 2026
  2. added theissue type on Nov 6, 2026
  3. added this to the LayerFS v2.0.0 milestone on Nov 6, 2026
  4. self-assigned this
    on Nov 6, 2026
  5. jiangkoumo commented on Nov 10, 2026

    @jiangkoumo
    Collaborator

    @ImmuneFOMO 感谢你的 sample 分析和 sync_volume_np 试验,按这个思路开了 #647,commit 里加了你的 Co-authored-by。

    改法:macOS 上 Commit 前只刷 export 挂载点所在卷(sync_volume_np + SYNC_VOLUME_WAIT),失败时回退 sync(),不跳过 flush。

    我们在 M2 Air 上测得:3 会话并发 commit p50 约 130 ms → 12 ms。方便的话请在你 5+ agent 的场景下复测一下,看卡顿是否消失。

  6. ImmuneFOMO commented on Nov 10, 2026

    @ImmuneFOMO
    ContributorAuthor

    Retested #647 (1c0a6fd) against its base bdeb9e9 on my M3 Max, macOS 26.6.2, with the scripts from #578 (each round writes 20 files and commits, all agents start at once). The stalls are gone:

                         commits   p50      p95      max      over 1 s
    5 agents   bdeb9e9     180     141 ms   172 ms   3513 ms   3
    5 agents   #647        144       6 ms    13 ms     20 ms   0
    10 agents  bdeb9e9      90     145 ms  1801 ms   2784 ms   8
    10 agents  #647         50      15 ms    38 ms     48 ms   0
    

    flushcheck passes 5 of 5 rounds on #647 (40/40 small files, the open 128 KiB file and the short line all read back).

    The commit counts differ because many commits on #647 failed: 96 of 240 in the 5-agent runs and 50 of 100 in the 10-agent run. The first failures are Store busy before effect or Service(Busy), and on some Sandboxes the later ones are admitted no connection for 60s. On bdeb9e9 with my volsync helper from #578 it's the same, and on an older #570 head (6be2f6c, on an older base) with the helper it was 0 of 400. So I think it's the #534 Busy, which now shows up because commits no longer wait in sync(2), and not #647 itself. 7 Sandboxes kept their daemon and NFS mount after the agent exited. Closing them later, 3 got Busy after 60 s (all 8 control slots taken, looks like #540), 2 got MountsActive and 2 closed. Maybe #570 should go in before #647 or together with it

  7. jiangkoumo commented on Nov 11, 2026

    @jiangkoumo
    Collaborator

    @ImmuneFOMO 感谢你昨天的复测,并发 commit 的 Store busy 已经修了,都合在 #647(头 7d2bfbd),#570 已关闭并入。

    改动:
    • 把 #570 的 macOS 写入准入(目录 flock,非阻塞)移植进来,拿不到锁返回确定的 Busy,不再落到 SQLITE_BUSY
    • digest 拒绝归为确定 Busy,不再判成「结果未知」去占控制槽,残留 daemon/挂载的问题一并消失

    我们用你的脚本在 macOS 27.2(M2)上跑了 5×20×3 和 10×10×2,共 500 次:0 失败、0 残留,flushcheck 5/5。Linux 全量无新增失败,并发 commit Busy 0/8700。

    方便的话请在 #647 新头上再跑一遍你那套(尤其 M3 Max 和 10 个以上 agent,我们没测到),有问题直接在 #647 说。

  8. Mola-maker commented on Nov 11, 2026

    @Mola-maker
    Member

    molamolamiao

  9. ImmuneFOMO commented on Nov 14, 2026

    @ImmuneFOMO
    ContributorAuthor

    Fixed by #647. On layerfs_v2_0_0 at 8cfb6e2 on my M3 Max, macOS 26.6.2, with 5 agents x 20 commits and then 10 x 10 all starting at once, 200 of 200 commits went through with no Busy and a max of 34 ms, and flushcheck passes 5 of 5. Closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingintegrationIntegrations with external tools and agent hosts

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions