fix(envd): mount sandbox volumes with nolock to stop flock hang (#3619) - #3623
Open
AdaAibaby wants to merge 2 commits into
Open
fix(envd): mount sandbox volumes with nolock to stop flock hang (#3619)#3623AdaAibaby wants to merge 2 commits into
AdaAibaby wants to merge 2 commits into
Conversation
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
September 7, 2026 10:03
Sandbox volume mounts set neither nolock nor local_lock in nfsOptions, so local_lock defaults to none and the kernel routes every advisory lock to the nfs proxy as an NLM request. The proxy has no lock manager and its portmap never registers NLM (100021), so the portmapper GETPORT lookup returns port 0. Because the mount is hard, the client retries the bind forever and the caller (e.g. flock in $CODEX_HOME) wedges in uninterruptible sleep (state D, wchan rpc_wait_bit_killable) until the sandbox is destroyed. Add nolock so locks resolve node-locally, which is the correct semantics given the proxy has no lock manager -- and matches how the same server is mounted everywhere else (nfsproxy e2e, Filestore chunk cache). As a side effect mount.nfs no longer starts rpc.statd during setupNFS, so the pause/resume freeze allowlist in cgroups/hierarchy.go no longer depends on the local portmapper; its comment is updated and the now-defensive rpcbind entries are kept and re-justified. Also add an flock assertion to the nfsproxy e2e POSIX suite so a future regression that drops nolock fails loudly (10s timeout) instead of hanging. Reproduced on a dev host with the real proxy + portmap: without nolock, flock on the volume hangs (state D, exact kernel stack from the issue: rpc_wait_bit_killable -> nlmclnt_call -> nlmclnt_lock -> nfs3_proc_lock -> nfs_flock -> __do_sys_flock) while a local-disk flock and all reads/writes succeed; with nolock the mount reports local_lock=all and flock returns 0. Fixes e2b-dev#3619 Signed-off-by: AdaAibaby <shaolila@buaa.edu.cn>
AdaAibaby
force-pushed
the
fix/nfs-volume-nolock-flock-hang-3619
branch
from
September 7, 2026 10:05
5b4439c to
cebb064
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
flock()on a mounted sandbox volume hangs forever in uninterruptible sleep (stateD), cannot be killed even withSIGKILL, and holds the sandbox until it is destroyed. Reads and writes are fine; only locking hangs.Root cause:
nfsOptionsinpackages/envd/internal/api/init.gosets neithernolocknorlocal_lock, solocal_lockdefaults tononeand the kernel sends every advisory lock to the nfs proxy as an NLM request. The proxy has no lock manager and its portmap (packages/orchestrator/pkg/portmap) never registers NLM (100021), soPMAPPROC_GETPORTreturns port0. Because the mount ishard, the client retries the bind forever.Fix: add
nolocktonfsOptions. Locks then resolve node-locally, which is the correct semantics given the proxy has no lock manager — and matches how the same server is mounted everywhere else in the repo (nfsproxy e2enolock, Filestore chunk cachenolock).Changes
packages/envd/internal/api/init.go— addnolocktonfsOptions(the fix).packages/envd/internal/services/cgroups/hierarchy.go— the pause/resume freeze allowlist documented itsrpcbind/rpc-statdentries as justified by "mounts have no nolock, so mount.nfs starts rpc.statd". Withnolock,mount.nfsno longer startsrpc.statd, so that justification is gone. Comment updated to reflect reality; the entries are kept defensively (re-justified inline) rather than pruned, to keep this PR's blast radius on freeze behavior at zero.packages/orchestrator/pkg/nfsproxy/e2e_test.sh— add anflockassertion (10s timeout) to the POSIX suite so a future regression that dropsnolockfails loudly instead of hanging the suite. This closes the gap that let the bug ship: the e2e harness (e2e_start.sh) mounts withnolock, so it never exercised the production config.Why this is not a duplicate
Checked before opening:
gh issue view 3619 --repo e2b-dev/infra --comments— no comments, no linked PR.gh pr list --state open --search '3619 in:body'— none.gh pr list --state open --search 'nolock' / 'flock' / 'nfsOptions'— no PR touching this.Tests run
Reproduced and verified on a dev host using the real proxy + portmap code (unmodified), two containers on a bridge network mirroring production topology (client with local rpcbind+statd; server = go-nfs proxy+portmap registering only NFS/mountd on 2049, never NLM):
Before (current prod options, no
nolock) — mount reportslocal_lock=none;flockon the volume hangs, killed by timeout (exit 124). Kernel evidence matches the issue exactly:Local-disk
flock(control) and all reads/writes succeed.After (with
nolock) — same unmodified proxy; mount reportslocal_lock=all;flockon the volume returns0(LOCKED_OK).Also:
go build ./...(envd) — OK.go vetoninternal/api,internal/services/cgroups,pkg/nfsproxy— OK.gofmt -lon changed Go files — clean.bash -n e2e_test.sh— OK; rane2e_test.shstandalone, new flock assertion passes.The full
TestIntegrationTestcould not run on this shared box because the hostrpcbindalready holds:111(a pre-existing environment constraint, not related to this change); the flock behavior was instead proven with the isolated two-container harness above.Model evaluation
Not applicable — no change to model output, accuracy, or serving. This only changes NFS mount options and a test/comment.
AI assistance
AI assistance was used to investigate, reproduce, and draft this change. A human submitter has reviewed every changed line and the reproduction.
Fixes #3619