Skip to content

fix(shared): avoid lost wakeups when canceling Acquire - #3629

Closed
HeyiSun wants to merge 3 commits into
e2b-dev:mainfrom
HeyiSun:codex/fix-semaphore-cancel-wakeup
Closed

fix(shared): avoid lost wakeups when canceling Acquire#3629
HeyiSun wants to merge 3 commits into
e2b-dev:mainfrom
HeyiSun:codex/fix-semaphore-cancel-wakeup

Conversation

@HeyiSun

@HeyiSun HeyiSun commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

I noticed this while reading through the project code. If cancellation broadcasts after the context check but before Wait, Acquire can stay blocked until another semaphore update wakes it. This can leave sandbox resume handlers blocked past the 15-second wait for a startup slot, even after the caller times out. Canceled storage uploads can get stuck waiting for a slot too.

The callback now takes the condition lock before broadcasting. I added a regression test for that ordering; it fails on the old code and passes with the fix.

Reproduction

The original callback allows this ordering:

Acquire holds s.mu and checks ctx.Err() == nil
-> the context is canceled
-> AfterFunc calls Broadcast() without taking s.mu
-> Acquire enters Wait(), registers its waiter, and releases s.mu
-> no later wakeup occurs, so Acquire stays blocked despite cancellation

The test pauses Acquire after its context check, cancels the context, then lets it enter Wait. Tested on Go 1.26.6:

go test -race -run '^TestAcquireCancellationBeforeWait$' -count=1 ./packages/shared/pkg/utils

With the original semaphore implementation substituted using Go's -overlay, keeping the test unchanged:

--- FAIL: TestAcquireCancellationBeforeWait (0.00s)
    resizable_semaphore_test.go:368: Acquire missed cancellation before entering Wait

With the fix, Acquire returns context.Canceled without an extra wakeup. The test passed 100 times each with 1, 2 and 8 processors. The full utils tests also pass with -race.

@jakubno

jakubno commented Sep 9, 2026

Copy link
Copy Markdown
Member

Merged in a586d1e

@jakubno jakubno closed this Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants