Skip to content

sdk: follow a sandbox that moved to another node - #483

Merged
CMGS merged 4 commits into
mainfrom
feat/sdk-rebind
Nov 6, 2026
Merged

CMGS merged 4 commits into
mainfrom
feat/sdk-rebind

Conversation

@CMGS

@CMGS CMGS commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

A handle kept the owner it was claimed on. After a live move (#445), an evacuation (#437) or the old node's retirement, every call answered 404 or failed to dial until the lease ran out; only Close looked the claim up. The MCP server and the LangChain and OpenAI adapters hold handles for a long time, so they broke until restart. This was left open by the 2026-11-06 audit (#481).

What changes

Both SDKs follow a moved sandbox:

  • Trigger. The owner's refused dial, its agent-upgrade 404 or its control-call 404 is tagged where it arises, with the sandbox id and the owner address it came from. Only a tag for this handle's sandbox at its current owner runs Lookup; a different owner then replaces the old one and parked connections are drained. A tag whose address a concurrent call has already replaced retries at the current owner without a second Lookup.
  • Replay only when nothing reached the old owner.
    • Go retries the control calls, the one-reply and streaming verbs, and Run without Stdin.
    • Python retries every call, since its inputs are bytes; find_iter is the one exception.
    • Calls that already streamed input or handed back a stream rebind and fail once, and the next call reaches the new owner: Go Run with Stdin, WriteFile, Push, OpenPty, Watch, DialPort, Lsp.Request, Logs and Attach.
  • Caller code is never retried. An error from the caller's own writer (Go Run stdout/stderr, ReadFileTo, Pull) or output callback (Python) is returned as it is, whatever tag it carries.
  • Derived handles from Fork, Checkpoint and Promote bind to the node that answered, not to a concurrently rebound owner.
  • A released sandbox still answers 404. The MCP server and both adapters inherit the change without code.

Numbers

4 commits. Production +366 −145 (net +221: Go +~110, Python +~70, and the rest moved or re-indented code); tests +245; docs +11.

Gates

  • make go-lint: 0 issues. ×10 (5 modules × linux/darwin).
  • make go-test green; asl clean on both GOOS.
  • sdk/python: ruff, mypy and 285 tests; langchain 15 tests.
  • Every new test fails on the code before its fix (negative control): moved, retired node, writer/callback errors tagged for another sandbox and for this one.
  • Outside review: Codex, four rounds on one thread.
    • Round 1: a caller writer/callback error could trigger a replay; derived handles bound to a rebound owner.
    • Round 2: a nested call's tag from another sandbox was accepted.
    • Round 3: a nested call on the same handle's find_iter was accepted.
    • All fixed with tests. Round 4: "No blocking findings."
  • No hardware run: a live move or evacuation with the MCP server attached is not covered here.

CMGS added 4 commits November 6, 2026 14:20
…ntil its lease ends

A handle kept the owner it was claimed on, so after a live move, an evacuation or the old node's retirement every call answered 404 or failed to dial until the lease ran out; only Close looked the claim up. Both SDKs now run Lookup once on a 404 or a refused dial from the owner and adopt a different owner, dropping parked connections to the old one. A call whose request is its whole input is sent again there: the control calls and the one-reply and streaming verbs in Go, and every call in Python, whose inputs are bytes. Calls that already streamed input (Go Run with Stdin, WriteFile, Push) or handed back a stream fail once, and the next call reaches the new owner. The MCP server and the LangChain and OpenAI adapters inherit it.
…s to the node that answered

Codex review. rebind judged errors by type, so a caller's own stdout writer failing with ECONNREFUSED, or a Python output callback raising a 404 from another sandbox call, made the SDK look the claim up and run the command again on another node. The owner's refused dial, its agent-upgrade 404 and its control-call 404 are now tagged where they arise, and only a tagged error rebinds. Fork, Checkpoint and Promote bound their new handles to the handle's current owner, which a concurrent rebind could have moved off the node that answered; they now take the address that answered.
Codex review round 2. An output callback or writer that itself called another sandbox could surface that call's tagged 404 or refused dial, and the running command would accept it and replay on its own new owner. The tag now names the sandbox id and the owner address it was raised for, and a handle rebinds only on its own tag at its current owner.
… callback, and let a call that lost a race to a move retry at the owner found

Codex review round 3. A Python output callback that used the same handle's find_iter could surface a 404 tagged for this very sandbox and owner, and the running command replayed on the new owner. Errors from the caller's code are now cut off where it is called: Go wraps a Run writer's or a ReadFileTo/Pull sink's error in callerError, Python clears the tag of an SDK error an output callback raises, and rebind ignores both. A tagged call whose old owner a concurrent call has already replaced retries at the new owner without a second Lookup.
@CMGS
CMGS merged commit 7ebdf84 into main Nov 6, 2026
2 checks passed
@CMGS
CMGS deleted the feat/sdk-rebind branch November 6, 2026 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant