feat(sandbox): driver-independent protected mock service for CLI fixtures - #90
Open
dmorosanu wants to merge 4 commits into
Open
feat(sandbox): driver-independent protected mock service for CLI fixtures#90dmorosanu wants to merge 4 commits into
dmorosanu wants to merge 4 commits into
Conversation
dmorosanu
requested review from
akshaylive,
bai-uipath,
tmatup and
uipreliga
as code owners
August 6, 2026 16:20
|
Claude finished @dmorosanu's task in 1m 21s —— View job Code Review
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What
Adds a driver-independent
protected_mocksfixture service: task fixtures are loaded by a small host-side per-run server, and the agent's sandbox only ever contains a thin client shim. Extracted from #87 (codex/uid-gid-agent-isolation) and made portable; supersedes the protected_mock portion of that PR, which will rebase onto this and re-add the Docker UID/GID wiring.src/coder_eval/protected_mock/{protocol,server,client,runtime}.py- JSON-over-socket request/response service with per-tool request budgets, bounded payload sizes, and prefix-limited passthrough (unchanged from the donor where it worked).match_mode: subsetadded alongsideexact/normalized: all rule tokens must appear in the invocation's normalized token set; rules scan in fixture-file order, first match wins; exact/normalized take precedence; duplicates allowed; empty subset argv rejected at load.driver: tempdir; underdriver: dockervalidation fails closed ("requires the UID/GID isolation layer; not yet available") until Add UID/GID isolation for evaluated agents #87 lands.protected_mocks/<tool>shim (+.cmdtwin) per entry, PATH-prepended likemock_path_dirs; each shim carries its endpoint/token/call-log path itself and does not rely on the agent process environment.uipath_eval.eval_set); fixture bytes are never copied into the sandbox.cli_called-schema JSONL written toprotected_mock_calls.jsonlnext totask.jsonin the run dir, outside the sandbox. Diagnostic-only for now:cli_calledresolves itslogfield sandbox-relative and cannot read the host-side file (documented)._cleanup, which runs on every exit path.environment_inforecords the endpoint kind and a SHA-256 digest of the fixture contents.Why
Eval fixtures are grading material and must not be readable in the agent workspace. The current approach ships them into the sandbox behind a reversible encoding, and evaluated agents have read them to pass tests they should have failed - an encoding is not a boundary. Host-side loading removes the data from the workspace entirely, and the docker-only version in #87 could not run on the tempdir driver the test suites actually use (or on Windows).
Validation
ruff format --check,ruff check,pyright: clean.tests/test_custom_lint.py): 171 passed.tests/test_sandbox.py(WinError 1314 without Developer Mode, unrelated).tests/test_protected_mock.py(28 tests): validation gates (tempdir OK, docker fails closed, name uniqueness, record_cli overlap), exact/normalized/subset matching (order, first-wins, precedence, budget exhaustion, duplicate rules), passthrough prefix+cache, TCP loopback end-to-end with token (mismatch rejected), AF_UNIX end-to-end where available, per-run endpoint isolation (two servers), runtime teardown (terminate, scratch dir removed), fixture resolution against task dir, shim generation (endpoint baked in, no fixture bytes anywhere in the sandbox tree), self-sufficient shim execution against a live service, and a fullOrchestrator.run()lifecycle test (audit record, seeded call log, server stopped, preserved sandbox carries only the shim).