URL blocklist network mode enforced inside the sandbox (Daytona, Docker) - #1115
Open
zhiheng-yang wants to merge 8 commits into
Open
URL blocklist network mode enforced inside the sandbox (Daytona, Docker)#1115zhiheng-yang wants to merge 8 commits into
zhiheng-yang wants to merge 8 commits into
Conversation
A task can now declare network_mode = "blocklist" with blocked_urls (hosts or host/path prefixes) beside the existing allowlist fields. NetworkPolicy resolves the agent's policy from the task config; the standard-library egress filter enforces it on loopback inside the sandbox: plain HTTP is decided on the full URL, CONNECT tunnels on the TLS server name (the upstream goes to that name, so aliases and numeric addresses cannot stand in for a blocked host), and names with path rules are inspected with a per-run certificate. The uid firewall from the no-web pipeline also fires under the policy marker, so loopback is the agent's only exit. Decisions are logged for the rollout; the policy, the log, and the key are readable by the filter's user only. The capability gate refuses filtering modes on backends that cannot run the filter.
The rollout resolves the task's network policy, marks the agent environment, forces the model proxy into the sandbox, starts the filter beside it once per sandbox, and threads the filter's proxy and CA variables into every model-driven agent environment on both connect paths; the oracle is exempt. The LiteLLM proxy receives the policy for provider-side tools and strips Gemini grounding and web_search_options under it, while never inheriting the agent's proxy variables. Cleanup stops the filter and collects its decision log into the rollout before the proxy goes down. Runs without a sandbox user or with a session-factory agent are refused rather than left advisory. Tests cover the config model, rule matching and canonicalisation, the live filter against local HTTP and TLS upstreams, the proxy hook, the production runtime wrapper, the firewall marker, filter start and stop against a fake sandbox, and the rollout wiring and refusals.
Docker withholds CAP_NET_ADMIN, which the agent-uid firewall needs, so a policy run now writes a per-rollout compose override that grants it and appends it last, before the sandbox starts; a sandbox started elsewhere cannot be changed and is refused. An integration canary, parametrised over Docker and Daytona and skipped by default, exercises the whole enforcement layer inside a real sandbox as the sandbox user.
A tunnel that did not start with TLS was closed with its peeked bytes still unread, which the kernel turns into a reset; macOS reports that to the peer as ConnectionResetError where Linux delivers the earlier FIN. Drain the bytes first so the close is orderly on every platform. Nothing is relayed either way. Replace the two mypy-style ignore comments ty does not honour with a None guard in _inspect and a cast in _relay.
Existing tests build a Rollout with __new__ and never set the new _network_policy attribute, and the connect path had dropped the getattr main used for _disallow_web_tools. Read both the way main does. The role-path session-factory refusal now formats the resolved per-role policy instead of reaching through an Optional, which ty rejected.
The gap test asserted that allowlists are parsed but not enforced, which the egress filter now closes. Docker reports no gap; agentcore still reports agent.network_mode, the scope the agent runs under.
…tion auth An agent on native subscription auth has no LiteLLM loopback proxy, so the firewall refused it at connect time. Under a filtering policy the egress filter is the agent's only loopback exit and its provider traffic is an ordinary tunnel through it, so accept a loopback HTTPS_PROXY in place of the provider URL when BENCHFLOW_NETWORK_POLICY is set. The no-web policy alone still requires the model proxy. Verified with claude-agent-acp on a CLAUDE_CODE_OAUTH_TOKEN on Docker and Daytona.
…scope The support table listed the network allowlist as parsed but not enforced. Allowlists and blocklists are now enforced for the agent by the in-sandbox egress filter and uid firewall on Docker, Daytona, and Modal; the verifier scope and other backends fail closed.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Devin Review found 5 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
Comment on lines
+127
to
+129
| names = [ | ||
| x509.DNSName(name) for host in hosts for name in (host, f"*.{host}") | ||
| ] |
Contributor
There was a problem hiding this comment.
🟡 Allowed deep subdomains fail HTTPS
A path rule inspects every nested subdomain, but issue_tls_material certifies only one wildcard level. HTTPS clients reject allowed deeper subdomains.
Prompt for agents
Path-scoped blocklist rules apply to a domain and all subdomains through Policy.inspects and host_within, but issue_tls_material creates SANs only for the base domain and *.<base>. TLS hostname validation therefore fails for hosts two or more labels below the rule, even when their requested path is allowed. Align the inspection scope and certificate coverage. Since standard wildcards cover one label only, this may require issuing certificates dynamically per observed SNI or narrowing which subdomains are inspected.
Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+471
to
+475
| return { | ||
| key: value | ||
| for key, value in headers.items() | ||
| if key.lower() not in _HOP_BY_HOP and key.lower() not in named | ||
| } |
Contributor
There was a problem hiding this comment.
🟡 Repeated HTTP headers are discarded
_end_to_end collapses repeated fields into one dictionary entry. Responses with multiple Set-Cookie headers lose cookies and can break authenticated requests.
Prompt for agents
The forwarding helper converts email.message-style headers into dict[str, str]. Duplicate request and response headers are therefore collapsed. This is especially visible for multiple Set-Cookie response fields, where only one cookie survives. Preserve headers as an ordered sequence of pairs while filtering hop-by-hop fields, then forward repeated fields individually. Update both request forwarding and response streaming callers.
Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+217
to
+222
| if log_destination is not None: | ||
| try: | ||
| log_destination.parent.mkdir(parents=True, exist_ok=True) | ||
| await self.sandbox.download_file(self.paths["log"], log_destination) | ||
| except Exception: | ||
| return |
Contributor
There was a problem hiding this comment.
🟡 Policy audit logs vanish silently
A failed decision-log download returns successfully from stop. Cleanup then deletes the sandbox, losing the audit record without reporting the failure.
Suggested change
| if log_destination is not None: | |
| try: | |
| log_destination.parent.mkdir(parents=True, exist_ok=True) | |
| await self.sandbox.download_file(self.paths["log"], log_destination) | |
| except Exception: | |
| return | |
| if log_destination is not None: | |
| log_destination.parent.mkdir(parents=True, exist_ok=True) | |
| await self.sandbox.download_file(self.paths["log"], log_destination) |
Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+336
to
+350
| def _target(self) -> tuple[str, str, int, str] | None: | ||
| if _SCHEME.match(self.path): | ||
| url = urlsplit(self.path) | ||
| if not url.hostname: | ||
| return None | ||
| default_port = 443 if url.scheme == "https" else 80 | ||
| path = url.path or "/" | ||
| if url.query: | ||
| path = f"{path}?{url.query}" | ||
| return url.scheme, url.hostname.lower(), url.port or default_port, path | ||
| authority = self.headers.get("Host") | ||
| if not authority: | ||
| return None | ||
| host, port = _authority(authority, 443 if self.scheme == "https" else 80) | ||
| return self.scheme, host, port, self.path |
Contributor
There was a problem hiding this comment.
🟥 Proxy permits arbitrary destination ports
Requests can target any port despite rules permitting only HTTP and HTTPS defaults. Agents can reach unlisted internal services through the proxy.
Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+319
to
+324
| path = self.rollout_paths.rollout_dir / "docker-compose-egress-firewall.json" | ||
| path.parent.mkdir(parents=True, exist_ok=True) | ||
| path.write_text( | ||
| json.dumps({"services": {"main": {"cap_add": ["NET_ADMIN"]}}}, indent=2) | ||
| ) | ||
| self._egress_firewall_compose_path = path |
Contributor
There was a problem hiding this comment.
🟨 Network isolation grants broad container capability
Policy runs grant NET_ADMIN to the entire main container. Any privileged process inside it can alter the firewall and bypass isolation.
Was this helpful? React with 👍 or 👎 to provide feedback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a URL blocklist network mode enforced inside the sandbox (Daytona, Docker)
Summary
Tasks can now block specific URLs or hosts for the agent (
network_mode: blocklist+blocked_urls), and the existingallowlistmode is now actually enforced instead of only parsed.Enforcement lives inside the sandbox: a stdlib egress filter on loopback decides every HTTP(S) request, and the per-uid firewall the no-web pipeline already uses makes loopback the agent's only exit.
The filter covers every harness's fetch tool and anything run in the shell, and the LiteLLM hook covers provider-side search.
Every decision is recorded in
network_policy.jsonlin the rollout, so a reviewer can audit what the agent asked for and what was refused.Behaviour changes
network_mode: allowlistwas parsed but not enforced; it is now enforced on Docker, Daytona, and Modal, and refused by the capability gate elsewhere.Tasks that relied on an allowlist being a no-op will now see it applied.
--sandbox-user.A policy on a root-run agent or a session-factory agent is refused at setup rather than run advisory.
CAP_NET_ADMIN(the firewall needs it); it is appended last, so it also covers a task's own compose file.CLAUDE_CODE_OAUTH_TOKEN) can run under a policy: the filter is their loopback exit and provider traffic passes as an ordinary tunnel.The provider-side-search hook does not run on that path (listed under limits).
How it works
host(subdomains included) orhost/path-prefix.Plain HTTP is decided on the full URL; HTTPS tunnels on the TLS server name (SNI, not the CONNECT target); hosts that carry path rules are TLS-inspected with a per-run CA that tools trust through
SSL_CERT_FILE,REQUESTS_CA_BUNDLE,CURL_CA_BUNDLE, andNODE_EXTRA_CA_CERTS.BENCHFLOW_NETWORK_POLICY=1plus the proxy and CA variables; the rules, the key, and the decision log are readable by root only, and a refused request gets a bare 403.web_search_optionsunder a policy, so search cannot be delegated to the provider.Usage
Declare the rules in the task.
Taking skillsbench
tasks/citation-checkunchanged, the whole edit totask.mdis:The
agentsection is the usual place; asandbox-level entry is the default the agent section overrides; averifier-level entry is reported as a gap because the policy is enforced for the agent only.The image needs
python3(the filter runs on it) andiptables(installed on the fly if the image has a package manager), and the task needsallow_internet: true.Or apply one rule set to a whole experiment:
What the rollout records (
network_policy.jsonl; query strings of allowed requests are not logged):{"time": "2026-09-09T01:37:19Z", "method": "CONNECT", "host": "arxiv.org", "path": null, "decision": "inspect", "rule": null} {"time": "2026-09-09T01:37:19Z", "method": "GET", "host": "arxiv.org", "path": "/abs/2401.01234", "decision": "block", "rule": "arxiv.org/abs/2401.01234"} {"time": "2026-09-09T01:37:19Z", "method": "GET", "host": "arxiv.org", "path": "/abs/2309.00001", "decision": "allow", "rule": null} {"time": "2026-09-09T01:40:42Z", "method": "CONNECT", "host": "api.semanticscholar.org", "path": null, "decision": "block", "rule": "api.semanticscholar.org"}What the agent sees (from the trajectories of the runs below):
Validation
Everything below has been run on this branch; the commands are given so a reviewer can repeat them.
Unit tests,
uv run pytest -q tests/test_network_policy.py, 38 passed:rule parsing and canonicalisation, policy resolution (agent overrides sandbox, oracle exempt), a live in-process filter (plain HTTP, Host override, HEAD, bad and chunked bodies, numeric hosts in five spellings, refused CONNECT, tunnels decided on SNI, TLS interception, refusal without a certificate, orderly close of a non-TLS tunnel), the LiteLLM hook, proxy-env isolation, the firewall marker and the no-proxy subscription case, the Docker override, the capability gate, and the rollout refusals.
Sandbox canaries,
uv run pytest -m integration -k canary tests/test_network_policy.py -o addopts="", passed on Docker and on Daytona:a real sandbox, filter and firewall started, then as the sandbox user: blocked path 403,
../and percent-encoded spelling 403, sibling path 200, blocked host CONNECT 403, numeric address 403, bypassing the proxy cannot resolve the host, policy file unreadable, and the collected log carries theblock.End-to-end runs with a real harness,
claude-agent-acp(claude-sonnet-4-6) on aCLAUDE_CODE_OAUTH_TOKEN,--sandbox-user agent; every run finished[PASS]with reward 1.0:arxiv.org/abs/2401.01234/abs/2401.01234block,/abs/2309.00001allow; WebFetch of the blocked paper "HTTP 403 Forbidden", the other paper's title returnedpypi.org/project/requests/requestsblock with WebFetch 403,pipallow with its summary returnedtask.md, no overridecitation-checkwithapi.semanticscholar.orgblocked intask.md; oracle and agentTunnel connection failed: 403 Forbidden, bypassing the proxy fails at DNS, the task is still solved through CrossRefRegression: every test module that imports a touched source module (95 modules) plus the new one, 2008 passed / 0 failed, the same counts as
origin/main.ruff check,ruff format --check, andty check src/pass.