feat(sandbox): enforce a URL denylist egress mode on docker and daytona

[Benjamin-eecs]

Problem

Research tasks built from a published paper need the agent online (docs, other papers, package indexes) but must keep it away from the paper itself, its mirrors and its code. Today network_mode offers only public, no-network, and an allowlist that is parsed but not enforced. benchflow-ai/FrontierPhysics#365 asks for a URL denylist that works across the ACP harnesses.

What this adds

network_mode: denylist with blocked_urls (URL prefixes) and blocked_hosts (hosts and their subdomains) on sandbox and agent:

sandbox:
  network_mode: denylist
  blocked_urls:
    - https://arxiv.org/abs/2401.12345
    - https://arxiv.org/pdf/2401.12345
    - github.com/example-org/paper-code
  blocked_hosts:
    - alphaxiv.org

Enforcement, for the agent only (oracle runs and the verifier are untouched):

1. A stdlib Python proxy (sandbox/_egress_denylist_proxy.py) runs as root on 127.0.0.1:18628 inside the sandbox. It refuses matching URLs and hosts with a 403, tunnels everything else, and only terminates TLS for hosts that appear in blocked_urls, using leaf certificates minted on the host with cryptography. The CA private key never enters the sandbox.
2. The existing uid-owner iptables firewall (lockdown.py) is armed by a second marker, BENCHFLOW_EGRESS_DENYLIST=1, so the sandbox user can reach loopback only. A client that ignores proxy variables fails closed instead of bypassing. Docker gets cap_add: NET_ADMIN from a benchflow compose overlay; Daytona direct sandboxes already allow iptables.
3. The agent env gets HTTP_PROXY/HTTPS_PROXY, NO_PROXY for loopback, and CA paths (SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, GIT_SSL_CAINFO, NODE_EXTRA_CA_CERTS). LiteLLM is forced sandbox-local, as in the no-web mode.
4. Hosted search tools are switched off per harness through new disallow_hosted_search_* fields on AgentConfig: Claude WebSearch (WebFetch stays on, it runs inside the sandbox), Codex tools.web_search, Gemini google_web_search and web_fetch, OpenCode/MiMo websearch.
5. Every refused attempt is logged and pulled into trajectory/egress_denylist.jsonl at cleanup (ts, action, method, url, rule).

modal, apple-container, and agentcore refuse the mode at preflight (enforces_denylist on the provider registry). Session-factory agents raise at connect time. sandbox/docker.py grows by 6 lines and sandbox/daytona.py is untouched; the mechanism lives in sandbox/egress_denylist.py and reuses the lockdown firewall.

Evidence

• Daytona end-to-end with bench eval run --sandbox daytona --agent codex-acp --model azure-foundry-openai/gpt-5.6-sol on a probe task that curls a blocked arXiv page, a sibling page, example.com, a blocked host, and the blocked page through python: verifier saw blocked=403 allowed=200 tunnel=200 host=403 python=403, reward 1.0, and the block log listed the three refused attempts.
• Daytona spike outside benchflow (python:3.12-slim): iptables via apt, agent uid direct egress and DNS refused, blocked 403, sibling 200, pip download through the proxy OK, policy file unreadable by the agent.
• Real-network smoke on macOS through the proxy script: 13 curl probes and 3 urllib probes (prefix, www, http scheme, host and subdomain, IP literal, tunnel, plain HTTP) all as expected.
• Tests: tests/test_egress_denylist.py (policy matcher, in-process proxy against a local TLS upstream, certificate material, env, start/stop with a fake sandbox, firewall gate), tests/test_network_denylist_config.py (validation, capability gate, rubric mirrors), tests/test_hosted_search_policy.py (per-harness switches executed with bash, planes, rollout wiring). Full suite, ruff check, ruff format --check, and ty check pass locally.

Hardening from an adversarial review pass

• Path matching normalizes before comparing: repeated percent-decoding, . and .. segments, duplicate slashes, backslashes, and ; path parameters, so /abs/../abs/<id>, //abs/<id> and /abs/%2e%2e/abs/<id> hit the same rule as /abs/<id> (GitHub serves such spellings directly).
• The address check refuses every resolver notation, not only canonical dotted quads: 3232235777, 0xc0a80101, 0300.0250.1.1, 127.1.
• blocked_hosts entries are matched exactly plus subdomains; only blocked_urls hosts strip a leading www..
• Daytona DinD (tasks that ship a docker-compose.yaml) gets the same NET_ADMIN overlay as Docker.
• The proxy is (re)started on every ACP connect, so a sandbox restored from a snapshot (branching) is never firewalled without a proxy.
• Proxy and CA variables are added to the agent env after the sandbox-local LiteLLM gateway starts; the gateway does not route model traffic through the egress proxy and does not see a CA bundle that does not exist yet (this one was caught by the end-to-end run).
• denylist is sandbox-level only: agent.network_mode and verifier.network_mode reject it instead of parsing a policy nothing enforces.
• Wildcard DNS names that embed an address (1-2-3-4.sslip.io, 1.2.3.4.nip.io, hex labels) are refused like addresses; a name the agent controls remains the inherent limit of a hostname denylist and is documented as such.
• The restart path tolerates a stale pid file, waits for the old proxy to exit, and no longer truncates the block log, so multi-scene, user-loop and branch rollouts keep every refused attempt.
• A scene whose primary role is the oracle still applies the denylist to later real-agent roles; --self-gen-no-internet wins over the denylist (no-web is stricter); the bash-primitive TaskRuntime refuses denylist tasks because it never arms the proxy or firewall; DockerSandbox.restore re-adds NET_ADMIN.
• The proxy rewrites the Host header from the request target, forwards only the declared request body after the head (a pipelined second request is dropped), half-closes each relay direction independently, and treats only http(s):// request targets as absolute-form.
• cryptography is imported lazily inside certificate minting with a clear error, since it is only an agentcore extra today; adding it to the core dependencies is a one-line follow-up if preferred.
• The proxy resolves every destination first and refuses names whose answers include a loopback, private, link-local or other non-global address (cloud metadata included), so a hostname the agent controls cannot turn the root proxy into a bridge to sandbox-internal or host services (from the Devin flag on this PR). The uid firewall persisting into later oracle roles or a same-user verifier matches the existing no-web mode and is documented.

Limits (documented in docs/sandbox-hardening.md)

• A blocklist hides pages, not knowledge. Mirrors, aggregator copies, and citing papers stay reachable unless listed.
• Hosted search tools are off in this mode because the proxy cannot see provider-side fetches.
• The task image needs python3; the run needs a non-root sandbox_user.
• The agent can tell intercepted hosts from the certificate issuer and sees a 403 that names the policy.

Refs: benchflow-ai/FrontierPhysics#365. Related: #785 (allowlist enforcement; this PR does not touch that mechanism and keeps the backend classes unchanged).

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[devin-ai-integration]

Devin Review found 2 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[devin-ai-integration]

🟡 Exempt runs retain agent firewall

After a denylisted agent runs, connect_as skips proxy setup for an oracle but leaves its UID firewall active. Oracle roles and same-user verifiers then lose internet access.

Prompt for agents

The denylist owner rules installed by enforce_agent_egress_firewall persist for the sandbox user's UID. In src/benchflow/rollout/__init__.py, connect_as excludes oracle roles from egress_denylist, but it does not remove those rules after a preceding denylisted role. Verification also runs after the rules were installed, and VerifierConfig permits verifier.user to equal the sandbox user. Add lifecycle management that disarms only BenchFlow's denylist firewall rules before exempt oracle or verifier execution, while preserving no-network behavior and safely rearming the denylist for later agent roles. Cover a denylisted-agent-to-oracle transition and a verifier running as sandbox_user.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟥 DNS aliases bypass network isolation

The root proxy checks hostname text, then socket.create_connection resolves it without validating the destination. An agent-controlled hostname can reach loopback or private services despite the UID firewall.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[Benjamin-eecs]

Follow-up on the two Devin flags, both addressed in the latest push (ed3f789):

• DNS aliases reaching loopback or private services: the proxy now resolves every destination before connecting and refuses names whose answers include any non-global address (loopback, RFC 1918, link-local incl. 169.254.169.254, CGNAT, unspecified). Refusals are logged with rule private-address. Covered by unit tests and re-verified end to end on Daytona.
• Firewall persisting into an oracle role or a same-user verifier: this matches how the existing no-web mode behaves (the uid rules are never removed once armed), so I kept it and documented it in docs/sandbox-hardening.md rather than adding disarm/rearm machinery. Happy to add that if you would rather have it in this PR.