feat(sandbox): enforce agent egress blocklist with in-container proxy and firewall

[questiondlmarks]

Summary

Implements layered, protocol-level egress blocklist enforcement (network_mode = "blocklist") to hide specific research papers, target repositories, and mirrors from web-enabled agents during evaluation, while keeping the rest of the internet open.

Unlike naive virtualization or coarse-grained container-level network blocks, this implementation:

1. Preserves native OS networking: Agents retain full, native access to tools like curl, Python requests/urllib, Node.js fetch, etc.
2. Lossless binary streaming: Proxies via raw TCP streaming so that PDF papers, dataset tarballs, and images download without corruption.
3. 404 Stealth behavior: Blocked endpoints return 404 Not Found instead of 403 Forbidden, making it indistinguishable from a missing page and preventing the agent from guessing blocked resources.
4. Leak-proof & zero-escape: Combines a root-run filtering proxy, kernel-level iptables agent-UID confinement, and LiteLLM pre-call model tool interception.

---

Architecture & Enforcement Layers

Enforcement is layered across three boundaries:

1. In-Sandbox Filtering Proxy (src/benchflow/sandbox/egress.py)

   • Runs as root on loopback (127.0.0.1:61380) with MAX_CONNECTIONS = 256 semaphore backpressure.
   • Transparent TCP CONNECT tunneling for unblocked hosts (zero overhead / end-to-end TLS preserved).
   • TLS MITM inspection exclusively for hosts carrying path-specific rules, using a per-run CA bundle installed into system trust stores and exported via SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, NODE_EXTRA_CA_CERTS, and GIT_SSL_CAINFO.
   • ALPN downgraded to http/1.1 to ensure reliable plaintext inspection without h2 binary frames.
   • SSRF protection: resolves upstream once, vetting every address against link-local (169.254.169.254), loopback, unspecified, and multicast, while keeping RFC1918 private networks reachable for Compose side-services.

2. Agent-UID Kernel Firewall (src/benchflow/sandbox/lockdown.py)

   • Stacks docker-compose-net-admin.yaml overlay when an agent network policy is active.
   • Post-ACP bootstrap, an iptables (and ip6tables if dual-stack) rule restricts the non-root sandbox_user UID strictly to loopback (-o lo). Any direct TCP/UDP packets bypassing HTTP(S)_PROXY fail closed.
   • Executes an automated self-check probe as sandbox_user before the first prompt, failing fast if the blocklist or firewall is breached.

3. Provider-Side Model Tool Rewriting (src/benchflow/providers/litellm_logging.py)

   • Server-side web tools run at the model provider (outside the sandbox). The LiteLLM pre-call hook dynamically merges the rule list into Anthropic web_search_*/web_fetch_* blocked_domains.
   • Strips OpenAI hosted search tools (which only support an allowlist).
   • Harness knobs switch off server-side search for Codex (-c tools.web_search=false) and Gemini.
   • Strips BENCHFLOW_EGRESS_BLOCKED_URLS from the agent process env so the agent never learns which URLs are hidden.

---

Batch Safety & CLI Overlays

• CLI Overlays: Adds --block-url and --block-url-file to bench eval run as C-axis config overlays.
• Preflight Check: Adds NetworkPolicyPreflightError to validate task runtime support across the entire batch up-front before rollouts start.
• Resume Protection: Adds ResumeMismatchError to prevent resuming jobs whose completed tasks ran under a different network policy.
• Audit Logs: Downloads agent/egress.jsonl containing structured logs of every proxy decision (allow/block/refuse/probe) for auditor verification.

---

Defensive Hardening (WAF & Cross-Platform)

• WAF Evasion Defense: Path matching uses urllib.parse.unquote and posixpath.normpath with case-folding, preventing bypasses via URL percent-encoding (%xx), duplicate slashes (//), or dot-segments (/../).
• Dual-Host Inspection: Validates both the URI host and the Host: request header to thwart IP-direct spoofing attempts.
• Docker Bridge IPv6 Fallback: resolve_upstream traverses vetted address candidates prioritizing IPv4 to avoid Errno 101 Network is unreachable in standard Docker bridge setups.
• Cross-Platform UTF-8: Explicitly sets encoding="utf-8" across all file reading utilities and shim loaders, eliminating Windows cp1252 encoding crashes.

---

Validation

Validation was conducted across 5 discrete layers — from pure functional logic up to live end-to-end forward proxying on Linux and full repository baseline regression — followed by explicit disclosure of testing boundaries.

---

1. Static Gates (L0 CI Parity)

Verified against BenchFlow's CI L0 test workflow requirements:

• ruff check . (Clean, 0 errors)
• ruff format --check . (Clean, all files compliant)
• ty check (Type checks passed)

---

2. Unit Tests: Pure Logic & Configuration Layer

Isolated unit tests in tests/test_egress_blocklist.py verifying protocol and policy logic without external network dependencies:

• Rule Matching: 15 parameterized cases covering host suffixes, path prefixes, case insensitivity, query/fragment stripping, percent-decoding (%xx), and path normalization (// and /../).
• Policy Resolution: Sandbox vs. agent precedence, oracle exemption, hard no-web priority over blocklist, and container provisioning following task definitions rather than primary roles.
• Environment Shaping: Agent processes receive proxy and CA routing vars while rule secret lists are stripped; LiteLLM processes retain the rule list without inheriting proxy routing.
• Harness Web Knobs: Verified Codex receives -c tools.web_search=false under blocklists; Gemini setup commands were executed in bash to parse emitted settings.json; non-applicable harnesses remain untouched.
• LiteLLM Pre-Call Hook: Directly executed embedded callback module source to ensure Anthropic tools receive blocked_domains, allowed_domains are narrowed, OpenAI hosted search tools are stripped, and baseline no-web dropping behavior remains intact.
• Batching & State: CLI overlays, Docker NET_ADMIN compose overlay, config.json schema updates, resume mismatch guards, and batch preflight conflict detection.

---

3. Orchestration Flow Verification (Mocked Sandbox)

Tested Rollout lifecycle using AsyncMock for env.exec and env.upload_file to assert ordering, privilege separation, and error contracts:

• Proxy starts only after health probe passes, uploading scripts/policies as root with chmod 600.
• CA setup runs only when path rules require TLS inspection.
• Firewall self-check probe executes under sandbox_user, logging outcomes as root.
• Harness-level web policies apply before proxy boot.
• Disconnect handler downloads agent/egress.jsonl whenever the proxy was started for any scene.
• Host-bound session-factory agents fail fast before launch.

---

4. Real-Proxy End-to-End Testing (Linux / WSL)

The in-sandbox stdlib proxy script was executed as an active subprocess in a Linux environment (Ubuntu 24.04 on WSL) with a live plaintext origin, an HTTPS origin with dynamic OpenSSL certificates, and client traffic sent via urllib:

Scenario	Expected Outcome	Verified
Plaintext HTTP filtered by host & path	Unblocked paths return 200; blocked return 404	✅
CONNECT tunnel to blocked host	Returns 404 immediately without upstream DNS resolution	✅
Opaque tunnel for hosts without path rules	Transparent passthrough; client observes origin's original cert	✅
TLS inspection for hosts with path rules	Mints leaf cert signed by run CA; blocked paths return 404; untrusted client fails handshake (proves MITM termination)	✅
SSRF refusal (169.254.169.254 & loopback)	Returns 403 Forbidden and logs refuse event	✅
Leaf certificate key reuse	Exactly one leaf.key generated across distinct inspected hosts	✅
WAF evasion (percent-encoding, //, .., case)	Blocked paths answer 404; neighboring paths answer 200	✅
Host header spoofing (http://<IP>/path + Host: target)	Returns 404; legitimate Host header returns 200	✅
Live probe script against running proxy	Completes with ok: true	✅

---

5. Full Repository Regression & Baseline Differential

To ensure zero regressions across BenchFlow's test suite, a full test rollout was executed in WSL and diffed against a clean checkout of main:

• Results: 6,044 tests passed.
• Differential: The failure set on this branch (15 tests) strictly matched the baseline failures on main (all 15 are known Windows/WSL CRLF checkout line-ending and symlink quirks). Zero new regressions introduced.

---

[devin-ai-integration]

Devin Review found 3 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)