feat(sandbox): enforce agent egress blocklist with in-container proxy and firewall

CHANGELOG.md

@@ -2,6 +2,28 @@
 
 ## [Unreleased]
 
+### Added
+- **Egress blocklist: `network_mode = "blocklist"` with `blocked_urls`.** The
+  agent keeps full internet access except for a declared list of `host` or
+  `host/path-prefix` entries — the inverse of `allowlist`, for experiments that
+  hide specific papers or pages from a web-enabled research agent. Declare it in
+  `task.md` (`agent`/`sandbox` sections) or per run with
+  `bench eval run --block-url https://arxiv.org/abs/2401.12345 --block-url-file hidden.txt`
+  (a C-axis overlay that replaces task-level `blocked_urls`). Enforcement is
+  layered so every path an agent has to the web is covered: a root-run
+  filtering proxy on the sandbox loopback (host rules reject `CONNECT`
+  tunnels; hosts with path rules are TLS-inspected under a per-run CA installed
+  into the sandbox trust store), the existing agent-UID iptables rule so tools
+  that ignore `HTTP(S)_PROXY` fail closed, Anthropic `blocked_domains`
+  injection plus OpenAI hosted-search stripping in the LiteLLM pre-call hook,
+  and narrow per-harness knobs (`blocklist_web_tools_*`) that switch off only
+  server-side search (Codex `web_search`, Gemini grounding). Blocked requests
+  answer `404`, the rule list never enters the agent process env, a self-check
+  runs as the sandbox user before the first prompt, and every decision lands in
+  `agent/egress.jsonl` next to a `network_policy` block in `config.json`.
+  Supported on docker (stacks a `NET_ADMIN` compose overlay) and daytona; modal,
+  apple-container, and agentcore refuse blocklist tasks before launch.
+
 ## 0.7.6 — 2026-09-04
 
 ### Added

docs/reference/cli.md

@@ -263,6 +263,8 @@ bench eval run --tasks-dir ./tasks --matrix matrix.yaml --trials 3
 | `--skill-creator-dir` | — | Path to a `skill-creator` directory (or a skills root containing it); used when `--skill-mode self-gen` |
 | `--self-gen-no-internet` | `false` | Disable web tools for the self-generated skill run |
 | `--agent-env` | — | Agent environment variable as `KEY=VALUE`; repeatable |
+| `--block-url` | — | Hide a host or `host/path-prefix` from the agent for this run (repeatable; pasted `https://` URLs accepted). Internet stays open otherwise. Sets `sandbox.network_mode=blocklist` via the C-axis overlay and replaces task-level `blocked_urls`; see [Sandbox hardening → Egress blocklist](../sandbox-hardening.md#egress-blocklist-network_mode--blocklist) |
+| `--block-url-file` | — | File with one host or `host/path-prefix` per line (`#` comments allowed); merged with `--block-url` |
 | `--include` | — | Only run these task names; repeatable (e.g. `--include jax-computing-basics --include data-to-d3`) |
 | `--exclude` | — | Skip these task names; repeatable (e.g. `--exclude quantum-numerical-simulation`) |
 | `--loop-strategy` | — | Wrap each rollout in a loop, e.g. `verify-retry:k=3,feedback=names` or `self-review:k=3` (omit for single-shot) |

docs/sandbox-hardening.md

@@ -52,6 +52,128 @@ Known residual risk:
 - An agent with sustained access can poison `__pycache__` for files that exist in the baseline (those caches aren't deleted because some tasks diff workspace against `/testbed_verify`). Mitigated by the workspace chown but not eliminated.
 - Tasks that don't ship a build-config snapshot can still be hijacked via `setup.py` edits. Snapshot is automatic for declared filenames — task authors don't need to opt in.
 
+## Egress blocklist (`network_mode = "blocklist"`)
+
+`no-network` and `allowlist` answer "may the agent reach the internet at all".
+The blocklist answers a different research question: **the agent may use the
+whole web except a list of URLs it must not discover** — e.g. hide the paper
+under evaluation (and its mirrors) from a deep-research agent while every
+other paper stays readable.
+
+```toml
+[sandbox]
+network_mode = "blocklist"
+blocked_urls = [
+  "https://arxiv.org/abs/2401.12345",   # normalized to arxiv.org/abs/2401.12345
+  "arxiv.org/pdf/2401.12345",           # list every mirror path you care about
+  "openreview.net",                     # a bare host blocks it and its subdomains
+]
+```
+
+Per run, without editing the task: `bench eval run … --block-url arxiv.org/abs/2401.12345 --block-url-file hidden.txt`
+(the run-level list replaces the task's `blocked_urls`; the config is
+re-validated, so a `no-network` or `allowlist` task fails loudly, and so does
+a task whose `agent` section pins its own `network_mode` — that override would
+otherwise shadow the sandbox blocklist).
+
+### How it is enforced
+
+A URL blocklist has to be enforced in three places, because no single layer
+sees every route an agent has to the web (`src/benchflow/sandbox/egress.py`):
+
+1. **Sandbox-local filtering proxy.** Before the agent launches, a root-run
+   stdlib proxy starts on the sandbox loopback and the agent env gets
+   `HTTP_PROXY`/`HTTPS_PROXY` (plus `NODE_USE_ENV_PROXY=1` for Node fetch).
+   Plain HTTP exposes the full URL, so host and path rules both apply. HTTPS
+   `CONNECT` only exposes the host: a host rule rejects the tunnel; a host that
+   carries **path** rules is TLS-inspected — the proxy terminates TLS with a
+   leaf certificate signed by a per-run CA (installed into the system trust
+   store and exported via `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`,
+   `CURL_CA_BUNDLE`, `NODE_EXTRA_CA_CERTS`, `GIT_SSL_CAINFO`), reads the path,
+   and forwards allowed requests upstream over a fresh verified TLS connection.
+   The CA key, the rule file, and the log are root-only.
+2. **Agent-UID firewall.** The same iptables rule the no-web policy uses
+   confines the agent UID to loopback, so anything that ignores the proxy env
+   (a hand-rolled socket, `curl --noproxy`, DoH) fails closed instead of
+   bypassing the filter. Under the blocklist it is applied **before the agent
+   process is launched** (the proxy is already listening), so there is no
+   pre-handshake window; the no-web policy keeps its post-handshake placement. The LiteLLM model proxy is forced
+   sandbox-local for the same reason. Docker stacks a `NET_ADMIN` compose
+   overlay for these runs; Daytona allows iptables natively.
+3. **Server-side web tools** run at the provider, outside the sandbox. The
+   LiteLLM pre-call hook merges the rules into Anthropic `web_search_*` /
+   `web_fetch_*` `blocked_domains` (domains and `domain/path` prefixes) and
+   strips OpenAI hosted `web_search` tools (allowlist-only filter). Harness
+   knobs cover the rest: Codex gets `-c tools.web_search=false`, Gemini
+   excludes `google_web_search` and `web_fetch`. Client-side fetchers
+   (OpenCode/MiMo `webfetch`, OpenHands browsing, Claude Code `WebFetch`) stay
+   enabled because their traffic goes through layer 1.
+
+Blocked requests answer **`404 Not Found`**, so a research agent cannot tell a
+hidden page from a missing one. The rule list is deliberately kept out of the
+agent process env (`BENCHFLOW_EGRESS_BLOCKED_URLS` reaches only the model
+proxy). Before the first prompt a self-check runs **as the sandbox user**: the
+first blocked host must answer 404 through the proxy and direct egress must be
+rejected; a failed check aborts the rollout rather than running it open.
+
+### Batch safety
+
+`bench eval run` resolves every selected task's network posture under the
+run's overlay and backend before the first rollout starts; a `--block-url`
+against a task that declares `no-network`/`allowlist`, or a blocklist task on
+a backend that cannot enforce it, fails the whole batch up front naming the
+offending tasks. Resuming a job whose completed tasks recorded a different
+`network_policy` (open vs. blocklisted, or a different list) is refused, the
+same way an agent mismatch is — those scores belong to different experiments.
+
+### Auditing a run
+
+- `config.json` carries a `network_policy` block (`mode`, `blocked_urls`,
+  `tls_inspection_hosts`, `blocked_status`).
+- `agent/egress.jsonl` (downloaded from the root-only log at disconnect) lists
+  every `allow` / `block` decision with host, path, and matched rule, plus the
+  `probe` self-check record — the evidence that the agent attempted (or never
+  attempted) the hidden URLs.
+- Server-side search really being off is verified from the request bodies in
+  `trajectory/llm_trajectory.jsonl`, not from config files.
+
+### Limits
+
+- The blocklist hides **URLs**, not knowledge: search-result snippets from
+  unblocked engines, citations in other papers, and the model's own training
+  data can still reveal that a paper exists. Block the mirrors you care about
+  (Semantic Scholar, OpenReview, alphaXiv, HF Papers, …) by host.
+- Path rules are prefix matches: `arxiv.org/abs/2401.12345` also hides
+  `…/abs/2401.123456` (and, usefully, `…/abs/2401.12345v2`). Matching runs on
+  the canonical path an upstream server would route — percent-decoded,
+  `..`/`.`/`//` collapsed, case-folded — and on both the URL host and the
+  `Host` header, so encoding tricks or an IP-literal URL with a spoofed
+  `Host` do not slip past. Percent-decoding is repeated until stable, so a
+  double-encoded separator cannot reach an upstream that decodes twice.
+- Only the agent phase is covered; `verifier.network_mode = "blocklist"` is
+  rejected before launch. Modal, Apple Container, and AgentCore cannot run
+  the root proxy + UID firewall and refuse blocklist tasks.
+- The proxy needs `python3` in the task image (and `openssl` for TLS
+  inspection; it is apt/dnf/apk-installed on demand).
+- The proxy runs as root, outside the agent-UID firewall, so it vets every
+  resolved upstream address and answers `403` for loopback, link-local (cloud
+  instance metadata such as `169.254.169.254`), unspecified, multicast,
+  reserved, and private ranges. The only private addresses it will reach are
+  the container's own directly-connected subnets (from `/proc/net/route`),
+  which is where compose side-services live; other RFC1918 space — the host
+  LAN behind the bridge, other projects' networks — is refused.
+- Session-factory agents run in-process on the host, outside the sandbox
+  proxy and firewall; a blocklist run refuses them before connecting.
+- TLS inspection speaks HTTP/1.1 only (ALPN advertises just `http/1.1`, so
+  HTTP/2-capable clients negotiate down).
+- The proxy handles at most 256 connections at once; a burst of concurrent
+  fetches queues in the listen backlog rather than exhausting threads or file
+  descriptors.
+- When the primary agent is the oracle, the oracle itself is exempt from the
+  blocklist, but the container is still provisioned for it (docker
+  `NET_ADMIN`, sandbox-local model proxy) so role agents connecting later are
+  covered.
+
 ## Related
 
 - [`progressive-disclosure.md`](./progressive-disclosure.md) — soft-verify (the relaxed hardening used between rounds in multi-round trials).

docs/task-authoring-task-md.md

@@ -74,9 +74,9 @@ so typos fail at parse time instead of becoming silently-ignored config:
 | `schema_version` (alias `version`) | Config schema version, currently `"1.3"` |
 | `task` | Package identity: `name` (`org/name` format), `description`, `authors`, `keywords`, `version` (informational Harbor 1.3 field, stored verbatim) |
 | `metadata` | Freeform mapping — difficulty, category, tags, anything descriptive |
-| `agent` | Agent run policy: `timeout_sec`, `user`, `network_mode`, `allowed_hosts` |
+| `agent` | Agent run policy: `timeout_sec`, `user`, `network_mode`, `allowed_hosts`, `blocked_urls` |
 | `verifier` | Verifier run policy: `timeout_sec` (default 600), `env`, `user`, `service`, … |
-| `sandbox` | Sandbox: `docker_image`, `cpus`, `memory_mb`, `storage_mb`, `network_mode`, `env`, `workdir`, … (legacy `task.toml` imports convert the Harbor `environment` table to this key; `environment:` in `task.md` is rejected with a rename hint) |
+| `sandbox` | Sandbox: `docker_image`, `cpus`, `memory_mb`, `storage_mb`, `network_mode`, `allowed_hosts`, `blocked_urls` (see [Sandbox hardening → Egress blocklist](./sandbox-hardening.md#egress-blocklist-network_mode--blocklist)), `env`, `workdir`, … (legacy `task.toml` imports convert the Harbor `environment` table to this key; `environment:` in `task.md` is rejected with a rename hint) |
 | `oracle` | Oracle run policy: `env`, `timeout_sec` (import alias: `solution`) |
 | `source`, `artifacts`, `steps`, `multi_step_reward_strategy`, `reward` | Provenance, artifact, and reward metadata |
 

src/benchflow/_utils/config_override.py

@@ -90,6 +90,43 @@ def load_config_override(value: str | None) -> dict[str, Any] | None:
     return _parse_overlay(value)
 
 
+def blocklist_override(
+    raw_override: str | None,
+    block_urls: list[str] | None,
+    block_url_file: str | Path | None,
+) -> str | None:
+    """Fold ``--block-url`` / ``--block-url-file`` into a C-axis overlay string.
+
+    The run-level list REPLACES any task-level ``blocked_urls`` (overlay lists
+    are not unioned) and forces ``sandbox.network_mode = "blocklist"``; the
+    task config is re-validated at rollout, so a task that declared
+    ``no-network`` or ``allowlist`` fails loudly instead of silently changing
+    posture. A task whose ``agent`` section pins its own ``network_mode`` is
+    refused as well (that override would otherwise shadow the sandbox
+    blocklist and serve the hidden URLs). Returns ``raw_override`` untouched
+    when no URLs were given.
+    """
+    entries: list[str] = []
+    for url in block_urls or []:
+        url = url.strip()
+        if url and url not in entries:
+            entries.append(url)
+    if block_url_file is not None:
+        listing = Path(block_url_file).expanduser().read_text(encoding="utf-8")
+        for line in listing.splitlines():
+            line = line.split("#", 1)[0].strip()
+            if line and line not in entries:
+                entries.append(line)
+    if not entries:
+        return raw_override
+    overlay = dict(load_config_override(raw_override) or {})
+    sandbox = dict(overlay.get("sandbox") or {})
+    sandbox["network_mode"] = "blocklist"
+    sandbox["blocked_urls"] = entries
+    overlay["sandbox"] = sandbox
+    return json.dumps(overlay)
+
+
 def deep_merge(base: dict[str, Any], overlay: dict[str, Any]) -> dict[str, Any]:
     """Recursively merge ``overlay`` into ``base``.
 

src/benchflow/acp/runtime.py

@@ -51,6 +51,11 @@
     TransportClosedDiagnostic,
     TransportClosedError,
 )
+from benchflow.sandbox.egress import (
+    blocklist_active,
+    strip_blocklist_secret,
+    verify_egress_blocklist,
+)
 from benchflow.sandbox.lockdown import (
     build_priv_drop_cmd,
     enforce_agent_egress_firewall,
@@ -644,6 +649,20 @@ async def connect_acp(
         agent_launch = build_priv_drop_cmd(agent_launch, sandbox_user)
         logger.info(f"Agent sandboxed as: {sandbox_user}")
 
+    # Egress blocklist: the filtering proxy was started in the install phase
+    # (Rollout.install_agent / connect_as); here the rule list is kept out of
+    # the agent process env and the post-firewall self-check runs below.
+    process_env = strip_blocklist_secret(agent_env)
+    # Under the blocklist the proxy is already up and HTTP(S)_PROXY is in the
+    # agent env, so the UID firewall goes up BEFORE the agent process exists:
+    # startup traffic that ignores the proxy fails closed instead of enjoying
+    # a pre-handshake window. The no-web policy keeps its post-handshake
+    # placement (its agents have no proxy to fall back on during bootstrap).
+    firewall_applied = False
+    if blocklist_active(agent_env):
+        await enforce_agent_egress_firewall(env, sandbox_user, agent_env)
+        firewall_applied = True
+
     acp_client: ACPClient | None = None
     session: object | None = None
     agent_name = agent
@@ -668,7 +687,7 @@ async def connect_acp(
             transport = ContainerTransport(
                 container_process=live_proc,
                 command=agent_launch,
-                env=agent_env,
+                env=process_env,
                 cwd=agent_cwd,
                 agent_log_path=agent_log,
             )
@@ -720,7 +739,9 @@ async def connect_acp(
             reasoning_effort=reasoning_effort,
             launch_config_owns_model=launch_config_owns_model,
         )
-        await enforce_agent_egress_firewall(env, sandbox_user, agent_env)
+        if not firewall_applied:
+            await enforce_agent_egress_firewall(env, sandbox_user, agent_env)
+        await verify_egress_blocklist(env, sandbox_user, agent_env)
     except Exception:
         with contextlib.suppress(Exception):
             await acp_client.close()

src/benchflow/agents/install.py

@@ -265,15 +265,28 @@ async def apply_web_tool_policy(
     home: str,
     *,
     disallow: bool,
+    blocklist: bool = False,
 ) -> None:
-    """Apply an agent-specific hard web-tool disable in the agent home."""
-    if not disallow or not agent_cfg or not agent_cfg.disallow_web_tools_setup_cmd:
+    """Apply an agent-specific web-tool policy in the agent home.
+
+    ``disallow`` applies the hard no-web disable; otherwise ``blocklist``
+    applies the narrower egress-blocklist switch (server-side search tools
+    only). A no-web run always wins over a blocklist.
+    """
+    if not agent_cfg:
+        return
+    if disallow:
+        setup_cmd = agent_cfg.disallow_web_tools_setup_cmd
+        policy_name = "no-web"
+    elif blocklist:
+        setup_cmd = agent_cfg.blocklist_web_tools_setup_cmd
+        policy_name = "egress-blocklist"
+    else:
+        return
+    if not setup_cmd:
         return
 
-    cmd = (
-        f"export BENCHFLOW_AGENT_HOME={shlex.quote(home)}; "
-        f"{agent_cfg.disallow_web_tools_setup_cmd}"
-    )
+    cmd = f"export BENCHFLOW_AGENT_HOME={shlex.quote(home)}; {setup_cmd}"
     owner = _owner_from_home(home)
     if owner:
         q_owner = shlex.quote(owner)
@@ -296,7 +309,7 @@ async def apply_web_tool_policy(
         if stderr:
             details.append(f"stderr: {stderr}")
         raise RuntimeError(
-            f"Failed to apply no-web policy for {agent}: {'; '.join(details)}"
+            f"Failed to apply {policy_name} policy for {agent}: {'; '.join(details)}"
         )
 
 

src/benchflow/agents/manifest.py

@@ -89,6 +89,8 @@
         "disallow_web_tools_setup_cmd",
         "disallow_web_tools_owned_paths",
         "disallow_web_tools_launch_suffix",
+        "blocklist_web_tools_setup_cmd",
+        "blocklist_web_tools_launch_suffix",
         "task_mcp_transport",
         "task_mcp_config_path",
     }