feat: add private research controls on top of URL denylist - #1112
feat: add private research controls on top of URL denylist#1112ElegantLin wants to merge 4 commits into
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 6 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
There was a problem hiding this comment.
🟥 Root setup bypasses research filtering
Task setup commands run as root before the user firewall activates. They can fetch blocked content into the workspace, bypassing all research-policy checks.
(Refers to this code)
Was this helpful? React with 👍 or 👎 to provide feedback.
| # Private operator-side policy file. Its contents and path are never copied | ||
| # into public rollout/worker artifacts. | ||
| research_policy_path: str | None = None |
There was a problem hiding this comment.
🟡 Resumes mix research policies
Resuming with a changed or removed policy leaves completed tasks skipped. _check_resume_mismatch ignores research_policy_path, so final scores combine different access rules.
Prompt for agents
Add research-policy compatibility to resume validation in src/benchflow/evaluation.py. _check_resume_mismatch currently compares only the agent and loop configuration, while _get_completed_tasks skips completed rollouts unconditionally. Compare the current per-task resolved policy state and SHA-256 against each completed rollout's config.json metadata. Reject resumes that add, remove, or change a policy, or ensure affected tasks are rerun. Preserve private policy paths and rule values in all errors and artifacts.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if cfg.research_policy_path is not None: | ||
| self._research_policy = load_research_policy( | ||
| cfg.research_policy_path, task_id=cfg.task_path.name | ||
| ) |
There was a problem hiding this comment.
🟡 Missing policies allow partial batches
An uncovered task fails only during its own setup. Concurrent covered tasks keep running, so policy validation produces a partial evaluation instead of stopping the batch.
Prompt for agents
Preflight research policies against the complete selected task set before Evaluation schedules any rollouts. Load and validate each selected task entry once, abort the batch before execution if any entry is missing or empty, and pass the resolved per-task policy or validated policy state into rollout setup without exposing private values in worker or run artifacts. Keep programmatic single-rollout validation as defense in depth.
Was this helpful? React with 👍 or 👎 to provide feedback.
| "content": ( | ||
| response.body.decode("utf-8", errors="replace") | ||
| if textual | ||
| else base64.b64encode(response.body).decode("ascii") |
There was a problem hiding this comment.
🟡 Text downloads corrupt original bytes
A response labeled textual is decoded with replacement by _result_payload. web_download re-encodes that string, corrupting non-UTF-8 files instead of preserving server bytes.
Prompt for agents
Preserve exact response bytes for web_download in src/benchflow/sandbox/_research_gateway_runtime.py. The shared _result_payload currently converts text bodies to a UTF-8 string, and _mcp_call converts that string back to bytes. Return or request base64 for downloads regardless of Content-Type while retaining readable text output for web_fetch. Add coverage using non-UTF-8 text bytes and verify the saved file matches byte-for-byte.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if cfg.primary_agent != "oracle" and cfg.sandbox_user is None: | ||
| raise ValueError("research policy requires a non-root sandbox_user") |
There was a problem hiding this comment.
🟥 Oracle-first scenes bypass user isolation
An oracle primary bypasses the non-root validation. A later non-oracle scene role then runs as root without firewall enforcement, allowing unrestricted network access.
Was this helpful? React with 👍 or 👎 to provide feedback.
| server = ThreadingHTTPServer(("127.0.0.1", port), GatewayHandler) | ||
| server.serve_forever() |
There was a problem hiding this comment.
🟨 Gateway requests create unbounded threads
Each loopback request creates a new ThreadingHTTPServer thread without a concurrency cap. An agent can exhaust container memory or descriptors with parallel requests.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
Claude live-validation follow-up ( The first real The follow-up routes native Anthropic traffic through a root-owned, fixed-destination loopback relay to Validation on the pushed tree:
One Sonnet attempt hit a transient ACP |
c683832 to
47cf868
Compare
ElegantLin
changed the title
|
Positioning update after #1113 merged:
The PR title and description now state this relationship explicitly. Head |
1 participant
Summary
Extends the generic URL/host denylist merged in #1113 with the private,
run-scoped research controls needed to address the discovery/privacy part of
benchflow-ai/FrontierPhysics#365. This PR is an additive research-benchmark
layer, not a competing replacement for
network_mode: denylist.bench eval run --research-policy /secure/policy.yamlplus SDK, YAML-config, matrix, and worker-sharding propagationbenchflow-researchMCP server (web_search,web_fetch,web_download) across ACP and native-MCP-config harnessesapi.anthropic.com; reject Anthropic server-side web tools and remote MCP requests before forwardingRelationship to #1113
#1113 provides a task-declared network denylist for ordinary HTTP clients: it
prevents the agent from retrieving configured URL prefixes or hosts while
leaving other network access available.
This PR covers the stricter research-evaluation case where the target list must
remain private and the agent should not discover the target paper while
searching:
When
--research-policyis absent, #1113's normalnetwork_mode: denylistbehavior is unchanged. When it is present, the stricter private research path
takes precedence instead of starting both proxy implementations.
Policy boundary
This guarantees that the sandbox user cannot directly retrieve listed resources or bypass the gateway with curl/raw sockets. It cannot erase knowledge already present in model pretraining or hide unlisted mirrors/citations; operators can add blocked terms, content hashes, and known mirror URLs when discovery leakage matters.
Validation
uv run pytest -qafter rebasing ontof396c355— 6146 passed, 86 skipped, 8 deselected in 12m18suv run pytest tests/test_research_policy.py -m integration -qafter the rebase — 1 passed, 32 deselected in 41.80suv run pytest tests/test_source_adapters.py -q— 21 passed; the previously observed full-suite startup timeout also passed in four isolated repetitions and in the final full suiteruff format --check, targetedruff check, and fulluv run ty check— passedclaude-agent-acp/claude-sonnet-4-6/ Docker / subscription-auth rollout — reward 1.0; blocked MCP URL rejected, allowed MCP URL fetched, direct curl failed with exit 7, and an Anthropic server-side web-search request was rejected with HTTP 403claude-agent-acp/claude-haiku-4-5-20251001/ Docker rollout exercised all three MCP tools — reward 1.0;web_fetchandweb_downloadsucceeded for an allowed URL, a blocked fetch was rejected, the download was verified, and direct curl failed with exit 7web_searchresults — reward 1.0codex-acpwas launched with recordedmodel: gpt-5.6-solandreasoning_effort: xhigh; sandbox-local LiteLLM startup, the UID firewall, the explicit provider route, and the outbound provider request were all exercised, and artifact scanning found zero API-key matchesThe Docker canary also proves the private root policy is unreadable by the agent and deterministically checks that a correctly formed Anthropic provider-side web-search request returns HTTP 403.
The Codex live run is not counted as a successful generation: the configured provider authenticated but returned
model_not_foundbecause no account in that provider group supportsgpt-5.6-sol. The requested model andxhigheffort were kept unchanged rather than silently substituting a different model. BenchFlow now records those two fields separately and routesxhighinto the LiteLLM deployment configuration.