SWE-bench accuracy for Agentic Inference depends heavily on agent network access #560

Summary

With the reference client's default SWE-bench setup, the agent containers have network access. We ran the same test twice, on two separate systems, with the recipe below (DeepSeek-V4.1-Flash, first 200 SWE-bench Verified instances, default mini-swe-agent template). In both tests most agents used the network to fetch upstream project code, and the accuracy score depends strongly on that access:

SWE-bench Verified, first 200 instances	Test 1	Test 2
Agent network on	98.5 % (197 / 200)	98.0 % (196 / 200)
Agent network off	83.0 % (166 / 200)	84.5 % (169 / 200)
Resolved only with network on / only with network off	31 / 0	27 / 0
Agents that fetched upstream code (network on)	173 of 200	184 of 200

The Agentic Inference gate is ≥ 96.4 % (reference 97.5 %).

What the agents did with the network

• In the network-on runs, 173 of 200 (test 1) and 184 of 200 (test 2) agents ran at least one command that
  fetches upstream code:
  • curl of files from raw.githubusercontent.com, including the current main version of the very file being
    fixed;
  • GitHub API issue searches for the task's own issue;
  • git ls-remote / git clone of the project;
  • pip download of later releases, which may already contain the fix.
• Every instance resolved with the network off was also resolved with it on, in both tests.
• All 31 (test 1) and all 27 (test 2) instances resolved only with the network on had fetched upstream code. In 16 of the 27 of test 2, a fetch returned new lines of the reference fix.
• 18 earlier network-on runs of the same setup as test 2 resolved 196–198 of 200. In those runs, 11.9 % of the trajectories got lines of the reference fix from the pull request of their own task.

Fetching proves the agent looked, not that it copied. But SWE-bench Verified tasks are historical issues whose fixes are public upstream, so with network access the score partly measures retrieval rather than the model's problem-solving.

Questions for the working group

1. Is network access for the agent containers intended in official runs? The rules text doesn't address it.
2. Was the 97.5 % reference score measured with the network on? If so, the gate also depends on network access.
3. If access should be limited, which policy?
   • (a) No network for agent containers (--network none). Simple, but it also blocks legitimate pip install of
     tools.
   • (b) Allow package indexes but block source hosts (github.com, githubusercontent.com, gitlab.com, git remotes).
   • (c) Keep it on, but require disclosing the fetch rate.

Recipe to reproduce

Reference client mlcommons/endpoints at d3c57f2, config examples/10_Agentic_Inference/dsv4_agentic_benchmark.yaml (SWE-bench Verified, num_instances: 200, workers: 16, default mini-swe-agent template), against any DeepSeek-V4.1-Flash endpoint.

1. Set up the client and SWE-bench service as in examples/10_Agentic_Inference/accuracy/RUNBOOK.md, and point the config's endpoint at your server.

2. Baseline (network on). Start the service as usual. The config expects port 18080 and the config's auth token.

   uv run --project src/inference_endpoint/evaluation/swebench_service \
     python -m swebench_service --host 127.0.0.1 --port 18080 \
     --auth-token dsv41-swebench-20260918 --artifact-root ./swe_on
   uv run inference-endpoint benchmark from-config \
     --config examples/10_Agentic_Inference/dsv4_agentic_benchmark.yaml --mode acc

3. Network off. mini-swe-agent ("MSWEA") reads its docker binary from MSWEA_DOCKER_EXECUTABLE, and the service passes its environment through. A small wrapper adds --network none to every agent docker run. The agent process and its model calls stay on the host, and the harness's evaluation containers (Docker SDK) are unaffected.

   cat > /tmp/docker_no_network <<'EOF'
   #!/bin/sh
   if [ "$1" = run ]; then shift; exec /usr/bin/docker run --network none "$@"; fi
   exec /usr/bin/docker "$@"
   EOF
   chmod +x /tmp/docker_no_network
   MSWEA_DOCKER_EXECUTABLE=/tmp/docker_no_network \
   uv run --project src/inference_endpoint/evaluation/swebench_service \
     python -m swebench_service --host 127.0.0.1 --port 18080 \
     --auth-token dsv41-swebench-20260918 --artifact-root ./swe_off
   # then rerun the same `inference-endpoint ... --mode acc` command

   Check while it runs: docker inspect -f '{{.HostConfig.NetworkMode}}' $(docker ps -q) prints none for the agent containers.

4. Count upstream fetches in either run's trajectories:

   python3 - ./swe_on <<'EOF'
   import json, re, sys, pathlib
   pat = re.compile(r"github\.com|githubusercontent\.com|gitlab\.com|bitbucket\.org|"
                    r"git\s+(fetch|pull|clone|remote\s+add)|pip\s+download", re.I)
   trajs = list(pathlib.Path(sys.argv[1]).glob("*/swe_bench_output/*/*.traj.json"))
   hit = [t.parent.name for t in trajs
          if any(pat.search(a.get("command", ""))
                 for m in json.load(open(t)).get("messages", [])
                 for a in ((m.get("extra") or {}).get("actions") or []) if isinstance(a, dict))]
   print(f"{len(hit)} of {len(trajs)} agents fetched upstream code")
   EOF

   Compare each run's resolved instances (resolved_ids in the SWE-bench results) to see which instances are resolved only with the network on.