[taylorg009]

What

This measures build plan item 2's open half, "ContextBench file precision >= 0.5", and ships it as a reproducible bench in bench/contextbench/, following the pattern of bench/arb/ (#184).

• Benchmark. ContextBench v3, all 500 ContextBench Lite instances: Verified 174, Multi 156, Poly 116, Pro 54; 8 languages; 59 repos.
• Pins.
  • Metric code: EuniAI/ContextBench@1436c28a8eb95496da4ea69ad458b9f8a8eb7d61.
  • Data: HF Contextbench/ContextBench@c2855792b006af41c67202d33883fb9d46362853, contextbench_verified.parquet, with its sha256 checked.
  • The 500 ids in that file are exactly selected_500_instances.csv.
• Metric. ContextBench's own compute_granularity_metrics, at the file level: recall |P∩G|/|G|, precision |P∩G|/|P|. Gold comes from ContextBench's GoldLoader.
  • The headline is the macro mean over instances, because the paper's Appendix H says "all reported metrics are averaged over tasks".
  • The micro aggregate from ContextBench's CLI aggregate_results is reported alongside.
  • CIs are percentile bootstraps over instances (10k resamples).
• What CGraph sees. Only problem_statement. instances.py exports repo, commit and issue text, and the labels stay behind.
  • Each repo is checked out at its base commit.
  • The real cgraph and graphd binaries index and serve it, and cgraph-client sends the queries.
• Cutoffs, fixed before scoring.
  • default (the gate row): the files of one context call at engine defaults (budget 6000). This is what CGraph hands an agent.
  • top-K: the full curve for K=1..20 over the ARB recipe's fused ranking. No K is picked from the labels.

Command

PYTHON=<python with pyarrow> bench/contextbench/run.sh --cgraph-bin-dir <dir with cgraph, cgraph-client, graphd> --jobs 8

The run used a build of main at 43453f3: cgraph-native 0.1.0 (bin-v0.9.4-1-g43453f3).

Results (500 instances, 499 scored)

Prediction	Mean files	Precision [95% CI]	Recall [95% CI]	F1	Micro P / R
default context	12.54	0.100 [0.089, 0.112]	0.371 [0.337, 0.407]	0.134	0.072 / 0.260
top-1	1	0.214 [0.178, 0.251]	0.106 [0.083, 0.129]	0.126	0.214 / 0.062
top-2	2	0.217 [0.191, 0.243]	0.200 [0.173, 0.229]	0.183	0.217 / 0.125
top-3	3	0.192 [0.172, 0.213]	0.262 [0.231, 0.295]	0.194	0.192 / 0.166
top-5	5	0.162 [0.146, 0.178]	0.347 [0.313, 0.382]	0.194	0.162 / 0.233
top-10	10	0.110 [0.100, 0.120]	0.441 [0.406, 0.477]	0.158	0.110 / 0.316
top-20	19.97	0.070 [0.065, 0.076]	0.555 [0.520, 0.591]	0.116	0.070 / 0.404

Published ContextBench Lite file-level baselines (paper v3, point estimates; no per-instance data is released):

System	Recall	Precision
mini-SWE-agent + GPT-5 (bash-only agent)	0.682	0.709
SWE-agent + GPT-5	0.726	0.537
OpenHands + GPT-5	0.733	0.400
Agentless + GPT-5	0.609	0.352
Prometheus + GPT-5 (graph retrieval)	0.717	0.336

The agents are not like-for-like with CGraph. Each runs a full GPT-5 session, then declares the files it used. CGraph answers the issue text once. The agent rows show where the gate sits, not what CGraph should match.

Gate verdict: FAIL

CGraph's default file precision is 0.100 (CI 0.089-0.112), against a gate of 0.5. The best point on the whole curve is 0.217 at K=2, with a CI upper bound of 0.243. The fail is not close.

• Bundle width. The default bundle averages 12.5 files (median 10), while gold averages 3.47 (median 2). Even with perfect selection inside bundles of those sizes, macro precision would be capped at 0.332. That ceiling is a post-hoc diagnostic only.
• Ranking. Precision is low at every K. Top-2 precision is 0.217.

Other findings

• Engine bug. The build of element-hq/element-web@c9d9c421 ran for 2 h 9 min and was killed, after about 2 h of CPU on one thread.
  • Bisecting the snapshot's files one at a time isolated docs/lib/mermaid.min.js, a 2.9 MB minified bundle.
  • The stack is in the HTTP-client contract extractor: http_call_handler -> analyze_client_call -> collect_url_template -> local_values -> exclusive_branches -> branch_path -> ts_node_parent.
  • That instance is scored as an error. Counting it as 0 moves no overall number by more than 0.001.
  • Fixed on main after this run by fix(js): a minified bundle no longer takes hours to extract #204 (a41425e). These numbers stay pinned to 43453f3; a 500/500 rerun has not been done.
• ContextBench path quirk. gold.py uses lstrip("./"), which turns .github/... into github/.... 18 of the 1,746 gold files are not in their tree. This applies to every system equally and is left unpatched.
• Determinism. A separate --limit 30 rerun matched the full run on 30 of 30 instances: default files, ranking and per-op lists.
• Shared adapter. bench/arb/cgraph_eval.py now has served_snapshot, the extracted checkout, build, serve and teardown lifecycle. ARB trace2code and edit2ripple one-sample runs before and after the change gave identical details.

Spend and time

• Model and API spend: $0. There is no model in the loop.
• Wall time. 2 h 28 min on 8 pinned cores (16132.80s user 755.04s system 190% cpu 2:28:03.77 total). 499 instances finished in the first 51 minutes; the rest was the one stuck build.
• CPU. 4.7 CPU-hours, about 2.0 of them the stuck build. Over the 499 instances, graph builds took 3.08 h and retrieval took 0.13 h.
• Data. A 1.4 GB snapshot cache.

Tests

• ctest on the release build: 100 of 100 passed.
• No C++ changed, and eval data did not change.

Full write-up: bench/contextbench/RESULTS.md.

🤖 Generated with Claude Code

[tgod009]

Non-author review: no blocking findings. Leakage boundary (problem_statement only), ContextBench metric calls, macro/micro aggregation, bootstrap, and the served_snapshot ARB refactor all check out. Stale #204 text, Appendix H quote, results.json pointer and run.sh interpreter fixed in ea271ba.