CGR-38: ARB result with no tuning on ARB samples (tests edges, file documents, anchor fusion)

bench/arb/RESULTS.md

@@ -6,6 +6,80 @@ The benchmark is [Agent Retrieval Bench](https://arxiv.org/abs/2607.24882) (ARB,
 repository is checked out at its base commit, because ARB's released corpus is chunked
 and truncated and CGraph has to index the real tree.
 
+## CGR-38: tuned without ARB samples (branch build, pending release)
+
+bin-v0.9.1's numbers below were tuned on a 40-sample ARB subset (clap, caddy, gin). This section
+reports work done without that: every change was developed on a declared dev side and the rest of
+ARB was scored only to measure it.
+
+**Protocol.**
+
+- **Split, declared before any change** (`score.py` `DEV_REPOS`, commit bee900e). Dev = every
+  gin, caddy and clap sample (111, a superset of CGR-25's 40). Held-out = the other 234 samples
+  from 22 repositories.
+- **External dev set.** The ARB dev split has no comment2context and only 19 code2test samples,
+  so it cannot drive tuning on its own. A second dev set was built with ARB's own miners at the
+  pinned commit from 10 repositories ARB never used (`extdev/`): 229 code2test and 187
+  edit2ripple samples.
+- **Held-out looks.** All diagnosis used dev and extdev only; held-out samples were never
+  inspected one by one. Held-out aggregates were read three times:
+  1. the bin-v0.9.5 baseline, before any change;
+  2. the branch build below;
+  3. the released binary.
+
+  One earlier held-out run of a pre-review build lost a snapshot to a host memory fault and was
+  discarded without its scores being read.
+
+**Changes** (openspec `test-subjects-and-file-documents`):
+
+1. **`tests` edges.** Each test file is linked to the source file its name pairs with.
+2. **File body documents.** A file node's BM25 document is now its text outside any symbol, plus
+   its path.
+3. **Tests in context.** `context` reaches a seed's tests at depth 1.
+4. **Anchor fusion (harness).** The per-file anchor op lists are fused among themselves and vote
+   once, not once per file.
+
+| Held-out, 234 samples | R@20 | MRR | BCY@8k |
+|---|---:|---:|---:|
+| RepoMap (rerun) | 0.5402 | 0.1785 | 0.3363 |
+| CGraph bin-v0.9.5 | 0.5164 | 0.2076 | 0.3184 |
+| **CGraph branch** | **0.5628** | **0.2076** | **0.3167** |
+
+Against RepoMap on held-out:
+
+- R@20 is +0.0226 and MRR +0.0291, both outside the noise floor.
+- BCY@8k is -0.0196.
+
+By task:
+
+| Held-out task | n | RepoMap R@20 / MRR / BCY@8k | CGraph branch R@20 / MRR / BCY@8k |
+|---|---:|---|---|
+| code2test | 87 | 0.4950 / 0.1667 / 0.3280 | 0.4927 / 0.1516 / 0.2188 |
+| comment2context | 80 | 0.4708 / 0.1527 / 0.2896 | 0.5125 / 0.2223 / 0.3104 |
+| trace2code | 37 | 0.8829 / 0.2773 / 0.5270 | 0.8694 / 0.3094 / 0.5811 |
+| edit2ripple | 30 | 0.4333 / 0.1597 / 0.2500 | 0.5222 / 0.2054 / 0.2917 |
+
+What changed against bin-v0.9.5 on held-out:
+
+- **code2test** gains the most: R@20 0.3870 to 0.4927.
+- **trace2code** falls: MRR 0.3551 to 0.3094, BCY@8k 0.6261 to 0.5811. The extra test files now
+  compete with the source files a stack trace names. This is why overall MRR is flat.
+
+| All 345 samples | R@20 | MRR | BCY@8k |
+|---|---:|---:|---:|
+| RepoMap (rerun) | 0.6311 | 0.2156 | 0.3846 |
+| CGraph bin-v0.9.5 | 0.6353 | 0.2759 | 0.4309 |
+| **CGraph branch** | **0.6692** | **0.2951** | **0.4395** |
+
+Dev sides, for reference (both were used for tuning, so they are not evidence on their own):
+
+| | n | bin-v0.9.5 R@20 / MRR / BCY@8k | branch R@20 / MRR / BCY@8k | RepoMap R@20 / MRR / BCY@8k |
+|---|---:|---|---|---|
+| ARB dev (gin, caddy, clap) | 111 | 0.8859 / 0.4199 / 0.6682 | 0.8934 / 0.4797 / 0.6982 | 0.8228 / 0.2939 / 0.4865 |
+| External dev set | 416 | 0.6891 / 0.3119 / 0.4752 | 0.7418 / 0.4101 / 0.5517 | 0.8099 / 0.4543 / 0.5815 |
+
+On the external dev set CGraph still trails RepoMap on every metric.
+
 ## bin-v0.9.1 (BM25 code-text search, CGR-25)
 
 Same harness and data pins as below, engine release bin-v0.9.1 (`cgraph --version`:

bench/arb/ablation.py

@@ -23,7 +23,7 @@
     "query(q) only": lambda lists: [lists["query_text"]],
     "context(q) + query(q)": lambda lists: [lists["context_text"], lists["query_text"]],
     "anchor ops only": lambda lists: [v for k, v in lists.items() if k.startswith("context_anchor") or k == "impact_changed"],
-    "all ops (scored recipe)": lambda lists: list(lists.values()),
+    "all ops, one vote each": lambda lists: list(lists.values()),
 }
 
 
@@ -46,9 +46,13 @@ def main(argv: list[str] | None = None) -> int:
                 candidates[chunks] = set(bcy_curve.load_file_texts(chunks))
             files = candidates[chunks]
             gold = set(row["gold_files"])
-            for name, pick in VARIANTS.items():
-                lists = pick(row["cgraph"]["lists"])
-                ranked = [p for p in cgraph_eval.rrf(lists) if p in files] if lists else []
+            for name, pick in [*VARIANTS.items(), ("scored recipe (cgraph_eval.fuse)", None)]:
+                if pick is None:
+                    fused = cgraph_eval.fuse(row["cgraph"]["lists"])
+                else:
+                    lists = pick(row["cgraph"]["lists"])
+                    fused = cgraph_eval.rrf(lists) if lists else []
+                ranked = [p for p in fused if p in files]
                 recall = len(gold & set(ranked[:20])) / len(gold)
                 rr = next((1.0 / i for i, p in enumerate(ranked, start=1) if p in gold), 0.0)
                 scores[name]["overall"].append((recall, rr))
@@ -60,6 +64,9 @@ def main(argv: list[str] | None = None) -> int:
         cells = []
         for g in groups:
             vals = by_group[g]
+            if not vals:  # a split run (--split dev) can hold no sample of a task
+                cells.append("-")
+                continue
             cells.append(f"{sum(v[0] for v in vals) / len(vals):.4f} / {sum(v[1] for v in vals) / len(vals):.4f}")
         print(f"| {name} | " + " | ".join(cells) + " |")
     return 0

bench/arb/cgraph_eval.py

@@ -48,6 +48,7 @@
 from typing import Any, Iterator
 
 import snapshots
+from score import SPLITS
 
 RRF_K = 60
 CONTEXT_BUDGET = 8000
@@ -316,6 +317,18 @@ def rrf(lists: list[list[str]]) -> list[str]:
     return sorted(score, key=lambda p: (-score[p], first[p], p))
 
 
+def fuse(lists: dict[str, list[str]]) -> list[str]:
+    """The scored recipe: every op list votes once by RRF, except the anchors.
+    They are views of one change (two build-tag twins, three locale files) whose
+    near-identical lists outvoted the query's own lists one vote each, so they
+    are fused among themselves first and vote once (CGR-38)."""
+    anchor_lists = [v for k, v in lists.items() if k.startswith("context_anchor_")]
+    voices = [v for k, v in lists.items() if not k.startswith("context_anchor_")]
+    if anchor_lists:
+        voices.append(rrf(anchor_lists))
+    return rrf(voices)
+
+
 def retrieve(daemon: Daemon, sample: dict[str, Any], index: SnapshotIndex, files: set[str]) -> dict[str, Any]:
     prefix = str(daemon.root.resolve()) + "/"
     text = query_text(sample)
@@ -346,7 +359,7 @@ def retrieve(daemon: Daemon, sample: dict[str, Any], index: SnapshotIndex, files
         impacted.sort(key=lambda node: int(node.get("depth") or 0))  # stable: per-seed order kept within a depth
         lists["impact_changed"] = node_files(impacted, prefix)
 
-    fused = rrf(list(lists.values()))
+    fused = fuse(lists)
     in_candidates = [path for path in fused if path in files]
     ops["dropped_non_candidate"] = len(fused) - len(in_candidates)
     return {"ranked_files": in_candidates, "lists": {k: v[:50] for k, v in lists.items()}, "ops": ops}
@@ -442,6 +455,8 @@ def main(argv: list[str] | None = None) -> int:
     parser.add_argument("--jobs", type=int, default=4)
     parser.add_argument("--limit", type=int, default=0, help="Evaluate at most N samples per task (0 = all).")
     parser.add_argument("--keep-snapshots", action="store_true")
+    parser.add_argument("--split", choices=sorted(SPLITS), default="all",
+                        help="Evaluate only the dev or held-out repos (score.DEV_REPOS).")
     args = parser.parse_args(argv)
     if args.cgraph_bin_dir is None:
         parser.error("no cgraph on PATH; pass --cgraph-bin-dir")
@@ -460,6 +475,7 @@ def main(argv: list[str] | None = None) -> int:
     skipped: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int))
     for task in args.tasks:
         details_path = args.out / f"{task}_details.jsonl"
+        details_path.touch()  # a split can hold no sample of a task; score.py still expects the file
         done[task] = set()
         if details_path.exists():
             done[task] = {json.loads(l)["sample_id"] for l in details_path.read_text().splitlines() if l.strip()}
@@ -468,7 +484,7 @@ def main(argv: list[str] | None = None) -> int:
         if args.limit:
             samples = samples[: args.limit]
         for sample in samples:
-            if sample["id"] in done[task]:
+            if sample["id"] in done[task] or not SPLITS[args.split](sample["repo"]):
                 continue
             if task != "abstention" and baseline.query_has_leakage(sample, baseline.query_text_for_eval(sample)):
                 skipped[task]["query_leakage"] += 1

bench/arb/extdev/.gitignore

@@ -0,0 +1,3 @@
+# finalize.py writes <task>.jsonl as a byte copy of samples.jsonl.
+data/benchmark/*/code2test.jsonl
+data/benchmark/*/edit2ripple.jsonl