[taylorg009]

CGR-38: a CGraph result on Agent Retrieval Bench (ARB) with no tuning on ARB samples.

Protocol

• Declared split before any change (score.py DEV_REPOS, bee900e). Dev = every gin, caddy and clap sample (111). Held-out = the other 234 samples from 22 repos.
• External dev set (bench/arb/extdev/): 229 code2test and 187 edit2ripple samples, mined with ARB's own miners from 10 repos ARB never used. The ARB dev split has no comment2context and only 19 code2test samples, so it could not drive tuning alone.
• Held-out samples never inspected individually. Aggregates were read for: the bin-v0.9.5 baseline, this branch build, and (still to do) the released binary.

Changes

1. tests edges from a test file to the source file its name pairs with. In-directory always. Package-unique or mirrored pairing only under a test root. Generic stems pair in-directory only.
2. File body documents. A file node's BM25 document = the text outside any symbol, plus its path.
3. Test counterparts in context. A seed's tests are reached at depth 1 (none when max_depth is 0).
4. Harness: anchor fusion. The per-file anchor lists are fused among themselves and vote once.

OpenSpec: test-subjects-and-file-documents (validates --strict). Index key logic-26.

Results (branch build; to be re-run on the released binary)

	R@20	MRR	BCY@8k
Held-out 234, RepoMap	0.5402	0.1785	0.3363
Held-out 234, bin-v0.9.5	0.5164	0.2076	0.3184
Held-out 234, branch	0.5628	0.2076	0.3167
All 345, RepoMap	0.6311	0.2156	0.3846
All 345, branch	0.6692	0.2951	0.4395

Against RepoMap on held-out:

• Ahead: R@20 by 0.0226 and MRR by 0.0291, both outside the noise floor.
• Behind: BCY@8k by 0.0196.
• trace2code regressed: MRR 0.3551 to 0.3094 against bin-v0.9.5.

Full tables are in bench/arb/RESULTS.md.

Verification

• 102/102 ctest pass. The new tests fail without their change (daemon_ops_test returns 130 when the counterpart block is reverted).
• An ASan build ran the full external dev set (416 samples) with 0 reports and rankings identical to the release build.
• A non-author review found one blocking bug, fixed in 4a93da7: a test outside a test root was paired across packages.
• tests pairs were checked by hand on the gin (27), clap (12) and caddy (67) snapshots.

🤖 Generated with Claude Code

[github-actions]

PR brief: 0 agent commits · run the full suite

Warning

Run the full suite. The graph cannot vouch for this diff, so every test file is selected. Nothing is skipped; this is the safe default, not a failure.

6 files have no graph node (config, asset, or unextracted).

If these files cannot change which tests should run, add a pattern for them to ignore. If they feed tests (fixtures, data), this outcome is correct.

bench/arb/extdev/data/benchmark/v2_code2test    ████████████████████████  2
bench/arb/extdev/data/benchmark/v2_edit2ripple  ████████████████████████  2
bench/arb/extdev                                ████████████  1
bench/arb/extdev/scripts                        ████████████  1

All 6 unmapped files

bench/arb/extdev/.gitignore
bench/arb/extdev/data/benchmark/v2_code2test/manifest.json
bench/arb/extdev/data/benchmark/v2_code2test/samples.jsonl
bench/arb/extdev/data/benchmark/v2_edit2ripple/manifest.json
bench/arb/extdev/data/benchmark/v2_edit2ripple/samples.jsonl
bench/arb/extdev/scripts/crawl_repo.sh

Computed at c374a8d against base 54da1b0.

Summary	PR #209
Intent	no checkpoints on this branch (10 commits)

What each commit did

Commit	Intent	Files	Reaches	Ran before push
bee900e bench(arb): declare the CGR-38 dev/held-out split and score…	no checkpoint	1 file
e79ec10 feat(graph): tests edges from a test file to the source fil…	no checkpoint	11 files
25933c2 fix(graph): scope cross-directory test pairing to the test'…	no checkpoint	4 files
968b654 feat(search): a file's body document holds its unenclosed t…	no checkpoint	2 files
a3f130d feat(context): a seed's tests are reached at depth 1	no checkpoint	2 files
053aba3 bench(arb): anchors vote once in the fused ranking	no checkpoint	2 files
4a93da7 fix(graph): a test outside any test root pairs only in its …	no checkpoint	6 files
67f994c docs(openspec): test-subjects-and-file-documents change	no checkpoint	4 files
87095cf bench(arb): CGR-38 results section (branch build, pending r…	no checkpoint	1 file
c374a8d bench(arb): commit the external dev set and score.py --tasks	no checkpoint	11 files

Claims checked

no checkpoint on this branch makes a claim the graph can check

_{blastline 0.13.0. Selection is advisory unless your workflow gates on it. How selection works. Intent: 0 of 10 commits. Not checked: symbol changes: no --change-context given.}

[tgod009]

Non-author review: one blocking finding (cross-package test pairing) fixed in 4a93da7 and re-verified on caddy/gin/clap; 102/102 ctest, ASan extdev clean, CI green.