graph scope stopword pollution, undocumented unresolved_refs fallback, and benchmark vs. grep gap (with repro + eval data)

[emmahyde]

Ran mex against a large (1600+ file) real-world Rails codebase with a Ruby extractor I contributed (Prism-based). Found three related issues while comparing mex against plain grep on real lookup tasks, with a blind eval to back it up.

1. graph scope's exact-match scoring has no stopword filtering

taskTokens() in src/graph/scope.ts splits the free-text task string on non-identifier chars and gives every token >=2 chars a flat 1.0 "exact-name-match" boost if it happens to collide with a real (often trivial) identifier in the repo — e.g. task phrasing containing "on", "to", "of", "up" collided with Ruby DSL params/block args literally named that, burying the actually-relevant symbol under stopword noise.

Repro: mex graph scope "find every call site that invokes mark_failed! on a model" returned on (score 1.0) x5 as top hits; the real target never appeared in the top 10.

I have a fix (stopword list + filter, with a regression test) ready as a diff/PR if useful.

2. who-calls/scope can't find dynamically-generated methods, with no documented recovery path

Ruby methods generated via metaprogramming (e.g. a state-machine DSL block) have real call sites but no literal def, so no resolved node exists. who-calls <method> returns TARGET_NOT_FOUND with no next step. The call sites are captured, just in the unresolved_refs table, which isn't exposed by any CLI command — recovering them required hand-writing SQL against internal schema that the contributor docs say requires a core/discuss-first issue to even touch.

Would who-calls falling through to unresolved_refs when no resolved node exists (clearly labeled as unresolved/dynamic) be in scope for a lighter-weight contribution?

3. The benchmark doesn't compare against grep, and the README implies it does

evaluate/README.md says outright:

The current harness does not compare an agent with the graph against the same agent using only Read/Grep/Glob. Therefore these results do not support an end-to-end graph-vs-no-graph token-savings claim.

That's accurate and appreciated, but the top-level README's "Results" table (10.74x, 916x) reads as exactly that claim to a new user, and it's measured against a favorable synthetic baseline ("grep's top-3 whole files") rather than a real grep-only agent run.

Our eval

Six blind subagents (fresh context each, no shared answers, no knowledge of the "correct" result), split into a grep-only condition and a mex condition, on the same three real questions against our codebase. Real reported token/tool-call/time usage per agent:

Task	Condition	Tokens	Tool calls	Time	Correct?
Find every class defining build_commands! (excl. tests)	grep	36,719	1	10.3s	19/19
	mex (graph query/sqlite)	61,736	2	27.5s	19/19
Find every call site of .mark_failed! in app/	grep	36,700	1	8.1s	15/23
	mex (who-calls + unresolved_refs)	63,481	5	46.3s	23/23
Trace CoderSession's superclass chain to the gem boundary	grep	47,714	4	63.1s	correct
	mex (graph scope + sqlite)	62,722	4	45.1s	correct

Takeaways:

• mex ran 1.3-1.7x grep's tokens on every task, and matched or lost on wall time too.
• mex tied grep on 2/3 tasks (no benefit for the extra cost) and won decisively on 1/3: grep's \.mark_failed! regex missed 8 real call sites that invoke the method without an explicit dot-receiver (bare self-calls inside the model classes); mex's AST-based edges don't have that blind spot. That's a real, non-obvious correctness edge — a naive regex can produce a confidently wrong "15 results, looks complete" answer that nothing flags.
• Getting that win required falling through to raw sqlite against unresolved_refs (issue Expand KNOWN_RUNTIMES allowlist in dependency checker #2 above) — not something a first-time CLI user would discover from who-calls alone returning TARGET_NOT_FOUND.

Net: mex earns its cost specifically on call-site enumeration where call syntax varies in ways a regex can't enumerate up front. For straightforward name/definition lookups and short inheritance walks, grep was cheaper and just as correct in this test. Worth either softening the README's headline claim or running the real 3-arm comparison the eval docs already call out as missing.

Happy to share the Ruby extractor, the scope.ts stopword fix, and the raw eval transcripts if useful.

[emmahyde]

@theDakshJaitly could you let me know who is the best reviewer for the attached PRs?

[theDakshJaitly]

Thanks for this — genuinely. Running a blind eval with a control arm, reporting the result that made mex look bad alongside the one that made it look good, and showing the methodology is considerably more work than filing a bug. All three findings hold up under review.

On your question: I'm the reviewer for both. There's no CODEOWNERS in the repo, which is why this wasn't discoverable — I'll add one. On src/graph/ specifically, the only other contributor with real context is @abhinav-phi.

Taking the findings in order.

1. Stopword pollution — confirmed, merging.

I traced it, and it's worse than the top-hits noise you saw. In selectScope, semantic matches cap at 0.6, exact-name-match is a flat 1.0, and directSeeds then takes the top 6 and expands one hop for callers/callees. So stopword nodes don't just clutter the ranking — they consume all six seed slots, and the neighborhood expansion gets built around on and to instead of the target. Your fix addresses the root of that.

One review note before I merge #117: the list includes call, calls, get, find, list, show, invoke. The filter is unconditional, so a symbol legitimately named one of those can never take the boost — and def call is the canonical Rails service-object entry point, which makes your own codebase the worst case. Semantic search still surfaces it at ≤0.6, so it degrades rather than breaks, but I'd prefer either dropping the verb tail and keeping only the true function words, or keeping the list and skipping the filter when a stopword is the only surviving token. Your call which — either works.

Worth recording: this bug was live in the default natural-language retrieval path, and our gated benchmark structurally could not see it, because evaluate/fixtures/symbol-tasks.json is six bare identifiers with no English around them. That's a gap in our eval, not just a bug in scope.ts.

2. unresolved_refs fallback — yes, in scope.

To answer directly: yes, who-calls falling through to unresolved_refs when no resolved node exists, clearly labeled as unresolved/dynamic, is exactly the kind of lighter-weight contribution I'd take. It touches none of identity, reconciliation, schema, or drift semantics, so CONTRIBUTING's discuss-first rule doesn't apply. The table is already indexed on reference_name (schema.sql:183), so the lookup is cheap. It's yours if you want it; if you'd rather hand it off after two PRs, say so and I'll pick it up.

This is also the fix that matters most from your eval. The 23/23 vs 15/23 on .mark_failed! is the strongest result in the writeup — a regex missing bare self-calls and returning a confidently complete-looking answer is precisely the failure mode the graph should catch — and right now it's only reachable by hand-writing SQL against internal schema. That's the difference between mex being useful and mex being usable.

3. README overclaim — you're right.

The numbers are real for what they measure, and what they measure is not what the Results table implies. The denominator in evaluate/lib/grep-baseline.mjs is ceil(chars/4) over the entire contents of three whole files — a payload-size ratio against a favorable baseline, with no agent in the harness at all. Your grep arm spending 36.7k tokens on a single tool call is mostly fixed session overhead, which the payload we optimize barely moves. Both sets of numbers are true at once; the honest framing is compact retrieval payload, which is what evaluate/README.md says and what the top-level README fails to.

I'm relabeling that section with the denominator stated inline and leading with the correctness argument rather than the ratios. The three-arm experiment evaluate/README.md already calls for is the real fix — agent-e2e-model.mjs drives claude -p and parses stream-json today, so it's mostly a third variant plus raw usage aggregation.

On #116.

I'm the reviewer, but I don't want to line-review 478 lines before settling the question underneath them: whether we take on a second parser architecture for one language. You made the case well and kept the frozen LanguageExtractor interface, node identity scheme, and schema untouched, which is the right instinct. My hesitation is the placeholder TSTree — it keeps extractFile's null gate looking uniform, but a Prism parse failure now surfaces inside extract() rather than at the gate, so Ruby's error path does diverge even where the happy path doesn't. And it sets the precedent for the next non-tree-sitter parser someone proposes.

That's my call to make, not a defect in your PR. I'm opening a separate issue for it so it doesn't sit blocked on me. Prism being more semantically accurate than tree-sitter-ruby isn't in dispute.

Finally — yes to the raw eval transcripts if you're still willing to share. Even at n=1 per cell they're the closest thing we have to a real three-arm baseline, and I'd like them in evaluate/ as reference data while we build the proper one.

[emmahyde]

At work ATM but will definitely get you these before midnight EST!

[theDakshJaitly]

Yeah yeah no worries, thanks again!