Filter English stopwords out of graph scope's exact-match boost

[emmahyde]

What

graph scope's exact-match scoring in src/graph/scope.ts had no stopword filtering. taskTokens() split the free-text task string on non-identifier characters and gave every token >=2 chars a flat 1.0 "exact-name-match" boost whenever it collided with any real identifier in the repo.

In practice, task phrasing is full of English function words and generic task-instruction verbs ("on", "to", "of", "up", "find", "trace"...) that routinely collide with trivial in-repo identifiers — Ruby DSL params (on:), block args, ActiveRecord's find — and those collisions outrank the actually relevant symbol.

Repro (see #115 for the full writeup): mex graph scope "find every call site that invokes mark_failed! on a model" returned the literal identifier on at score 1.0 five times as the top hits; the real target never made it into the top 10 at all.

Fix

Added a TASK_STOPWORDS set (English function words + generic task-instruction verbs) and filtered them out of taskTokens() before the exact-match loop runs. Whole-task semantic search (bm25-ranked, via graph.searchNodes(task, ...)) is untouched — this only scopes the flat-1.0 exact-match boost, which is the specific mechanism that was letting stopwords dominate.

Verification

• Added a regression test (scope.test.ts) reproducing the exact collision.
• npx vitest run — 11/11 in scope.test.ts; full suite 376/377 (the one failure is the pre-existing, unrelated tui.test.ts issue also called out in Add Ruby extraction via Prism, not tree-sitter #116).
• npx tsc --noEmit — clean.
• Rebuilt dist/ and re-ran the three real repro queries against a ~1600-file Rails codebase: all three now surface the correct symbol at or near the top of the ranking with no stopword noise.

Related

• graph scope stopword pollution, undocumented unresolved_refs fallback, and benchmark vs. grep gap (with repro + eval data) #115 — full writeup with repro steps and a blind grep-vs-mex eval.
• Add Ruby extraction via Prism, not tree-sitter #116 — the Ruby/Prism extractor PR from the same investigation (independent change, no overlap with this one).

[theDakshJaitly]

Confirmed the root cause over in #115 — the flat 1.0 boost lets stopword nodes take all six directSeeds slots, so the one-hop expansion gets built around on/to rather than the target. This is the right place to fix it. One thing to settle before I merge.

The filter is unconditional, so a symbol legitimately named one of the verb-tail entries can never take the exact-match boost. call, get, find, list, show, invoke are all real method names, and def call is the canonical Rails service-object entry point — so "who calls the call method on PhaseRunner" loses the boost on the one token carrying the query. It degrades rather than breaks, since whole-task semantic search still surfaces the node at ≤0.6, but it's a silent recall regression in exactly the class of codebase this fix came from.

Two ways out, either is fine by me:

A. Drop the verb tail. Remove find, list, trace, locate, show, get, invoke, invokes, call, calls and keep only the true English function words. The function words are what actually collided in your repro; the verbs are the entries most likely to be real symbol names.

B. Down-weight instead of filtering. Keep the full list, but give stopword exact-matches a reduced score (~0.4) rather than dropping them. They can no longer hijack the six seed slots, but a genuinely-named call still ranks. Strictly better on recall, slightly more change — it means threading a weight through the add() call rather than filtering the token list.

I lean A for minimality since it keeps the diff at its current size, but B is the more correct fix if you'd rather do it once. The regression test works unchanged under either.

Non-blocking: how, does, and what aren't in the list, so natural-language phrasing like our own evaluate/fixtures/nl-tasks.json queries still leaks a couple of tokens. Not worth holding this PR for.