Phase 5: Go, Rust, Python, JavaScript and Java - #31
Merged
Conversation
internal/ts becomes internal/extract. The per-file loop was already language-agnostic -- walk, hash, split lines, three capture passes, number overloads -- so the four things that were not move into a Spec: the grammar, the .scm, how a member call's receiver is read, and what an import binds. ScopeSegment replaces the node-kind switch that qualifiedName and enclosingDecl each kept a copy of. A language now states its scope rules once, and both walks go through it. files.language is the spec's name rather than a constant, so .tsx reports "tsx" -- the value says which grammar actually read the file, which is the failure the .tsx bug was invisible for. utils.IsSourceFile had no callers and is gone; the registry decides what is a source file, by exact extension rather than a suffix scan.
byName and byPkgName were repo-wide and keyed on name alone, so a call in main.go would have matched a same-named function in main.py. Both are now keyed by resolution group, and package_path is scoped to it too -- a polyglot repository puts main.go and main.py in the same directory. Partitioning at index-build time rather than filtering at lookup: there is then no code path that can reach a foreign-language candidate and forget to check. .ts and .tsx share the one group with more than one language in it. Everything else is its own, so the boundary holds by default for a language nobody has added yet. Imports are dropped at index time for languages whose specifiers name no file here. Go, Rust, Python and Java resolve through package clauses, crate paths, sys.path and the classpath, none of which this package models; entering rule 2 would answer unresolved for a module it cannot reach, throwing away the name_match rule 3 still has.
tree-sitter-javascript, one grammar for both -- unlike .ts and .tsx it parses JSX in any file. Scope rules, receivers and ESM imports are TypeScript's unchanged: same grammar family, same node kinds. require() is the addition, and it is a call rather than an import statement. The .scm cannot say "only require" without a predicate the Go binding does not evaluate, so it over-captures every single-string call argument and jsImports returns nil for the ones that are not. The driver treats a nil symbol list as "not an import" and drops the match. .js and .jsx join the ECMAScript resolution group, so a .js file importing a .ts file still resolves exact. That exposed a latent bug in ModuleCandidates: it stripped .js to find the .ts behind it and never restored it, so a specifier naming a real .js file matched nothing. Both are candidates now, stem first. TestExtract_OnlyTypeScriptExtensions is replaced by a registry-driven one, so the next language cannot leave it asserting last month's set.
Methods are named after their receiver type, so Repo.Sync never collides with a package-level Sync -- and because ScopeSegment now gets first say on a declaration's own name, the definition and the call site agree on it. The .scm deliberately does not capture a generic call with one type argument. Map[int](xs) parses as a type_conversion_expression, the same shape as int(x); capturing it would invent a call for every conversion in the repository. Two or more type arguments are unambiguous and do parse as a call. The fixture pins both halves, because a limit nobody wrote down is a limit that changes by accident. Imports bind a qualifier, never the symbols behind it. Recorded for a later Go resolver; the current one does not follow them, since a Go path names a module rather than a file here.
Methods are named after the type their impl block targets, so Repo.sync never
collides with a free sync.
Calls inside a macro are not captured, and the fixture pins that rather than
wishing otherwise. println!("{}", describe(&label)) has a token_tree body:
tree-sitter does not parse expressions in it, so describe is a bare identifier
beside a token_tree. Matching identifiers there would invent a call for every
name mentioned in every macro in the repository.
Rust has no quoted import specifier -- a `use` is one nested path expression --
so Spec.Imports now takes the captured node and returns both the module and the
symbols. The driver had been reading a quoted string out of the capture and
guessing at the statement above it, which only ever worked because every
language so far quoted its specifier.
A decorated definition wraps the function_definition rather than replacing it, so the captured name's parent is still the definition and the recorded source is the function's rather than the decorator's. async def is an ordinary function_definition. Both are pinned by the fixture. f-string interpolation and comprehension clauses are parsed, so calls in them are real calls -- unlike Rust's macro bodies. The fixture asserts each one, since these are exactly the places a query stops matching without saying so. Class nesting is part of the name: Repo.Nested.deep never collides with a module-level deep. The whole import statement is captured, because `import a.b` and `from .m import x as y` share no node to point at. `import a.b` binds a, not a.b, because a is the only name a call site can then write.
Every enclosing type names a method, and both hidden scopes are named too: a
method inside `new Runnable(){...}` is Repo.task.<anonymous>.run, and a lambda
body is anonymous the same way.
Java is the first language here with real overloads -- sync() and sync(int) in
one class -- so this is where overload_index stops being theoretical.
TypeScript's overload signatures parse as function_signature and were never
captured, so the post-pass had nothing to number. The resolver test asserts the
consequence: a call to sync resolves to nothing, because choosing between two
methods needs argument types this parser does not have.
Also asserts what the phase promises across all four new languages: an exact
edge outside the ECMAScript family is always same-file.
One directory, a file per language, each defining and calling a function named helper. One directory on purpose: package_path is identical across all of them, so a symbol table keyed on name and package alone links Go to Python. helper alone is not enough to prove the boundary. It exists in seven files, so ambiguity answers unresolved and the assertion passes whether the partition works or not. main.go therefore calls python_only and main.py calls go_only -- names that are unambiguous everywhere, so the only thing stopping them is the language partition. The assertions compare language names from a literal set rather than asking utils.ResolutionGroup. A test that measures the boundary with the function under test agrees with itself: collapsing every language into one group makes main.go resolve into main.py, and the group-based version noticed nothing. Verified by breaking ResolutionGroup and watching these fail. Skip-path defaults gain vendor, target, __pycache__, .venv, venv, .gradle, .mypy_cache and .tox -- the new languages' node_modules.
A badge on every node saying "typescript" on a TypeScript map is noise on a canvas whose three edge styles are already only a dash apart. What is worth showing is the boundary -- which is also where the resolver stops being able to say anything exact, so the badge and the dotted edge explain each other. Filled in MindMap rather than buildGraph: a language belongs to a file and buildGraph is given functions, and only the canvas knows which file the reader opened, which is what "another language" is measured against. The tree already carries every file's language, so it costs no request. The card is re-measured when the badge appears. A card that grows a badge it was not sized for truncates its own name instead, which is the same class of bug as the "start" badge squeezing cloneRetryOptions to cloneRet... Sidebar's empty state no longer claims TypeScript is the only language, and Shiki learns java.
A re-parse reinserts a changed file's functions under new ids while the file row keeps its own, so the ids zustand/persist restores can point at rows that are gone -- and since Phase 4 a webhook does that with nobody touching the browser. Nothing can be checked at rehydrate, because nothing is loaded yet. The check runs when the open file's function list arrives, which is the first moment there is anything to check against, and that list is authoritative for branch roots because a root is only ever opened from its card. An expanded id deeper in the map, in another file, is left to its own query: it 404s and that branch is not drawn. Judging it here would mean fetching every file's functions to answer a question the query already answers.
CLAUDE.md gains the language list, the Spec layout, and the two conventions that bite: the resolver partitions rather than filters, and a test must not decide what to allow by calling the function it is testing. PARSING_STRATEGY closes the cross-language limitation it recorded as unreachable, and gains a table of what each language does not capture and why. Every row was found by dumping the tree and is pinned by a fixture assertion -- they are limits, not bugs, and the failure mode of an unrecorded one is a parser that quietly produces less. RISKS closes R34 and opens R36 (a self-referential test) and R37 (six languages through one resolver, with only one language's rules modelled).
The recipe cds into services/parser, so the path documented in CLAUDE.md -- written relative to the repo root, like every other path in that file -- was being looked up one level too deep and failed with lstat. abspath resolves it against the Makefile's own directory and leaves an absolute path alone.
ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Free Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (11)
🚧 Files skipped from review as they are similar to previous changes (6)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe parser now supports specification-driven extraction for TypeScript, JavaScript, Go, Rust, Python, and Java. Resolver indexes use language groups. The web canvas displays foreign-language badges and removes stale persisted roots. ChangesPolyglot extraction and resolution
Estimated code review effort: 4 (Complex) | ~60 minutes Merge Risk: ⚪ Minimal · up to The PR expands language extraction and same-file resolution with documented verification; no actionable merge-blocking risk remains beyond normal checks and review. Sequence Diagram(s)sequenceDiagram
participant Parser
participant Extract
participant Resolver
participant MindMap
participant FunctionCard
Parser->>Extract: parse registered source extensions
Extract->>Resolver: provide language-tagged functions, calls, and imports
Resolver->>Resolver: resolve within language groups
Resolver-->>MindMap: return graph nodes and edges
MindMap->>FunctionCard: render localized nodes and foreign-language badges
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/settings/billing. Comment |
go mod tidy had not been re-run since JavaScript went in, so tree-sitter-go, -rust, -python and -java were still listed as indirect and go.sum still carried the versions other grammars pull in transitively. CI's tidy check caught it.
Twenty-seven tree-sitter field names were still spelled inline across the seven language specs -- "name" six times, "alias" four, and so on. They join the node kinds in internal/utils/constants.go, in their own block: a field is not a kind, and every grammar names its fields from the same small vocabulary, so they are shared rather than grouped per language. Worth having as constants for the same reason the kinds are. ChildByFieldName returns nil for a field that does not exist, so a grammar that renames one drops whatever read it and says nothing. cmd/parser also spelled out the three confidence tiers a second time, next to the constants that exist so they cannot drift from the database CHECK. It reads utils.ConfidenceTiers now. Its two --format values become package-local constants, since nothing outside that command reads them. No behaviour change: the polyglot summary is identical before and after.
It was named the branch after the 3b gate, in two places. Phases 4 and 5 went first, so the page is overdue rather than upcoming and the doc was telling the next reader something that stopped being true two phases ago.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extraction and same-file resolution for five more languages, and a language boundary the resolver cannot cross by accident.
What changed
internal/tsbecomesinternal/extract. The per-file loop was already language-agnostic — walk, hash, split lines, three capture passes, number overloads — so the four things that were not moved into aSpec: the grammar, the.scm, how a member call's receiver is read, and what an import binds. Adding a language is now a Spec, a.scm, and a fixture.files.languageis the spec's name rather than a constant, so.tsxreportstsx— the value now says which grammar actually read the file, which is precisely what the.tsxbug was invisible for.files.languageexact?.ts/.tsx/.js/.jsx/.mjs/.cjstypescript/tsx/javascript/jsxecmascript.gogogo.rsrustrust.pypythonpython.javajavajavaJavaScript came along because the TypeScript spec already knew how to read it and
.js/.jsxwere being skipped entirely. Java came along because it is the first language here with genuine overloads, which is whatoverload_indexwas always for.The boundary
byNameandbyPkgNamewere repo-wide and keyed on name alone, so a call inmain.gowould have matched a same-named function inmain.py. Both are now keyed by resolution group,package_pathincluded — a polyglot repository putsmain.goandmain.pyin the same directory. Partitioned at index-build time rather than filtered at lookup, so there is no code path that can reach a foreign-language candidate and forget to check.Imports are recorded for every language but dropped at index time for the ones whose specifiers name no file here. Go, Rust, Python and Java resolve through package clauses, crate paths,
sys.pathand the classpath; entering the import rule for them would answerunresolvedfor a module it cannot reach, throwing away thename_matchthe later rules still have.Limits found by dumping the tree, not by assuming
Each is pinned by an assertion, because a parser that quietly produces less reads as one that worked.
Map[int](xs), one type argument, parses astype_conversion_expression: the same shape asint(x). Not captured. Two or more type arguments are unambiguous and are.token_tree. Nothing insideprintln!("{}", helper())is parsed as an expression.sync()andsync(int)needs argument types, so a call to an overloaded method resolvesunresolved.awaitare all parsed, anddecorated_definitionwraps the definition rather than replacing it.Two bugs the exit test found in itself
The polyglot fixture is one directory with a file per language, each defining and calling
helper.helperalone proves nothing — it exists in seven files, so ambiguity answersunresolvedwhether the partition works or not.main.gotherefore callspython_onlyandmain.pycallsgo_only: names unambiguous everywhere, so the only thing stopping them is the partition.And the assertions first compared
ResolutionGroup(caller)withResolutionGroup(callee)— which passed withResolutionGroupreturning a constant, because both sides moved together. They compare language names from a literal set now. Verified by breaking the function on purpose and watching the tests fail. Recorded as R36.Also in here
zustand/persistrehydrates, sodropMissingRootsruns when the open file's function list arrives — the first moment there is anything to check against.ModuleCandidatesstripped.jsto find the.tsbehind it and never restored it, so a specifier naming a real.jsfile matched nothing. Latent until.jsfiles existed.make go-run REPO=...resolved its path againstservices/parserrather than the repo root, so the usage documented inCLAUDE.mdfailed withlstat.Verification
Written through to Postgres as well, since the IR passing is not the same as the write path handling it: seven
files.languagevalues, and the only cross-language edge istsx -> typescript.Not checked here: a real polyglot public repository charted end to end in a browser. That needs a GitHub OAuth sign-in, which is yours to do — the tree's language labels, the per-node badge and Shiki highlighting for Go/Rust/Python/Java are worth a look before this is called finished.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation