diff --git a/CLAUDE.md b/CLAUDE.md index ff00b7e..c1f7903 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,8 +7,9 @@ contract is in `PRD.md` and the phase plan is in `PLAN.md`. An interactive visual map of a codebase. Clone a repo → tree-sitter extracts functions and call sites → resolve calls to definitions → store the graph in Postgres → explore it on a React Flow -canvas (file → card → function mind-map → code block). Language: **TypeScript** through Phase 4; -Go, Rust and Python are added in Phase 5, extraction only. +canvas (file → card → function mind-map → code block). Languages: **TypeScript, TSX, JavaScript, +JSX, Go, Rust, Python and Java**. Everything past the ECMAScript family is extraction plus +same-file resolution only — see "Per-language extraction limits" in `docs/PARSING_STRATEGY.md`. The product is called **funcatlas** everywhere — repo, module path, npm scope, database, cookie. The old working name "CodeCanvas" is retired; do not reintroduce it. @@ -21,7 +22,7 @@ The old working name "CodeCanvas" is retired; do not reintroduce it. - [x] Phase 3a — API and auth - [x] Phase 3b — Canvas and search - [x] Phase 4 — Webhooks, queue, hardening -- [ ] Phase 5 — Go, Rust, Python (extraction only; per-language resolution stays cut) ← next +- [x] Phase 5 — Go, Rust, Python, JavaScript, Java (extraction only; per-language resolution stays cut) Active task list: `TASKLIST.md`. @@ -60,7 +61,8 @@ You implement, phase by phase, with tests. The user reviews at each phase gate. | Concern | Home | |---|---| -| All parser constants — confidence tiers, node kinds, import kinds, limits | `services/parser/internal/utils/constants.go` | +| All parser constants — language names, resolution groups, node kinds by language, confidence tiers, import kinds, limits | `services/parser/internal/utils/constants.go` | +| One language's grammar, `.scm`, scope rules, receiver and imports | `services/parser/internal/extract/.go`, registered in `spec.go` | | Tree-sitter node traversal | `services/parser/internal/utils/nodes.go` | | Repo-relative paths, module specifier resolution | `services/parser/internal/utils/paths.go` | | Qualified-name building and scope candidates | `services/parser/internal/utils/qualnames.go` | @@ -91,8 +93,9 @@ You implement, phase by phase, with tests. The user reviews at each phase gate. lucide-react, Zustand + TanStack Query. - **API:** Fastify + Drizzle + postgres.js + Zod, arctic/oslo for GitHub OAuth, Redis sessions, `@fastify/rate-limit`. -- **Parser:** Go + `tree-sitter/go-tree-sitter` v0.25.0 + `tree-sitter/tree-sitter-typescript` - v0.23.2 (both pinned in `go.mod`), pgx with explicit SQL, zap. +- **Parser:** Go + `tree-sitter/go-tree-sitter` v0.25.0, plus one pinned grammar per language: + `tree-sitter-typescript` v0.23.2, `-javascript` v0.25.0, `-go` v0.25.0, `-rust` v0.24.2, + `-python` v0.25.0, `-java` v0.23.5. pgx with explicit SQL, zap. - **Database:** Postgres — edge tables and recursive CTEs. Neo4j deferred indefinitely. - **Queue:** Redis + BullMQ, consumed by a **Node worker that spawns the Go binary** (`pnpm worker`). The parser has no Redis dependency. @@ -132,7 +135,17 @@ files over 1 MB skipped. - **One grammar per extension, never shared.** `.ts` uses `LanguageTypescript()`, `.tsx` uses `LanguageTSX()`. A mismatched grammar fails *silently*: the body becomes an `ERROR` node, the declaration still matches, and every call inside is dropped. Any new language needs a fixture that - pins the **calls** inside its hardest construct, not just the function names. + pins the **calls** inside its hardest construct, not just the function names. `tree-sitter-javascript` + is the exception that proves it: one grammar reads JSX in any file, so `.js` and `.jsx` share it. +- **Adding a language is a `Spec` in `internal/extract/`, a `.scm` in `queries/`, and a fixture.** + The third is not optional. `internal/extract/spec_test.go` fails if a `.scm` is missing any of the + three captures, and the extension registry is driven off `registry` so a new language cannot leave + a test asserting last month's set. +- **The resolver partitions by language group; it does not filter by it.** `byName` and `byPkgName` + are keyed on the group so there is no code path that can reach a foreign-language candidate at + all. `.ts`/`.tsx`/`.js`/`.jsx`/`.mjs`/`.cjs` share the one group with more than one language in + it. **Do not write a test that decides what to allow by calling `utils.ResolutionGroup`** — it + agrees with itself when broken. See R36. ## Known gaps @@ -142,10 +155,10 @@ Phase 3b is closed. `TASKLIST.md` is the chunk-level truth; this is what outlive any change, so a `memo` keyed on reference re-rendered every card whenever one changed — and each card showing source re-ran its highlighted block, which the reader saw as untouched cards blinking. `sameNodeData` in `lib/graph.ts` is what both node memos use; keep it that way. -- **Canvas state is persisted but never re-validated.** `store/ui.ts` restores the repository, file - and open branches through `zustand/persist`. A re-parse reinserts changed files' functions under - new ids, so a restored branch can point at rows that are gone — and a webhook now does that - without anyone touching the browser. See R34. +- **Restored canvas state is checked late, not at rehydrate.** Nothing is loaded when + `zustand/persist` rehydrates, so `dropMissingRoots` runs when the open file's function list + arrives. That list is authoritative for branch roots only; an expanded id in another file is left + to its own query, which 404s. See R34. - **A webhook's HMAC covers the raw bytes, so the JSON parser is scoped, not global.** Fastify's default parser drops the text, and re-serialising `req.body` does not reproduce what GitHub sent — the digest then fails in a way that reads as a wrong secret. A test that signs *compact* JSON will @@ -184,7 +197,11 @@ Phase 3b is closed. `TASKLIST.md` is the chunk-level truth; this is what outlive `node dist/index.js` cannot follow them. `pnpm dev` (tsx) works. Fix before containerising. - **Compiled test files land in `apps/api/dist`.** Harmless locally, wrong in an image. - **Resolution limits that are honest, not broken:** barrel re-export chains, default imports, and - `tsconfig` path aliases all resolve to `unresolved`. See `docs/PARSING_STRATEGY.md`. + `tsconfig` path aliases all resolve to `unresolved`. So does every per-language construct in + "Per-language extraction limits" — Go's single-type-argument generic call (ambiguous with a + conversion), anything inside a Rust macro (a `token_tree` is not parsed), and which Java overload + a call meant. Each is pinned by an assertion, because a parser that quietly produces *less* reads + as one that worked. See `docs/PARSING_STRATEGY.md`. ## Verify state before trusting this file @@ -204,7 +221,7 @@ gh run list --branch $(git branch --show-current) --limit 2 # is CI green? Parser on its own, when the question is about extraction rather than the app: ```bash -make go-run REPO=./services/parser/testdata/resolve +make go-run REPO=./services/parser/testdata/polyglot cd services/parser && go run ./cmd/parser --repo ./testdata/resolve --format summary ``` diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 9d0133b..628bd6a 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -55,7 +55,8 @@ on a reachable host was a session for the asking. /cmd/parser entry point /internal/clone local path or shallow git clone /internal/security path containment, size and depth caps, symlink rejection - /internal/ts tree-sitter extraction and the qualified-name scope walk + /internal/extract tree-sitter extraction: the language-agnostic driver, the + qualified-name scope walk, and one Spec per language /internal/ir Go-native intermediate representation /internal/resolver call resolution and confidence tagging /internal/db Postgres writer diff --git a/Makefile b/Makefile index 536f714..9cd7db3 100644 --- a/Makefile +++ b/Makefile @@ -106,7 +106,9 @@ parser-isolated: ## Run the parser in its container with no network at all (docs parser --repo /fixture --format summary --out - go-run: ## Run the parser against a local repo (usage: make go-run REPO=./path) - cd services/parser && go run ./cmd/parser --repo "$(REPO)" + # abspath, because the recipe cds into services/parser and REPO is written + # relative to the repo root -- which is where everything else in here is. + cd services/parser && go run ./cmd/parser --repo "$(abspath $(REPO))" --format summary clean: ## Clean up generated artifacts and caches pnpm store prune diff --git a/PLAN.md b/PLAN.md index ec8340c..15e9748 100644 --- a/PLAN.md +++ b/PLAN.md @@ -38,9 +38,9 @@ Delivered: - `internal/clone` — local path, or `git clone --depth 1`; never runs the repo's install or build scripts. - `internal/security` — env-driven caps, `ContainsRoot` path containment, and a `Walk` that hard-fails on symlinks, sniffs for binary content, and enforces file-count, per-file-size, and depth limits. -- `internal/ts` — tree-sitter TypeScript extraction: function declarations, methods, arrow and +- `internal/ts` (now `internal/extract`) — tree-sitter TypeScript extraction: function declarations, methods, arrow and function expressions assigned to variables; call sites with their enclosing caller; imports. -- `internal/ts/scope.go` — the dot-joined qualified-name walk (`Repo.sync`, `getUser.inner`). +- `internal/ts/scope.go` (now `internal/extract/scope.go`) — the dot-joined qualified-name walk (`Repo.sync`, `getUser.inner`). - `internal/ir` — Go-native `File` / `Function` / `CallSite` / `Import` / `Graph`. - `queries/typescript.scm` — the query patterns, embedded at build time with `//go:embed`. - `Dockerfile` — multi-stage, non-root; `docker-compose.yml` runs the parser with @@ -155,10 +155,11 @@ throttled; the parser still works with no network egress. — passed; see `TASKL **Known carry-over into Phase 5** — R34: `store/ui.ts` restores a file and its open branches through `zustand/persist`, and this is the first phase whose re-parse can delete the rows behind them. +Closed in Phase 5. --- -## Phase 5 — Go, Rust and Python · not started +## Phase 5 — Go, Rust, Python, JavaScript and Java · done Extraction only. Each language gets functions, call sites and imports in the IR, and same-file resolution, which is language-agnostic. Cross-file calls resolve to `name_match` or `unresolved`, @@ -167,7 +168,8 @@ never `exact`. **Why the split.** Extraction is a grammar, a `.scm`, and a set of node kinds — cheap and testable. Resolution is not, and it is the whole product. Go resolves through package clauses and capitalisation-based export, Rust through `mod`/`use`/crate paths and `impl` blocks, Python through -`sys.path` and `__init__.py`. Sharing one resolver across them would emit confident wrong edges, +`sys.path` and `__init__.py`, Java through the classpath and argument types. Sharing one resolver +across them would emit confident wrong edges, which is worse than admitting ignorance — see [`PRD.md`](PRD.md#8-the-design-commitment). The confidence tiers already carry that admission to the user honestly. @@ -180,13 +182,23 @@ parsed while three of four calls inside its JSX were dropped. One wrong grammar, edges gone, no error anywhere. Every language added multiplies that risk, which is why each one needs its own fixture pinning *calls*, not just functions. -**Builds:** `tree-sitter-go`, `tree-sitter-rust`, `tree-sitter-python`; one `.scm` per language; -per-language node-kind constants; `files.language` populated per file rather than hardcoded; -extraction fixtures per language. The UI learns to show language per node. +**Shipped:** `tree-sitter-javascript`, `-go`, `-rust`, `-python`, `-java`; one `.scm` per language; +per-language node-kind constants; `files.language` populated per file rather than hardcoded, so +`.tsx` now reports `tsx`; extraction fixtures per language. The UI badges a callee whose language +differs from the file being read. -**Exit test:** a polyglot fixture repo produces correct functions and call sites for all four -languages, no edge crosses a language boundary, and `.tsx`-style silent grammar mismatch is -impossible because every language's fixture pins the calls inside its hardest construct. +JavaScript was added alongside the three, because the TypeScript spec already knew how to read it +and `.js`/`.jsx` were being skipped entirely. It joins TypeScript's resolution group, so a `.js` +file importing a `.ts` file still resolves `exact` -- the one cross-file case that survives. +Java was added because it is the first language here with genuine overloads, which is what +`overload_index` was always for. + +**Exit test:** `testdata/polyglot` produces functions *and* call sites for every language, no edge +crosses a language boundary, and cross-file resolution is never `exact` outside the ECMAScript +family. A `.tsx`-style silent mismatch is impossible because every language's fixture pins the +calls inside its hardest construct -- and each of those turned out to be a real limit worth +recording rather than a hypothetical: Go's single-type-argument generic call, Rust's macro bodies, +Python's decorated definitions, Java's overloads. --- @@ -198,7 +210,7 @@ Each of these was considered and deliberately deferred, not forgotten. |---|---| | LSP-based resolution | Accurate but slow to build and RPC-heavy on re-export chains. Name/scope resolution ships first, honestly tagged; LSP upgrades `name_match` to `exact` later. | | Freehand annotation layer | Pure nice-to-have. Zero bearing on whether the core graph is trustworthy. | -| ~~Languages beyond TypeScript~~ | **Reinstated as Phase 5** (extraction only). Full per-language resolution is still cut. | +| ~~Languages beyond TypeScript~~ | **Done in Phase 5** (extraction only). Full per-language resolution is still cut. | | Neo4j | Recursive CTEs over an edge table handle this scale. Revisit when traversal is measurably the bottleneck. | | Saved canvas layouts | Positions resetting on reload is a papercut, not a blocker. | | Multi-tenancy and RBAC | Single-user MVP. Each repo is already an isolated workspace, so this stays cheap to add. | diff --git a/TASKLIST.md b/TASKLIST.md index 6f50e03..db60e3b 100644 --- a/TASKLIST.md +++ b/TASKLIST.md @@ -1,25 +1,19 @@ -# Phase 4 — Webhooks, Queue and Hardening +# Phase 5 — Go, Rust, Python, JavaScript and Java -The live task list. Phase 3b gave the graph a face. It also proved the ceiling: charting a -repository holds the HTTP request open for the whole clone-and-parse, and once charted the graph is -a snapshot of one commit forever. Phase 4 makes it self-updating and closes the security checklist. +The live task list. Phase 4 made the graph self-updating. It also left the parser reading exactly +one language, with `files.language` hardcoded to `"typescript"` and a resolver that had never heard +of a language at all. Phase 5 teaches it five more — extraction and same-file resolution only — and +makes the language boundary something the resolver cannot cross by accident. -**Branch:** `phase-4/webhooks-queue-hardening` -**Reference:** [`PLAN.md`](PLAN.md) Phase 4 · [`docs/SECURITY.md`](docs/SECURITY.md) · -[`docs/RISKS.md`](docs/RISKS.md) R12–R14, R27, R30–R31 +**Branch:** `phase-5/polyglot-extraction` +**Reference:** [`PLAN.md`](PLAN.md) Phase 5 · [`docs/PARSING_STRATEGY.md`](docs/PARSING_STRATEGY.md) · +[`docs/RISKS.md`](docs/RISKS.md) R34, R36 ## How to work through this -Claude implements; you review at the phase gate. One chunk at a time, top to bottom — the order is -load-bearing: D0 produces the hashes D1 diffs, D1 makes the write safe before anything triggers it -automatically, D2 gives D3 and D4 something to enqueue onto, and D4 is the only reason the queue -exists. Each chunk lists: - -- **Why** — the reason it exists, so it can be pushed back on if the reason is wrong. -- **Where** — the files it touches. -- **Do** — the work, broken into steps. -- **Done when** — the objective test. Not "the code compiles". -- **Watch for** — the specific bugs to check for before the chunk is committed. +Claude implements; you review at the phase gate. The order is load-bearing: E0 makes a language a +value rather than a hardcode, E1 closes the boundary before any second language exists to leak +across it, E2–E6 add one language each, and E7 is the gate. Write the test in the same commit as the code it tests. Commit one chunk at a time with an imperative message. `[ ]` todo · `[~]` in progress · `[x]` done. @@ -28,406 +22,133 @@ imperative message. `[ ]` todo · `[~]` in progress · `[x]` done. | Chunks | Concept | Where else you'll meet it | |---|---|---| -| D0 | Content-addressing as change detection, when the obvious source of truth is unavailable | Build caches, CDNs, every incremental compiler | -| D1 | A cascade delete reaching rows you did not think you were touching | Any schema with FKs and a partial rewrite | -| D2 | Idempotency keys collapsing a storm into one unit of work | Every queue, every webhook consumer | -| D4 | Verifying a signature over bytes you must not re-serialise | Stripe, GitHub, every HMAC webhook | -| D3 | A synchronous contract becoming asynchronous, and what the UI owes the reader afterwards | Every "we'll email you when it's ready" flow | - -D1 is the one worth slowing down for. It is where a re-parse stops being "write the files that -changed" and becomes "write the files whose *rows* changed" — a set that includes files nobody -edited. Get it wrong and edges vanish silently, which is the same class of failure as the `.tsx` -grammar mismatch in Phase 2: the data looks written, and part of it is simply gone. - -## Decisions taken before starting - -Asked and answered at planning time, so they are not re-litigated mid-phase. - -- **Full extract, full resolve, scoped write.** `PRD.md` FR-9 says "re-parse only the changed - files", and that cannot be done as written: `internal/resolver` builds six whole-repo maps, and - two of the three resolution passes need them. Given only the changed files, cross-file calls - degrade from `exact` to `unresolved` — and a partial candidate set can emit a **false `exact`** - where the whole repo would correctly say ambiguous. That is exactly the guess `PRD.md` §8 forbids. - So resolution stays byte-identical to today and only the write is scoped. The phase buys database - churn and edge correctness, **not** parse time, and `PLAN.md`'s exit test is amended in D7 to say - so honestly. -- **A Node worker that spawns the Go binary.** BullMQ is consumed in `apps/api`; the worker - `execFile`s the parser exactly as registration does today. The parser gains no Redis dependency - and `go-ci.yml` needs no Redis service. This contradicts `docs/ARCHITECTURE.md`'s claim that the - worker talks only through the queue and Postgres; D7 corrects the document. -- **Content hash, not `git diff`.** A `--depth 1` clone has one commit and no history, deepening the - fetch costs network on every parse, and a force-push invalidates a commit range anyway. The walk - already reads every byte. -- **Security scope.** Do L34 (webhook), L39 (clone cleanup), R30 (`/auth/dev-login`), R31 (Redis - hang). Tick L35, L36, L38 — already true, never ticked. **Defer** L32 (TOCTOU symlinks) and L37 - (token refresh), with reasons recorded rather than left blank. - -## What this phase does not build - -Multi-tenancy. Private repositories. A retry or backoff UI. Per-language resolution — that is -Phase 5. Saved canvas layouts. Deleting a repository from the UI: R14's database half already ships -and is tested, and this phase adds only the disk half. - ---- - -## D0 — Change detection: hash every file `[x]` - -**Why.** Everything downstream needs the answer to "which files actually changed", and it has to -work for a shallow clone and a bare local path alike. - -**Where.** `services/parser/migrations/0003_file_content_hash.*.sql`, `internal/ir/ir.go`, -`internal/ts/extract.go`, `internal/db/writer.go`, `packages/shared/src/schema.ts`, `.gitattributes`. - -**Do.** - -1. Migration `0003` adds `files.content_hash TEXT`, nullable, with its `.down.sql` in the same - commit. ✅ -2. `ir.File.ContentHash`, filled in the extractor where the bytes are already in hand — `sha256`, - hex, stdlib only. ✅ -3. `upsertFiles` writes it and the `ON CONFLICT` path updates it. ✅ -4. Mirror the column in the Drizzle schema. The SQL and Drizzle disagreeing is R20, and it cost a - whole chunk last time. ✅ - -**Done when.** Parsing a fixture twice leaves every hash identical; editing one file moves exactly -one hash. — `TestWriteGraph_ContentHashTracksEdits`, passing. - -**What it turned up.** Two things beyond the chunk. `make test` never sourced `.env`, so -`dbtest.URL` found no `DATABASE_URL`, every database integration test skipped, and the command -reported a green run that had not touched Postgres — fixed in its own commit. And pinning hashes in -the golden fixture makes its bytes load-bearing, so `.gitattributes` now marks `testdata` binary: -without it a clone with `core.autocrlf=true` rewrites every line ending and breaks every hash for a -reason the diff does not show. - ---- - -## D1 — Scope the write to the files whose rows changed `[x]` - -**Why.** The substance of "incremental", and the reason a naive per-file write is unsafe. - -**The cascade trap.** `deleteFunctions` deletes a file's functions and `edges.callee_function_id` -is `ON DELETE CASCADE`, so deleting file B's functions **also deletes file A's edge pointing into -B**. If A is not in the write set that edge is never recreated: not orphaned, not unresolved, -absent. `TestWriteGraph_RenameLeavesNoOrphanEdges` passes today only because it rewrites both files. - -So the write set is not "files whose hash changed": - -``` -changed = files whose content hash moved, plus new files, plus files gone from the walk -rewriteEdges = changed ∪ callers(changed), as the new graph sees them AND as the database still does -rewriteFuncs = changed -``` - -**Planned as one set; it had to be two.** The first cut deleted functions for every file it wanted -to rewrite edges for — and deleting a file's functions cascades its *incoming* edges too, so each -caller pulled in its own callers, and so on: a transitive closure that reaches most of a repository -and gives back nothing. Deleting edges explicitly, by caller, stops the closure at one level. A file -that merely calls a changed file keeps its function rows and their ids, which is the whole point. - -The database half of `callers` is one `SELECT DISTINCT` across `edges` → `functions` → `files`, and -it is the half that is easy to forget and impossible to notice: it catches the call that was -*deleted* in this commit, whose only remaining trace is the row. - -**Where.** `internal/db/writer.go`, `cmd/parser/main.go`, `internal/db/writer_test.go`. - -**Do.** - -1. `WriteGraph` gains an `Incremental bool` option. Off ⇒ today's behaviour exactly, so a first - parse and a hand-run `--write` are unchanged. -2. Inside the existing transaction, after `upsertFiles` — which must still run for every file, so - new files get ids — read the stored hashes, diff, and build `writeSet` by the rule above. -3. Pass `writeSet`'s ids to `deleteFunctions`. It already takes a file-id list, so this is the seam - and needs no new SQL shape. -4. Filter `insertFunctions` and `insertEdges` to `writeSet`. -5. Delete `files` rows, and cascade, for paths in the database but absent from the walk. Nothing - prunes today, so a file deleted upstream keeps its functions forever. -6. A `--incremental` flag on the parser, off by default. - -**Done when.** A test that writes a two-file fixture, edits only the callee's file, re-writes -incrementally, and asserts: the caller's edge still exists and still points at a live function; the -caller's function rows kept their ids; and the resulting row counts match a from-scratch full write -of the same tree. The last assertion is the one that catches everything. — -`TestWriteGraph_IncrementalKeepsCrossFileEdges`, `_IncrementalMatchesFullWrite`, -`_IncrementalPrunesDeletedFiles`, all passing. The first was checked by neutering both caller -expansions and watching it fail: the edge count drops to zero, which is the bug it exists to catch. - -**Watch for.** - -- `edges` has **no unique constraint**. Duplicate protection is only the cascade delete — insert - edges for a file whose functions you did not delete and they silently double. -- `edges_callee_consistency` forbids a null callee on a confident edge, so an edge cannot be parked - as "relink later" without downgrading it to `unresolved`. Do not park edges; rewrite the caller. -- A file whose own bytes are unchanged but whose callee moved still needs rewriting. That is what - clauses two and three of the write set are for. - ---- - -## D2 — Queue and worker `[x]` - -**Why.** Removes the ceiling `register.ts` already marks with a `ponytail:` comment. - -**Where.** `apps/api/src/queue/` (new), `apps/api/src/redis.ts`, `apps/api/package.json`. - -**Do.** +| E0 | Pulling a hardcoded assumption out into data, without inventing a plugin framework for it | Every "we only support X" that becomes "we support X, Y, Z" | +| E1 | Partitioning an index instead of filtering its results | Multi-tenant queries, per-shard caches, anything where a missed filter leaks | +| E2–E6 | A parser that silently produces *less* rather than failing | Every codegen, every linter rule, every migration that skips rows | +| E7 | A test that measures the thing under test with the thing under test | Assertions built on the helper they are supposed to be checking | -1. Add `bullmq`, on its own connection with `maxRetriesPerRequest: null`. `redis.ts` already - reserves exactly this and explains why it must not go on the request path. -2. `queue/parse.ts` — the queue, named from `env.QUEUE_NAME` (declared and set in CI, read by - nothing today), plus `enqueueParse(repoId)`. -3. `queue/worker.ts` — a `Worker` with a concurrency cap whose processor is today's `runParser`, - unchanged apart from the incremental flag. -4. **R12 and R13 both fall out of `jobId`.** `jobId = repo:` makes BullMQ ignore a duplicate - while one is waiting or active: no Redlock, no debounce timer. -5. Close the gap that leaves — a push arriving *while* a parse runs is dropped, because the job is - active. Set a `dirty:` key when `add()` collapses onto a live job, and re-enqueue once on - completion if it is set. -6. A separate worker entrypoint, not the Fastify process. A CPU-heavy parse does not belong on the - request event loop. - -**Done when.** Two enqueues for one repository run the parser once. An enqueue during a run causes -exactly one follow-up run. Killing the worker mid-job leaves the repository re-parseable. — -`src/queue/parse.test.ts`, seven tests, skipping as a group when Redis is unreachable the way the Go -integration tests skip without Postgres. - -**What it turned up.** `pnpm install` now fails outright until the project records a decision about -BullMQ's optional native msgpack accelerator. Declined in `pnpm-workspace.yaml`: it compiles at -install time, which is a build script nobody here has read, and BullMQ falls back to the JavaScript -encoder without it. - -**Watch for.** Two Redis connections exist now and `app.ts`'s `onClose` closes one. The worker must -close its own or vitest hangs — `auth/routes.test.ts` has the existing pattern for that class of bug. +E7 is the one worth slowing down for. The first version of the exit test compared +`ResolutionGroup(caller)` with `ResolutionGroup(callee)` — and passed with `ResolutionGroup` +returning a constant, because both sides moved together. Breaking the function on purpose is what +found it. --- -## D3 — Registration enqueues, and the UI stops blocking `[x]` - -**Why.** The route returning only when parsing finishes *is* the ceiling. This also closes the dead -end the 3b gate found: charting a repository left the sidebar on "Nothing charted", because nothing -selected it. - -**Where.** `apps/api/src/routes/repos.ts`, `repos/register.ts`, migration `0004`, -`packages/shared/src/{schema,types}.ts`, `apps/web/src/components/RepoPicker.tsx`, `lib/api.ts`. - -**Do.** - -1. Migration `0004`: `repos.parse_status TEXT NOT NULL DEFAULT 'ready'` (`queued` / `parsing` / - `ready` / `failed`) and `repos.parse_error TEXT`. Default `ready` so existing rows stay correct. - Mirror in Drizzle and in the shared types. -2. The route inserts `queued` and returns **202** with the row. The worker moves it through - `parsing` → `ready` / `failed`, writing `parse_error` from `failureReason`, which already reduces - the parser's zap JSON to one sentence. -3. `GET /api/repos` returns the status; the client polls with a `refetchInterval` that is live only - while some repository is non-terminal. -4. `RepoPicker` closes on 202 and **selects the new repository**. A `failed` row shows its reason in - the existing `break-words` error style. - -**Done when.** Charting returns in well under a second, the dialog closes, the new repository is -selected, and the tree fills in when the parse lands — with no reload. A repository that fails to -clone shows one sentence naming it and why. — `routes/repos.test.ts`, fifteen tests, including one -that asserts the row is written *before* the job is queued and one that asserts the request does not -wait on a parse. - -**What it turned up.** The tree is cached with `staleTime: Infinity`, which was correct while the -only way to re-parse was to register again. A webhook re-parse replaces the graph under a cache with -no reason to suspect it, so the transition into `ready` now invalidates that repository's tree. And -the tree query is disabled while a parse is queued or running: left enabled it returns an empty tree -and the sidebar says the repository has no files, which is a different claim from "not yet". - -**Watch for.** `routes/graph.test.ts` keeps a `GATED` list of every `/api` route; keep it current. -Do not let the poll run forever — `failed` is terminal, and a skeleton that never resolves is worse -than a spinner (`docs/UI_GUIDE.md` §3.3). +## E0 — Generalise the extractor behind a Spec + +- [x] **Why.** `internal/ts` was the only extractor and `files.language` was a constant. Everything + about the per-file loop was already language-agnostic; four things were not. +- [x] **Where.** `services/parser/internal/extract/` (was `internal/ts/`), + `internal/utils/constants.go`, `internal/utils/nodes.go`, `cmd/parser/main.go`. +- [x] **Do.** Rename the package. Add `Spec`: name, extensions, grammar, `.scm`, `ScopeSegment`, + `CalleeReceiver`, `Imports`. Re-express TypeScript as two specs. Delete `utils.Language` and + the callerless `utils.IsSourceFile`. +- [x] **Done when.** Every existing TypeScript test passes unchanged, and `.tsx` reports + `files.language = "tsx"`. +- [x] **Watch for.** `forFile` was a suffix scan; with several languages registered one extension + swallows another. It is an exact `filepath.Ext` lookup now. + +## E1 — Partition resolution by language group + +- [x] **Why.** `byName` and `byPkgName` were repo-wide and keyed on name alone. A call in `main.go` + would have matched a same-named function in `main.py`. +- [x] **Where.** `internal/resolver/resolver.go`, `internal/utils/constants.go`. +- [x] **Do.** Key both maps by resolution group, `package_path` included. Gate imports at index + time on `utils.ResolvesModules`. +- [x] **Done when.** Same-named functions in two languages never link, in either direction. +- [x] **Watch for.** Partition at build time, not filter at lookup: a filter is something a later + code path can forget. + +## E2 — JavaScript and JSX + +- [x] **Why.** `.js` and `.jsx` were skipped entirely, and the TypeScript spec already knew how to + read them. +- [x] **Where.** `queries/javascript.scm`, `internal/extract/javascript.go`, `internal/utils/paths.go`. +- [x] **Do.** Reuse the TypeScript scope, receiver and import functions. Add CommonJS `require()`. + Join the ECMAScript resolution group. +- [x] **Done when.** Calls inside a `.jsx` JSX body are all present, and `require()` binds what it + destructures. +- [x] **Watch for.** `ModuleCandidates` stripped `.js` to find the `.ts` behind it and never + restored it, so a specifier naming a real `.js` file matched nothing. Both are candidates now. + +## E3 — Go + +- [x] **Why.** Extraction only; Go resolves through package clauses and capitalisation. +- [x] **Where.** `queries/go.scm`, `internal/extract/golang.go`. +- [x] **Do.** Methods named after their receiver type. Imports bind a qualifier. +- [x] **Done when.** Calls inside a goroutine literal, a `defer`, and a two-type-argument generic + call are all present. +- [x] **Watch for.** `Map[int](xs)` — one type argument — parses as `type_conversion_expression`, + the same shape as `int(x)`. Not captured, on purpose, and the fixture pins that. + +## E4 — Rust + +- [x] **Why.** Extraction only; Rust resolves through `mod`, `use`, crate paths and traits. +- [x] **Where.** `queries/rust.scm`, `internal/extract/rust.go`, `internal/extract/spec.go`. +- [x] **Do.** Methods named after their `impl` target. `use` declarations into the IR. +- [x] **Done when.** Calls in `impl` blocks, closures, match arms and method chains are present. +- [x] **Watch for.** A macro body is a `token_tree` — nothing inside `println!` is parsed as an + expression. Pinned rather than wished away. Rust also has no quoted specifier, which is why + `Spec.Imports` returns the module as well as the symbols. + +## E5 — Python + +- [x] **Why.** Extraction only; Python resolves through `sys.path` and `__init__.py`. +- [x] **Where.** `queries/python.scm`, `internal/extract/python.go`. +- [x] **Do.** Class nesting in the qualified name. Both import statement forms. +- [x] **Done when.** Calls in decorators, f-string interpolations, comprehension filters and behind + `await` are all present. +- [x] **Watch for.** `decorated_definition` wraps the `function_definition`, so the recorded source + must be the function's and not the decorator's. + +## E6 — Java + +- [x] **Why.** Extraction only; Java resolves through the classpath and picks overloads by type. +- [x] **Where.** `queries/java.scm`, `internal/extract/java.go`. +- [x] **Do.** Every enclosing type in the name; anonymous inner classes and lambdas as anonymous + scopes. +- [x] **Done when.** A call inside `new Runnable(){...}` is present, and a call to a genuinely + overloaded method resolves `unresolved`. +- [x] **Watch for.** This is the first language where `overload_index` does real work — TypeScript's + overload *signatures* were never captured, so the post-pass had nothing to number. + +## E7 — The polyglot fixture and the exit test + +- [x] **Why.** The phase's whole claim is that no edge crosses a language boundary. +- [x] **Where.** `testdata/polyglot/`, `internal/resolver/polyglot_test.go`, + `internal/security/config.go`. +- [x] **Do.** One directory, a file per language, each defining and calling `helper`. `main.go` + calls `python_only`; `main.py` calls `go_only`. Skip-path defaults for the new languages. +- [x] **Done when.** Every language yields functions **and** calls; no edge crosses a boundary; + cross-file is never `exact` outside the ECMAScript family. +- [x] **Watch for.** Two traps, both hit. `helper` alone proves nothing — it exists in seven files, + so ambiguity answers `unresolved` whether the partition works or not. And an assertion built + on `ResolutionGroup` agrees with itself: verified by breaking that function and watching the + test fail. + +## E8 — Language per node, and R34 + +- [x] **Why.** A reader following a call out of their language should see that they have, since that + is also where the resolver stops being able to say anything exact. And R34 has been open since + Phase 4 made it reachable without anyone touching the browser. +- [x] **Where.** `apps/web/src/components/MindMap.tsx`, `FunctionNode.tsx`, `Sidebar.tsx`, + `lib/graph.ts`, `lib/graph-constants.ts`, `lib/highlight.ts`, `store/ui.ts`. +- [x] **Do.** Badge a callee only when its language differs from the file being read. Drop branch + roots the open file no longer has. +- [x] **Done when.** `make test`, `make lint`, `make typecheck`, `make go-vet` clean. +- [x] **Watch for.** A card that grows a badge it was not measured for truncates its own name — + the same bug the "start" badge caused. `functionCardWidth` allows for it. --- -## D4 — The webhook `[x]` - -**Why.** FR-9, and the only reason the queue exists. - -**Where.** `apps/api/src/routes/webhook.ts` (new), `apps/api/src/app.ts`. - -**Do.** - -1. **Raw body first.** Nothing in `apps/api` handles a raw body today — the bytes are parsed and - discarded before a handler sees them, so HMAC has nothing to hash. Use Fastify's own - `addContentTypeParser("application/json", { parseAs: "buffer" })` inside the webhook's own - encapsulated scope. No new dependency, and the encapsulation keeps the raw parser off every other - route. -2. Verify `x-hub-signature-256` with `crypto.timingSafeEqual` against `env.GITHUB_WEBHOOK_SECRET` - — declared, set in CI, read by nothing yet. -3. Replay window: `SET webhook: NX EX `. A delivery id already seen is a - 200 with no work — never a 4xx, or GitHub disables the hook. -4. Per-repo throttle through `@fastify/rate-limit`'s route-level `config.rateLimit` with a - `keyGenerator` on the repository. Already a dependency; the global limiter stays. -5. **Outside the session gate** — GitHub sends no cookie. Register it as a sibling of - `registerAuth`, and add it to the `OPEN` list in `routes/graph.test.ts`. -6. `push` events only. Map the repository by canonical URL through the existing `normaliseRepoUrl`; - an unknown repository is a 200 and no job. - -**Done when.** A signed request enqueues one job; a tampered body is 401; a replayed delivery id is -200 and enqueues nothing; a flood for one repository is throttled. — `routes/webhook.test.ts`, -twelve tests against `buildApp`, no tunnel needed. - -**What it turned up.** Two things, both found by trying to break the tests rather than by reading -the code. - -The throttle keyed on the repository read out of the body, and **never throttled**: the limiter runs -`onRequest`, before the body is parsed, so the key was `undefined` for every request and each one -got its own bucket. It keys on `x-github-hook-id` now — a header, available that early, and stable -per configured webhook. - -The byte-exactness test was worthless as first written. Deleting the raw-body parser and -re-serialising `req.body` still passed, because a compact JSON fixture round-trips through -`JSON.stringify` unchanged. A pretty-printed body does not, and with one the test fails under -exactly that mutation — which is what makes the raw parser provably load-bearing rather than -merely commented as such. - -**Watch for.** The signature covers the **exact bytes**, so anything that re-serialises before the -check breaks it in a way that looks like a wrong secret. Length-check before `timingSafeEqual`; it -throws on mismatched lengths. - ---- - -## D5 — Hardening `[x]` - -**Why.** R30, R31 and R14's disk half have each been deferred once already. - -**Where.** `apps/api/src/auth/routes.ts`, `src/test-helpers.ts`, `src/redis.ts`, -`apps/web/src/components/LoginScreen.tsx`, `services/parser/internal/clone/clone.go`, -`cmd/parser/main.go`, `DEVELOPMENT.md`. - -**Do.** - -1. **Delete `/auth/dev-login` and `DEV_USER`.** Every route test gets its cookie from - `devLogin(app)`, so that helper's body becomes a direct `createSession` call — tests get a - session without the route existing. Drop the sign-in screen's button and fix `DEVELOPMENT.md`, - which currently tells the reader to use it. -2. **R31 — Redis timeouts.** `new Redis(url)` is constructed with no options, so ioredis retries - forever and a request touching Redis never settles: sign-in sits on "Signing in…" with no error, - ever. Add `connectTimeout`, `commandTimeout` and a finite `maxRetriesPerRequest`, and answer - **503**. The queue connection keeps `maxRetriesPerRequest: null` — that one is required. -3. **L39 / R14 disk half.** The clone's `os.MkdirTemp` is never removed, so every parse leaks a - checkout. `Prepare` returns a cleanup func — a no-op for a local path — and `main.go` defers it - after the write. Not a `defer` inside `Prepare`: the directory has to outlive the call. - -**Done when.** `grep -r dev-login apps/src` is empty and every API test still passes. Stopping Redis -makes sign-in fail with a 503 within seconds instead of hanging. Two consecutive parses leave no -`funcatlas-clone-*` directory behind. — three tests in `internal/clone`, and the dev-login test -inverted: it now asserts the route answers 404. - -**What it turned up.** `logger.Fatal` calls `os.Exit`, which skips every deferred call — so the -cleanup would have run on success and never on failure, which is the path that repeats when a -repository cannot be reached. `main` returns an error now. And `make dev` had to learn to start the -worker: without it a registered repository sits at `queued` forever, which reads as a hung interface -rather than a process nobody started. - -**Watch for.** `auth/routes.test.ts`'s `vi.stubEnv` + `vi.resetModules()` + re-import pattern — the -rebuilt module graph carries its own Redis and must be quit or vitest hangs. - ---- - -## D6 — The tests this phase cannot close without `[x]` - -**Why.** Most of these are failure modes that stay invisible until production. - -**Do.** - -1. Incremental write and full write produce **identical database state** for the same tree. The - single most valuable test in the phase. -2. A cross-file edge survives a re-parse of only the callee's file — the D1 cascade trap. -3. A file deleted upstream loses its rows. -4. Webhook: valid, tampered, replayed, flooded. -5. `jobId` collapsing: two enqueues ⇒ one run; enqueue-during-run ⇒ exactly one follow-up. -6. The parser still runs with `--network none`. `docker-compose.yml` already runs it that way and - this phase must not quietly regress it. — `make parser-isolated`, a new target that builds the - image and parses a fixture with no network, read-only rootfs and every capability dropped: - 6 files, 13 functions, 14 edges, all three tiers. - -**Five of the six were already covered** by the chunk that introduced them — D1 for the write, D2 for -the job ids, D4 for the webhook. Only the isolation check needed anything new, and it needed a -Makefile target rather than a test, because what is being asserted is a container's configuration. - ---- - -## D7 — Docs `[x]` - -**Do.** - -1. `PLAN.md` — the Phase 4 result, and **amend the exit test**. "Without a full re-parse" becomes - "without rewriting unchanged rows", with the resolver reason recorded. A phase must not close - against a test its own design decision made unpassable; 3b shipped exactly that contradiction and - it had to be fixed at the gate. -2. `docs/SECURITY.md` — tick L34 and L39, and the three already true (L35, L36, L38). Record L32 and - L37 as deferred **with reasons**, not left blank. -3. `docs/RISKS.md` — close R5, R12, R13, R14, R27, R30, R31. Add: whole-repo resolution forcing a - full re-parse; content-hash change detection; the dropped-push-during-parse gap and how `dirty` - closes it. -4. `docs/DATA_MODEL.md` says scoping the rewrite to changed files is Phase 4 work — make it describe - what shipped. `docs/ARCHITECTURE.md` says the parser worker talks only through the queue and - Postgres; that is now wrong and must say so. -5. `CLAUDE.md` status and known gaps. -6. **R34 stays open** and moves to Phase 5: `store/ui.ts` restores file and branch ids through - `zustand/persist`, and this is the first phase that can delete the rows behind them. - ---- - -## D8 — Exit gate `[x]` - -**Done when — the phase exit test.** Against a real repository: - -- register a repository; the request returns immediately and the row appears `queued`; -- the worker parses it and the tree fills in without a reload; -- push a commit to a fixture repository; the graph updates, and only the changed files' rows are - rewritten — check `functions.id` stability on untouched files; -- a replayed delivery is rejected, and a flood is throttled; -- the parser still runs with no network egress; -- `make test`, `make lint`, `make typecheck` and `make go-vet` all clean. - -**R5** — GitHub cannot reach localhost, so the live half needs `smee.io`. The deliveries below were -signed and posted locally with `curl`, which exercises everything except GitHub's own transport. - -### The run — 2026-08-21 - -Against `pmndrs/zustand` (fresh), `honojs/hono` (re-parse), and a purpose-built three-file fixture. - -**Registration returns before the parse.** `POST /api/repos` for `honojs/hono`: **202 in 26 ms**, -`parseStatus: "queued"`. The same repository held the request for roughly thirty seconds at the 3b -gate. The dialog closes immediately and the new repository is selected — the dead end 3b found, where -charting left the sidebar on "Nothing charted", is gone. - -**The tree fills in.** `pmndrs/zustand` went `queued → parsing → ready` and landed at 34 files and -187 functions, `.tsx` files carrying their counts, which is the JSX grammar path still working. - -**Unchanged rows are not rewritten.** A full webhook-driven re-parse of hono left the fingerprint of -all 1,460 `(id, qualified_name)` pairs **byte-identical** (`200defe0…`), and the confidence -distribution unchanged at 1,002 exact / 149 name_match / 4,141 unresolved. - -**An edited file rewrites itself and nothing else.** Through the real binary, on a fixture where -`main.ts` calls into `helpers.ts`: - -| file | before | after | | -|---|---|---|---| -| `src/helpers.ts` — edited | 3444, 3445 | **3448, 3449** | rewritten | -| `src/main.ts` — calls into it | 3446 | **3446** | id kept | -| `src/other.ts` — unrelated | 3447 | **3447** | id kept | - -and `main.run → helpers.helper` survived as `exact`, repointed at the new id. That is the cascade -trap not biting, end to end rather than in a unit test. - -**The webhook.** Valid push → 200, one job. Replayed delivery id → 200 `{replay:true}`, no job. -Tampered body → **401**. Unregistered repository → 200, ignored. Non-push event → 200, ignored. -Flood of 45 deliveries on one hook id → **30 through, 15 × 429**, exactly `WEBHOOK_RATE_MAX`; all of -them collapsed onto a single parse. - -**No network egress.** `make parser-isolated` — the image, `network_mode: none`, read-only rootfs, -every capability dropped: 6 files, 13 functions, 14 edges, all three tiers. - -**No clones left behind.** Every parse in this run left the temp directory empty. The one directory -present predated the fix and was itself empty. +## The exit gate -`make test` (127 api / 143 web / Go), `make lint`, `make typecheck`, `make go-vet` — all clean. +`make go-run REPO=./services/parser/testdata/polyglot` — 7 files, 7 languages, every one yielding +functions and calls, no edge crossing a boundary. -### What could not be checked here +`make test` (127 api / 146 web / Go, Postgres up), `make lint`, `make typecheck`, `make go-vet` — +all clean. -The status **poll** could not be watched live: the automated tab reports `visibilityState: hidden`, -and TanStack pauses `refetchInterval` while a document is hidden — by design, and it catches up on -refocus. The transitions were read from the API instead. Worth a human's eye on a real tab before -this is called finished. +Still to do by hand at the gate: a real polyglot public repository charted end to end in real +Chrome, checking the tree language labels, the per-node badge, and Shiki highlighting for each +language. diff --git a/apps/web/src/components/FunctionNode.tsx b/apps/web/src/components/FunctionNode.tsx index 98eeb36..e847c8e 100644 --- a/apps/web/src/components/FunctionNode.tsx +++ b/apps/web/src/components/FunctionNode.tsx @@ -128,6 +128,19 @@ function FunctionCard({ data }: NodeProps) { ) : null} + {/* Only when it differs from the file being read -- see + `foreignLanguage`. This is where a reader finds out that the call + they followed left the language, which is also where the resolver + stops being able to say anything exact. */} + {data.foreignLanguage !== null ? ( + + {data.foreignLanguage} + + ) : null} + {/* Whether clicking does anything. Without this a leaf looks exactly like an unopened function, and clicking it reads as a broken canvas rather than as a function that calls nothing. */} diff --git a/apps/web/src/components/MindMap.tsx b/apps/web/src/components/MindMap.tsx index f75fa39..873baf3 100644 --- a/apps/web/src/components/MindMap.tsx +++ b/apps/web/src/components/MindMap.tsx @@ -12,7 +12,13 @@ import { ApiError } from "../lib/api"; import { useFileFunctions } from "../lib/files"; import { useExpansions, useSources } from "../lib/functions"; import { useRepoTree } from "../lib/repos"; -import { buildGraph, fileCardPosition, fileCardSize, type GraphNodeData } from "../lib/graph"; +import { + buildGraph, + fileCardPosition, + fileCardSize, + functionCardWidth, + type GraphNodeData, +} from "../lib/graph"; import { FILE_CARD_NODE, FUNCTION_NODE, GHOST_NODE, NODE_HEIGHT, NODE_WIDTH } from "../lib/graph-constants"; import { useTheme } from "../lib/theme"; import { useAnimatedNodes } from "../lib/useAnimatedNodes"; @@ -58,6 +64,7 @@ export function MindMap() { const codeFunctionIds = useUiStore((state) => state.codeFunctionIds); const fileListExpanded = useUiStore((state) => state.fileListExpanded); const fullSourceIds = useUiStore((state) => state.fullSourceIds); + const dropMissingRoots = useUiStore((state) => state.dropMissingRoots); const selectedFileId = useUiStore((state) => state.selectedFileId); const selectedRepoId = useUiStore((state) => state.selectedRepoId); @@ -112,6 +119,62 @@ export function MindMap() { () => tree.data?.files.find((candidate) => candidate.id === selectedFileId), [tree.data, selectedFileId], ); + + /** + * A callee in another language wears its language; one in the same language + * as the file being read does not. + * + * The tree already has every file and its language, so this costs no request. + * It is done here rather than in `buildGraph` because a language belongs to a + * file and buildGraph is given functions -- and because only the canvas knows + * which file the reader opened, which is what "another language" is measured + * against. + */ + const languages = useMemo(() => { + const out = new Map(); + for (const candidate of tree.data?.files ?? []) { + out.set(candidate.id, candidate.language); + } + return out; + }, [tree.data]); + + const home = file?.language; + + const localised = useMemo(() => { + if (home === undefined) return graph; + return { + ...graph, + nodes: graph.nodes.map((node) => { + const language = + node.data.fileId === null ? undefined : languages.get(node.data.fileId); + const foreign = + language !== undefined && language !== home ? language : null; + // The node object is kept when nothing changed, so the common case -- + // a single-language repository -- allocates nothing per render. + if (foreign === node.data.foreignLanguage) return node; + + // Re-measured, because the badge takes width the card was not sized + // for and the name is what gets truncated instead. A card showing its + // source is already sized to the source, so it is left alone. + const size = node.data.showCode + ? node.data.size + : { + ...node.data.size, + width: functionCardWidth( + node.data.label, + node.data.qualifiedName, + node.data.depth === 0, + foreign, + ), + }; + return { + ...node, + data: { ...node.data, foreignLanguage: foreign, size }, + }; + }), + }; + }, [graph, home, languages]); + // The same query the card itself runs, so the canvas can size the node to // its contents without a second request. const fileFunctions = useFileFunctions(file?.id ?? null); @@ -120,6 +183,20 @@ export function MindMap() { const names = (fileFunctions.data?.functions ?? []).map((fn) => fn.name).join("|"); const pendingFunctions = fileFunctions.isPending; + /** + * A restored branch whose function is gone gets forgotten (R34). + * + * A re-parse reinserts a changed file's functions under new ids, and since + * Phase 4 a webhook does that with nobody touching the browser -- so the ids + * `zustand/persist` brings back can point at rows that no longer exist. This + * is the first moment there is anything to check them against. + */ + const ids = (fileFunctions.data?.functions ?? []).map((fn) => fn.id).join(","); + useEffect(() => { + if (ids === "") return; + dropMissingRoots(ids.split(",").map(Number)); + }, [ids, dropMissingRoots]); + const fileData = useMemo(() => { if (file === undefined) return null; const size = fileCardSize( @@ -139,16 +216,16 @@ export function MindMap() { const withFile = useMemo(() => { if (fileData === null) { - return graph; + return localised; } const fileNodeId = `file-${fileData.fileId}`; // Every branch root, not one: the reader can open several functions out of // one file and explore each independently. - const rootNodes = graph.nodes.filter((node) => node.data.depth === 0); + const rootNodes = localised.nodes.filter((node) => node.data.depth === 0); return { - ...graph, + ...localised, nodes: [ { id: fileNodeId, @@ -161,7 +238,7 @@ export function MindMap() { data: fileData, // A file is not a call, so it carries none of the graph node's data. } as unknown as Node, - ...graph.nodes, + ...localised.nodes, ], edges: [ ...rootNodes.map((rootNode) => ({ @@ -173,10 +250,10 @@ export function MindMap() { type: "smoothstep", style: { stroke: palette.surface.border, strokeWidth: 1.5 }, })), - ...graph.edges, + ...localised.edges, ], }; - }, [graph, fileData, palette.surface.border]); + }, [localised, fileData, palette.surface.border]); /** * A new file frames itself. diff --git a/apps/web/src/components/Sidebar.tsx b/apps/web/src/components/Sidebar.tsx index b50866a..3349c4d 100644 --- a/apps/web/src/components/Sidebar.tsx +++ b/apps/web/src/components/Sidebar.tsx @@ -187,8 +187,8 @@ function TreeBody({ No files to chart - The parser found nothing it can read here. TypeScript is the only - language it extracts today. + The parser found nothing it can read here. It reads TypeScript, + JavaScript, Go, Rust, Python and Java. diff --git a/apps/web/src/lib/graph-constants.ts b/apps/web/src/lib/graph-constants.ts index b9422fd..59085de 100644 --- a/apps/web/src/lib/graph-constants.ts +++ b/apps/web/src/lib/graph-constants.ts @@ -48,6 +48,9 @@ export const LABEL_CHAR = 7.2; export const SUBLABEL_CHAR = 5.5; export const CARD_CHROME = 118; export const ROOT_BADGE = 50; +/** A language badge's chrome, on top of its text. Measured like ROOT_BADGE: + * a card that grows a badge it was not measured for truncates its own name. */ +export const LANGUAGE_BADGE = 20; // --- Code cards ----------------------------------------------------------- diff --git a/apps/web/src/lib/graph.test.ts b/apps/web/src/lib/graph.test.ts index 33162fe..d15622c 100644 --- a/apps/web/src/lib/graph.test.ts +++ b/apps/web/src/lib/graph.test.ts @@ -731,6 +731,7 @@ describe("sameNodeData", () => { depth: 0, isRoot: true, callLines: [], + foreignLanguage: null, expanded: true, isLeaf: false, showCode: false, diff --git a/apps/web/src/lib/graph.ts b/apps/web/src/lib/graph.ts index e60a0fa..9d872ab 100644 --- a/apps/web/src/lib/graph.ts +++ b/apps/web/src/lib/graph.ts @@ -23,6 +23,7 @@ import { GUTTER, HEADER, LABEL_CHAR, + LANGUAGE_BADGE, LINE_HEIGHT, LIST_PADDING, MORE_ROW, @@ -79,13 +80,19 @@ export function functionCardWidth( label: string, qualifiedName: string | null, isRoot = false, + /** Shown only on a callee in another language; see `foreignLanguage`. */ + foreignLanguage: string | null = null, ): number { const sublabel = qualifiedName !== null && qualifiedName !== label ? qualifiedName.length : 0; const text = Math.max(label.length * LABEL_CHAR, sublabel * SUBLABEL_CHAR); - const chrome = CARD_CHROME + (isRoot ? ROOT_BADGE : 0); + const badge = + foreignLanguage === null + ? 0 + : foreignLanguage.length * SUBLABEL_CHAR + LANGUAGE_BADGE; + const chrome = CARD_CHROME + (isRoot ? ROOT_BADGE : 0) + badge; // No ceiling. A name is what the reader navigates by, and a card that caps // its width just moves the truncation somewhere less obvious. @@ -235,6 +242,20 @@ export interface GraphNodeData { * size, which is what the layout spaces around. */ showCode: boolean; + /** + * The node's language, but only when it differs from the file being read. + * + * Null on a node in the same language, and on a ghost -- which has no file + * and so no language. Labelling every node "typescript" on a TypeScript map + * is noise on a canvas whose three edge styles are already only a dash apart; + * what is worth a badge is the boundary, because the resolver will not cross + * one and the reader should be able to see why. + * + * Filled by `MindMap`, not here: a language belongs to a file, and buildGraph + * is given functions. + */ + foreignLanguage: string | null; + /** Whether that source is every line or the first `CODE_PREVIEW_LINES` of * them. Nothing scrolls here; the card grows instead. */ showAllSource: boolean; @@ -580,6 +601,7 @@ function toFunctionNode( position: { x: 0, y: 0 }, data: { kind: "function", + foreignLanguage: null, label: fn.name, functionId: fn.id, qualifiedName: fn.qualifiedName, @@ -610,6 +632,7 @@ function toGhostNode(ghost: Ghost, depth: number): Node { position: { x: 0, y: 0 }, data: { kind: "ghost", + foreignLanguage: null, label: ghost.name, functionId: null, qualifiedName: null, diff --git a/apps/web/src/lib/highlight.ts b/apps/web/src/lib/highlight.ts index a4657c7..ec530b7 100644 --- a/apps/web/src/lib/highlight.ts +++ b/apps/web/src/lib/highlight.ts @@ -19,6 +19,7 @@ const GRAMMARS: Record = { go: "go", rust: "rust", python: "python", + java: "java", }; /** diff --git a/apps/web/src/store/ui.test.ts b/apps/web/src/store/ui.test.ts index 048f718..2b05603 100644 --- a/apps/web/src/store/ui.test.ts +++ b/apps/web/src/store/ui.test.ts @@ -176,3 +176,49 @@ describe("selection", () => { expect(state().selectedRepoId).toBe(1); }); }); + +describe("dropMissingRoots (R34)", () => { + it("forgets a restored branch whose function no longer exists", () => { + const store = useUiStore.getState(); + store.selectFile(3); + store.toggleRoot(10); + store.toggleRoot(20); + store.toggleFunction(99); // a callee, in some other file + store.toggleCode(20); + + // A re-parse reinserted this file's functions: 20 is gone, 10 survived. + useUiStore.getState().dropMissingRoots([10, 11]); + + const after = useUiStore.getState(); + expect(after.rootFunctionIds).toEqual([10]); + expect(after.expandedFunctionIds).not.toContain(20); + expect(after.codeFunctionIds).not.toContain(20); + // A callee in another file is not this list's to judge; its own query + // answers 404 and that branch is simply not drawn. + expect(after.expandedFunctionIds).toContain(99); + }); + + it("clears the selection only when it was one of the dropped roots", () => { + const store = useUiStore.getState(); + store.selectFile(3); + store.toggleRoot(10); + store.toggleRoot(20); + + useUiStore.getState().dropMissingRoots([10]); + expect(useUiStore.getState().selectedFunctionId).toBeNull(); + + useUiStore.getState().toggleRoot(10); + useUiStore.getState().dropMissingRoots([10]); + expect(useUiStore.getState().selectedFunctionId).toBe(10); + }); + + it("changes nothing when every root still exists", () => { + const store = useUiStore.getState(); + store.selectFile(3); + store.toggleRoot(10); + const before = useUiStore.getState().rootFunctionIds; + + useUiStore.getState().dropMissingRoots([10, 11, 12]); + expect(useUiStore.getState().rootFunctionIds).toBe(before); + }); +}); diff --git a/apps/web/src/store/ui.ts b/apps/web/src/store/ui.ts index e4a03c6..faa77cc 100644 --- a/apps/web/src/store/ui.ts +++ b/apps/web/src/store/ui.ts @@ -82,6 +82,17 @@ interface UiState { toggleFullSource: (id: number) => void; /** Clears the map. Used when the file or repository changes. */ selectFunction: (id: number | null) => void; + + /** + * Forgets branch roots the open file no longer has (R34). + * + * A re-parse reinserts a changed file's functions under new ids while the + * file row keeps its own, so a restored branch points at rows that are gone + * -- and since Phase 4 a webhook does that without anyone touching the + * browser. `existingIds` is the open file's current function list, which is + * authoritative for roots because a root is only ever opened from its card. + */ + dropMissingRoots: (existingIds: number[]) => void; /** Signing out, where nothing selected should survive the next session. */ clearSelection: () => void; @@ -209,6 +220,32 @@ export const useUiStore = create()( : [...state.fullSourceIds, id], })), + dropMissingRoots: (existingIds) => + set((state) => { + const exists = new Set(existingIds); + const gone = state.rootFunctionIds.filter((id) => !exists.has(id)); + if (gone.length === 0) return {}; + + // Everything the dropped roots left behind goes with them. A root's + // own id in expandedFunctionIds would otherwise keep asking for a + // traversal from a function that no longer exists. + const kept = (ids: number[]) => + ids.filter((id) => exists.has(id) || !gone.includes(id)); + + return { + rootFunctionIds: state.rootFunctionIds.filter((id) => exists.has(id)), + expandedFunctionIds: kept(state.expandedFunctionIds), + collapsedFunctionIds: kept(state.collapsedFunctionIds), + codeFunctionIds: kept(state.codeFunctionIds), + fullSourceIds: kept(state.fullSourceIds), + selectedFunctionId: + state.selectedFunctionId !== null && + gone.includes(state.selectedFunctionId) + ? null + : state.selectedFunctionId, + }; + }), + clearSelection: () => set({ ...empty() }), }), { @@ -216,10 +253,11 @@ export const useUiStore = create()( // Everything the reader built, except the palette -- reloading into an // open search box is a state they never asked for. // - // Restored ids are not re-validated. A re-parse reinserts functions under - // new ids (Phase 4), so a restored branch can point at rows that no - // longer exist; the tree and graph queries answer 404 and the surfaces - // show their empty state, which is the honest outcome and not a crash. + // Restored ids cannot be checked here -- nothing is loaded yet at + // rehydrate. `dropMissingRoots` does it once the open file's function + // list arrives, which is the first moment there is anything to check + // against. An expanded id deeper in the map, in some other file, is left + // to its own query: it 404s and that branch simply is not drawn. partialize: (state) => ({ selectedRepoId: state.selectedRepoId, selectedFileId: state.selectedFileId, diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md index 57b53ca..f67e878 100644 --- a/docs/DATA_MODEL.md +++ b/docs/DATA_MODEL.md @@ -25,7 +25,8 @@ CREATE TABLE files ( id SERIAL PRIMARY KEY, repo_id INTEGER NOT NULL REFERENCES repos(id) ON DELETE CASCADE, path TEXT NOT NULL, -- repo-relative, e.g. src/services/auth.ts - language TEXT NOT NULL, -- 'typescript' (only language in the MVP) + language TEXT NOT NULL, -- one name per grammar: typescript, tsx, + -- javascript, jsx, go, rust, python, java content_hash TEXT, -- sha256 of the bytes; null = "changed" (migration 0003) UNIQUE (repo_id, path) -- constraint name: files_repo_id_path_key ); diff --git a/docs/PARSING_STRATEGY.md b/docs/PARSING_STRATEGY.md index a068d52..d4950ae 100644 --- a/docs/PARSING_STRATEGY.md +++ b/docs/PARSING_STRATEGY.md @@ -160,7 +160,32 @@ Every one of these lands as `unresolved` rather than a wrong answer — see the - **Re-exports / barrel files** — a symbol may pass through several intermediate files before its real definition. The re-export edges are recorded in the IR (`KindReExport`, carrying the original name and no local binding) but not yet followed. - **Overloading / shadowing** — multiple definitions sharing a name are not disambiguated by name and scope alone. -- **Cross-language repos** — name-only matching could wire a call in one language to an unrelated same-named function in another. Not reachable today: only `.ts` and `.tsx` are walked, and `.js`, `.jsx`, and every non-TypeScript extension are skipped entirely. +- **Cross-language repos** — closed in Phase 5. The resolver's two repo-wide lookups are keyed by **resolution group**, so a call in `main.go` cannot reach a same-named function in `main.py`. `.ts`, `.tsx`, `.js`, `.jsx`, `.mjs` and `.cjs` share one group and resolve across files normally; `go`, `rust`, `python` and `java` are each their own, and cross-file calls in them are `name_match` or `unresolved`, never `exact`. + +## Per-language extraction limits + +Every language below is **extraction only**: functions, call sites and imports in the IR, plus +same-file resolution, which is language-agnostic. Full per-language resolution is deliberately cut +(`PLAN.md` Phase 5). + +Each of these was found by dumping the tree rather than assumed, and each is pinned by an assertion +in that language's fixture — because the failure mode is a parser that quietly produces *less*, +which reads as a working parse. + +| Language | What is not captured | Why | +|---|---|---| +| **Go** | a generic call with **one** type argument — `Map[int](xs)` | It parses as `type_conversion_expression`, the same shape as `int(x)`. Capturing it would invent a call for every conversion in the repository. Two or more type arguments are unambiguous and *are* captured. | +| **Go** | a call through an index — `handlers[0]()` | The callee has no name to record. | +| **Rust** | anything inside a macro — `println!("{}", helper())` | A macro body is a `token_tree`; tree-sitter does not parse expressions in it, so `helper` is a bare identifier beside a token tree. Matching identifiers there would invent a call for every name mentioned in every macro. | +| **Java** | which overload a call meant | Choosing between `sync()` and `sync(int)` needs argument types. `uniqueQualified` answers "many, therefore none" — see the refusal table. | +| **Python** | nothing structural | Decorators, f-string interpolations, comprehension clauses and `await` are all parsed, so calls inside them are real calls. `decorated_definition` wraps the definition rather than replacing it, so the recorded source is the function's. | +| **JavaScript** | nothing structural | One grammar handles JSX in any file, unlike TypeScript's `.ts`/`.tsx` split. `require()` is captured by over-matching in the `.scm` and discarding non-`require` calls in Go. | + +**Imports are recorded for every language but only followed for the ECMAScript family.** A Go path +names a module, a Rust `use` names a crate path, a Python import names something on `sys.path`, a +Java import names a classpath entry — none of which this parser models. Entering the import rule +for them would answer `unresolved` for a module it cannot reach, throwing away the `name_match` the +package and repo-wide rules still have. See `utils.ResolvesModules`. ## v2: LSP-based resolution diff --git a/docs/RISKS.md b/docs/RISKS.md index fc6f22c..6f4a45c 100644 --- a/docs/RISKS.md +++ b/docs/RISKS.md @@ -57,8 +57,10 @@ Nothing outstanding -- R19 through R22 and R26 through R29 all closed; see Decid | | Risk | Notes | |---|---|---| -| **R34** | **Canvas state is persisted but never re-validated.** `store/ui.ts` restores the repository, the open file and every open branch through `zustand/persist`. Phase 4's re-parse reinserts functions under new ids, so a restored branch can point at rows that are gone. | Phase 4 made this reachable rather than theoretical: a webhook re-parse changes ids without anyone touching the browser. Scoping the write narrowed it -- an unedited file keeps its function ids now -- but an edited one does not. The fix is to drop unknown ids on restore rather than to stop persisting. | +| **R34** | **Canvas state is persisted but never re-validated.** `store/ui.ts` restores the repository, the open file and every open branch through `zustand/persist`. Phase 4's re-parse reinserts functions under new ids, so a restored branch can point at rows that are gone. | **Closed in Phase 5.** Nothing can be checked at rehydrate -- nothing is loaded yet -- so `dropMissingRoots` runs when the open file's function list arrives, which is the first moment there is anything to check against. That list is authoritative for branch roots, because a root is only ever opened from its card. An expanded id deeper in the map, in another file, is left to its own query: it 404s and that branch is not drawn. Judging it here would mean fetching every file's functions to answer what the query already answers. | | **R35** | **Resolution is whole-repo, so an incremental re-parse still re-parses everything.** The write is scoped; the clone, the extract and the resolve are not. | Accepted in Phase 4 and recorded rather than hidden, because `PRD.md` FR-9 says otherwise. Making it real means hydrating a repo-wide symbol table out of Postgres for the unchanged files, in the shape `newIndex` builds -- and keeping the two in step, or confidence drifts silently. Worth it when parse time is measured to be the problem, not before. | +| **R36** | **A test can measure the boundary with the function that defines it.** The Phase 5 exit test first compared `ResolutionGroup(caller)` with `ResolutionGroup(callee)` -- and passed with `ResolutionGroup` returning a constant, because both sides moved together. | Found by breaking the function on purpose and watching nothing fail. The assertions now compare language names from a literal set written in the test file. The same shape is worth suspecting anywhere a test derives its expectation from the code under test -- and the fixture had a second version of the problem: every file defined `helper`, so ambiguity answered `unresolved` whether the partition worked or not. It now also carries names defined in exactly one language. | +| **R37** | **Six languages share one resolver, and only TypeScript's rules are modelled.** Same-file resolution is language-agnostic and stays `exact`; everything else outside the ECMAScript family is `name_match` or `unresolved`. | Deliberate, and the reason Phase 5 was scoped to extraction (`PLAN.md`). The risk is drift: a later change that widens a rule for one language widens it for all six, because there is one `resolve`. The guard is `TestPolyglot_CrossFileIsNeverExactOutsideECMAScript`, which fails the moment any non-ECMAScript language starts answering confidently across files. | --- diff --git a/docs/UI_GUIDE.md b/docs/UI_GUIDE.md index 64a27f9..28d744b 100644 --- a/docs/UI_GUIDE.md +++ b/docs/UI_GUIDE.md @@ -167,10 +167,10 @@ A single centred card: wordmark, one line saying what the tool does, the confide The legend earns its place here where a background texture did not: it is the notation the canvas is about to use, and reading it once beats decoding it later. -**The marketing landing page gets its own PR, opened after the Phase 3b gate.** It is a second full -surface and the 3b exit test does not touch any of it, so it does not belong in the same review. It -was requested during 3b and is no longer "someday" — it is the next branch after the gate, and it -follows §1 like everything else. +**The marketing landing page gets its own PR, still unopened.** It is a second full surface that no +phase's exit test touches, so it has never belonged in a phase review. It was requested during 3b +and named the next branch after that gate; Phases 4 and 5 went first, so it is now overdue rather +than upcoming. It follows §1 like everything else. The landing page is the one surface that takes the maximal spatial treatment: section padding at `py-24` and above, nested double-bezel cards, and a hero that is a live drawing graph rather than a @@ -260,7 +260,7 @@ reachable tab order through tree, palette and canvas, and `prefers-reduced-motio **No longer deferred.** The light theme shipped in Phase 3b alongside the dark one — both palettes are in §1.1 and both are enforced by `confidence.test.ts`. The marketing landing page moved from -"post-MVP" to "the branch after the 3b gate" (§3.1). +"post-MVP" to a branch of its own, which has not been opened yet (§3.1). ## 7. What this must not look like diff --git a/services/parser/cmd/parser/main.go b/services/parser/cmd/parser/main.go index 69b3ef3..87913fb 100644 --- a/services/parser/cmd/parser/main.go +++ b/services/parser/cmd/parser/main.go @@ -13,13 +13,20 @@ import ( "github.com/ARCoder181105/funcatlas/parser/internal/clone" "github.com/ARCoder181105/funcatlas/parser/internal/db" + "github.com/ARCoder181105/funcatlas/parser/internal/extract" "github.com/ARCoder181105/funcatlas/parser/internal/ir" "github.com/ARCoder181105/funcatlas/parser/internal/resolver" "github.com/ARCoder181105/funcatlas/parser/internal/security" - "github.com/ARCoder181105/funcatlas/parser/internal/ts" "github.com/ARCoder181105/funcatlas/parser/internal/utils" ) +// Output formats accepted by --format. Package-local: nothing outside this +// command reads them. +const ( + formatJSON = "json" + formatSummary = "summary" +) + func main() { _ = godotenv.Load() @@ -41,7 +48,7 @@ func run(logger *zap.Logger) error { repo := flag.String("repo", "", "local path or git URL to parse") out := flag.String("out", "out.json", "output file path, or - for stdout") - format := flag.String("format", "json", "output format: json|summary") + format := flag.String("format", formatJSON, "output format: json|summary") write := flag.Bool("write", false, "write the graph to Postgres (needs DATABASE_URL)") repoURL := flag.String("repo-url", "", "repo identity for --write; defaults to --repo") branch := flag.String("branch", "", "default branch recorded on the repo row; detected when empty") @@ -62,7 +69,7 @@ func run(logger *zap.Logger) error { return fmt.Errorf("clone/prepare failed: %w", err) } - graph, err := ts.Extract(logger, root, cfg) + graph, err := extract.Extract(logger, root, cfg) if err != nil { return fmt.Errorf("parse failed: %w", err) } @@ -104,10 +111,10 @@ func run(logger *zap.Logger) error { // report prints the graph. Works with no database configured, so --format json // stays usable for inspection. func report(format, out string, graph ir.Graph, edges []ir.Edge) error { - if format == "summary" { + if format == formatSummary { fmt.Printf("files: %d\nfunctions: %d\ncalls: %d\nimports: %d\nedges: %d\n", len(graph.Files), len(graph.Functions), len(graph.Calls), len(graph.Imports), len(edges)) - for _, c := range []string{"exact", "name_match", "unresolved"} { + for _, c := range utils.ConfidenceTiers { fmt.Printf(" %-11s %d\n", c, countConfidence(edges, c)) } return nil diff --git a/services/parser/go.mod b/services/parser/go.mod index 3e48040..39ac8a4 100644 --- a/services/parser/go.mod +++ b/services/parser/go.mod @@ -7,6 +7,11 @@ require ( github.com/joho/godotenv v1.5.1 github.com/stretchr/testify v1.11.1 github.com/tree-sitter/go-tree-sitter v0.25.0 + github.com/tree-sitter/tree-sitter-go v0.25.0 + github.com/tree-sitter/tree-sitter-java v0.23.5 + github.com/tree-sitter/tree-sitter-javascript v0.25.0 + github.com/tree-sitter/tree-sitter-python v0.25.0 + github.com/tree-sitter/tree-sitter-rust v0.24.2 github.com/tree-sitter/tree-sitter-typescript v0.23.2 go.uber.org/zap v1.28.0 ) diff --git a/services/parser/go.sum b/services/parser/go.sum index 1f95ba8..8799b5d 100644 --- a/services/parser/go.sum +++ b/services/parser/go.sum @@ -35,24 +35,24 @@ github.com/tree-sitter/tree-sitter-cpp v0.23.4 h1:LaWZsiqQKvR65yHgKmnaqA+uz6tlDJ github.com/tree-sitter/tree-sitter-cpp v0.23.4/go.mod h1:doqNW64BriC7WBCQ1klf0KmJpdEvfxyXtoEybnBo6v8= github.com/tree-sitter/tree-sitter-embedded-template v0.23.2 h1:nFkkH6Sbe56EXLmZBqHHcamTpmz3TId97I16EnGy4rg= github.com/tree-sitter/tree-sitter-embedded-template v0.23.2/go.mod h1:HNPOhN0qF3hWluYLdxWs5WbzP/iE4aaRVPMsdxuzIaQ= -github.com/tree-sitter/tree-sitter-go v0.23.4 h1:yt5KMGnTHS+86pJmLIAZMWxukr8W7Ae1STPvQUuNROA= -github.com/tree-sitter/tree-sitter-go v0.23.4/go.mod h1:Jrx8QqYN0v7npv1fJRH1AznddllYiCMUChtVjxPK040= +github.com/tree-sitter/tree-sitter-go v0.25.0 h1:cEB0Q3LHgZtS+ECHx9wcP7AwzoOddJFQCVmytX42cVU= +github.com/tree-sitter/tree-sitter-go v0.25.0/go.mod h1:Jrx8QqYN0v7npv1fJRH1AznddllYiCMUChtVjxPK040= github.com/tree-sitter/tree-sitter-html v0.23.2 h1:1UYDV+Yd05GGRhVnTcbP58GkKLSHHZwVaN+lBZV11Lc= github.com/tree-sitter/tree-sitter-html v0.23.2/go.mod h1:gpUv/dG3Xl/eebqgeYeFMt+JLOY9cgFinb/Nw08a9og= github.com/tree-sitter/tree-sitter-java v0.23.5 h1:J9YeMGMwXYlKSP3K4Us8CitC6hjtMjqpeOf2GGo6tig= github.com/tree-sitter/tree-sitter-java v0.23.5/go.mod h1:NRKlI8+EznxA7t1Yt3xtraPk1Wzqh3GAIC46wxvc320= -github.com/tree-sitter/tree-sitter-javascript v0.23.1 h1:1fWupaRC0ArlHJ/QJzsfQ3Ibyopw7ZfQK4xXc40Zveo= -github.com/tree-sitter/tree-sitter-javascript v0.23.1/go.mod h1:lmGD1EJdCA+v0S1u2fFgepMg/opzSg/4pgFym2FPGAs= +github.com/tree-sitter/tree-sitter-javascript v0.25.0 h1:ZkWETb66/w8cc13yhfnNuHOLDQWl3BnKlH6f9AdR88c= +github.com/tree-sitter/tree-sitter-javascript v0.25.0/go.mod h1:lmGD1EJdCA+v0S1u2fFgepMg/opzSg/4pgFym2FPGAs= github.com/tree-sitter/tree-sitter-json v0.24.8 h1:tV5rMkihgtiOe14a9LHfDY5kzTl5GNUYe6carZBn0fQ= github.com/tree-sitter/tree-sitter-json v0.24.8/go.mod h1:F351KK0KGvCaYbZ5zxwx/gWWvZhIDl0eMtn+1r+gQbo= github.com/tree-sitter/tree-sitter-php v0.23.11 h1:iHewsLNDmznh8kgGyfWfujsZxIz1YGbSd2ZTEM0ZiP8= github.com/tree-sitter/tree-sitter-php v0.23.11/go.mod h1:T/kbfi+UcCywQfUNAJnGTN/fMSUjnwPXA8k4yoIks74= -github.com/tree-sitter/tree-sitter-python v0.23.6 h1:qHnWFR5WhtMQpxBZRwiaU5Hk/29vGju6CVtmvu5Haas= -github.com/tree-sitter/tree-sitter-python v0.23.6/go.mod h1:cpdthSy/Yoa28aJFBscFHlGiU+cnSiSh1kuDVtI8YeM= +github.com/tree-sitter/tree-sitter-python v0.25.0 h1:O6XD9v8U1LOcRc3cNj9nM7XufrtEBezE6VrpRrHZDf0= +github.com/tree-sitter/tree-sitter-python v0.25.0/go.mod h1:cpdthSy/Yoa28aJFBscFHlGiU+cnSiSh1kuDVtI8YeM= github.com/tree-sitter/tree-sitter-ruby v0.23.1 h1:T/NKHUA+iVbHM440hFx+lzVOzS4dV6z8Qw8ai+72bYo= github.com/tree-sitter/tree-sitter-ruby v0.23.1/go.mod h1:kUS4kCCQloFcdX6sdpr8p6r2rogbM6ZjTox5ZOQy8cA= -github.com/tree-sitter/tree-sitter-rust v0.23.2 h1:6AtoooCW5GqNrRpfnvl0iUhxTAZEovEmLKDbyHlfw90= -github.com/tree-sitter/tree-sitter-rust v0.23.2/go.mod h1:hfeGWic9BAfgTrc7Xf6FaOAguCFJRo3RBbs7QJ6D7MI= +github.com/tree-sitter/tree-sitter-rust v0.24.2 h1:NL4nF67ib21RMzzfvkmXlVwe45vvhW10DVyO+D0z/W0= +github.com/tree-sitter/tree-sitter-rust v0.24.2/go.mod h1:hfeGWic9BAfgTrc7Xf6FaOAguCFJRo3RBbs7QJ6D7MI= github.com/tree-sitter/tree-sitter-typescript v0.23.2 h1:/Odvphn18PniVixb9e97X0DbNVsU6Qocv9mfkyzdXwU= github.com/tree-sitter/tree-sitter-typescript v0.23.2/go.mod h1:zjzMXT/Ulffel2xfOcAkQQkiAkmgnbtPGlFQw/5X4xA= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= diff --git a/services/parser/internal/ts/extract.go b/services/parser/internal/extract/extract.go similarity index 61% rename from services/parser/internal/ts/extract.go rename to services/parser/internal/extract/extract.go index fee6b6b..99003a6 100644 --- a/services/parser/internal/ts/extract.go +++ b/services/parser/internal/extract/extract.go @@ -1,4 +1,9 @@ -package ts +// Package extract walks a repository and turns each source file into the IR. +// +// The per-file loop is language-agnostic: walk, hash, split lines, run three +// capture passes, number overloads. Everything a language does differently +// lives in its Spec. +package extract import ( "bytes" @@ -19,7 +24,8 @@ import ( "github.com/ARCoder181105/funcatlas/parser/internal/utils" ) -// Extract walks the repo, parses each TypeScript file, and returns the IR. +// Extract walks the repo, parses every file a registered spec claims, and +// returns the IR. func Extract(logger *zap.Logger, root string, cfg security.Config) (ir.Graph, error) { grammars, err := loadGrammars() if err != nil { @@ -54,7 +60,7 @@ func Extract(logger *zap.Logger, root string, cfg security.Config) (ir.Graph, er // the usual cause is a grammar mismatch. if tree.RootNode().HasError() { logger.Warn("file parsed with errors; some calls may be missing", - zap.String("path", p)) + zap.String("path", p), zap.String("language", g.spec.Name)) } rel, _ := filepath.Rel(root, p) @@ -66,7 +72,7 @@ func Extract(logger *zap.Logger, root string, cfg security.Config) (ir.Graph, er sum := sha256.Sum256(src) graph.Files = append(graph.Files, ir.File{ Path: rel, - Language: utils.Language, + Language: g.spec.Name, ContentHash: hex.EncodeToString(sum[:]), }) @@ -75,7 +81,7 @@ func Extract(logger *zap.Logger, root string, cfg security.Config) (ir.Graph, er // Split once per file, not once per function found in it. lines := strings.Split(string(src), "\n") - eachCapture(g.queries.def, rootNode, src, "function.def", func(nameNode tree_sitter.Node) { + eachCapture(g.queries.def, rootNode, src, utils.CaptureFunctionDef, func(nameNode tree_sitter.Node) { declNode := nameNode.Parent() if declNode == nil { return @@ -94,7 +100,7 @@ func Extract(logger *zap.Logger, root string, cfg security.Config) (ir.Graph, er FileID: fileID, PackagePath: pkgPath, Name: funcName, - QualifiedName: qualifiedName(*declNode, src, funcName), + QualifiedName: qualifiedName(*declNode, src, g.spec, funcName), StartLine: startLine, EndLine: endLine, Source: strings.Join(lines[startLine-1:endLine], "\n"), @@ -102,25 +108,25 @@ func Extract(logger *zap.Logger, root string, cfg security.Config) (ir.Graph, er }) assignOverloadIndices(graph.Functions[startLen:]) - eachCapture(g.queries.call, rootNode, src, "function.call", func(callNode tree_sitter.Node) { + eachCapture(g.queries.call, rootNode, src, utils.CaptureFunctionCall, func(callNode tree_sitter.Node) { graph.Calls = append(graph.Calls, ir.CallSite{ FileID: fileID, - CallerQualified: enclosingQualifiedName(callNode, src), - CalleeObject: calleeObject(callNode, src), + CallerQualified: enclosingQualifiedName(callNode, src, g.spec), + CalleeObject: g.spec.CalleeReceiver(callNode, src), CalleeName: callNode.Utf8Text(src), Line: int(callNode.StartPosition().Row) + 1, }) }) - eachCapture(g.queries.imp, rootNode, src, "import.from", func(sourceNode tree_sitter.Node) { - stmt := sourceNode.Parent() - if stmt == nil { - return + eachCapture(g.queries.imp, rootNode, src, utils.CaptureImportFrom, func(sourceNode tree_sitter.Node) { + from, symbols := g.spec.Imports(sourceNode, src) + if symbols == nil { + return // not an import; see Spec.Imports } graph.Imports = append(graph.Imports, ir.Import{ FileID: fileID, - From: strings.Trim(sourceNode.Utf8Text(src), "\"'`"), - Symbols: importSymbols(stmt, src), + From: from, + Symbols: symbols, }) }) @@ -185,79 +191,6 @@ func eachCapture(q *tree_sitter.Query, root *tree_sitter.Node, src []byte, name } } -// calleeObject returns a member call's receiver: Repo.sync() -> "Repo", -// a.b.c() -> "a.b". Empty for a bare call. -// -// Read from the tree, not captured in the .scm: an `object:` field would make -// the member_expression pattern require it, so a.b.c() would stop matching -// and the call site would vanish. -func calleeObject(callNode tree_sitter.Node, src []byte) string { - parent := callNode.Parent() - if parent == nil || parent.Kind() != utils.KindMemberExpression { - return "" - } - obj := parent.ChildByFieldName("object") - if obj == nil { - return "" - } - return obj.Utf8Text(src) -} - -// importSymbols collects only the names an import binds locally -- what a call -// site here can actually reference. Walking by node kind rather than grabbing -// every identifier is what keeps `import { a as b }` from yielding both a and b. -func importSymbols(stmt *tree_sitter.Node, src []byte) []ir.ImportedSymbol { - // A re-export binds nothing locally; recorded for barrel-following later. - var out []ir.ImportedSymbol - - if stmt.Kind() == utils.KindExportStatement { - clause := utils.ChildByKind(stmt, utils.KindExportClause) - if clause == nil { - return []ir.ImportedSymbol{{Kind: utils.KindReExport}} // export * from "m" - } - utils.NamedChildren(clause, func(spec *tree_sitter.Node) { - if spec.Kind() == utils.KindExportSpecifier { - out = append(out, ir.ImportedSymbol{ - Original: utils.FieldText(spec, "name", src), - Kind: utils.KindReExport, - }) - } - }) - return out - } - - clause := utils.ChildByKind(stmt, utils.KindImportClause) - if clause == nil { - return []ir.ImportedSymbol{{Kind: utils.KindSideEffect}} // import "m" - } - - utils.NamedChildren(clause, func(child *tree_sitter.Node) { - switch child.Kind() { - case utils.KindIdentifier: // import def from "m" - out = append(out, ir.ImportedSymbol{Local: child.Utf8Text(src), Kind: utils.KindDefault}) - - case utils.KindNamespaceImport: // import * as ns from "m" - if id := utils.ChildByKind(child, utils.KindIdentifier); id != nil { - out = append(out, ir.ImportedSymbol{Local: id.Utf8Text(src), Kind: utils.KindNamespace}) - } - - case utils.KindNamedImports: // import { a, b as c } from "m" - utils.NamedChildren(child, func(spec *tree_sitter.Node) { - if spec.Kind() != utils.KindImportSpecifier { - return - } - name := utils.FieldText(spec, "name", src) - local := utils.FieldText(spec, "alias", src) - if local == "" { - local = name - } - out = append(out, ir.ImportedSymbol{Local: local, Original: name, Kind: utils.KindNamed}) - }) - } - }) - return out -} - // assignOverloadIndices numbers functions sharing a qualified_name in one file, // by start_line so it stays stable across re-parses. Keeps the uniqueness key // collision-free, which the delete-and-reinsert relink depends on. diff --git a/services/parser/internal/ts/extract_test.go b/services/parser/internal/extract/extract_test.go similarity index 87% rename from services/parser/internal/ts/extract_test.go rename to services/parser/internal/extract/extract_test.go index bff63d7..b74e779 100644 --- a/services/parser/internal/ts/extract_test.go +++ b/services/parser/internal/extract/extract_test.go @@ -1,10 +1,9 @@ -package ts_test +package extract_test import ( "encoding/json" "flag" "os" - "path/filepath" "testing" "github.com/stretchr/testify/assert" @@ -38,7 +37,7 @@ func TestExtract_Golden(t *testing.T) { } expected, err := os.ReadFile(goldenJSON) - require.NoError(t, err, "run `go test ./internal/ts -run Golden -update` to create it") + require.NoError(t, err, "run `go test ./internal/extract -run Golden -update` to create it") if !assert.JSONEq(t, string(expected), string(actual)) { out := t.TempDir() + "/extract_actual.json" @@ -217,32 +216,18 @@ func TestExtract_TSXCallsInsideJSX(t *testing.T) { } } -// Both extensions are parsed, and nothing else is. -func TestExtract_OnlyTypeScriptExtensions(t *testing.T) { - dir := t.TempDir() - for name, body := range map[string]string{ - "a.ts": "export function fromTs() {}\n", - "b.tsx": "export function FromTsx() { return
; }\n", - "c.js": "export function fromJs() {}\n", - "d.jsx": "export function FromJsx() { return
; }\n", - "e.py": "def from_python(): pass\n", - "f.go": "package main\nfunc FromGo() {}\n", - "g.json": `{"not": "source"}`, - } { - require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644)) +// files.language names the grammar that actually read the file, so .tsx is +// "tsx" rather than "typescript". Shiki then highlights JSX correctly, and a +// mismatched grammar is visible in the data rather than only in lost calls. +func TestExtract_LanguagePerExtension(t *testing.T) { + byPath := map[string]string{} + for _, f := range testutil.Extract(t, "../../testdata/tsx").Files { + byPath[f.Path] = f.Language } + assert.Equal(t, utils.LangTSX, byPath["Card.tsx"]) + assert.Equal(t, utils.LangTypeScript, byPath["helpers.ts"]) - g := testutil.Extract(t, dir) - - var paths []string - for _, f := range g.Files { - paths = append(paths, f.Path) - } - assert.ElementsMatch(t, []string{"a.ts", "b.tsx"}, paths) - - var names []string - for _, fn := range g.Functions { - names = append(names, fn.Name) + for _, f := range extractGolden(t).Files { + assert.Equal(t, utils.LangTypeScript, f.Language, "%s", f.Path) } - assert.ElementsMatch(t, []string{"fromTs", "FromTsx"}, names) } diff --git a/services/parser/internal/extract/golang.go b/services/parser/internal/extract/golang.go new file mode 100644 index 0000000..ca59fa9 --- /dev/null +++ b/services/parser/internal/extract/golang.go @@ -0,0 +1,91 @@ +package extract + +import ( + "path" + + tree_sitter "github.com/tree-sitter/go-tree-sitter" + bindings "github.com/tree-sitter/tree-sitter-go/bindings/go" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" + "github.com/ARCoder181105/funcatlas/parser/queries" +) + +// Extraction only. Go resolves callees through package clauses and +// capitalisation-based export, neither of which this parser models, so a +// cross-file call gets name_match or unresolved and never exact. + +var golang = Spec{ + Name: utils.LangGo, + Extensions: []string{utils.ExtGo}, + Language: func() *tree_sitter.Language { return tree_sitter.NewLanguage(bindings.Language()) }, + Query: queries.GoSCM, + ScopeSegment: goScopeSegment, + CalleeReceiver: goCalleeReceiver, + Imports: goImports, +} + +// goScopeSegment names a method after its receiver type: `func (r *Repo) Sync` +// is Repo.Sync. A closure is anonymous -- Go has no name to give one. +func goScopeSegment(node *tree_sitter.Node, src []byte) (string, bool) { + switch node.Kind() { + case utils.KindGoFuncDecl: + return utils.DeclName(node, src), true + + case utils.KindGoMethodDecl: + name := utils.DeclName(node, src) + if receiver := goReceiverType(node, src); receiver != "" { + return utils.Join(receiver, name), true + } + return name, true + + case utils.KindGoFuncLiteral: + return utils.Anonymous, true + } + return "", false +} + +// goReceiverType is the bare type name a method hangs off. Written *Repo, Repo +// or Repo[T]; only the type_identifier inside is the name. +func goReceiverType(method *tree_sitter.Node, src []byte) string { + receiver := method.ChildByFieldName(utils.FieldReceiver) + if receiver == nil { + return "" + } + ident := utils.FirstDescendantByKind(receiver, utils.KindGoTypeIdentifier) + if ident == nil { + return "" + } + return ident.Utf8Text(src) +} + +// goCalleeReceiver returns a selector call's operand: r.unlock() -> "r", +// fmt.Errorf() -> "fmt". Empty for a bare call. +func goCalleeReceiver(callNode tree_sitter.Node, src []byte) string { + return utils.ParentFieldText(&callNode, utils.KindGoSelectorExpression, utils.FieldOperand, src) +} + +// goImports records what an import binds locally: the package name, which is +// the alias when there is one and the last path segment otherwise. +// +// Namespace rather than named, because a Go import binds a qualifier and never +// the symbols behind it. The resolver does not follow these -- a Go path names +// a module, not a file here -- but they are the IR a later Go resolver needs. +func goImports(specifier tree_sitter.Node, src []byte) (string, []ir.ImportedSymbol) { + stmt := specifier.Parent() + if stmt == nil { + return "", nil + } + from := utils.StringLiteralText(specifier, src) + + alias := utils.FieldText(stmt, utils.FieldName, src) + switch alias { + case utils.GoBlankImport, utils.GoDotImport: + // `_` binds nothing; `.` binds every exported name under no qualifier, + // which is not a local name this can record. + return from, []ir.ImportedSymbol{{Kind: utils.KindSideEffect}} + case "": + alias = path.Base(from) + } + return from, []ir.ImportedSymbol{{Local: alias, Kind: utils.KindNamespace}} +} diff --git a/services/parser/internal/extract/golang_test.go b/services/parser/internal/extract/golang_test.go new file mode 100644 index 0000000..509c9ff --- /dev/null +++ b/services/parser/internal/extract/golang_test.go @@ -0,0 +1,108 @@ +package extract_test + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/testutil" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" +) + +func goFixture(t *testing.T) ir.Graph { + t.Helper() + return testutil.Extract(t, "../../testdata/lang/go") +} + +// qualifiedNames lists every function's qualified name in the graph. +func qualifiedNames(g ir.Graph) []string { + out := make([]string, 0, len(g.Functions)) + for _, fn := range g.Functions { + out = append(out, fn.QualifiedName) + } + return out +} + +// Go's hardest constructs are the bodies a call can hide in: a goroutine +// literal, a defer, and a generic call. Asserting on the calls is the point -- +// a wrong grammar leaves every declaration matching and drops all of these. +func TestExtract_GoCallsInsideHardConstructs(t *testing.T) { + calls := callsIn(t, goFixture(t), "repo.go") + + for _, name := range []string{ + "unlock", // inside a defer + "notify", // inside a goroutine's function literal + "Errorf", // nested inside another call's arguments + "Wrap", // a selector call on an imported package + "Unlock", // a chained selector, r.mu.Unlock() + "Map", // a generic call with two type arguments + "println", // a builtin + } { + assert.Contains(t, calls, name, "call %q lost", name) + } +} + +// A generic call with ONE type argument parses as a type_conversion_expression, +// indistinguishable in shape from int(x). Capturing it would invent a call for +// every conversion in the repository, so it is dropped. +// +// This is a limit, not a bug, and it is pinned here so it cannot change by +// accident: the product's promise is that ambiguity is admitted, never guessed. +func TestExtract_GoSingleTypeArgumentCallIsNotCaptured(t *testing.T) { + assert.NotContains(t, callsIn(t, goFixture(t), "repo.go"), "Identity", + "Map[int](xs) is ambiguous with a conversion; capturing it would invent calls") +} + +// A method is named after its receiver type, so Repo.Sync never collides with +// a package-level Sync -- pointer and value receivers alike. +func TestExtract_GoMethodsCarryTheirReceiver(t *testing.T) { + names := qualifiedNames(goFixture(t)) + + assert.Contains(t, names, "Repo.Sync", "pointer receiver") + assert.Contains(t, names, "Repo.unlock", "value receiver") + assert.Contains(t, names, "notify", "a package-level function has no prefix") +} + +// A call inside a goroutine's closure is attributed to the enclosing method, +// through the anonymous literal. +func TestExtract_GoCallerInsideAFuncLiteral(t *testing.T) { + g := goFixture(t) + fileID := fileIDOf(t, g, "repo.go") + + for _, c := range g.Calls { + if c.FileID == fileID && c.CalleeName == "notify" { + assert.Equal(t, "Repo.Sync."+utils.Anonymous, c.CallerQualified) + return + } + } + t.Fatal("no call to notify in repo.go") +} + +func TestExtract_GoImports(t *testing.T) { + g := goFixture(t) + + byFrom := map[string][]ir.ImportedSymbol{} + for _, imp := range g.Imports { + byFrom[imp.From] = imp.Symbols + } + + // A Go import binds a qualifier, never the symbols behind it. + assert.Equal(t, + []ir.ImportedSymbol{{Local: "fmt", Kind: utils.KindNamespace}}, + byFrom["fmt"], `import "fmt" binds fmt`) + + assert.Equal(t, + []ir.ImportedSymbol{{Local: "stdsync", Kind: utils.KindNamespace}}, + byFrom["sync"], `stdsync "sync" binds the alias, not the last path segment`) + + assert.Equal(t, + []ir.ImportedSymbol{{Local: "util", Kind: utils.KindNamespace}}, + byFrom["example.com/app/internal/util"], "a module path binds its last segment") +} + +func TestExtract_GoLanguage(t *testing.T) { + for _, f := range goFixture(t).Files { + assert.Equal(t, utils.LangGo, f.Language, "%s", f.Path) + } +} diff --git a/services/parser/internal/extract/grammar.go b/services/parser/internal/extract/grammar.go new file mode 100644 index 0000000..7d765d8 --- /dev/null +++ b/services/parser/internal/extract/grammar.go @@ -0,0 +1,73 @@ +package extract + +import ( + "fmt" + "path/filepath" + + tree_sitter "github.com/tree-sitter/go-tree-sitter" +) + +// grammar pairs a spec with a parser and the queries compiled against its +// language. One per spec, reused across every file it claims. +type grammar struct { + spec *Spec + parser *tree_sitter.Parser + queries *compiledQueries +} + +func (g *grammar) Close() { + g.queries.Close() + g.parser.Close() +} + +// grammars maps a file extension to the grammar that reads it. Several +// extensions may point at one grammar; .ts and .tsx never may. +type grammars map[string]*grammar + +func (g grammars) Close() { + seen := make(map[*grammar]bool, len(g)) + for _, entry := range g { + if !seen[entry] { + seen[entry] = true + entry.Close() + } + } +} + +// forFile returns the grammar for a path, or nil to skip it. +// +// An exact extension lookup, not a suffix scan: with several languages +// registered a suffix match would let one extension swallow another. +func (g grammars) forFile(path string) *grammar { + return g[filepath.Ext(path)] +} + +// loadGrammars compiles every registered language and its queries once per run. +func loadGrammars() (grammars, error) { + out := make(grammars, len(registry)) + + for _, spec := range registry { + lang := spec.Language() + + parser := tree_sitter.NewParser() + if err := parser.SetLanguage(lang); err != nil { + out.Close() + parser.Close() + return nil, fmt.Errorf("set language for %s: %w", spec.Name, err) + } + + qs, err := loadQueries(lang, spec.Query) + if err != nil { + out.Close() + parser.Close() + return nil, fmt.Errorf("queries for %s: %w", spec.Name, err) + } + + entry := &grammar{spec: spec, parser: parser, queries: qs} + for _, ext := range spec.Extensions { + out[ext] = entry + } + } + + return out, nil +} diff --git a/services/parser/internal/extract/java.go b/services/parser/internal/extract/java.go new file mode 100644 index 0000000..92525ba --- /dev/null +++ b/services/parser/internal/extract/java.go @@ -0,0 +1,88 @@ +package extract + +import ( + "strings" + + tree_sitter "github.com/tree-sitter/go-tree-sitter" + bindings "github.com/tree-sitter/tree-sitter-java/bindings/go" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" + "github.com/ARCoder181105/funcatlas/parser/queries" +) + +// Extraction only. Java resolves through packages and the classpath, and picks +// between overloads by argument type -- none of which this parser models, so a +// cross-file call gets name_match or unresolved and never exact. +// +// Java is the first language here with real overloads: two methods of one name +// in one class. That is what overload_index has always been for, and what +// makes uniqueQualified answer unresolved rather than picking the first. + +var java = Spec{ + Name: utils.LangJava, + Extensions: []string{utils.ExtJava}, + Language: func() *tree_sitter.Language { return tree_sitter.NewLanguage(bindings.Language()) }, + Query: queries.JavaSCM, + ScopeSegment: javaScopeSegment, + CalleeReceiver: javaCalleeReceiver, + Imports: javaImports, +} + +// javaScopeSegment names methods after every type that encloses them, so +// Repo.Nested.deep is distinct from a Repo.deep. A lambda body and an +// anonymous inner class are both anonymous scopes. +func javaScopeSegment(node *tree_sitter.Node, src []byte) (string, bool) { + switch node.Kind() { + case utils.KindJavaMethodDecl, utils.KindJavaConstructorDecl, + utils.KindJavaClassDecl, utils.KindJavaInterfaceDecl, + utils.KindJavaEnumDecl, utils.KindJavaRecordDecl: + return utils.DeclName(node, src), true + + case utils.KindJavaLambda: + return utils.Anonymous, true + + case utils.KindJavaObjectCreation: + // `new Runnable(){ ... }` opens a scope; `new Repo("x")` does not. + if utils.ChildByKind(node, utils.KindJavaClassBody) != nil { + return utils.Anonymous, true + } + } + return "", false +} + +// javaCalleeReceiver returns a method invocation's object: items.get() -> +// "items", System.out.println() -> "System.out". Empty for a bare call. +// +// Read off the invocation itself rather than a parent, because Java puts the +// callee name and its object in one node. +func javaCalleeReceiver(callNode tree_sitter.Node, src []byte) string { + return utils.ParentFieldText(&callNode, utils.KindJavaMethodInvocation, utils.FieldObject, src) +} + +// javaImports records the name an import binds. +// +// `import java.util.List` binds List; `import static com.example.util.Text.wrap` +// binds wrap. Both are the last segment, so the two need no distinguishing -- +// what differs is only whether the prefix is a package or a class, and the +// resolver does not follow either. +func javaImports(stmt tree_sitter.Node, src []byte) (string, []ir.ImportedSymbol) { + path := utils.FirstDescendantByKind(&stmt, utils.KindJavaScopedIdentifier) + if path == nil { + return "", nil + } + text := path.Utf8Text(src) + + // `import java.util.*` binds every type in the package under no name of its + // own, which is not a local binding this can record. + if strings.HasSuffix(stmt.Utf8Text(src), utils.JavaWildcardImport) { + return text, []ir.ImportedSymbol{{Kind: utils.KindSideEffect}} + } + + i := strings.LastIndex(text, ".") + if i < 0 { + return "", []ir.ImportedSymbol{{Local: text, Original: text, Kind: utils.KindNamed}} + } + name := text[i+1:] + return text[:i], []ir.ImportedSymbol{{Local: name, Original: name, Kind: utils.KindNamed}} +} diff --git a/services/parser/internal/extract/java_test.go b/services/parser/internal/extract/java_test.go new file mode 100644 index 0000000..92b4fa9 --- /dev/null +++ b/services/parser/internal/extract/java_test.go @@ -0,0 +1,96 @@ +package extract_test + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/testutil" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" +) + +func javaFixture(t *testing.T) ir.Graph { + t.Helper() + return testutil.Extract(t, "../../testdata/lang/java") +} + +// Java's hidden scopes are the anonymous inner class and the lambda. A call in +// either is still a call, and asserting on the calls is what catches a query +// that stopped descending into them. +func TestExtract_JavaCallsInsideHardConstructs(t *testing.T) { + calls := callsIn(t, javaFixture(t), "Repo.java") + + for _, name := range []string{ + "helper", // inside an anonymous inner class's method + "describe", // inside a lambda body + "wrap", // a statically imported free function + "valueOf", // a static method on a type + "println", // a chained receiver, System.out.println + "get", // a method on a generic parameter + "sync", // one overload calling the other + } { + assert.Contains(t, calls, name, "call %q lost", name) + } +} + +// Every enclosing type names a method, and the anonymous scopes are named too, +// so a method inside `new Runnable(){...}` cannot collide with anything. +func TestExtract_JavaQualifiedNames(t *testing.T) { + names := qualifiedNames(javaFixture(t)) + + for _, want := range []string{ + "Repo.Repo", // the constructor + "Repo.sync", // both overloads share this + "Repo.Nested.deep", // a static nested class + "Repo.task." + utils.Anonymous + ".run", // inside new Runnable(){...} + } { + assert.Contains(t, names, want) + } +} + +// TypeScript's overload *signatures* never produced a genuine duplicate, so +// this is the first language where overload_index does real work: two methods +// of one name in one class, numbered by start_line. +func TestExtract_JavaOverloadsAreNumbered(t *testing.T) { + var syncs []ir.Function + for _, fn := range javaFixture(t).Functions { + if fn.QualifiedName == "Repo.sync" { + syncs = append(syncs, fn) + } + } + require.Len(t, syncs, 2, "sync() and sync(int) are two functions with one name") + + first, second := syncs[0], syncs[1] + if first.StartLine > second.StartLine { + first, second = second, first + } + assert.Equal(t, 0, first.OverloadIndex) + assert.Equal(t, 1, second.OverloadIndex) +} + +func TestExtract_JavaImports(t *testing.T) { + g := javaFixture(t) + + byFrom := map[string][]ir.ImportedSymbol{} + for _, imp := range g.Imports { + byFrom[imp.From] = imp.Symbols + } + + assert.Equal(t, + []ir.ImportedSymbol{{Local: "List", Original: "List", Kind: utils.KindNamed}}, + byFrom["java.util"], "import java.util.List") + + // A static import binds a method name rather than a type, but the shape is + // the same and the resolver follows neither. + assert.Equal(t, + []ir.ImportedSymbol{{Local: "wrap", Original: "wrap", Kind: utils.KindNamed}}, + byFrom["com.example.util.Text"], "import static com.example.util.Text.wrap") +} + +func TestExtract_JavaLanguage(t *testing.T) { + for _, f := range javaFixture(t).Files { + assert.Equal(t, utils.LangJava, f.Language, "%s", f.Path) + } +} diff --git a/services/parser/internal/extract/javascript.go b/services/parser/internal/extract/javascript.go new file mode 100644 index 0000000..5632dc6 --- /dev/null +++ b/services/parser/internal/extract/javascript.go @@ -0,0 +1,100 @@ +package extract + +import ( + tree_sitter "github.com/tree-sitter/go-tree-sitter" + bindings "github.com/tree-sitter/tree-sitter-javascript/bindings/go" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" + "github.com/ARCoder181105/funcatlas/parser/queries" +) + +// One grammar, two names. tree-sitter-javascript parses JSX in any file, so +// unlike .ts and .tsx these can share it -- but files.language still says which +// it is, because Shiki highlights .jsx differently and the reader wants to see +// what a file is. +// +// Scope rules, receivers and ESM imports are TypeScript's: the node kinds are +// the same grammar family. Only require() is new. + +var javaScript = Spec{ + Name: utils.LangJavaScript, + Extensions: []string{utils.ExtJS, utils.ExtMJS, utils.ExtCJS}, + Language: newJavaScriptLanguage, + Query: queries.JavaScriptSCM, + ScopeSegment: tsScopeSegment, + CalleeReceiver: tsCalleeReceiver, + Imports: jsImports, +} + +var jsx = Spec{ + Name: utils.LangJSX, + Extensions: []string{utils.ExtJSX}, + Language: newJavaScriptLanguage, + Query: queries.JavaScriptSCM, + ScopeSegment: tsScopeSegment, + CalleeReceiver: tsCalleeReceiver, + Imports: jsImports, +} + +func newJavaScriptLanguage() *tree_sitter.Language { + return tree_sitter.NewLanguage(bindings.Language()) +} + +// jsImports handles ESM exactly as TypeScript does, plus CommonJS. +// +// `const { helper } = require("./util.js")` is a call, not an import +// statement, so the .scm can only capture the specifier of *some* call. Which +// call it was is read from the tree here -- returning nil for anything that is +// not require() drops the match rather than recording a phantom import. +func jsImports(specifier tree_sitter.Node, src []byte) (string, []ir.ImportedSymbol) { + stmt := specifier.Parent() + if stmt == nil { + return "", nil + } + if stmt.Kind() != utils.KindArguments { + return tsImports(specifier, src) + } + + from := utils.StringLiteralText(specifier, src) + + call := stmt.Parent() + if call == nil || utils.FieldText(call, utils.FieldFunction, src) != utils.RequireCallee { + return "", nil + } + + // require("m") on its own binds nothing; the binding is the declarator + // wrapping it, which is where the local names live. + decl := call.Parent() + if decl == nil || decl.Kind() != utils.KindVariableDeclarator { + return from, []ir.ImportedSymbol{{Kind: utils.KindSideEffect}} + } + + name := decl.ChildByFieldName(utils.FieldName) + if name == nil { + return from, []ir.ImportedSymbol{{Kind: utils.KindSideEffect}} + } + + switch name.Kind() { + case utils.KindIdentifier: // const m = require("m") + return from, []ir.ImportedSymbol{{Local: name.Utf8Text(src), Kind: utils.KindNamespace}} + + case utils.KindObjectPattern: // const { a, b: c } = require("m") + var out []ir.ImportedSymbol + utils.NamedChildren(name, func(prop *tree_sitter.Node) { + switch prop.Kind() { + case utils.KindShorthandPropertyIdentifierPattern: + local := prop.Utf8Text(src) + out = append(out, ir.ImportedSymbol{Local: local, Original: local, Kind: utils.KindNamed}) + case utils.KindPairPattern: + out = append(out, ir.ImportedSymbol{ + Local: utils.FieldText(prop, utils.FieldValue, src), + Original: utils.FieldText(prop, utils.FieldKey, src), + Kind: utils.KindNamed, + }) + } + }) + return from, out + } + return "", nil +} diff --git a/services/parser/internal/extract/javascript_test.go b/services/parser/internal/extract/javascript_test.go new file mode 100644 index 0000000..a075b2a --- /dev/null +++ b/services/parser/internal/extract/javascript_test.go @@ -0,0 +1,113 @@ +package extract_test + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/testutil" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" +) + +func javaScriptFixture(t *testing.T) ir.Graph { + t.Helper() + return testutil.Extract(t, "../../testdata/lang/javascript") +} + +// callsIn lists the callee names recorded for one fixture file. +func callsIn(t *testing.T, g ir.Graph, path string) []string { + t.Helper() + fileID := fileIDOf(t, g, path) + + var out []string + for _, c := range g.Calls { + if c.FileID == fileID { + out = append(out, c.CalleeName) + } + } + return out +} + +// JSX is JavaScript's hardest construct for the same reason it is TypeScript's: +// a grammar that cannot read it drops every call inside while the declarations +// still match, so the file looks parsed. These assertions are on the calls. +func TestExtract_JSXCallsInsideMarkup(t *testing.T) { + calls := callsIn(t, javaScriptFixture(t), "Card.jsx") + + for _, name := range []string{ + "formatLabel", // above the return, outside any JSX + "cx", // inside a JSX attribute expression + "renderTitle", // inside a JSX child expression + "map", // a member call on a JSX child + "describeItem", // inside a callback nested two JSX levels down + "countItems", // inside a template literal inside JSX + "shout", // inside JSX in a different function + } { + assert.Contains(t, calls, name, "call %q lost inside JSX", name) + } +} + +func TestExtract_JavaScriptLanguagePerExtension(t *testing.T) { + byPath := map[string]string{} + for _, f := range javaScriptFixture(t).Files { + byPath[f.Path] = f.Language + } + + assert.Equal(t, utils.LangJavaScript, byPath["util.js"]) + assert.Equal(t, utils.LangJavaScript, byPath["legacy.js"]) + assert.Equal(t, utils.LangJSX, byPath["Card.jsx"]) +} + +// require() is a call, so the import query has to over-capture and let +// jsImports throw the rest away. Both halves are asserted here: the requires +// are recorded with the names they bind, and cx("title") -- an ordinary call +// with a string argument -- is not recorded as an import at all. +func TestExtract_CommonJSRequires(t *testing.T) { + g := javaScriptFixture(t) + legacyID := fileIDOf(t, g, "legacy.js") + + byFrom := map[string][]ir.ImportedSymbol{} + for _, imp := range g.Imports { + if imp.FileID == legacyID { + byFrom[imp.From] = imp.Symbols + } + } + + assert.Equal(t, + []ir.ImportedSymbol{{Local: "helper", Original: "helper", Kind: utils.KindNamed}}, + byFrom["./util.js"], `const { helper } = require("./util.js")`) + + assert.Equal(t, + []ir.ImportedSymbol{{Local: "path", Kind: utils.KindNamespace}}, + byFrom["node:path"], `const path = require("node:path") binds the whole module`) + + require.Len(t, byFrom, 2, "only require() calls are imports") + + for _, imp := range g.Imports { + assert.NotEqual(t, "title", imp.From, `cx("title") is a call, not an import`) + } +} + +// A .js file importing a .ts file resolves like any other ECMAScript import: +// the two are one resolution group, and ModuleCandidates has to try the literal +// ./util.js as well as the ./util.ts it might have stood for. +func TestModuleCandidates_KeepsTheLiteralJSPath(t *testing.T) { + candidates := utils.ModuleCandidates("src/legacy.js", "./util.js") + + assert.Contains(t, candidates, "src/util.ts", "ESM TypeScript writes .js for .ts") + assert.Contains(t, candidates, "src/util.js", "a real .js file is a candidate too") + assert.Less(t, + indexOf(candidates, "src/util.ts"), indexOf(candidates, "src/util.js"), + "the TypeScript stem is still tried first") +} + +func indexOf(values []string, want string) int { + for i, v := range values { + if v == want { + return i + } + } + return -1 +} diff --git a/services/parser/internal/extract/python.go b/services/parser/internal/extract/python.go new file mode 100644 index 0000000..a4bc227 --- /dev/null +++ b/services/parser/internal/extract/python.go @@ -0,0 +1,120 @@ +package extract + +import ( + "strings" + + tree_sitter "github.com/tree-sitter/go-tree-sitter" + bindings "github.com/tree-sitter/tree-sitter-python/bindings/go" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" + "github.com/ARCoder181105/funcatlas/parser/queries" +) + +// Extraction only. Python resolves through sys.path and __init__.py, neither +// of which this parser models, so a cross-file call gets name_match or +// unresolved and never exact. + +var python = Spec{ + Name: utils.LangPython, + Extensions: []string{utils.ExtPython}, + Language: func() *tree_sitter.Language { return tree_sitter.NewLanguage(bindings.Language()) }, + Query: queries.PythonSCM, + ScopeSegment: pyScopeSegment, + CalleeReceiver: pyCalleeReceiver, + Imports: pyImports, +} + +// pyScopeSegment names methods after their class, nesting included, so +// Repo.Nested.deep is distinct from a module-level deep. A lambda has no name. +func pyScopeSegment(node *tree_sitter.Node, src []byte) (string, bool) { + switch node.Kind() { + case utils.KindPyFunctionDef, utils.KindPyClassDef: + return utils.DeclName(node, src), true + + case utils.KindPyLambda: + return utils.Anonymous, true + } + return "", false +} + +// pyCalleeReceiver returns an attribute call's object: self.label() -> "self", +// osp.basename() -> "osp". Empty for a bare call. +func pyCalleeReceiver(callNode tree_sitter.Node, src []byte) string { + return utils.ParentFieldText(&callNode, utils.KindPyAttribute, utils.FieldObject, src) +} + +// pyImports records what an import binds locally. +// +// The capture is the whole statement: `import a.b` and `from .m import x as y` +// share no node to point at. `import a.b` binds the top-level name `a`, not +// `a.b` -- so the module recorded is the full path and the local name is what +// a call site here can actually write. +func pyImports(stmt tree_sitter.Node, src []byte) (string, []ir.ImportedSymbol) { + if stmt.Kind() == utils.KindPyImportFrom { + from := utils.FieldText(&stmt, utils.FieldModuleName, src) + + var out []ir.ImportedSymbol + utils.NamedChildren(&stmt, func(child *tree_sitter.Node) { + // The module_name is a sibling of the imported names, so it has to + // be skipped by identity rather than by kind: `from a import b` + // spells both as dotted_name. + if child.Id() == stmt.ChildByFieldName(utils.FieldModuleName).Id() { + return + } + if symbol, ok := pyImportedName(child, src); ok { + out = append(out, symbol) + } + }) + if out == nil { + return from, []ir.ImportedSymbol{{Kind: utils.KindSideEffect}} // from m import * + } + return from, out + } + + // `import a.b as c` binds c; `import a.b` binds a. + var from string + var out []ir.ImportedSymbol + utils.NamedChildren(&stmt, func(child *tree_sitter.Node) { + path, local := pyModuleBinding(child, src) + if local == "" { + return + } + from = path + out = append(out, ir.ImportedSymbol{Local: local, Kind: utils.KindNamespace}) + }) + return from, out +} + +// pyImportedName turns one name in a `from ... import ...` list into what it +// binds here. +func pyImportedName(node *tree_sitter.Node, src []byte) (ir.ImportedSymbol, bool) { + switch node.Kind() { + case utils.KindPyAliasedImport: // unwrap as peel + return ir.ImportedSymbol{ + Local: utils.FieldText(node, utils.FieldAlias, src), + Original: utils.FieldText(node, utils.FieldName, src), + Kind: utils.KindNamed, + }, true + + case utils.KindPyDottedName: + name := node.Utf8Text(src) + return ir.ImportedSymbol{Local: name, Original: name, Kind: utils.KindNamed}, true + } + return ir.ImportedSymbol{}, false +} + +// pyModuleBinding returns a plain import's module path and the name it binds. +func pyModuleBinding(node *tree_sitter.Node, src []byte) (path, local string) { + switch node.Kind() { + case utils.KindPyAliasedImport: + return utils.FieldText(node, utils.FieldName, src), utils.FieldText(node, utils.FieldAlias, src) + + case utils.KindPyDottedName: + path = node.Utf8Text(src) + // Only the head is bound: `import os.path` makes `os` callable, not + // `os.path`, and a call site writes os.path.basename through it. + return path, strings.SplitN(path, ".", 2)[0] + } + return "", "" +} diff --git a/services/parser/internal/extract/python_test.go b/services/parser/internal/extract/python_test.go new file mode 100644 index 0000000..c99ba05 --- /dev/null +++ b/services/parser/internal/extract/python_test.go @@ -0,0 +1,101 @@ +package extract_test + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/testutil" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" +) + +func pythonFixture(t *testing.T) ir.Graph { + t.Helper() + return testutil.Extract(t, "../../testdata/lang/python") +} + +// Python hides calls in places a naive query misses: inside a decorator, inside +// an f-string's interpolation, inside a comprehension's filter, and behind +// await. All are asserted on the calls. +func TestExtract_PythonCallsInsideHardConstructs(t *testing.T) { + calls := callsIn(t, pythonFixture(t), "repo.py") + + for _, name := range []string{ + "wraps", // inside a decorator expression + "describe", // inside an f-string interpolation + "label", // an attribute call on self + "sync", // behind await + "render", // the element of a list comprehension + "keep", // the comprehension's if-clause + "basename", // an attribute call on an aliased module + "peel", // called under its import alias + } { + assert.Contains(t, calls, name, "call %q lost", name) + } +} + +// A decorated definition wraps the function_definition rather than replacing +// it, so the captured name's parent is still the definition -- and the line +// range is the function's, not the decorator's. +func TestExtract_PythonDecoratedFunction(t *testing.T) { + g := pythonFixture(t) + + for _, fn := range g.Functions { + if fn.QualifiedName == "Repo.sync" { + assert.NotContains(t, fn.Source, "@trace", + "the decorator is above the definition, not part of it") + assert.Contains(t, fn.Source, "def sync(self):") + return + } + } + t.Fatal("Repo.sync not extracted") +} + +// Class nesting is part of the name, so Repo.Nested.deep never collides with a +// module-level deep -- and async def is an ordinary function_definition. +func TestExtract_PythonQualifiedNames(t *testing.T) { + names := qualifiedNames(pythonFixture(t)) + + for _, want := range []string{ + "Repo.sync", + "Repo.Nested.deep", + "trace.inner", // a closure defined inside a decorator factory + "fetch", // async def + "describe", + } { + assert.Contains(t, names, want) + } +} + +func TestExtract_PythonImports(t *testing.T) { + g := pythonFixture(t) + + byFrom := map[string][]ir.ImportedSymbol{} + for _, imp := range g.Imports { + byFrom[imp.From] = imp.Symbols + } + + assert.Equal(t, + []ir.ImportedSymbol{{Local: "functools", Kind: utils.KindNamespace}}, + byFrom["functools"], "import functools") + + // `import os.path as osp` binds osp; without the alias it would bind os, + // because that is the only name a call site could then write. + assert.Equal(t, + []ir.ImportedSymbol{{Local: "osp", Kind: utils.KindNamespace}}, + byFrom["os.path"], "import os.path as osp") + + assert.Equal(t, + []ir.ImportedSymbol{ + {Local: "wrap", Original: "wrap", Kind: utils.KindNamed}, + {Local: "peel", Original: "unwrap", Kind: utils.KindNamed}, + }, + byFrom[".util"], "from .util import wrap, unwrap as peel") +} + +func TestExtract_PythonLanguage(t *testing.T) { + for _, f := range pythonFixture(t).Files { + assert.Equal(t, utils.LangPython, f.Language, "%s", f.Path) + } +} diff --git a/services/parser/internal/extract/queries.go b/services/parser/internal/extract/queries.go new file mode 100644 index 0000000..df70ae7 --- /dev/null +++ b/services/parser/internal/extract/queries.go @@ -0,0 +1,70 @@ +package extract + +import ( + "fmt" + + tree_sitter "github.com/tree-sitter/go-tree-sitter" + + "github.com/ARCoder181105/funcatlas/parser/internal/utils" +) + +// compiledQueries holds one language's compiled tree-sitter queries. +// Compiled once per run, reused across every file. +type compiledQueries struct { + def *tree_sitter.Query // @function.def + call *tree_sitter.Query // @function.call + imp *tree_sitter.Query // @import.from +} + +func (q *compiledQueries) Close() { + q.def.Close() + q.call.Close() + q.imp.Close() +} + +// loadQueries compiles a language's .scm and verifies it declares all three +// captures. A half-written query fails here rather than yielding a file that +// looks parsed and has no calls. +func loadQueries(lang *tree_sitter.Language, src string) (*compiledQueries, error) { + out := &compiledQueries{} + for _, target := range []struct { + capture string + into **tree_sitter.Query + }{ + {utils.CaptureFunctionDef, &out.def}, + {utils.CaptureFunctionCall, &out.call}, + {utils.CaptureImportFrom, &out.imp}, + } { + q, err := compileOne(lang, src, target.capture) + if err != nil { + out.closePartial() + return nil, fmt.Errorf("%s: %w", target.capture, err) + } + *target.into = q + } + return out, nil +} + +// closePartial releases whichever queries were compiled before one failed. +func (q *compiledQueries) closePartial() { + for _, compiled := range []*tree_sitter.Query{q.def, q.call, q.imp} { + if compiled != nil { + compiled.Close() + } + } +} + +// compileOne compiles the whole .scm against the language and verifies the +// requested capture name exists. The returned Query runs ALL patterns; the +// extractor filters by capture index. +func compileOne(lang *tree_sitter.Language, src, capture string) (*tree_sitter.Query, error) { + q, qerr := tree_sitter.NewQuery(lang, src) + if qerr != nil { + return nil, qerr + } + if _, ok := q.CaptureIndexForName(capture); !ok { + q.Close() + return nil, fmt.Errorf("capture @%s not found in query", capture) + } + return q, nil +} diff --git a/services/parser/internal/extract/rust.go b/services/parser/internal/extract/rust.go new file mode 100644 index 0000000..f16831e --- /dev/null +++ b/services/parser/internal/extract/rust.go @@ -0,0 +1,134 @@ +package extract + +import ( + "strings" + + tree_sitter "github.com/tree-sitter/go-tree-sitter" + bindings "github.com/tree-sitter/tree-sitter-rust/bindings/go" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" + "github.com/ARCoder181105/funcatlas/parser/queries" +) + +// Extraction only. Rust resolves through mod, use and crate paths, and through +// impl blocks and traits -- none of which this parser models, so a cross-file +// call gets name_match or unresolved and never exact. + +var rust = Spec{ + Name: utils.LangRust, + Extensions: []string{utils.ExtRust}, + Language: func() *tree_sitter.Language { return tree_sitter.NewLanguage(bindings.Language()) }, + Query: queries.RustSCM, + ScopeSegment: rustScopeSegment, + CalleeReceiver: rustCalleeReceiver, + Imports: rustImports, +} + +// rustScopeSegment names a method after the type its impl block targets, so +// Repo.sync never collides with a free function called sync. A closure has no +// name to give. +func rustScopeSegment(node *tree_sitter.Node, src []byte) (string, bool) { + switch node.Kind() { + case utils.KindRustFunctionItem: + return utils.DeclName(node, src), true + + case utils.KindRustImplItem: + // `impl Repo`, `impl Repo` and `impl Trait for Repo` all name the + // target in the `type` field; only the type_identifier in it is a name. + if ident := utils.FirstDescendantByKind( + node.ChildByFieldName(utils.FieldType), utils.KindRustTypeIdentifier, + ); ident != nil { + return ident.Utf8Text(src), true + } + return utils.Anonymous, true + + case utils.KindRustClosureExpression: + return utils.Anonymous, true + } + return "", false +} + +// rustCalleeReceiver returns a method call's receiver: self.label() -> "self", +// values.iter() -> "values". Empty for a bare or path call. +func rustCalleeReceiver(callNode tree_sitter.Node, src []byte) string { + return utils.ParentFieldText(&callNode, utils.KindRustFieldExpression, utils.FieldValue, src) +} + +// rustImports records what a `use` binds locally. +// +// The capture is the declaration's argument, because Rust has no quoted +// specifier to point at: `use crate::util::{wrap, unwrap as peel}` is one +// nested path expression. From is the module prefix, and the symbols are the +// leaf names -- the alias where there is one, since that is what a call site +// here writes. +func rustImports(argument tree_sitter.Node, src []byte) (string, []ir.ImportedSymbol) { + switch argument.Kind() { + case utils.KindRustScopedUseList: // use a::b::{c, d as e} + from := utils.FieldText(&argument, utils.FieldPath, src) + list := argument.ChildByFieldName(utils.FieldList) + if list == nil { + return from, []ir.ImportedSymbol{{Kind: utils.KindSideEffect}} + } + var out []ir.ImportedSymbol + utils.NamedChildren(list, func(item *tree_sitter.Node) { + if symbol, ok := rustUseLeaf(item, src); ok { + out = append(out, symbol) + } + }) + return from, out + + case utils.KindRustUseAsClause, utils.KindRustScopedIdentifier, utils.KindRustIdentifier: + symbol, ok := rustUseLeaf(&argument, src) + if !ok { + return "", nil + } + // The path names the item itself, so the module is everything above it. + return rustModulePrefix(&argument, src), []ir.ImportedSymbol{symbol} + + case utils.KindRustUseWildcard: // use a::b::* + return rustModulePrefix(&argument, src), []ir.ImportedSymbol{{Kind: utils.KindSideEffect}} + } + return "", nil +} + +// rustUseLeaf turns one item of a use path into the name it binds. +func rustUseLeaf(node *tree_sitter.Node, src []byte) (ir.ImportedSymbol, bool) { + switch node.Kind() { + case utils.KindRustUseAsClause: // unwrap as peel + original := lastPathSegment(utils.FieldText(node, utils.FieldPath, src)) + return ir.ImportedSymbol{ + Local: utils.FieldText(node, utils.FieldAlias, src), + Original: original, + Kind: utils.KindNamed, + }, true + + case utils.KindRustIdentifier, utils.KindRustScopedIdentifier, utils.KindRustTypeIdentifier: + name := lastPathSegment(node.Utf8Text(src)) + return ir.ImportedSymbol{Local: name, Original: name, Kind: utils.KindNamed}, true + + case utils.KindRustUseWildcard, utils.KindRustSelf: + return ir.ImportedSymbol{Kind: utils.KindSideEffect}, true + } + return ir.ImportedSymbol{}, false +} + +// rustModulePrefix is a path with its last segment removed: the module an item +// was imported from. +func rustModulePrefix(node *tree_sitter.Node, src []byte) string { + path := node.Utf8Text(src) + if node.Kind() == utils.KindRustUseAsClause { + path = utils.FieldText(node, utils.FieldPath, src) + } + if i := strings.LastIndex(path, utils.RustPathSeparator); i >= 0 { + return path[:i] + } + return "" +} + +func lastPathSegment(path string) string { + if i := strings.LastIndex(path, utils.RustPathSeparator); i >= 0 { + return path[i+len(utils.RustPathSeparator):] + } + return path +} diff --git a/services/parser/internal/extract/rust_test.go b/services/parser/internal/extract/rust_test.go new file mode 100644 index 0000000..0e7d1f9 --- /dev/null +++ b/services/parser/internal/extract/rust_test.go @@ -0,0 +1,115 @@ +package extract_test + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/testutil" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" +) + +func rustFixture(t *testing.T) ir.Graph { + t.Helper() + return testutil.Extract(t, "../../testdata/lang/rust") +} + +// Closures, match arms and method chains are where a Rust call hides. All +// three are asserted on the calls, because a grammar that cannot read them +// leaves every function_item matching and drops the bodies. +func TestExtract_RustCallsInsideHardConstructs(t *testing.T) { + calls := callsIn(t, rustFixture(t), "repo.rs") + + for _, name := range []string{ + "label", // a method call on self + "wrap", // a free function imported from another module + "peel", // called under its use-alias, not its original name + "iter", // the head of a method chain + "map", // the middle of one + "collect", // the tail + "render", // inside a closure passed to map + "empty", // inside a match arm + } { + assert.Contains(t, calls, name, "call %q lost", name) + } +} + +// A macro body is a token_tree: tree-sitter does not parse expressions inside +// it, so describe(&label) in println! is a bare identifier beside a token_tree +// and not a call at all. +// +// Matching identifiers there would invent a call for every name mentioned in +// every macro. This test pins the limit rather than the wish -- describe IS +// called elsewhere in the fixture, from a match arm, so the assertion is that +// the macro contributes no *second* call site rather than that describe is +// absent entirely. +func TestExtract_RustMacroBodiesAreNotParsed(t *testing.T) { + g := rustFixture(t) + fileID := fileIDOf(t, g, "repo.rs") + + var lines []int + for _, c := range g.Calls { + if c.FileID == fileID && c.CalleeName == "describe" { + lines = append(lines, c.Line) + } + } + + assert.NotContains(t, lines, 11, + "repo.rs:11 is describe(&label) inside println!; a macro body is a token_tree") + assert.NotEmpty(t, lines, "describe is still called from the match arm at least") + + assert.NotContains(t, callsIn(t, g, "repo.rs"), "println", + "a macro invocation is not a call either") +} + +// A method is named after the type its impl block targets. +func TestExtract_RustMethodsCarryTheirImplType(t *testing.T) { + names := qualifiedNames(rustFixture(t)) + + assert.Contains(t, names, "Repo.sync") + assert.Contains(t, names, "Repo.label") + assert.Contains(t, names, "describe", "a free function has no prefix") +} + +// A call inside a closure is attributed through the anonymous closure to the +// function that owns it. +func TestExtract_RustCallerInsideAClosure(t *testing.T) { + g := rustFixture(t) + fileID := fileIDOf(t, g, "repo.rs") + + for _, c := range g.Calls { + if c.FileID == fileID && c.CalleeName == "render" { + assert.Equal(t, "apply."+utils.Anonymous, c.CallerQualified) + return + } + } + t.Fatal("no call to render in repo.rs") +} + +// A `use` binds the alias, because that is the name a call site here writes. +func TestExtract_RustUseDeclarations(t *testing.T) { + g := rustFixture(t) + + byFrom := map[string][]ir.ImportedSymbol{} + for _, imp := range g.Imports { + byFrom[imp.From] = imp.Symbols + } + + assert.Equal(t, + []ir.ImportedSymbol{ + {Local: "wrap", Original: "wrap", Kind: utils.KindNamed}, + {Local: "peel", Original: "unwrap", Kind: utils.KindNamed}, + }, + byFrom["crate::util"], "use crate::util::{wrap, unwrap as peel}") + + assert.Equal(t, + []ir.ImportedSymbol{{Local: "FmtWrite", Original: "Write", Kind: utils.KindNamed}}, + byFrom["std::fmt"], "use std::fmt::Write as FmtWrite") +} + +func TestExtract_RustLanguage(t *testing.T) { + for _, f := range rustFixture(t).Files { + assert.Equal(t, utils.LangRust, f.Language, "%s", f.Path) + } +} diff --git a/services/parser/internal/extract/scope.go b/services/parser/internal/extract/scope.go new file mode 100644 index 0000000..bee0230 --- /dev/null +++ b/services/parser/internal/extract/scope.go @@ -0,0 +1,60 @@ +package extract + +import ( + "strings" + + tree_sitter "github.com/tree-sitter/go-tree-sitter" + + "github.com/ARCoder181105/funcatlas/parser/internal/utils" +) + +// Qualified names are dot-joined and scope-aware: `getUser` at top level, +// `Repo.sync` for a method, `getUser.inner` nested. Which nodes count as a +// scope is the one part a language decides, through Spec.ScopeSegment. + +// qualifiedName builds the dot-joined scope path of a declaration. The walk +// starts at its parent so its own name is not counted twice. +// +// ScopeSegment gets first say on that name. Usually it agrees with the +// captured identifier, but a Go method is `Repo.Sync` and only the declaration +// knows its receiver -- and running it here is what keeps the definition and +// the call site naming the same scope the same way. +func qualifiedName(decl tree_sitter.Node, src []byte, spec *Spec, baseName string) string { + if segment, ok := spec.ScopeSegment(&decl, src); ok { + baseName = segment + } + return strings.Join(append(scopePath(decl.Parent(), src, spec), baseName), ".") +} + +// enclosingQualifiedName returns the qualified name of the nearest enclosing +// scope, or utils.ModuleCaller when the node sits at the top level of the file. +// +// Used for call-site attribution; the definition path calls qualifiedName +// directly because it already knows its own declaration node. +func enclosingQualifiedName(node tree_sitter.Node, src []byte, spec *Spec) string { + parts := scopePath(node.Parent(), src, spec) + if len(parts) == 0 { + return utils.ModuleCaller + } + return strings.Join(parts, ".") +} + +// scopePath walks up from node collecting scope segments, outermost first. +// Stops at a missing or errored ancestor: a malformed file must yield a short +// name, not a panic. +func scopePath(node *tree_sitter.Node, src []byte, spec *Spec) []string { + var parts []string + + for node != nil && !node.IsMissing() && !node.HasError() && node.Id() != 0 { + if segment, ok := spec.ScopeSegment(node, src); ok { + parts = append(parts, segment) + } + node = node.Parent() + } + + // Collected innermost-first; a qualified name reads outermost-first. + for i, j := 0, len(parts)-1; i < j; i, j = i+1, j-1 { + parts[i], parts[j] = parts[j], parts[i] + } + return parts +} diff --git a/services/parser/internal/extract/spec.go b/services/parser/internal/extract/spec.go new file mode 100644 index 0000000..6ec2310 --- /dev/null +++ b/services/parser/internal/extract/spec.go @@ -0,0 +1,41 @@ +package extract + +import ( + tree_sitter "github.com/tree-sitter/go-tree-sitter" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" +) + +// Spec is everything the driver needs to read one language. The per-file loop +// in extract.go is language-agnostic; these are the four things that are not. +type Spec struct { + Name string // written to files.language + Extensions []string + Language func() *tree_sitter.Language + Query string // the embedded .scm + + // ScopeSegment reports whether node names a scope and, if so, the segment it + // contributes to a qualified name. Both the definition walk and the + // call-site walk go through it, so a language states its scope rules once. + ScopeSegment func(node *tree_sitter.Node, src []byte) (string, bool) + + // CalleeReceiver returns a member call's receiver text, "" for a bare call. + // Read from the tree rather than captured in the .scm: an `object:` field + // would make the pattern require one, and bare calls would stop matching. + CalleeReceiver func(callNode tree_sitter.Node, src []byte) string + + // Imports maps an @import.from capture to the module it names and the + // names it binds locally -- what a call site in this file can reference. + // + // It takes the captured node and finds its own way up, because there is no + // shape shared across languages: TypeScript captures a quoted string, Go a + // path inside an import_spec, Rust a `use` argument with no quotes at all. + // A nil symbol list means the match was not an import; some queries have to + // over-capture, since JavaScript's require() is an ordinary call. + Imports func(captured tree_sitter.Node, src []byte) (string, []ir.ImportedSymbol) +} + +// registry is every language the parser reads. Adding one is a Spec here, a +// .scm in queries/, and a fixture pinning the calls inside its hardest +// construct -- the third is not optional; see docs/PARSING_STRATEGY.md. +var registry = []*Spec{&typeScript, &tsx, &javaScript, &jsx, &golang, &rust, &python, &java} diff --git a/services/parser/internal/extract/spec_test.go b/services/parser/internal/extract/spec_test.go new file mode 100644 index 0000000..8af6ce8 --- /dev/null +++ b/services/parser/internal/extract/spec_test.go @@ -0,0 +1,99 @@ +package extract + +import ( + "os" + "path/filepath" + "slices" + "sort" + "testing" + + "go.uber.org/zap" + + "github.com/ARCoder181105/funcatlas/parser/internal/security" +) + +// Every registered language compiles, and its .scm declares all three +// captures. A language whose query is missing @function.call parses files into +// functions with no edges at all, which reads as a working parse -- the exact +// failure .tsx had through Phase 2. This is where that fails loudly instead. +func TestRegistryCompiles(t *testing.T) { + grammars, err := loadGrammars() + if err != nil { + t.Fatalf("loadGrammars: %v", err) + } + defer grammars.Close() + + for _, spec := range registry { + for _, ext := range spec.Extensions { + if grammars[ext] == nil { + t.Errorf("%s claims %q but no grammar is registered for it", spec.Name, ext) + } + } + } +} + +// One extension, one language. Two specs claiming the same extension means one +// silently wins the map, and files of the loser's language are read with the +// wrong grammar -- which fails by dropping calls, not by erroring. +func TestRegistryExtensionsAreUnique(t *testing.T) { + owner := map[string]string{} + for _, spec := range registry { + for _, ext := range spec.Extensions { + if prev, taken := owner[ext]; taken { + t.Errorf("extension %q claimed by both %s and %s", ext, prev, spec.Name) + } + owner[ext] = spec.Name + if ext != filepath.Ext("x"+ext) { + t.Errorf("extension %q is not a bare extension; forFile looks it up with filepath.Ext", ext) + } + } + } +} + +// Exactly the registered extensions are read, and nothing else. +// +// Driven by the registry rather than a hand-kept list, so adding a language +// cannot leave this test asserting last month's set. The unregistered +// extensions are the point: a file type nobody claimed must produce no file +// row at all, not a row parsed with whatever grammar sorted first. +func TestExtract_ReadsExactlyTheRegisteredExtensions(t *testing.T) { + dir := t.TempDir() + + var want []string + for _, spec := range registry { + for _, ext := range spec.Extensions { + name := "sample" + ext + write(t, dir, name) + want = append(want, name) + } + } + for _, ext := range []string{".json", ".md", ".css", ".html", ".txt", ".yaml"} { + write(t, dir, "sample"+ext) + } + + graph, err := Extract(zap.NewNop(), dir, security.Config{ + MaxFiles: 100, MaxFileBytes: 1 << 20, MaxDepth: 10, + }) + if err != nil { + t.Fatalf("Extract: %v", err) + } + + var got []string + for _, f := range graph.Files { + got = append(got, f.Path) + } + sort.Strings(got) + sort.Strings(want) + if !slices.Equal(got, want) { + t.Errorf("read %v, want %v", got, want) + } +} + +func write(t *testing.T, dir, name string) { + t.Helper() + // Contents do not matter: this asserts which files are read, not what is + // found in them. Every language's own fixture covers that. + if err := os.WriteFile(filepath.Join(dir, name), []byte("\n"), 0o644); err != nil { + t.Fatal(err) + } +} diff --git a/services/parser/internal/extract/typescript.go b/services/parser/internal/extract/typescript.go new file mode 100644 index 0000000..6904bc1 --- /dev/null +++ b/services/parser/internal/extract/typescript.go @@ -0,0 +1,125 @@ +package extract + +import ( + tree_sitter "github.com/tree-sitter/go-tree-sitter" + bindings "github.com/tree-sitter/tree-sitter-typescript/bindings/go" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" + "github.com/ARCoder181105/funcatlas/parser/queries" +) + +// .ts and .tsx are one language with two grammars, and they must never share +// one. The TypeScript grammar cannot parse JSX: a component body becomes an +// ERROR node, the function declaration still matches, and every call inside +// the JSX is silently lost. The result looks like a working parse and is +// missing most of its edges. + +var typeScript = Spec{ + Name: utils.LangTypeScript, + Extensions: []string{utils.ExtTS}, + Language: func() *tree_sitter.Language { return tree_sitter.NewLanguage(bindings.LanguageTypescript()) }, + Query: queries.TypeScriptSCM, + ScopeSegment: tsScopeSegment, + CalleeReceiver: tsCalleeReceiver, + Imports: tsImports, +} + +var tsx = Spec{ + Name: utils.LangTSX, + Extensions: []string{utils.ExtTSX}, + Language: func() *tree_sitter.Language { return tree_sitter.NewLanguage(bindings.LanguageTSX()) }, + Query: queries.TypeScriptSCM, + ScopeSegment: tsScopeSegment, + CalleeReceiver: tsCalleeReceiver, + Imports: tsImports, +} + +// tsScopeSegment names the scopes a qualified name is built from: classes, +// functions and methods by their own name, closures by the variable they are +// bound to and otherwise. +func tsScopeSegment(node *tree_sitter.Node, src []byte) (string, bool) { + switch node.Kind() { + case utils.KindClassDecl, utils.KindFunctionDecl, utils.KindMethodDefinition: + return utils.DeclName(node, src), true + + case utils.KindArrowFunction, utils.KindFunctionExpression: + // Only named when bound to a variable: const f = () => {} + if p := node.Parent(); p != nil && p.Kind() == utils.KindVariableDeclarator { + return utils.DeclName(p, src), true + } + return utils.Anonymous, true + } + return "", false +} + +// tsCalleeReceiver returns a member call's receiver: Repo.sync() -> "Repo", +// a.b.c() -> "a.b". Empty for a bare call. +func tsCalleeReceiver(callNode tree_sitter.Node, src []byte) string { + return utils.ParentFieldText(&callNode, utils.KindMemberExpression, utils.FieldObject, src) +} + +// tsImports collects only the names an import binds locally. Walking by node +// kind rather than grabbing every identifier is what keeps `import { a as b }` +// from yielding both a and b. +// +// The capture is the quoted specifier; the statement around it is the parent. +func tsImports(specifier tree_sitter.Node, src []byte) (string, []ir.ImportedSymbol) { + stmt := specifier.Parent() + if stmt == nil { + return "", nil + } + return utils.StringLiteralText(specifier, src), tsImportSymbols(stmt, src) +} + +func tsImportSymbols(stmt *tree_sitter.Node, src []byte) []ir.ImportedSymbol { + // A re-export binds nothing locally; recorded for barrel-following later. + var out []ir.ImportedSymbol + + if stmt.Kind() == utils.KindExportStatement { + clause := utils.ChildByKind(stmt, utils.KindExportClause) + if clause == nil { + return []ir.ImportedSymbol{{Kind: utils.KindReExport}} // export * from "m" + } + utils.NamedChildren(clause, func(spec *tree_sitter.Node) { + if spec.Kind() == utils.KindExportSpecifier { + out = append(out, ir.ImportedSymbol{ + Original: utils.FieldText(spec, utils.FieldName, src), + Kind: utils.KindReExport, + }) + } + }) + return out + } + + clause := utils.ChildByKind(stmt, utils.KindImportClause) + if clause == nil { + return []ir.ImportedSymbol{{Kind: utils.KindSideEffect}} // import "m" + } + + utils.NamedChildren(clause, func(child *tree_sitter.Node) { + switch child.Kind() { + case utils.KindIdentifier: // import def from "m" + out = append(out, ir.ImportedSymbol{Local: child.Utf8Text(src), Kind: utils.KindDefault}) + + case utils.KindNamespaceImport: // import * as ns from "m" + if id := utils.ChildByKind(child, utils.KindIdentifier); id != nil { + out = append(out, ir.ImportedSymbol{Local: id.Utf8Text(src), Kind: utils.KindNamespace}) + } + + case utils.KindNamedImports: // import { a, b as c } from "m" + utils.NamedChildren(child, func(spec *tree_sitter.Node) { + if spec.Kind() != utils.KindImportSpecifier { + return + } + name := utils.FieldText(spec, utils.FieldName, src) + local := utils.FieldText(spec, utils.FieldAlias, src) + if local == "" { + local = name + } + out = append(out, ir.ImportedSymbol{Local: local, Original: name, Kind: utils.KindNamed}) + }) + } + }) + return out +} diff --git a/services/parser/internal/resolver/polyglot_test.go b/services/parser/internal/resolver/polyglot_test.go new file mode 100644 index 0000000..b846770 --- /dev/null +++ b/services/parser/internal/resolver/polyglot_test.go @@ -0,0 +1,127 @@ +package resolver_test + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/ARCoder181105/funcatlas/parser/internal/ir" + "github.com/ARCoder181105/funcatlas/parser/internal/utils" +) + +// The Phase 5 exit test. +// +// testdata/polyglot is one directory holding a file per language, each +// defining and calling a function named `helper`. One directory on purpose: +// package_path is identical across all of them, so a symbol table keyed on +// name and package alone would happily link Go to Python. + +const polyglotDir = "../../testdata/polyglot" + +// ecmaScriptFamily is spelled out here rather than read from +// utils.ResolutionGroup, because a test that measures the boundary with the +// function under test passes when that function is broken. Collapsing every +// language into one group makes src/main.go resolve into src/main.py -- and a +// group-based assertion agrees with itself and notices nothing. +var ecmaScriptFamily = map[string]bool{ + "typescript": true, + "tsx": true, + "javascript": true, + "jsx": true, +} + +// sameLanguageFamily is the only pairing an edge is allowed to span. +func sameLanguageFamily(caller, callee string) bool { + return caller == callee || (ecmaScriptFamily[caller] && ecmaScriptFamily[callee]) +} + +func polyglot(t *testing.T) (ir.Graph, []ir.Edge) { + t.Helper() + return resolveFixture(t, polyglotDir) +} + +// Every language is read, and every one yields both a function and a call. +// Functions alone would pass with a grammar that drops every body -- which is +// exactly how the .tsx bug survived two phases. +func TestPolyglot_EveryLanguageYieldsFunctionsAndCalls(t *testing.T) { + g, _ := polyglot(t) + + functions := map[string]int{} + for _, fn := range g.Functions { + functions[g.Files[fn.FileID].Language]++ + } + calls := map[string]int{} + for _, c := range g.Calls { + calls[g.Files[c.FileID].Language]++ + } + + for _, language := range []string{ + utils.LangTypeScript, utils.LangTSX, utils.LangJavaScript, + utils.LangGo, utils.LangRust, utils.LangPython, utils.LangJava, + } { + assert.NotZero(t, functions[language], "%s yielded no functions", language) + assert.NotZero(t, calls[language], "%s yielded no calls", language) + } +} + +// No edge crosses a resolution group. +// +// Every file here defines `helper`, so a resolver that matched on name alone +// would link main.go's call to main.py's definition. This is the promise the +// phase turns on, and it is asserted over every edge rather than a sample. +func TestPolyglot_NoEdgeCrossesALanguageBoundary(t *testing.T) { + g, edges := polyglot(t) + require.NotEmpty(t, edges) + + for i, e := range edges { + if e.CalleeFuncIdx == utils.NoFunc { + continue + } + caller := g.Files[g.Calls[i].FileID] + callee := g.Files[g.Functions[e.CalleeFuncIdx].FileID] + + assert.True(t, sameLanguageFamily(caller.Language, callee.Language), + "%s:%d calling %q resolved into %s (%s -> %s)", + caller.Path, g.Calls[i].Line, g.Calls[i].CalleeName, callee.Path, + caller.Language, callee.Language) + } +} + +// Outside the ECMAScript family, an exact edge is always same-file. Those +// languages resolve through package clauses, crate paths, sys.path and the +// classpath, and this parser models none of it -- so cross-file gets +// name_match or unresolved, and never a confident answer it cannot back. +func TestPolyglot_CrossFileIsNeverExactOutsideECMAScript(t *testing.T) { + g, edges := polyglot(t) + + for i, e := range edges { + if e.Confidence != utils.Exact { + continue + } + callerFile := g.Calls[i].FileID + calleeFile := g.Functions[e.CalleeFuncIdx].FileID + if callerFile == calleeFile { + continue + } + assert.True(t, ecmaScriptFamily[g.Files[callerFile].Language], + "%s resolved a cross-file call exactly", g.Files[callerFile].Path) + } +} + +// The ECMAScript half of that: .tsx importing .ts does resolve exactly, so the +// partition stopped cross-language matches without breaking cross-file ones. +func TestPolyglot_ECMAScriptStillResolvesAcrossFiles(t *testing.T) { + g, edges := polyglot(t) + + for i, c := range g.Calls { + if g.Files[c.FileID].Path != "src/Widget.tsx" || c.CalleeName != "helper" { + continue + } + assert.Equal(t, utils.Exact, edges[i].Confidence, "Widget.tsx imports helper from ./main") + require.NotEqual(t, utils.NoFunc, edges[i].CalleeFuncIdx) + assert.Equal(t, "src/main.ts", g.Files[g.Functions[edges[i].CalleeFuncIdx].FileID].Path) + return + } + t.Fatal("no call to helper in src/Widget.tsx") +} diff --git a/services/parser/internal/resolver/resolver.go b/services/parser/internal/resolver/resolver.go index ccb1db6..ee95ec6 100644 --- a/services/parser/internal/resolver/resolver.go +++ b/services/parser/internal/resolver/resolver.go @@ -35,6 +35,11 @@ func Resolve(g ir.Graph) []ir.Edge { } // index holds the lookup tables, built once per repo. +// +// The two tables that reach past a single file are keyed by resolution group, +// not just by name. A call in main.go must never match a same-named function +// in main.py, and partitioning here rather than filtering at lookup time means +// there is no code path that can reach a foreign-language candidate at all. type index struct { g ir.Graph @@ -42,12 +47,17 @@ type index struct { funcsInFile map[int][]int // fileID -> function indexes byFileQualified map[int]map[string][]int // fileID -> qualified_name -> indexes - byPkgName map[string]map[string][]int // package_path -> name -> indexes - byName map[string][]int // name -> indexes, repo-wide + byPkgName map[pkgKey]map[string][]int // group+package_path -> name -> indexes + byName map[string]map[string][]int // group -> name -> indexes importsByFile map[int]map[string]importedFrom // fileID -> local -> import } +// pkgKey scopes a package path to its language. A polyglot repository puts +// main.go and main.py in the same directory, so package_path alone is not +// enough to keep their symbols apart. +type pkgKey struct{ group, pkg string } + // importedFrom pairs a local binding with the module it came from. type importedFrom struct { sym ir.ImportedSymbol @@ -60,8 +70,8 @@ func newIndex(g ir.Graph) *index { fileIDByPath: make(map[string]int, len(g.Files)), funcsInFile: make(map[int][]int), byFileQualified: make(map[int]map[string][]int), - byPkgName: make(map[string]map[string][]int), - byName: make(map[string][]int), + byPkgName: make(map[pkgKey]map[string][]int), + byName: make(map[string]map[string][]int), importsByFile: make(map[int]map[string]importedFrom), } @@ -78,15 +88,31 @@ func newIndex(g ir.Graph) *index { x.byFileQualified[fn.FileID][fn.QualifiedName] = append( x.byFileQualified[fn.FileID][fn.QualifiedName], i) - if x.byPkgName[fn.PackagePath] == nil { - x.byPkgName[fn.PackagePath] = make(map[string][]int) + group := x.groupOf(fn.FileID) + + pkg := pkgKey{group: group, pkg: fn.PackagePath} + if x.byPkgName[pkg] == nil { + x.byPkgName[pkg] = make(map[string][]int) } - x.byPkgName[fn.PackagePath][fn.Name] = append(x.byPkgName[fn.PackagePath][fn.Name], i) + x.byPkgName[pkg][fn.Name] = append(x.byPkgName[pkg][fn.Name], i) - x.byName[fn.Name] = append(x.byName[fn.Name], i) + if x.byName[group] == nil { + x.byName[group] = make(map[string][]int) + } + x.byName[group][fn.Name] = append(x.byName[group][fn.Name], i) } + // Imports are recorded in the IR for every language, but only consulted + // where a specifier actually names a file here. Go, Rust, Python and Java + // resolve through package clauses, crate paths, sys.path and the classpath + // -- none of which this package models. Following them by name would answer + // unresolved for a module it cannot reach, where rule 3 still has an honest + // name_match to give. Skipping them at the source keeps that one decision + // in one place, covering resolveMember as well as rule 2. for _, imp := range g.Imports { + if !utils.ResolvesModules(g.Files[imp.FileID].Language) { + continue + } if x.importsByFile[imp.FileID] == nil { x.importsByFile[imp.FileID] = make(map[string]importedFrom) } @@ -114,17 +140,19 @@ func (x *index) resolve(c ir.CallSite) (int, string) { return i, utils.Exact } - // 2. A symbol this file imports. + // 2. A symbol this file imports. Empty for languages whose specifiers name + // no file in this repository -- see newIndex. if imp, ok := x.importsByFile[c.FileID][c.CalleeName]; ok { return x.resolveImported(c.FileID, imp) } - // 3. Package, then repo-wide. Name matching only, hence the weaker tag. - pkg := utils.PackagePath(x.g.Files[c.FileID].Path) + // 3. Package, then group-wide. Name matching only, hence the weaker tag. + group := x.groupOf(c.FileID) + pkg := pkgKey{group: group, pkg: utils.PackagePath(x.g.Files[c.FileID].Path)} if i, ok := utils.Only(x.byPkgName[pkg][c.CalleeName]); ok && !x.overloaded(i) { return i, utils.NameMatch } - if i, ok := utils.Only(x.byName[c.CalleeName]); ok && !x.overloaded(i) { + if i, ok := utils.Only(x.byName[group][c.CalleeName]); ok && !x.overloaded(i) { return i, utils.NameMatch } @@ -206,6 +234,11 @@ func (x *index) lookupScoped(fileID int, callerQualified, name string) (int, boo return utils.NoFunc, false } +// groupOf is the resolution group of the file's language. +func (x *index) groupOf(fileID int) string { + return utils.ResolutionGroup(x.g.Files[fileID].Language) +} + // uniqueQualified returns the one function with this qualified name in this // file. Two means an overload, which name matching cannot choose between. func (x *index) uniqueQualified(fileID int, qualified string) (int, bool) { diff --git a/services/parser/internal/resolver/resolver_test.go b/services/parser/internal/resolver/resolver_test.go index 077a007..12e8285 100644 --- a/services/parser/internal/resolver/resolver_test.go +++ b/services/parser/internal/resolver/resolver_test.go @@ -146,3 +146,125 @@ func TestResolve_ConfidenceMatchesCalleePresence(t *testing.T) { } } } + +// crossLanguage is two files in one directory, each defining and calling a +// function named helper. Built by hand rather than from a fixture directory: +// this is the resolver's own guarantee, and it has to hold for every language +// the extractor learns, including ones it does not read yet. +func crossLanguage(callerLang, calleeLang string) ir.Graph { + return ir.Graph{ + Files: []ir.File{ + {Path: "src/main." + callerLang, Language: callerLang}, + {Path: "src/other." + calleeLang, Language: calleeLang}, + }, + Functions: []ir.Function{ + {FileID: 0, PackagePath: "src", Name: "run", QualifiedName: "run", StartLine: 1, EndLine: 3}, + {FileID: 1, PackagePath: "src", Name: "helper", QualifiedName: "helper", StartLine: 1, EndLine: 2}, + }, + Calls: []ir.CallSite{ + {FileID: 0, CallerQualified: "run", CalleeName: "helper", Line: 2}, + }, + } +} + +// A call in one language must never match a same-named function in another. +// The two repo-wide lookups are keyed by resolution group for exactly this, +// and the same package path is the case that would otherwise slip through. +func TestResolve_NeverCrossesALanguageBoundary(t *testing.T) { + languages := []string{ + utils.LangTypeScript, utils.LangTSX, utils.LangJavaScript, utils.LangJSX, + utils.LangGo, utils.LangRust, utils.LangPython, utils.LangJava, + } + + for _, caller := range languages { + for _, callee := range languages { + // Spelled out in sameLanguageFamily rather than asked of + // ResolutionGroup: a test that decides what to skip by calling the + // function under test skips everything when that function breaks. + if sameLanguageFamily(caller, callee) { + continue + } + g := crossLanguage(caller, callee) + edges := resolver.Resolve(g) + require.Len(t, edges, 1) + assert.Equal(t, utils.Unresolved, edges[0].Confidence, + "a call in %s resolved to a function in %s", caller, callee) + assert.Equal(t, utils.NoFunc, edges[0].CalleeFuncIdx) + assert.Equal(t, "helper", edges[0].CalleeName, + "an unresolved edge still says what it failed to resolve") + } + } +} + +// Within a group it still resolves, or the partition would have broken .ts +// calling .tsx rather than only stopping cross-language matches. +func TestResolve_MatchesWithinAResolutionGroup(t *testing.T) { + g := crossLanguage(utils.LangTypeScript, utils.LangTSX) + edges := resolver.Resolve(g) + + require.Len(t, edges, 1) + assert.Equal(t, utils.NameMatch, edges[0].Confidence) + assert.Equal(t, "helper", target(g, edges[0])) +} + +// Imports are extracted for every language but only followed where a specifier +// names a file here. Without that, a Python `from utils import helper` would +// enter rule 2, fail to find a module, and answer unresolved -- throwing away +// the name_match rule 3 would have given. +func TestResolve_ImportsAreOnlyFollowedWhereTheyNameAFile(t *testing.T) { + g := crossLanguage(utils.LangPython, utils.LangPython) + g.Imports = []ir.Import{{ + FileID: 0, + From: "utils", + Symbols: []ir.ImportedSymbol{{Local: "helper", Original: "helper", Kind: utils.KindNamed}}, + }} + + edges := resolver.Resolve(g) + require.Len(t, edges, 1) + assert.Equal(t, utils.NameMatch, edges[0].Confidence, + "an unfollowable import must not downgrade a call rule 3 can still answer") + assert.Equal(t, "helper", target(g, edges[0])) +} + +// A genuine overload resolves to nothing. +// +// Java's Repo.sync() and Repo.sync(int) share a qualified name, and picking +// between them needs argument types this parser does not have. uniqueQualified +// answers "many, therefore none" -- the whole point of Only over indexing. +func TestResolve_JavaOverloadIsUnresolved(t *testing.T) { + g, edges := resolveFixture(t, "../../testdata/lang/java") + + var found bool + for i, c := range g.Calls { + if c.CalleeName != "sync" { + continue + } + found = true + assert.Equal(t, utils.Unresolved, edges[i].Confidence, + "two overloads of sync; choosing one would be a guess") + assert.Equal(t, utils.NoFunc, edges[i].CalleeFuncIdx) + } + require.True(t, found, "no call to sync in the Java fixture") +} + +// Cross-file calls outside the ECMAScript family are never exact: those +// languages resolve through packages, crate paths, sys.path and the classpath, +// and none of that is modelled here. +func TestResolve_NonECMAScriptCrossFileIsNeverExact(t *testing.T) { + for _, dir := range []string{ + "../../testdata/lang/go", + "../../testdata/lang/rust", + "../../testdata/lang/python", + "../../testdata/lang/java", + } { + g, edges := resolveFixture(t, dir) + for i, e := range edges { + if e.CalleeFuncIdx == utils.NoFunc || e.Confidence != utils.Exact { + continue + } + assert.Equal(t, + g.Calls[i].FileID, g.Functions[e.CalleeFuncIdx].FileID, + "%s: an exact edge outside ECMAScript must be same-file", dir) + } + } +} diff --git a/services/parser/internal/security/config.go b/services/parser/internal/security/config.go index f83706e..55a81fa 100644 --- a/services/parser/internal/security/config.go +++ b/services/parser/internal/security/config.go @@ -23,7 +23,12 @@ func ConfigFromEnv() Config { MaxFileBytes: envInt64("PARSER_MAX_FILE_BYTES", 1<<20), // 1MB MaxFiles: envInt("PARSER_MAX_FILES", 50000), MaxDepth: envInt("PARSER_MAX_DEPTH", 25), - SkipPaths: envSet("PARSER_SKIP_PATHS", "node_modules,.git,dist,build,coverage,.next"), + // Dependency and build directories, one set per language. Vendored code + // is not this repository's graph, and it is what the file cap is spent + // on if it is walked. + SkipPaths: envSet("PARSER_SKIP_PATHS", + "node_modules,.git,dist,build,coverage,.next,"+ + "vendor,target,__pycache__,.venv,venv,.gradle,.mypy_cache,.tox"), } } diff --git a/services/parser/internal/testutil/testutil.go b/services/parser/internal/testutil/testutil.go index c1ee6fe..c908899 100644 --- a/services/parser/internal/testutil/testutil.go +++ b/services/parser/internal/testutil/testutil.go @@ -7,9 +7,9 @@ import ( "github.com/stretchr/testify/require" "go.uber.org/zap" + "github.com/ARCoder181105/funcatlas/parser/internal/extract" "github.com/ARCoder181105/funcatlas/parser/internal/ir" "github.com/ARCoder181105/funcatlas/parser/internal/security" - "github.com/ARCoder181105/funcatlas/parser/internal/ts" ) // Config is the security config for fixtures: limits high enough that nothing @@ -29,7 +29,7 @@ func Config() security.Config { // Extract parses a fixture directory into the IR. func Extract(t *testing.T, dir string) ir.Graph { t.Helper() - g, err := ts.Extract(zap.NewNop(), dir, Config()) + g, err := extract.Extract(zap.NewNop(), dir, Config()) require.NoError(t, err) return g } diff --git a/services/parser/internal/ts/grammar.go b/services/parser/internal/ts/grammar.go deleted file mode 100644 index 5c2ea81..0000000 --- a/services/parser/internal/ts/grammar.go +++ /dev/null @@ -1,73 +0,0 @@ -package ts - -import ( - "fmt" - - tree_sitter "github.com/tree-sitter/go-tree-sitter" - bindings "github.com/tree-sitter/tree-sitter-typescript/bindings/go" - - "github.com/ARCoder181105/funcatlas/parser/internal/utils" -) - -// grammar pairs a parser with the queries compiled against its language. -// -// .ts and .tsx need separate ones. The TypeScript grammar cannot parse JSX: a -// component body becomes an ERROR node, the function declaration still matches, -// and every call inside the JSX is silently lost. The result looks like a -// working parse and is missing most of its edges. -type grammar struct { - parser *tree_sitter.Parser - queries *compiledQueries -} - -func (g *grammar) Close() { - g.queries.Close() - g.parser.Close() -} - -// grammars holds one entry per source extension. -type grammars map[string]*grammar - -func (g grammars) Close() { - for _, entry := range g { - entry.Close() - } -} - -// forFile returns the grammar for a file's extension, or nil to skip it. -func (g grammars) forFile(path string) *grammar { - for ext, entry := range g { - if len(path) > len(ext) && path[len(path)-len(ext):] == ext { - return entry - } - } - return nil -} - -// loadGrammars compiles every grammar and its queries once per run. -func loadGrammars() (grammars, error) { - out := make(grammars, 2) - - for ext, lang := range map[string]*tree_sitter.Language{ - utils.ExtTS: tree_sitter.NewLanguage(bindings.LanguageTypescript()), - utils.ExtTSX: tree_sitter.NewLanguage(bindings.LanguageTSX()), - } { - parser := tree_sitter.NewParser() - if err := parser.SetLanguage(lang); err != nil { - out.Close() - parser.Close() - return nil, fmt.Errorf("set language for %s: %w", ext, err) - } - - qs, err := loadQueries(lang) - if err != nil { - out.Close() - parser.Close() - return nil, fmt.Errorf("queries for %s: %w", ext, err) - } - - out[ext] = &grammar{parser: parser, queries: qs} - } - - return out, nil -} diff --git a/services/parser/internal/ts/queries.go b/services/parser/internal/ts/queries.go deleted file mode 100644 index cd46554..0000000 --- a/services/parser/internal/ts/queries.go +++ /dev/null @@ -1,61 +0,0 @@ -package ts - -import ( - "fmt" - - tree_sitter "github.com/tree-sitter/go-tree-sitter" - - "github.com/ARCoder181105/funcatlas/parser/queries" -) - -// compiledQueries holds the compiled tree-sitter queries for TypeScript. -// One loader call per run (per language); reused across every file. -type compiledQueries struct { - def *tree_sitter.Query // @function.def - call *tree_sitter.Query // @function.call - imp *tree_sitter.Query // @import.from -} - -func (q *compiledQueries) Close() { - q.def.Close() - q.call.Close() - q.imp.Close() -} - -// loadQueries compiles the bundled TypeScript .scm once per run. -// Returns the compiled queries or an error if any capture name is missing. -func loadQueries(lang *tree_sitter.Language) (*compiledQueries, error) { - src := queries.TypeScriptSCM - - def, err := compileOne(lang, src, "function.def") - if err != nil { - return nil, fmt.Errorf("function.def: %w", err) - } - call, err := compileOne(lang, src, "function.call") - if err != nil { - def.Close() - return nil, fmt.Errorf("function.call: %w", err) - } - imp, err := compileOne(lang, src, "import.from") - if err != nil { - def.Close() - call.Close() - return nil, fmt.Errorf("import.from: %w", err) - } - return &compiledQueries{def: def, call: call, imp: imp}, nil -} - -// compileOne compiles the whole .scm against the language and verifies the -// requested capture name exists. Returns the compiled Query (which runs ALL -// patterns; we filter by capture index in C2/C5/C6). -func compileOne(lang *tree_sitter.Language, src, capture string) (*tree_sitter.Query, error) { - q, qerr := tree_sitter.NewQuery(lang, src) - if qerr != nil { - return nil, qerr - } - if _, ok := q.CaptureIndexForName(capture); !ok { - q.Close() - return nil, fmt.Errorf("capture @%s not found in query", capture) - } - return q, nil -} diff --git a/services/parser/internal/ts/queries_test.go b/services/parser/internal/ts/queries_test.go deleted file mode 100644 index 25456f1..0000000 --- a/services/parser/internal/ts/queries_test.go +++ /dev/null @@ -1,17 +0,0 @@ -package ts - -import ( - "testing" - - tree_sitter "github.com/tree-sitter/go-tree-sitter" - bindings "github.com/tree-sitter/tree-sitter-typescript/bindings/go" -) - -func TestLoadQueriesCompiles(t *testing.T) { - lang := tree_sitter.NewLanguage(bindings.LanguageTypescript()) - qs, err := loadQueries(lang) - if err != nil { - t.Fatalf("loadQueries: %v", err) - } - defer qs.Close() -} diff --git a/services/parser/internal/ts/scope.go b/services/parser/internal/ts/scope.go deleted file mode 100644 index a0cd5a4..0000000 --- a/services/parser/internal/ts/scope.go +++ /dev/null @@ -1,77 +0,0 @@ -package ts - -import ( - "strings" - - tree_sitter "github.com/tree-sitter/go-tree-sitter" - - "github.com/ARCoder181105/funcatlas/parser/internal/utils" -) - -// qualifiedName walks up the AST from a node to build a dot-joined scope path. -// The node passed in is the declaration itself; its own name comes in as -// baseName, so the walk starts at the parent and never double-counts it. -func qualifiedName(node tree_sitter.Node, src []byte, baseName string) string { - parts := []string{baseName} - - parent := node.Parent() - for parent != nil && !parent.IsMissing() && !parent.HasError() && parent.Id() != 0 { - curr := *parent - - switch curr.Kind() { - case utils.KindClassDecl, utils.KindFunctionDecl, utils.KindMethodDefinition: - parts = append(parts, utils.DeclName(&curr, src)) - case utils.KindArrowFunction, utils.KindFunctionExpression: - // Only named when bound to a variable: const f = () => {} - if p := curr.Parent(); p != nil && p.Kind() == utils.KindVariableDeclarator { - parts = append(parts, utils.DeclName(p, src)) - } else { - parts = append(parts, utils.Anonymous) - } - } - - parent = curr.Parent() - } - - // Collected innermost-first; a qualified name reads outermost-first. - for i, j := 0, len(parts)-1; i < j; i, j = i+1, j-1 { - parts[i], parts[j] = parts[j], parts[i] - } - - return strings.Join(parts, ".") -} - -// enclosingQualifiedName returns the qualified name of the nearest function, -// method, or variable-bound closure containing node, or utils.ModuleCaller when -// the node sits at the top level of the file. -// -// Used for call-site attribution; the definition path calls qualifiedName -// directly because it already knows its own declaration node. -func enclosingQualifiedName(node tree_sitter.Node, src []byte) string { - decl := enclosingDecl(node) - if decl == nil { - return utils.ModuleCaller - } - return qualifiedName(*decl, src, utils.DeclName(decl, src)) -} - -// enclosingDecl walks up to the nearest node that names a callable scope. An -// arrow function or function expression bound to a variable reports the -// variable_declarator, so it is named after the variable rather than being -// anonymous. -func enclosingDecl(node tree_sitter.Node) *tree_sitter.Node { - parent := node.Parent() - for parent != nil && !parent.IsMissing() && !parent.HasError() && parent.Id() != 0 { - switch parent.Kind() { - case utils.KindFunctionDecl, utils.KindMethodDefinition: - return parent - case utils.KindArrowFunction, utils.KindFunctionExpression: - if p := parent.Parent(); p != nil && p.Kind() == utils.KindVariableDeclarator { - return p - } - return parent - } - parent = parent.Parent() - } - return nil -} diff --git a/services/parser/internal/utils/constants.go b/services/parser/internal/utils/constants.go index d8763eb..a4d975b 100644 --- a/services/parser/internal/utils/constants.go +++ b/services/parser/internal/utils/constants.go @@ -3,8 +3,57 @@ package utils // Every shared literal in the parser. One home, so a string like "unresolved" // is never spelled twice and cannot drift from the database CHECK constraint. -// Language is the only language the MVP parses. -const Language = "typescript" +// Languages, as written to files.language. One name per grammar-and-extension +// pairing, so the value always says which grammar actually read the file. +const ( + LangTypeScript = "typescript" + LangTSX = "tsx" + LangJavaScript = "javascript" + LangJSX = "jsx" + LangGo = "go" + LangRust = "rust" + LangPython = "python" + LangJava = "java" +) + +// GroupECMAScript is the one resolution group holding more than one language. +// Every other language is its own group, named after itself. +const GroupECMAScript = "ecmascript" + +// resolutionGroups lists only the languages that share a group with another. +var resolutionGroups = map[string]string{ + LangTypeScript: GroupECMAScript, + LangTSX: GroupECMAScript, + LangJavaScript: GroupECMAScript, + LangJSX: GroupECMAScript, +} + +// ResolutionGroup partitions the resolver's symbol table. A call in main.go +// must never match a same-named function in main.py, and partitioning by this +// at index-build time is what makes that structural rather than a filter +// somebody can forget to apply. +func ResolutionGroup(language string) string { + if group, ok := resolutionGroups[language]; ok { + return group + } + return language +} + +// ResolvesModules reports whether a language's import specifiers name files in +// this repository. Only the ECMAScript family's do. For the rest, consulting +// imports would answer "unresolved" for a module the resolver simply cannot +// follow, where falling through to name matching is the more honest answer. +func ResolvesModules(language string) bool { + return ResolutionGroup(language) == GroupECMAScript +} + +// Capture names every language's .scm must declare. Missing one is a compile +// error at load time rather than a file that parses to nothing. +const ( + CaptureFunctionDef = "function.def" + CaptureFunctionCall = "function.call" + CaptureImportFrom = "import.from" +) // DefaultBranch is recorded when a checkout reports no branch of its own -- // a detached head, or a path that is not a git repository at all. @@ -24,6 +73,11 @@ const ( Unresolved = "unresolved" ) +// ConfidenceTiers is every tier, most confident first -- the order a summary +// reads in. Here rather than at the call site, so the three are never spelled +// out a second time and cannot drift from the constants above. +var ConfidenceTiers = []string{Exact, NameMatch, Unresolved} + // NoFunc marks an edge endpoint that is not a function in this repo: // an unresolved callee, or a caller at module level. const NoFunc = -1 @@ -38,19 +92,60 @@ const ( ) // Source extensions. .tsx needs its own grammar -- the TypeScript grammar -// cannot parse JSX, and fails silently by dropping calls inside it. +// cannot parse JSX, and fails silently by dropping calls inside it. The +// JavaScript grammar has no such split. const ( - ExtTS = ".ts" - ExtTSX = ".tsx" + ExtTS = ".ts" + ExtTSX = ".tsx" + ExtJS = ".js" + ExtJSX = ".jsx" + ExtMJS = ".mjs" + ExtCJS = ".cjs" + ExtGo = ".go" + ExtRust = ".rs" + ExtPython = ".py" + ExtJava = ".java" ) -// Extensions a module specifier is resolved against, in order. +// Extensions a module specifier is resolved against, in order. TypeScript +// first: ESM TypeScript writes ./foo.js for what is really ./foo.ts, so the +// stripped candidates have to be tried before the literal one. var ( - SourceExtensions = []string{ExtTS, ExtTSX} - IndexFiles = []string{"/index" + ExtTS, "/index" + ExtTSX} + SourceExtensions = []string{ExtTS, ExtTSX, ExtJS, ExtJSX, ExtMJS, ExtCJS} + IndexFiles = indexFilesFor(SourceExtensions) +) + +func indexFilesFor(extensions []string) []string { + out := make([]string, 0, len(extensions)) + for _, ext := range extensions { + out = append(out, "/index"+ext) + } + return out +} + +// Tree-sitter *field* names, as opposed to node kinds below. Every grammar +// names its fields from the same small vocabulary, so these are shared rather +// than grouped by language -- what differs is which node carries which field. +// +// One home for the same reason as the kinds: a grammar that renames a field +// drops whatever read it, silently, since ChildByFieldName just returns nil. +const ( + FieldAlias = "alias" + FieldFunction = "function" + FieldKey = "key" + FieldList = "list" + FieldModuleName = "module_name" + FieldName = "name" + FieldObject = "object" + FieldOperand = "operand" + FieldPath = "path" + FieldReceiver = "receiver" + FieldType = "type" + FieldValue = "value" ) // Tree-sitter node kinds, so a grammar rename breaks in one place. +// Grouped by language: these are TypeScript's, shared with JSX. const ( KindIdentifier = "identifier" KindMemberExpression = "member_expression" @@ -69,6 +164,84 @@ const ( KindExportSpecifier = "export_specifier" ) +// JavaScript adds CommonJS: an import that is a call, and destructuring on the +// left of it rather than an import clause. +const ( + KindArguments = "arguments" + KindObjectPattern = "object_pattern" + KindPairPattern = "pair_pattern" + KindShorthandPropertyIdentifierPattern = "shorthand_property_identifier_pattern" +) + +// RequireCallee is the only call the import query's specifier capture keeps. +const RequireCallee = "require" + +// Go. A method's name is a field_identifier, and its receiver type is what +// makes Repo.Sync distinguishable from a package-level Sync. +const ( + KindGoFuncDecl = "function_declaration" + KindGoMethodDecl = "method_declaration" + KindGoFuncLiteral = "func_literal" + KindGoSelectorExpression = "selector_expression" + KindGoTypeIdentifier = "type_identifier" +) + +// Import aliases that bind no usable local name. +const ( + GoBlankImport = "_" + GoDotImport = "." +) + +// Rust. A `use` is a nested path expression rather than a quoted specifier, +// and a method's owner is the type its impl block targets. +const ( + KindRustFunctionItem = "function_item" + KindRustImplItem = "impl_item" + KindRustClosureExpression = "closure_expression" + KindRustFieldExpression = "field_expression" + KindRustTypeIdentifier = "type_identifier" + KindRustIdentifier = "identifier" + KindRustScopedIdentifier = "scoped_identifier" + KindRustScopedUseList = "scoped_use_list" + KindRustUseAsClause = "use_as_clause" + KindRustUseWildcard = "use_wildcard" + KindRustSelf = "self" +) + +// RustPathSeparator joins the segments of a use path. +const RustPathSeparator = "::" + +// Python. A decorated definition wraps the function_definition rather than +// replacing it, and an import statement has no single specifier node. +const ( + KindPyFunctionDef = "function_definition" + KindPyClassDef = "class_definition" + KindPyLambda = "lambda" + KindPyAttribute = "attribute" + KindPyImportFrom = "import_from_statement" + KindPyAliasedImport = "aliased_import" + KindPyDottedName = "dotted_name" +) + +// Java. Every enclosing type names a method, and an anonymous inner class is a +// class_body hanging off an object_creation_expression. +const ( + KindJavaMethodDecl = "method_declaration" + KindJavaConstructorDecl = "constructor_declaration" + KindJavaClassDecl = "class_declaration" + KindJavaInterfaceDecl = "interface_declaration" + KindJavaEnumDecl = "enum_declaration" + KindJavaRecordDecl = "record_declaration" + KindJavaLambda = "lambda_expression" + KindJavaObjectCreation = "object_creation_expression" + KindJavaClassBody = "class_body" + KindJavaMethodInvocation = "method_invocation" + KindJavaScopedIdentifier = "scoped_identifier" +) + +// JavaWildcardImport binds every type in a package under no local name. +const JavaWildcardImport = ".*;" + // InsertChunkSize caps rows per multi-row INSERT. Postgres allows 65535 bind // parameters; functions is 9 columns wide, so 500 rows is ~4500. const InsertChunkSize = 500 diff --git a/services/parser/internal/utils/nodes.go b/services/parser/internal/utils/nodes.go index 03ccc55..12fc1ff 100644 --- a/services/parser/internal/utils/nodes.go +++ b/services/parser/internal/utils/nodes.go @@ -2,7 +2,11 @@ // concern: constants.go, nodes.go, paths.go, qualnames.go, slices.go. package utils -import tree_sitter "github.com/tree-sitter/go-tree-sitter" +import ( + "strings" + + tree_sitter "github.com/tree-sitter/go-tree-sitter" +) // Tree-sitter traversal. Every helper tolerates a nil node -- a malformed file // from an untrusted repo must skip, not panic. @@ -34,12 +38,51 @@ func FieldText(node *tree_sitter.Node, field string, src []byte) string { // DeclName returns a declaration's name, or Anonymous if it has none. func DeclName(node *tree_sitter.Node, src []byte) string { - if name := FieldText(node, "name", src); name != "" { + if name := FieldText(node, FieldName, src); name != "" { return name } return Anonymous } +// ParentFieldText returns a field of node's parent, but only when the parent is +// of the given kind. How every language reads a member call's receiver: +// Repo.sync() -> "Repo", with the callee capture pointing at `sync`. +func ParentFieldText(node *tree_sitter.Node, parentKind, field string, src []byte) string { + if node == nil { + return "" + } + parent := node.Parent() + if parent == nil || parent.Kind() != parentKind { + return "" + } + return FieldText(parent, field, src) +} + +// FirstDescendantByKind returns the first named descendant of the given kind, +// depth-first. How a receiver or an impl target is named: Go writes *Repo, +// Repo or Repo[T] and only the type_identifier inside is the name. +func FirstDescendantByKind(node *tree_sitter.Node, kind string) *tree_sitter.Node { + if node == nil { + return nil + } + if node.Kind() == kind { + return node + } + for i := uint(0); i < node.NamedChildCount(); i++ { + if found := FirstDescendantByKind(node.NamedChild(i), kind); found != nil { + return found + } + } + return nil +} + +// StringLiteralText is a string literal's contents, without the quotes +// tree-sitter includes in the node. Covers every quoting style the parsed +// languages use for an import specifier. +func StringLiteralText(node tree_sitter.Node, src []byte) string { + return strings.Trim(node.Utf8Text(src), "\"'`") +} + // NamedChildren calls fn for each named child, skipping nils. func NamedChildren(node *tree_sitter.Node, fn func(*tree_sitter.Node)) { if node == nil { diff --git a/services/parser/internal/utils/paths.go b/services/parser/internal/utils/paths.go index 4eb3569..2d14fa3 100644 --- a/services/parser/internal/utils/paths.go +++ b/services/parser/internal/utils/paths.go @@ -30,19 +30,26 @@ func ModuleCandidates(fromPath, spec string) []string { return nil // escaped the repo root } - // ESM TypeScript writes ./foo.js for what is really ./foo.ts. - if ext := path.Ext(base); ext == ".js" || ext == ".jsx" { - base = strings.TrimSuffix(base, ext) + // ESM TypeScript writes ./foo.js for what is really ./foo.ts, so the + // stripped stem is tried first -- but the literal path has to stay a + // candidate too, or a specifier naming a real .js file resolves to nothing. + stem := base + if ext := path.Ext(base); ext == ExtJS || ext == ExtJSX { + stem = strings.TrimSuffix(base, ext) } - out := make([]string, 0, len(SourceExtensions)+len(IndexFiles)+1) + out := make([]string, 0, len(SourceExtensions)+len(IndexFiles)+2) for _, ext := range SourceExtensions { - out = append(out, base+ext) + out = append(out, stem+ext) } for _, index := range IndexFiles { - out = append(out, base+index) + out = append(out, stem+index) } - return append(out, base) // may already carry an explicit extension + out = append(out, base) // may already carry an explicit extension + if stem != base { + out = append(out, stem) + } + return out } // IsRelativeSpecifier reports whether a specifier points inside the repo. @@ -50,13 +57,3 @@ func IsRelativeSpecifier(spec string) bool { return spec == "." || spec == ".." || strings.HasPrefix(spec, "./") || strings.HasPrefix(spec, "../") } - -// IsSourceFile reports whether a path is a TypeScript file the parser handles. -func IsSourceFile(filePath string) bool { - for _, ext := range SourceExtensions { - if strings.HasSuffix(filePath, ext) { - return true - } - } - return false -} diff --git a/services/parser/queries/embed.go b/services/parser/queries/embed.go index cfcd737..81741a0 100644 --- a/services/parser/queries/embed.go +++ b/services/parser/queries/embed.go @@ -5,4 +5,19 @@ package queries import _ "embed" //go:embed typescript.scm -var TypeScriptSCM string \ No newline at end of file +var TypeScriptSCM string + +//go:embed javascript.scm +var JavaScriptSCM string + +//go:embed go.scm +var GoSCM string + +//go:embed rust.scm +var RustSCM string + +//go:embed python.scm +var PythonSCM string + +//go:embed java.scm +var JavaSCM string diff --git a/services/parser/queries/go.scm b/services/parser/queries/go.scm new file mode 100644 index 0000000..23dec2e --- /dev/null +++ b/services/parser/queries/go.scm @@ -0,0 +1,25 @@ +; tree-sitter queries for Go extraction. +; +; Deliberately absent: a generic call with one type argument. Map[int](xs) +; parses as a type_conversion_expression, the same shape as int(x) -- capturing +; it would invent a call for every conversion in the repository. Two or more +; type arguments are unambiguous and do parse as a call_expression, so those +; are matched by the plain identifier pattern below. + +(function_declaration + name: (identifier) @function.def + body: (block) @function.block) + +(method_declaration + name: (field_identifier) @function.def + body: (block) @function.block) + +(call_expression + function: [ + (identifier) @function.call + (selector_expression + field: (field_identifier) @function.call) + ]) + +(import_spec + path: (_) @import.from) diff --git a/services/parser/queries/java.scm b/services/parser/queries/java.scm new file mode 100644 index 0000000..a3fb9d7 --- /dev/null +++ b/services/parser/queries/java.scm @@ -0,0 +1,21 @@ +; tree-sitter queries for Java extraction. +; +; A method inside an anonymous inner class is an ordinary method_declaration in +; the class_body of an object_creation_expression, so it matches here and the +; scope walk gives it the segment. Same for a lambda body. +; +; The whole import declaration is captured: it has no quoted specifier, only a +; scoped_identifier that may or may not be static and may end in an asterisk. + +(method_declaration + name: (identifier) @function.def + body: (block) @function.block) + +(constructor_declaration + name: (identifier) @function.def + body: (constructor_body) @function.block) + +(method_invocation + name: (identifier) @function.call) + +(import_declaration) @import.from diff --git a/services/parser/queries/javascript.scm b/services/parser/queries/javascript.scm new file mode 100644 index 0000000..8f7cfe1 --- /dev/null +++ b/services/parser/queries/javascript.scm @@ -0,0 +1,37 @@ +; tree-sitter queries for JavaScript and JSX extraction. +; Shares TypeScript's node kinds; require() is the one addition. + +(function_declaration + name: (identifier) @function.def + body: (statement_block) @function.block) + +(method_definition + name: (property_identifier) @function.def + body: (statement_block) @function.block) + +(call_expression + function: [ + (identifier) @function.call + (member_expression + property: (property_identifier) @function.call) + ]) + +(import_statement + source: (string) @import.from) + +(export_statement + source: (string) @import.from) + +(variable_declarator + name: (identifier) @function.def + value: [ + (arrow_function) + (function_expression) + ]) + +; CommonJS. The .scm cannot say "only require()" without a predicate the Go +; binding does not evaluate, so it captures any single-string call argument and +; jsImports discards the ones that are not require. +(call_expression + function: (identifier) + arguments: (arguments (string) @import.from)) diff --git a/services/parser/queries/python.scm b/services/parser/queries/python.scm new file mode 100644 index 0000000..465fdc6 --- /dev/null +++ b/services/parser/queries/python.scm @@ -0,0 +1,24 @@ +; tree-sitter queries for Python extraction. +; +; `async def` is a function_definition like any other, and a decorated one is a +; function_definition wrapped in a decorated_definition -- so the name's parent +; is still the definition and the line range is still the body's, not the +; decorator's. Both are pinned by the fixture. +; +; The whole import statement is captured rather than a specifier: `import a.b` +; and `from .m import x as y` share no node to point at. + +(function_definition + name: (identifier) @function.def + body: (block) @function.block) + +(call + function: [ + (identifier) @function.call + (attribute + attribute: (identifier) @function.call) + ]) + +(import_statement) @import.from + +(import_from_statement) @import.from diff --git a/services/parser/queries/rust.scm b/services/parser/queries/rust.scm new file mode 100644 index 0000000..a71d098 --- /dev/null +++ b/services/parser/queries/rust.scm @@ -0,0 +1,23 @@ +; tree-sitter queries for Rust extraction. +; +; Deliberately absent: anything inside a macro. println!("{}", helper()) has a +; token_tree body -- tree-sitter does not parse expressions in it, so helper() +; is a bare identifier next to a token_tree and not a call at all. Matching +; identifiers there would invent a call for every name mentioned in every +; macro. The limit is pinned by the fixture; see docs/PARSING_STRATEGY.md. + +(function_item + name: (identifier) @function.def + body: (block) @function.block) + +(call_expression + function: [ + (identifier) @function.call + (field_expression + field: (field_identifier) @function.call) + (scoped_identifier + name: (identifier) @function.call) + ]) + +(use_declaration + argument: (_) @import.from) diff --git a/services/parser/testdata/lang/go/repo.go b/services/parser/testdata/lang/go/repo.go new file mode 100644 index 0000000..dd011c8 --- /dev/null +++ b/services/parser/testdata/lang/go/repo.go @@ -0,0 +1,59 @@ +package store + +import ( + "fmt" + stdsync "sync" + + "example.com/app/internal/util" +) + +type Repo struct { + mu stdsync.Mutex +} + +// Sync is a pointer-receiver method: its qualified name has to carry Repo. +func (r *Repo) Sync(id string) error { + defer r.unlock() + go func() { + notify(id) + }() + return util.Wrap(fmt.Errorf("sync %s", id)) +} + +func (r Repo) unlock() { + r.mu.Unlock() +} + +func notify(id string) { + println(id) +} + +// Map is generic. A call to it is written Map[int](xs, f), which parses as a +// call over an index_expression rather than over a plain identifier. +func Map[T any, U any](in []T, fn func(T) U) []U { + out := make([]U, 0, len(in)) + for _, v := range in { + out = append(out, fn(v)) + } + return out +} + +func useGenerics(xs []int) []string { + return Map[int, string](xs, describe) +} + +func describe(v int) string { + return fmt.Sprint(v) +} + +// A single type argument is genuinely ambiguous with a conversion: tree-sitter +// parses Map[int](xs) as a type_conversion_expression, identical in shape to +// int(x). Capturing it would invent a call for every conversion in the repo, +// so it is dropped -- unresolved by omission rather than a guess. +func useSingleTypeArgument(xs []int) []int { + return Identity[int](xs) +} + +func Identity[T any](in []T) []T { + return in +} diff --git a/services/parser/testdata/lang/java/Repo.java b/services/parser/testdata/lang/java/Repo.java new file mode 100644 index 0000000..84d1dd5 --- /dev/null +++ b/services/parser/testdata/lang/java/Repo.java @@ -0,0 +1,50 @@ +package com.example.store; + +import java.util.List; +import java.util.function.Supplier; +import static com.example.util.Text.wrap; + +public class Repo { + private final String id; + + public Repo(String id) { + this.id = id; + } + + // Two genuine overloads: same name, same class, different signatures. + // A call to sync() cannot be attributed to one of them by name alone. + public String sync() { + return sync(1); + } + + public String sync(int attempts) { + return wrap(describe(attempts)); + } + + public Runnable task() { + return new Runnable() { + @Override + public void run() { + helper(); + } + }; + } + + public Supplier lazy() { + return () -> describe(0); + } + + static String describe(int value) { + return String.valueOf(value); + } + + static void helper() { + System.out.println("helper"); + } + + static class Nested { + String deep(List items) { + return items.get(0); + } + } +} diff --git a/services/parser/testdata/lang/javascript/Card.jsx b/services/parser/testdata/lang/javascript/Card.jsx new file mode 100644 index 0000000..5b7e3c7 --- /dev/null +++ b/services/parser/testdata/lang/javascript/Card.jsx @@ -0,0 +1,36 @@ +import { shout } from "./util.js"; + +function renderTitle(title) { + return

{shout(title)}

; +} + +export default function Card({ title, items }) { + const label = formatLabel(title); + return ( +
+ {renderTitle(label)} +
    + {items.map((item) => ( +
  • {describeItem(item)}
    • + ))} +
+

{`total ${countItems(items)}`}

+
+ ); +} + +function formatLabel(title) { + return title.trim(); +} + +function cx(name) { + return name; +} + +function describeItem(item) { + return item.name; +} + +function countItems(items) { + return items.length; +} diff --git a/services/parser/testdata/lang/javascript/legacy.js b/services/parser/testdata/lang/javascript/legacy.js new file mode 100644 index 0000000..30d5975 --- /dev/null +++ b/services/parser/testdata/lang/javascript/legacy.js @@ -0,0 +1,14 @@ +const { helper } = require("./util.js"); +const path = require("node:path"); + +function describe(value) { + return helper(value); +} + +const registry = { + register(value) { + return describe(value); + }, +}; + +module.exports = { describe, registry }; diff --git a/services/parser/testdata/lang/javascript/util.js b/services/parser/testdata/lang/javascript/util.js new file mode 100644 index 0000000..640ba13 --- /dev/null +++ b/services/parser/testdata/lang/javascript/util.js @@ -0,0 +1,5 @@ +export function helper(value) { + return String(value); +} + +export const shout = (value) => helper(value).toUpperCase(); diff --git a/services/parser/testdata/lang/python/repo.py b/services/parser/testdata/lang/python/repo.py new file mode 100644 index 0000000..6b87ba5 --- /dev/null +++ b/services/parser/testdata/lang/python/repo.py @@ -0,0 +1,48 @@ +import functools +import os.path as osp +from .util import wrap, unwrap as peel + + +def trace(fn): + @functools.wraps(fn) + def inner(*args): + return fn(*args) + + return inner + + +class Repo: + def __init__(self, ident): + self.ident = ident + + @trace + def sync(self): + label = self.label() + return wrap(f"{describe(label)}") + + def label(self): + return peel(self.ident) + + class Nested: + def deep(self): + return describe("nested") + + +async def fetch(repo): + return await repo.sync() + + +def describe(value): + return str(value) + + +def apply(values): + return [render(v) for v in values if keep(v)] + + +def render(value): + return osp.basename(str(value)) + + +def keep(value): + return bool(value) diff --git a/services/parser/testdata/lang/rust/repo.rs b/services/parser/testdata/lang/rust/repo.rs new file mode 100644 index 0000000..557fa31 --- /dev/null +++ b/services/parser/testdata/lang/rust/repo.rs @@ -0,0 +1,37 @@ +use std::fmt::Write as FmtWrite; +use crate::util::{wrap, unwrap as peel}; + +pub struct Repo { + id: String, +} + +impl Repo { + pub fn sync(&self) -> String { + let label = self.label(); + println!("{}", describe(&label)); + wrap(label) + } + + fn label(&self) -> String { + peel(self.id.clone()) + } +} + +pub fn describe(value: &str) -> String { + value.to_owned() +} + +pub fn apply(values: Vec) -> Vec { + values.iter().map(|v| render(*v)).collect() +} + +fn render(value: i32) -> String { + match value { + 0 => empty(), + _ => describe(&value.to_string()), + } +} + +fn empty() -> String { + String::new() +} diff --git a/services/parser/testdata/polyglot/src/Main.java b/services/parser/testdata/polyglot/src/Main.java new file mode 100644 index 0000000..499a676 --- /dev/null +++ b/services/parser/testdata/polyglot/src/Main.java @@ -0,0 +1,11 @@ +package com.example.src; + +public class Main { + static String helper() { + return "java"; + } + + static String run() { + return helper(); + } +} diff --git a/services/parser/testdata/polyglot/src/Widget.tsx b/services/parser/testdata/polyglot/src/Widget.tsx new file mode 100644 index 0000000..5e2cbca --- /dev/null +++ b/services/parser/testdata/polyglot/src/Widget.tsx @@ -0,0 +1,5 @@ +import { helper } from "./main"; + +export function Widget() { + return {helper()}; +} diff --git a/services/parser/testdata/polyglot/src/main.go b/services/parser/testdata/polyglot/src/main.go new file mode 100644 index 0000000..eb523ab --- /dev/null +++ b/services/parser/testdata/polyglot/src/main.go @@ -0,0 +1,14 @@ +package src + +func helper() string { + return "go" +} + +func go_only() string { + return "only defined in go" +} + +func run() string { + // python_only is defined only in main.py. + return helper() + python_only() +} diff --git a/services/parser/testdata/polyglot/src/main.js b/services/parser/testdata/polyglot/src/main.js new file mode 100644 index 0000000..4420665 --- /dev/null +++ b/services/parser/testdata/polyglot/src/main.js @@ -0,0 +1,7 @@ +export function helper() { + return "javascript"; +} + +export function run() { + return helper(); +} diff --git a/services/parser/testdata/polyglot/src/main.py b/services/parser/testdata/polyglot/src/main.py new file mode 100644 index 0000000..e7212bb --- /dev/null +++ b/services/parser/testdata/polyglot/src/main.py @@ -0,0 +1,12 @@ +def helper(): + return "python" + + +def python_only(): + return "only defined in python" + + +def run(): + # go_only is defined only in main.go. Resolving it would mean crossing a + # language boundary on a name that is unambiguous everywhere else. + return helper() + go_only() diff --git a/services/parser/testdata/polyglot/src/main.rs b/services/parser/testdata/polyglot/src/main.rs new file mode 100644 index 0000000..36eadd1 --- /dev/null +++ b/services/parser/testdata/polyglot/src/main.rs @@ -0,0 +1,7 @@ +fn helper() -> &'static str { + "rust" +} + +fn run() -> &'static str { + helper() +} diff --git a/services/parser/testdata/polyglot/src/main.ts b/services/parser/testdata/polyglot/src/main.ts new file mode 100644 index 0000000..1e72612 --- /dev/null +++ b/services/parser/testdata/polyglot/src/main.ts @@ -0,0 +1,7 @@ +export function helper(): string { + return "typescript"; +} + +export function run(): string { + return helper(); +}