From 932eea87e895d3d010209ee6974cf89bb09b73fa Mon Sep 17 00:00:00 2001 From: zaebee Date: Wed, 14 Oct 2026 11:54:40 +0000 Subject: [PATCH 1/6] =?UTF-8?q?docs(plan):=20language-packs=20step=203=20?= =?UTF-8?q?=E2=80=94=20per-pack=20index=20facts=20and=20classification?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/plans/2026-10-14-language-packs-step3.md | 732 ++++++++++++++++++ 1 file changed, 732 insertions(+) create mode 100644 docs/plans/2026-10-14-language-packs-step3.md diff --git a/docs/plans/2026-10-14-language-packs-step3.md b/docs/plans/2026-10-14-language-packs-step3.md new file mode 100644 index 00000000..31d8bc0f --- /dev/null +++ b/docs/plans/2026-10-14-language-packs-step3.md @@ -0,0 +1,732 @@ +# Language Packs — Step 3: Index Facts and Classification — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Each language pack computes the index facts it classifies with and classifies FQNs itself. The resolver's `classify_fqn` stops containing Python or TypeScript rules, and every stored graph stays byte-identical. + +**Architecture:** +- `cgis.core.index` defines `IndexFacts` (a protocol) and `ClassifyContext` (a frozen dataclass). +- `LanguagePack` gains two methods: + - `index_facts(nodes, state) -> IndexFacts`; + - `classify(fqn, ctx) -> NodeNamespace`. +- The registry adds two functions: + - `index_facts(nodes, state) -> dict[str, IndexFacts]`; + - `classifying_pack(source_file)`, which returns the Python pack when there is no source. That is the preserved quirk. +- `IndexBuilder` keeps the language-free indices and asks every pack for its facts. +- `SymbolIndex.classify_fqn` keeps the two language-free rules (relative `.` and `self.`), then hands off to the source file's pack. A source no pack reads gets the generic tail. +- The resolution-only facts stay on `SymbolIndex` until step 4 moves their consumers: `layout_prefixes`, `workspace_packages`, `path_aliases`, `file_bare_bindings` and the star-import expansion. + +**Tech Stack:** Python 3.12, tree-sitter, SQLite, pytest, ruff, mypy strict, interrogate. + +**Spec:** `docs/specs/2026-10-14-language-packs-design.md` (rev 3), "Index and classification" and Migration row 3. Outline: `docs/plans/2026-10-14-language-packs.md` "Step 3". Previous steps: #597, #599 (their plans' Rulings and the spec's implementation notes). + +## Global Constraints + +- `make format && make lint && make type-check && make pytest && make doc-coverage` passes before every commit. Gate with `&&`, never with `;`, a newline after a heredoc, or a pipe on the gating command. +- mypy strict on `src/`, docstring coverage ≥ 90%, Sonar cognitive complexity ≤ 15 per function. +- **Behaviour unchanged:** `tests/fixtures/graph_identity/snapshot.json` is not modified, and the main-vs-branch `graph_identity diff` reads `identical` on all three trees: cgis `src` @715cd9a, owner-api @8de2245, and this repo's `ui/`. +- The existing classification tests stay as they are, assertions included: + - `tests/unit/test_resolver_indices.py` + - `tests/unit/test_resolver_language_scope.py` + - `tests/unit/test_resolver.py` + + They may only be re-pointed where they read a `SymbolIndex` field that moves: `external_roots` and `dependency_roots`. +- Ingest time: median of three owner-api ingests, branch vs main, within +5%. That is the spec's step-4 budget, applied here because `classify_fqn` is on the hot path. Over budget: **stop and report**. +- Drift tolerances only go down. If `extraction` drift exceeds 0.40 or unresolved exceeds 0.37, or `resolution` drift exceeds 0.40 or unresolved exceeds 0.45: **stop and report**. Main measures `resolution` at 0.385 / 0.433, close to its lock. +- Test-count floor: raise `.github/test-count-baseline` (now 3095) if the suite passes 3245. +- Commits end with `Co-Authored-By: Claude Opus 5.5 `. The PR body ends with `🤖 Generated with [Claude Code](https://claude.com/claude-code)`. +- Worktree `/home/zaebee/projects/ownima/cgb-step3`, branch `feat/language-packs-step3`. + +## Rulings (where this plan departs from the spec's or outline's text) + +- **C1: `classify` returns `NodeNamespace`, not `NodeNamespace | None`.** + - Today's order interleaves the languages: TS bare imports, then internal roots (with Python's declared-dependency rule), then the JS root, then Python stdlib, then Python external roots. + - A pack that answered only "its" part could not preserve that order. For example, a project package named `json` must stay INTERNAL, not STDLIB. + - So each pack's `classify` runs its whole chain, and the shared rules (internal root, JS root) appear in both packs. + - *Cost if wrong:* the duplication is two `if`s per pack. +- **C2: `index_facts(nodes, state)` takes the merged project state, keyed by `ingest_state` key, not the pack's own `ProjectFacts` object.** + - The resolver receives project facts as values (`ResolverEngine(..., dependency_roots=..., npm_dependencies=...)`), not as pack objects. + - Rebuilding pack objects there would make the engine name pack classes. + - *Cost if wrong:* a signature change when step 5 rebuilds the pipeline. +- **C3: facts that only resolution consumes stay on `SymbolIndex` until step 4.** + - These are `layout_prefixes`, star-import expansion (`reexports`), `workspace_packages`, `path_aliases` and `file_bare_bindings`. + - Step 4 is where resolution becomes per-pack (`reconcile_import`, positioned steps). Moving the data now would leave the resolver reaching into a pack's facts for each one. + - The Python facts still compute the corroborated prefixes, because that is the same pass as `external_roots`. `IndexFacts.layout_prefixes` hands them back. +- **C4: the outline's "`resolve_layout`/`map_to_node_fqn` gated by pack" is not done.** + - Both run for every language today. Gating them would change TypeScript resolution, and this step promises no change. + - It stays a separate, measured change. +- **C5: one transitional engine-to-pack edge.** + - `symbols._package_call`, a TypeScript rule inside the resolver, needs the TS pack's `package_namespace`. + - It imports it directly, recorded under a new forbidden-edge rule `engine-packs` (`cgis.resolver` → `cgis.languages.`). Step 4 empties that allowlist when `_package_call` becomes the TS pack's positioned step. +- **C6: the JS constants move to core.** `JS_BUILTINS_ROOT`, `NODE_SCHEME` and `NODE_BUILTIN_MODULES` move from `cgis.resolver.js_builtins` to `cgis.core.js_globals`, so the TS pack does not import the engine. + +## Review Focus + +- **A project package that shares a stdlib or third-party name** (`app/json/`, `app/alembic/` plus a declared `alembic`) keeps today's namespace. Covered by the existing `test_resolver_indices.py` cases and the owner-api identity diff. +- **`classify_fqn(fqn)` with no source** still applies the Python reading. It is called that way from `resolve_layout` and from two sites in `symbols.py` (Task 3, `test_no_source_classifies_as_python`). +- **A source path no pack reads** (`.js`, `.pyi`, `Foo.PY`) gets the old generic tail: internal root → INTERNAL, JS root → STDLIB, anything else → UNKNOWN (Task 3, `test_a_source_no_pack_reads_gets_the_generic_tail`). +- **Hot-path cost.** The per-call pack lookup is cached, with no `PurePosixPath` on every call. Measured in Task 4. +- **TS bare imports in a file with no bindings, and a Python file with none.** Classification must not depend on which pack's facts happen to hold an entry for the path (Task 2, `test_only_typescript_files_carry_bare_imports`). + +--- + +### Task 1: The Python pack's index facts and classification + +**Files:** +- Create: `src/cgis/core/index.py`, `src/cgis/languages/python/index.py`, `tests/unit/test_python_index.py` +- Modify: `src/cgis/core/js_globals.py`, `src/cgis/resolver/js_builtins.py`, `src/cgis/resolver/indices.py` (C6: import the moved constants) + +**Interfaces:** +- Produces: + - **`cgis.core.index.IndexFacts`** (Protocol). Two read-only properties: + - `internal_roots: frozenset[str]`: roots the pack adds to the language-free set; + - `layout_prefixes: frozenset[str]`: corroborated prefixes before internal roots are subtracted. + - **`cgis.core.index.ClassifyContext`** (frozen dataclass): `source_file: str | None`, `internal_roots: frozenset[str]`, `nodes: Mapping[str, Node]`, `facts: IndexFacts`. + - **`cgis.languages.python.index`:** + - `PythonIndexFacts`, with fields `internal_roots`, `layout_prefixes`, `external_roots`, `dependency_roots`, `dependency_heads`, all `frozenset[str]`; + - `index_facts(nodes: Mapping[str, Node], state: Mapping[str, object]) -> PythonIndexFacts`; + - `classify(fqn: str, ctx: ClassifyContext) -> NodeNamespace`. + - **`cgis.core.js_globals`:** `JS_BUILTINS_ROOT`, `NODE_SCHEME`, `NODE_BUILTIN_MODULES`. + +- [ ] **Step 1: Move the JS constants to core (C6).** + 1. Cut these from `src/cgis/resolver/js_builtins.py` and paste them, verbatim with their comments, at the end of `src/cgis/core/js_globals.py`: + - `JS_BUILTINS_ROOT = "js_builtins"` + - `NODE_SCHEME = "node:"` + - the whole `NODE_BUILTIN_MODULES` frozenset + 2. In `js_builtins.py`, add `from cgis.core.js_globals import JS_BUILTINS_ROOT, NODE_BUILTIN_MODULES, NODE_SCHEME`. Drop names it no longer uses: ruff tells you which. + 3. In `indices.py`, import the three constants from `cgis.core.js_globals`. + 4. Run `git grep -n "js_builtins import" -- src tests scripts`. Re-point every import of a moved name to `cgis.core.js_globals`. + 5. Run `uv run pytest tests/unit -q -k "js or builtin or language_scope"`. Expected: all pass. + +- [ ] **Step 2: Write the failing tests.** Create `tests/unit/test_python_index.py`: + +```python +"""The Python pack's index facts and classification (language-packs step 3). + +Values are those the pre-step-3 `IndexBuilder`/`classify_fqn` produced for the +same nodes; the equivalence over real graphs is the identity snapshot's job. +""" + +from types import MappingProxyType + +from cgis.core.index import ClassifyContext +from cgis.core.models import Node, NodeNamespace, NodeType +from cgis.languages.python.index import PythonIndexFacts, classify, index_facts + + +def _node(fqn: str, node_type: NodeType = NodeType.FUNCTION, **kw: object) -> Node: + return Node( + id=fqn, type=node_type, name=fqn.rsplit(".", 1)[-1], + file_path=kw.pop("file_path", f"{fqn.replace('.', '/')}.py"), # type: ignore[arg-type] + start_line=1, end_line=1, **kw, # type: ignore[arg-type] + ) # fmt: skip + + +def _facts(nodes: list[Node], deps: list[str] | None = None) -> PythonIndexFacts: + by_id = {n.id: n for n in nodes} + return index_facts(MappingProxyType(by_id), {"python_dependencies": deps or []}) + + +def _ctx(facts: PythonIndexFacts, nodes: list[Node], roots: set[str], source: str | None) -> ClassifyContext: + return ClassifyContext( + source_file=source, + internal_roots=frozenset(roots) | facts.internal_roots, + nodes=MappingProxyType({n.id: n for n in nodes}), + facts=facts, + ) + + +def test_import_roots_split_into_external_and_first_party() -> None: + """#424: `app` reaches our nodes once stripped (twice: corroborated); `pydantic` never does.""" + file_node = _node( + "domains.views", NodeType.FILE, file_path="domains/views.py", + metadata={"import_map": { + "A": "app.domains.models.A", "B": "app.domains.models.B", "C": "pydantic.BaseModel", + }}, + ) # fmt: skip + nodes = [file_node, _node("domains.models.A"), _node("domains.models.B")] + facts = _facts(nodes) + assert facts.external_roots == frozenset({"pydantic"}) + assert "app" in facts.internal_roots + assert facts.layout_prefixes == frozenset({"app"}) + + +def test_src_and_lib_layouts_add_their_second_segment() -> None: + facts = _facts([_node("src.cgis.pipeline.run"), _node("lib.util.x")]) + assert {"cgis", "util"} <= facts.internal_roots + + +def test_only_colliding_dependencies_are_kept() -> None: + """#495: a declared dependency matters only where an internal root shares its name.""" + nodes = [_node("alembic.env.run_migrations"), _node("app.main")] + facts = _facts(nodes, deps=["alembic", "fastapi"]) + assert facts.dependency_roots == frozenset({"alembic"}) + assert facts.dependency_heads == frozenset({"alembic.env"}) + + +def test_classification_chain() -> None: + nodes = [_node("alembic.env.run_migrations"), _node("json.helpers.f")] + facts = _facts(nodes, deps=["alembic"]) + ctx = _ctx(facts, nodes, {"alembic", "json"}, "app/x.py") + assert classify("alembic.env.run_migrations", ctx) is NodeNamespace.INTERNAL + assert classify("alembic.op.add_column", ctx) is NodeNamespace.EXTERNAL # declared, not ours + assert classify("json.helpers.f", ctx) is NodeNamespace.INTERNAL # our package beats stdlib + assert classify("os.path.join", ctx) is NodeNamespace.STDLIB + assert classify("len", ctx) is NodeNamespace.STDLIB + assert classify("js_builtins.Math", ctx) is NodeNamespace.STDLIB + assert classify("nothing.known", ctx) is NodeNamespace.UNKNOWN +``` + +- [ ] **Step 3: Run them and watch them fail.** + - Run: `uv run pytest tests/unit/test_python_index.py -q` + - Expected: collection error, `ModuleNotFoundError: cgis.core.index`. + +- [ ] **Step 4: The contract.** Create `src/cgis/core/index.py`: + +```python +"""What a language pack contributes to the symbol index and to classification. + +Spec: Index and classification. The resolver keeps the language-free indices; +each pack derives what only its language's nodes and configs can say, and +classifies an FQN referenced from one of its files (language-packs step 3). +""" + +from collections.abc import Mapping +from dataclasses import dataclass +from typing import Protocol + +from cgis.core.models import Node + + +class IndexFacts(Protocol): + """One pack's facts over the whole node set; frozen.""" + + @property + def internal_roots(self) -> frozenset[str]: + """Roots this pack adds to the language-free set (first segment of every node id).""" + ... + + @property + def layout_prefixes(self) -> frozenset[str]: + """Prefixes sources write that node ids do not carry, before internal roots are removed.""" + ... + + +@dataclass(frozen=True) +class ClassifyContext: + """What a pack's `classify` may read about one reference.""" + + #: The file the reference comes from; None applies the Python reading (spec quirk). + source_file: str | None + #: Every internal root, all packs' contributions included. + internal_roots: frozenset[str] + nodes: Mapping[str, Node] + #: This pack's own facts. + facts: IndexFacts +``` + +- [ ] **Step 5: The Python pack's facts.** Create `src/cgis/languages/python/index.py`. + - **Move these verbatim out of `src/cgis/resolver/indices.py`, with their docstrings and comments:** + - `_LAYOUT_PREFIX_EVIDENCE` + - `_strips_to_a_node` + - `_heads_under` + - `_is_declared_dependency` + - the body of `IndexBuilder._build_external_roots`, as a module-level `_split_import_roots` + - `_BUILTINS` + - Rewrite `_is_declared_dependency` to take `(fqn, root, facts, nodes)`. It is only ever called for a Python source or for none, so the source-file check goes. + - Do not delete the originals from `indices.py` yet. Task 3 does that once nothing calls them. + +```python +"""The Python pack's index facts and classification (spec: Index and classification). + +Moved from `cgis.resolver.indices` in language-packs step 3; the history of each +rule is in its docstring and issue number. +""" + +import builtins +import os +import sys +from collections.abc import Iterable, Mapping +from dataclasses import dataclass +from typing import cast + +from cgis.core.index import ClassifyContext +from cgis.core.js_globals import JS_BUILTINS_ROOT +from cgis.core.models import Node, NodeNamespace, NodeType + +_BUILTINS: frozenset[str] = frozenset(dir(builtins)) +_LAYOUT_DIRS = frozenset({"src", "lib"}) +# ... _LAYOUT_PREFIX_EVIDENCE, _strips_to_a_node, _heads_under moved verbatim ... + + +@dataclass(frozen=True) +class PythonIndexFacts: + """What Python import maps and manifests say about the whole node set.""" + + #: Second segments of `src`/`lib` layouts and first-party import prefixes (#424). + internal_roots: frozenset[str] + #: Import roots with enough evidence to strip off a target (#498 review). + layout_prefixes: frozenset[str] + #: Import roots no node of ours sits under. + external_roots: frozenset[str] + #: Declared dependencies that collide with an internal root (#495). + dependency_roots: frozenset[str] + #: `root.second` heads our own nodes have under those roots. + dependency_heads: frozenset[str] + + +def index_facts(nodes: Mapping[str, Node], state: Mapping[str, object]) -> PythonIndexFacts: + """Derive the facts from every node and the declared `python_dependencies`.""" + node_ids = set(nodes) + local_roots: set[str] = set() + layout_roots: set[str] = set() + file_imports: dict[str, dict[str, str]] = {} + for node in nodes.values(): + parts = node.id.split(".") + local_roots.add(parts[0]) + if len(parts) > 1 and parts[0] in _LAYOUT_DIRS: + layout_roots.add(parts[1]) + if node.type == NodeType.FILE: + file_imports[os.path.normpath(node.file_path)] = node.metadata.get("import_map") or {} + external, first_party, corroborated = _split_import_roots(file_imports, node_ids) + declared = set(cast("list[str]", state.get("python_dependencies", []))) + colliding = (local_roots | layout_roots) & declared + return PythonIndexFacts( + internal_roots=frozenset(layout_roots | first_party), + layout_prefixes=frozenset(corroborated), + external_roots=frozenset(external), + dependency_roots=frozenset(colliding), + dependency_heads=_heads_under(colliding, node_ids), + ) + + +def classify(fqn: str, ctx: ClassifyContext) -> NodeNamespace: + """The Python reading of a root: ours, a declared dependency, stdlib, or a known import.""" + facts = cast("PythonIndexFacts", ctx.facts) + root = fqn.split(".", maxsplit=1)[0] + if root in ctx.internal_roots: + if _is_declared_dependency(fqn, root, facts, ctx.nodes): + return NodeNamespace.EXTERNAL + return NodeNamespace.INTERNAL + if root == JS_BUILTINS_ROOT: + return NodeNamespace.STDLIB + if root in sys.stdlib_module_names or root in _BUILTINS: + return NodeNamespace.STDLIB + if root in facts.external_roots: + return NodeNamespace.EXTERNAL + return NodeNamespace.UNKNOWN +``` + + - `colliding` is `local_roots | layout_roots` because the old builder intersected the declared roots with `internal_roots` *before* adding first-party prefixes. At that point it held the first segments plus the `src`/`lib` second segments. Keep that exact set. + - If ruff flags `Iterable` as unused after the move, drop it. + +- [ ] **Step 6: Run the tests.** + - Run: `uv run pytest tests/unit/test_python_index.py tests/unit/test_resolver_indices.py -q` + - Expected: all pass. The old index is still the one the resolver uses. + +- [ ] **Step 7: Gate and commit.** Message: `feat(languages): the Python pack's index facts and classification (not wired yet)`, with a body naming C6 and the Co-Authored-By line. Use the same gate line as step 2's plan, with scratchpad dir `step3`. + +--- + +### Task 2: The TypeScript pack's index facts and classification + +**Files:** +- Create: `src/cgis/languages/typescript/index.py`, `tests/unit/test_typescript_index.py` + +**Interfaces:** +- Consumes: `cgis.core.index.ClassifyContext` and `IndexFacts` (Task 1); `JS_BUILTINS_ROOT`, `NODE_SCHEME` and `NODE_BUILTIN_MODULES` from `cgis.core.js_globals`. +- Produces, in `cgis.languages.typescript.index`: + - `TypeScriptIndexFacts`, with fields `internal_roots`, `layout_prefixes` (both always empty), `npm_dependencies: frozenset[str]` and `file_bare_imports: Mapping[str, frozenset[str]]`; + - `index_facts(nodes, state) -> TypeScriptIndexFacts`; + - `package_namespace(fqn: str, source_file: str | None, facts: TypeScriptIndexFacts) -> NodeNamespace | None`; + - `classify(fqn, ctx) -> NodeNamespace`. + +- [ ] **Step 1: Write the failing tests.** Create `tests/unit/test_typescript_index.py`: + +```python +"""The TypeScript pack's index facts and classification (language-packs step 3).""" + +from types import MappingProxyType + +from cgis.core.index import ClassifyContext +from cgis.core.models import Node, NodeNamespace, NodeType +from cgis.languages.typescript.index import ( + TypeScriptIndexFacts, + classify, + index_facts, + package_namespace, +) + + +def _file(fqn: str, path: str, **metadata: object) -> Node: + return Node( + id=fqn, type=NodeType.FILE, name=path.rsplit("/", 1)[-1], file_path=path, + start_line=1, end_line=1, language="typescript", metadata=metadata, + ) # fmt: skip + + +def _facts(nodes: list[Node], npm: list[str]) -> TypeScriptIndexFacts: + return index_facts(MappingProxyType({n.id: n for n in nodes}), {"npm_dependencies": npm}) + + +def test_only_typescript_files_carry_bare_imports() -> None: + nodes = [ + _file("app.h", "app/h.ts", bare_imports=["react", "node:fs"]), + _file("app.plain", "app/plain.ts"), + ] + facts = _facts(nodes, ["react"]) + assert facts.file_bare_imports == {"app/h.ts": frozenset({"react", "node:fs"})} + assert facts.internal_roots == frozenset() == facts.layout_prefixes + + +def test_a_package_the_file_imports_and_the_manifest_declares_is_external() -> None: + facts = _facts([_file("app.h", "app/h.ts", bare_imports=["react", "fs", "lodash"])], ["react"]) + assert package_namespace("react.useState", "app/h.ts", facts) is NodeNamespace.EXTERNAL + assert package_namespace("fs.readFile", "app/h.ts", facts) is NodeNamespace.STDLIB + assert package_namespace("lodash.map", "app/h.ts", facts) is None # undeclared: a lost edge + assert package_namespace("react.useState", "app/other.ts", facts) is None # not imported there + + +def test_classification_chain() -> None: + facts = _facts([_file("app.h", "app/h.ts", bare_imports=["react"])], ["react"]) + ctx = ClassifyContext( + source_file="app/h.ts", internal_roots=frozenset({"app", "json"}), + nodes=MappingProxyType({}), facts=facts, + ) # fmt: skip + assert classify("react.useState", ctx) is NodeNamespace.EXTERNAL + assert classify("app.util", ctx) is NodeNamespace.INTERNAL + assert classify("js_builtins.Math", ctx) is NodeNamespace.STDLIB + assert classify("json.parse", ctx) is NodeNamespace.INTERNAL # an internal root, not Python's json + assert classify("os.path", ctx) is NodeNamespace.UNKNOWN # #454: Python names mean nothing here +``` + +- [ ] **Step 2: Run them and watch them fail.** + - Run: `uv run pytest tests/unit/test_typescript_index.py -q` + - Expected: `ModuleNotFoundError: cgis.languages.typescript.index`. + +- [ ] **Step 3: Write the module.** + - Move `bare_import_namespace`'s docstring and body from `indices.py` into `package_namespace`. Read `facts.file_bare_imports` and `facts.npm_dependencies` instead of the index's. + - Leave the original in place until Task 3. + +```python +"""The TypeScript pack's index facts and classification (spec: Index and classification).""" + +import os +from collections.abc import Mapping +from dataclasses import dataclass +from types import MappingProxyType +from typing import cast + +from cgis.core.index import ClassifyContext +from cgis.core.js_globals import JS_BUILTINS_ROOT, NODE_BUILTIN_MODULES, NODE_SCHEME, specifier_fqn +from cgis.core.models import Node, NodeNamespace, NodeType + + +@dataclass(frozen=True) +class TypeScriptIndexFacts: + """Declared npm packages and each file's bare imports (#556).""" + + internal_roots: frozenset[str] + layout_prefixes: frozenset[str] + #: npm packages the manifests declare, as written, less our workspace packages. + npm_dependencies: frozenset[str] + #: normalized file_path -> the bare specifiers that file imports, as written. + file_bare_imports: Mapping[str, frozenset[str]] + + +def index_facts(nodes: Mapping[str, Node], state: Mapping[str, object]) -> TypeScriptIndexFacts: + """Read the bare imports the extractor recorded and the declared npm packages.""" + bare = { + os.path.normpath(n.file_path): frozenset(n.metadata["bare_imports"]) + for n in nodes.values() + if n.type == NodeType.FILE and n.metadata.get("bare_imports") + } + return TypeScriptIndexFacts( + internal_roots=frozenset(), + layout_prefixes=frozenset(), + npm_dependencies=frozenset(cast("list[str]", state.get("npm_dependencies", []))), + file_bare_imports=MappingProxyType(bare), + ) + + +def package_namespace( + fqn: str, source_file: str | None, facts: TypeScriptIndexFacts +) -> NodeNamespace | None: + """""" + # + + +def classify(fqn: str, ctx: ClassifyContext) -> NodeNamespace: + """A package the file imports, then our own roots, then the JS builtins; else UNKNOWN (#454).""" + package = package_namespace(fqn, ctx.source_file, cast("TypeScriptIndexFacts", ctx.facts)) + if package is not None: + return package + root = fqn.split(".", maxsplit=1)[0] + if root in ctx.internal_roots: + return NodeNamespace.INTERNAL + if root == JS_BUILTINS_ROOT: + return NodeNamespace.STDLIB + return NodeNamespace.UNKNOWN +``` + + - Check that the dict comprehension matches the old builder. `_index_metadata` keyed bare imports by `os.path.normpath(node.file_path)` and recorded them only when `specifiers` was truthy, which is the same condition used here. + +- [ ] **Step 4: Run the tests.** + - Run: `uv run pytest tests/unit/test_typescript_index.py tests/unit/test_resolver_language_scope.py -q` + - Expected: all pass. + +- [ ] **Step 5: Gate and commit.** Message: `feat(languages): the TypeScript pack's index facts and classification (not wired yet)`. + +--- + +### Task 3: Wire the packs in; the resolver's classification becomes language-free + +**Files:** +- Modify: + - `src/cgis/core/extractor.py` (two `LanguagePack` methods) + - both pack `__init__.py` + - `src/cgis/languages/__init__.py` (`index_facts`, `classifying_pack`) + - `src/cgis/resolver/indices.py` + - `src/cgis/resolver/symbols.py` + - `tests/self_parsing/test_architecture.py` (rule `engine-packs`) + - `tests/self_parsing/forbidden_edges_allowlist.json` + - `tests/unit/test_resolver_indices.py:64,213,454,459` (re-point moved fields only) +- Test: `tests/unit/test_languages.py` (append) + +**Interfaces:** +- Consumes: Task 1 and Task 2 modules. +- Produces: + - `LanguagePack.index_facts(nodes: Mapping[str, Node], state: Mapping[str, object]) -> IndexFacts` + - `LanguagePack.classify(fqn: str, ctx: ClassifyContext) -> NodeNamespace` + - `cgis.languages.index_facts(nodes, state) -> dict[str, IndexFacts]` + - `cgis.languages.classifying_pack(source_file: str | None) -> LanguagePack | None` + - `SymbolIndex.facts: Mapping[str, IndexFacts]` + + `SymbolIndex` loses `external_roots`, `dependency_roots`, `dependency_heads`, `npm_dependencies` and `file_bare_imports`. + +- [ ] **Step 1: Write the failing tests.** Append to `tests/unit/test_languages.py`: + +```python +from cgis.core.models import NodeNamespace +from cgis.languages import classifying_pack +from cgis.resolver.indices import IndexBuilder + + +def test_no_source_classifies_as_python() -> None: + """Spec quirk, kept in the registry: classify_fqn(fqn) without a source reads it as Python.""" + pack = classifying_pack(None) + assert pack is not None + assert pack.name == "python" + index = IndexBuilder().build([]) + assert index.classify_fqn("os.path") is NodeNamespace.STDLIB + assert index.classify_fqn("os.path", "app/x.ts") is NodeNamespace.UNKNOWN + + +@pytest.mark.parametrize("source", ["web/a.js", "types/x.pyi", "Foo.PY"]) +def test_a_source_no_pack_reads_gets_the_generic_tail(source: str) -> None: + """Internal root → INTERNAL, the JS root → STDLIB, anything else UNKNOWN — as before.""" + assert classifying_pack(source) is None + index = IndexBuilder().build( + [Node(id="app.f", type=NodeType.FUNCTION, name="f", file_path="app/f.py", + start_line=1, end_line=1)] + ) # fmt: skip + assert index.classify_fqn("app.g", source) is NodeNamespace.INTERNAL + assert index.classify_fqn("js_builtins.Math", source) is NodeNamespace.STDLIB + assert index.classify_fqn("os.path", source) is NodeNamespace.UNKNOWN +``` + +Move the imports to the file's import block (`Node` and `NodeType` are already imported there, so check), and add `import pytest` if it is missing. + +- [ ] **Step 2: Run them and watch them fail.** + - Run: `uv run pytest tests/unit/test_languages.py -q -k "no_source or generic_tail"` + - Expected: `ImportError: cannot import name 'classifying_pack'`. + +- [ ] **Step 3: The contract and the packs.** + 1. In `src/cgis/core/extractor.py`, import `IndexFacts` and `ClassifyContext` from `cgis.core.index`, plus `NodeNamespace`. Add to `LanguagePack`: + +```python + def index_facts(self, nodes: Mapping[str, Node], state: Mapping[str, object]) -> IndexFacts: + """This pack's facts over every node, given the merged project state (step 3, C2).""" + ... + + def classify(self, fqn: str, ctx: ClassifyContext) -> NodeNamespace: + """The namespace of `fqn` referenced from one of this pack's files — the whole chain (C1).""" + ... +``` + + 2. In each pack's `__init__.py`, implement both with `@override`, delegating to the pack's `index` module: + +```python + @override + def index_facts(self, nodes: Mapping[str, Node], state: Mapping[str, object]) -> IndexFacts: + """See `cgis.languages.python.index.index_facts`.""" + return index.index_facts(nodes, state) + + @override + def classify(self, fqn: str, ctx: ClassifyContext) -> NodeNamespace: + """See `cgis.languages.python.index.classify`.""" + return index.classify(fqn, ctx) +``` + + Use `from cgis.languages.python import index` (and the same for typescript). The modules import no grammar. + +- [ ] **Step 4: The registry.** In `src/cgis/languages/__init__.py`, add `from functools import cache, lru_cache`, import `IndexFacts` from `cgis.core.index` and `Node` from core models, then: + +```python +def index_facts(nodes: Mapping[str, Node], state: Mapping[str, object]) -> dict[str, IndexFacts]: + """Every pack's index facts, by pack name.""" + return {m.name: _load(m.name).index_facts(nodes, state) for m in MANIFESTS} + + +@lru_cache(maxsize=4096) +def classifying_pack(source_file: str | None) -> LanguagePack | None: + """The pack that classifies a reference from `source_file`; None when no pack reads it. + + No source means the Python pack: `classify_fqn(fqn)` has always applied the + Python reading, and removing that is a separate, measured change (spec, + Index and classification). Cached: classification is on the resolver's hot + path, and a path's suffix never changes. + """ + return _load("python") if source_file is None else pack_for_path(source_file) +``` + +- [ ] **Step 5: The resolver.** In `src/cgis/resolver/indices.py`: + 1. `SymbolIndex`: + - delete the fields `external_roots`, `dependency_roots`, `dependency_heads`, `npm_dependencies` and `file_bare_imports`, with their comments; + - add `facts: Mapping[str, IndexFacts] = MappingProxyType({})`, commented `# pack name -> that pack's index facts (language-packs step 3)`; + - keep `internal_roots` and `layout_prefixes`. + 2. Replace `classify_fqn`'s body after the two generic rules: + +```python + pack = classifying_pack(source_file) + if pack is None: + # A source no pack reads: the rules every language shared before step 3. + root = fqn.split(".", maxsplit=1)[0] + if root in self.internal_roots: + return NodeNamespace.INTERNAL + return NodeNamespace.STDLIB if root == JS_BUILTINS_ROOT else NodeNamespace.UNKNOWN + context = ClassifyContext( + source_file=source_file, + internal_roots=self.internal_roots, + nodes=self.nodes, + facts=self.facts[pack.name], + ) + return pack.classify(fqn, context) +``` + + Keep its docstring. Add: `The language rules are the source's pack's (language-packs step 3).` + 3. `IndexBuilder.build`: + - keep the language-free loop; + - change `_add_node_to_suffix_map` so it adds only `parts[0]` to `internal_roots` (the `src`/`lib` second segment is now Python's); + - drop the `file_bare_imports` accumulation from `_index_metadata`, but **keep** `bare_bindings`, populated under the same `if specifiers:` condition; + - replace the external-roots and dependency block with: + +```python + state = { + "python_dependencies": list(dependency_roots), + "npm_dependencies": list(npm_dependencies), + } + facts = pack_index_facts(MappingProxyType(nodes_by_id), state) + roots = frozenset(internal_roots.union(*(f.internal_roots for f in facts.values()))) + corroborated = frozenset().union(*(f.layout_prefixes for f in facts.values())) + self._expand_star_imports(star_imports, reexports, set(nodes_by_id)) +``` + + then build `SymbolIndex(..., internal_roots=roots, layout_prefixes=corroborated - roots, facts=MappingProxyType(facts), ...)`, without the deleted fields. Import `from cgis.languages import classifying_pack, index_facts as pack_index_facts` and `from cgis.core.index import ClassifyContext, IndexFacts`. + 4. Delete what Tasks 1 and 2 moved and nothing now calls: `bare_import_namespace`, `_is_declared_dependency`, `_strips_to_a_node`, `_heads_under`, `_LAYOUT_PREFIX_EVIDENCE`, `_build_external_roots`, `_BUILTINS`, and the `builtins`/`sys` imports if unused. Check with `git grep -n "" -- src tests` before each deletion. + +- [ ] **Step 6: `symbols._package_call` (C5).** + - Import `from cgis.languages.typescript.index import TypeScriptIndexFacts, package_namespace` and drop `bare_import_namespace` from the `indices` import. + - Replace the check with: + +```python + facts = index.facts.get("typescript") + if not isinstance(facts, TypeScriptIndexFacts): + return None + if package_namespace(candidate, file_path, facts) is None: + return None +``` + + - Add one comment line above it: `# A TypeScript rule inside the engine until step 4 makes it the TS pack's step (C5).` + +- [ ] **Step 7: Record the transitional edge in the gate.** + 1. In `tests/self_parsing/test_architecture.py` `RULES`, add, after `resolver-storage`: + +```python + _Rule("engine-packs", ("cgis.resolver",), ("cgis.languages",), + exempt_modules=frozenset({"cgis.languages"})), +``` + + 2. Add `"engine-packs": {}` to `forbidden_edges_allowlist.json`. + 3. Run `uv run pytest tests/self_parsing/test_architecture.py -q -k allowlist`. It fails and prints the actual entries. + 4. Expected: exactly one key, `"cgis.resolver.symbols -> cgis.languages.typescript.index"`, with a small count (an import plus a call or two). Copy that entry into the JSON. + 5. Any other `engine-packs` entry means the resolver reaches a pack somewhere else: fix the code, not the allowlist. + 6. If `RULES[0]` is referenced by index in a test (`grep -n "RULES\[" tests/self_parsing/test_architecture.py`), check that inserting the rule after `resolver-storage` leaves index 0 unchanged. + +- [ ] **Step 8: Re-point the moved-field reads in tests.** In `tests/unit/test_resolver_indices.py`: + - `index.external_roots` → `index.facts["python"].external_roots`, at :64 and :213; + - `.dependency_roots` → `.facts["python"].dependency_roots`, at :454 and :459. + + Change nothing else: no assertion and no fixture. Then run `git grep -nE "\.(external_roots|dependency_roots|dependency_heads|npm_dependencies|file_bare_imports)\b" -- tests src`. Expected: only uses on pack facts objects, and `ImportNames.npm_dependencies`. + +- [ ] **Step 9: Run the tests.** + - Run: `uv run pytest tests/unit/test_languages.py tests/unit/test_resolver_indices.py tests/unit/test_resolver_language_scope.py tests/unit/test_resolver.py tests/unit/test_python_index.py tests/unit/test_typescript_index.py tests/unit/test_graph_identity.py tests/self_parsing -q` + - Expected: all pass. + - A failure in the resolver tests is a behaviour change. Find it with `superpowers:systematic-debugging`; never edit the assertion. + +- [ ] **Step 10: Gate and commit.** Message: `refactor(resolver): classification and its index facts come from the language packs`, with a body naming C1–C5 and the Co-Authored-By line. + +--- + +### Task 4: Measure, record, document + +**Files:** +- `tests/unit/review_path_inventory.txt` (regenerate if the closure changed) +- `docs/ontology/tolerances.lock` and `patterns.yaml`, only if a tolerance may go down +- `docs/specs/2026-10-14-language-packs-design.md` (implementation note) +- `CLAUDE.md` ("Adding a Language Pack": `index_facts`, `classify`) + +- [ ] **Step 1: Drift.** Ingest `src` with the branch's cgis and run `cgis drift --profile python --format json`, as in step 2's plan. Read `extraction`, `resolution` and `cgis-project`. Stop if any exceeds the bounds in Global Constraints. Record the numbers in the ledger. + +- [ ] **Step 2: Hot-path timing.** + +```bash +S=/tmp/claude-1001/-home-zaebee-projects-ownima-codegraph-brain/97c475b5-6bc7-4231-a643-ff81f3f651ec/scratchpad/step3/perf && /bin/rm -rf $S && mkdir -p $S/oa && git -C /home/zaebee/projects/ownima/owner-api archive 8de2245 ownima-backend | tar -x -C $S/oa && for side in main branch; do P=$([ $side = main ] && echo /home/zaebee/projects/ownima/codegraph-brain || echo /home/zaebee/projects/ownima/cgb-step3); for i in 1 2 3; do /usr/bin/time -f "$side %e" -a -o $S/times.txt uv run --project $P cgis ingest $S/oa/ownima-backend --output $S/$side-$i.db > /dev/null 2>&1 || exit 1; done; done && sort $S/times.txt +``` + + - Expected: the branch median is within +5% of main's. Report both medians. + - If it is over budget, profile `classify_fqn` before anything else (`python -m cProfile -s cumtime`). + +- [ ] **Step 3: Inventory.** + - Regenerate it as in step 2's plan, Task 2 Step 3. + - Expected: nothing gone, and new entries for the two `index.py` modules and `core/index.py`. + +- [ ] **Step 4: Docs.** + 1. In the spec's "Index and classification" section, add an `> **Implementation note (step 3):**` covering four points: + - `classify` returns the whole chain, not `None` (C1); + - `index_facts` takes the merged project state (C2); + - resolution-only facts and star expansion stay on `SymbolIndex` until step 4 (C3); + - the outline's `resolve_layout` gating was dropped as a behaviour change (C4). + 2. In CLAUDE.md "Adding a Language Pack" step 2, add `index_facts` and `classify` to the list of members. Add one line: `classify answers for references from the pack's own files and runs the whole chain; a source with no pack gets the generic tail in SymbolIndex.classify_fqn.` + +- [ ] **Step 5: Gate and commit.** Message: `docs: step 3 implementation notes; inventory`, with the measurements in the body. + +--- + +### Task 5: Identity report, review, PR + +- [ ] **Step 1: Identity.** Run the same commands as step 2's plan, Task 6, Steps 1–2, with `step3` paths and `BR=/home/zaebee/projects/ownima/cgb-step3`. + - Expected: `identical` three times. + - Before ingesting, check that main has not moved under the branch: `git -C
diff --stat main -- src` should be empty. + +- [ ] **Step 2: Independent review.** Dispatch a reviewer with execution tools on the most capable model. Give it the branch diff, this plan, the spec, the ledger rulings and the Review Focus. Fix Critical/Important findings test-first, then rerun the gate and the identity step. + +- [ ] **Step 3: PR.** Push and open the PR, titled `refactor(languages): step 3 — per-pack index facts and classification`. The body carries: + - the module map; + - rulings C1–C6; + - three identity results; + - drift and timing numbers; + - the `engine-packs` allowlist entry and that step 4 empties it; + - the reviewer's deferred minors. + + Then check the Gemini and Guardian comments and Sonar, and reply in every thread. From efe6a20fc2859a9bd6fea5e9d0698b49fdcf1cfd Mon Sep 17 00:00:00 2001 From: zaebee Date: Wed, 14 Oct 2026 12:10:18 +0000 Subject: [PATCH 2/6] feat(languages): the Python pack's index facts and classification (not wired yet) cgis.core.index holds the contract (IndexFacts, ClassifyContext). The Python pack's facts and classification chain are moved from cgis.resolver.indices with their docstrings; the resolver still uses its own copies until step 3 Task 3. JS_BUILTINS_ROOT and the Node builtin names move to cgis.core.js_globals so the TypeScript pack never imports the engine (ruling C6). Co-Authored-By: Claude Opus 5.5 --- src/cgis/core/index.py | 39 +++++ src/cgis/core/js_globals.py | 57 +++++++ src/cgis/languages/python/index.py | 186 +++++++++++++++++++++++ src/cgis/resolver/indices.py | 3 +- src/cgis/resolver/js_builtins.py | 57 +------ tests/self_parsing/test_self_parse_ts.py | 2 +- tests/unit/test_python_index.py | 91 +++++++++++ 7 files changed, 376 insertions(+), 59 deletions(-) create mode 100644 src/cgis/core/index.py create mode 100644 src/cgis/languages/python/index.py create mode 100644 tests/unit/test_python_index.py diff --git a/src/cgis/core/index.py b/src/cgis/core/index.py new file mode 100644 index 00000000..68a7916c --- /dev/null +++ b/src/cgis/core/index.py @@ -0,0 +1,39 @@ +"""What a language pack contributes to the symbol index and to classification. + +Spec: Index and classification. The resolver keeps the language-free indices; +each pack derives what only its language's nodes and configs can say, and +classifies an FQN referenced from one of its files (language-packs step 3). +""" + +from collections.abc import Mapping +from dataclasses import dataclass +from typing import Protocol + +from cgis.core.models import Node + + +class IndexFacts(Protocol): + """One pack's facts over the whole node set; frozen.""" + + @property + def internal_roots(self) -> frozenset[str]: + """Roots this pack adds to the language-free set (first segment of every node id).""" + ... + + @property + def layout_prefixes(self) -> frozenset[str]: + """Prefixes sources write that node ids do not carry, before internal roots are removed.""" + ... + + +@dataclass(frozen=True) +class ClassifyContext: + """What a pack's `classify` may read about one reference.""" + + #: The file the reference comes from; None applies the Python reading (spec quirk). + source_file: str | None + #: Every internal root, all packs' contributions included. + internal_roots: frozenset[str] + nodes: Mapping[str, Node] + #: This pack's own facts. + facts: IndexFacts diff --git a/src/cgis/core/js_globals.py b/src/cgis/core/js_globals.py index ff7e3b95..6d69464a 100644 --- a/src/cgis/core/js_globals.py +++ b/src/cgis/core/js_globals.py @@ -120,3 +120,60 @@ def specifier_fqn(specifier: str) -> str: """A bare module specifier dotted the way import targets are: `@x/a/b` -> `@x.a.b`.""" return specifier.replace("/", ".") + + +#: The synthetic root calls on a JS runtime global resolve into (#111). +JS_BUILTINS_ROOT = "js_builtins" + +#: The `node:` scheme, which names a Node builtin and nothing else (#556). +NODE_SCHEME = "node:" + +#: Node's builtin modules importable without the `node:` scheme — Node's own +#: `module.builtinModules`, minus internals and the scheme-only ones (`node:test`, +#: `node:sqlite`). Matched on the specifier's root, so `fs/promises` counts (#556). +NODE_BUILTIN_MODULES: frozenset[str] = frozenset( + { + "assert", + "async_hooks", + "buffer", + "child_process", + "cluster", + "console", + "constants", + "crypto", + "dgram", + "diagnostics_channel", + "dns", + "domain", + "events", + "fs", + "http", + "http2", + "https", + "inspector", + "module", + "net", + "os", + "path", + "perf_hooks", + "process", + "punycode", + "querystring", + "readline", + "repl", + "stream", + "string_decoder", + "sys", + "timers", + "tls", + "trace_events", + "tty", + "url", + "util", + "v8", + "vm", + "wasi", + "worker_threads", + "zlib", + } +) diff --git a/src/cgis/languages/python/index.py b/src/cgis/languages/python/index.py new file mode 100644 index 00000000..0a315f8b --- /dev/null +++ b/src/cgis/languages/python/index.py @@ -0,0 +1,186 @@ +"""The Python pack's index facts and classification (spec: Index and classification). + +Moved from `cgis.resolver.indices` in language-packs step 3; the history of each +rule is in its docstring and issue number. +""" + +import builtins +import os +import sys +from collections.abc import Iterable, Mapping +from dataclasses import dataclass +from typing import cast + +from cgis.core.index import ClassifyContext +from cgis.core.js_globals import JS_BUILTINS_ROOT +from cgis.core.models import Node, NodeNamespace, NodeType + +_BUILTINS: frozenset[str] = frozenset(dir(builtins)) +_LAYOUT_DIRS = frozenset({"src", "lib"}) + +#: How many of a root's imports must reach a node before it may be stripped off a +#: target (#498 review). One is not evidence: a third-party module that happens to +#: share a path with ours — `from boto3.utils.helpers import x` against our own +#: `utils/helpers.py` — made `boto3` first-party, and stripping it wired +#: `import boto3.config` into this project's `config.py` at full confidence. +#: Classification itself still accepts one, because there the cost of a wrong call +#: is a namespace label rather than a fabricated internal edge. +_LAYOUT_PREFIX_EVIDENCE = 2 + + +def _strips_to_a_node(imported_fqn: str, node_ids: set[str]) -> bool: + """True when dropping leading segments of `imported_fqn` reaches a real node. + + `app.domains.models.Thing` against a graph rooted at `domains` strips to + `domains.models.Thing`, which exists — so `app` is this project's own name, + not a package it depends on. + + The remainder must keep at least two segments. Stripping to a single one + matches any library whose submodule shares a name with a top-level module of + ours — `from pydantic import config` against a project that has `config.py` + strips to `config`, and the whole of pydantic would then classify INTERNAL, + breaking resolution for every symbol in it. Two segments cannot collide by + accident, and a genuine first-party prefix always has deeper imports than + that: on owner-api 3346 values resolve, none of them needing this. + """ + parts = imported_fqn.split(".") + return any(".".join(parts[i:]) in node_ids for i in range(1, len(parts) - 1)) + + +def _heads_under(roots: set[str], node_ids: Iterable[str]) -> frozenset[str]: + """The `root.second` heads of node ids under `roots`, read past a `src`/`lib` layout. + + The same two spellings `_add_node_to_suffix_map` makes a root internal by. + """ + if not roots: + return frozenset() + heads: set[str] = set() + for node_id in node_ids: + parts = node_id.split(".") + start = 1 if len(parts) > 2 and parts[0] in {"src", "lib"} else 0 + for offset in {0, start}: + if len(parts) > offset + 1 and parts[offset] in roots: + heads.add(f"{parts[offset]}.{parts[offset + 1]}") + return frozenset(heads) + + +def _is_declared_dependency( + fqn: str, root: str, facts: "PythonIndexFacts", nodes: Mapping[str, Node] +) -> bool: + """True when an internal-root `fqn` names a declared dependency, not our package. + + The root alone cannot say: `app/alembic/` makes `alembic` internal, while + `from alembic import op` is the library (#495). Our package is what it + holds, so `alembic.env.run_migrations` stays internal and + `alembic.op.add_column` — `op` being nothing of ours — is the dependency. + + The manifest is what licenses the second reading, never the graph alone: + a graph-only rule declared `scripts.ci.openapi_snapshot` third-party + because that module lives outside the ingest root, though it is ours and + `scripts` is no dependency. The roots come from Python manifests, so only + a Python source (or an unscoped call) reads them, as in #454. + """ + if root not in facts.dependency_roots or fqn in nodes: + return False + return ".".join(fqn.split(".", maxsplit=2)[:2]) not in facts.dependency_heads + + +def _split_import_roots( + file_imports: dict[str, dict[str, str]], node_ids: set[str] +) -> tuple[set[str], set[str], set[str]]: + """Split import roots into genuinely external ones and first-party prefixes. + + Returns (external_roots, first_party_prefixes, corroborated_prefixes) — + the third being those with enough evidence to strip off a target, not only + to classify (see `_LAYOUT_PREFIX_EVIDENCE`). + + A root is a *first-party prefix* when some import under it reaches a real + node once its head is stripped. That happens whenever a project is + ingested at a subdirectory: `cgis ingest ownima-backend/app` roots its + nodes at `domains`, `api`, … while the code imports `from app.models + import X`, so `app` looks like a third-party package. Every `app.*` FQN + without a node then classified EXTERNAL, and the resolver minted a + boundary node for a class of ours that does not exist — 2 461 confident + edges to nothing on that codebase (#424). + + The discriminator is unambiguous in practice: measured there, `app` had + 3 346 of 4 885 import values reach a node while pydantic, sqlalchemy, + grpc, fastapi and httpx had zero each. It asks whether the *import value* + resolves, not whether some node's id ends with the root's name — the + latter would match the library `grpc` against a project's own + `api/dependencies/grpc/` package. + """ + imports_by_root: dict[str, set[str]] = {} + for import_map in file_imports.values(): + for val in import_map.values(): + if val and not val.startswith("."): + imports_by_root.setdefault(val.split(".", maxsplit=1)[0], set()).add(val) + + external: set[str] = set() + first_party: set[str] = set() + corroborated: set[str] = set() + for root, values in imports_by_root.items(): + resolving = sum(1 for v in values if _strips_to_a_node(v, node_ids)) + (first_party if resolving else external).add(root) + if resolving >= _LAYOUT_PREFIX_EVIDENCE: + corroborated.add(root) + return external, first_party, corroborated + + +@dataclass(frozen=True) +class PythonIndexFacts: + """What Python import maps and manifests say about the whole node set.""" + + #: Second segments of `src`/`lib` layouts and first-party import prefixes (#424). + internal_roots: frozenset[str] + #: Import roots with enough evidence to strip off a target (#498 review). + layout_prefixes: frozenset[str] + #: Import roots no node of ours sits under. + external_roots: frozenset[str] + #: Declared dependencies that collide with an internal root (#495). + dependency_roots: frozenset[str] + #: `root.second` heads our own nodes have under those roots. + dependency_heads: frozenset[str] + + +def index_facts(nodes: Mapping[str, Node], state: Mapping[str, object]) -> PythonIndexFacts: + """Derive the facts from every node and the declared `python_dependencies`.""" + node_ids = set(nodes) + local_roots: set[str] = set() + layout_roots: set[str] = set() + file_imports: dict[str, dict[str, str]] = {} + for node in nodes.values(): + parts = node.id.split(".") + local_roots.add(parts[0]) + if len(parts) > 1 and parts[0] in _LAYOUT_DIRS: + layout_roots.add(parts[1]) + if node.type == NodeType.FILE: + file_imports[os.path.normpath(node.file_path)] = node.metadata.get("import_map") or {} + external, first_party, corroborated = _split_import_roots(file_imports, node_ids) + declared = set(cast("list[str]", state.get("python_dependencies", []))) + # Before first-party prefixes join: the set the old builder intersected (#495). + colliding = (local_roots | layout_roots) & declared + return PythonIndexFacts( + internal_roots=frozenset(layout_roots | first_party), + layout_prefixes=frozenset(corroborated), + external_roots=frozenset(external), + dependency_roots=frozenset(colliding), + dependency_heads=_heads_under(colliding, node_ids), + ) + + +def classify(fqn: str, ctx: ClassifyContext) -> NodeNamespace: + """The Python reading of a root: ours, a declared dependency, stdlib, or a known import.""" + facts = cast("PythonIndexFacts", ctx.facts) + root = fqn.split(".", maxsplit=1)[0] + if root in ctx.internal_roots: + if _is_declared_dependency(fqn, root, facts, ctx.nodes): + return NodeNamespace.EXTERNAL + return NodeNamespace.INTERNAL + if root == JS_BUILTINS_ROOT: + return NodeNamespace.STDLIB + if root in sys.stdlib_module_names or root in _BUILTINS: + return NodeNamespace.STDLIB + if root in facts.external_roots: + return NodeNamespace.EXTERNAL + return NodeNamespace.UNKNOWN diff --git a/src/cgis/resolver/indices.py b/src/cgis/resolver/indices.py index 342fff49..bcc02f60 100644 --- a/src/cgis/resolver/indices.py +++ b/src/cgis/resolver/indices.py @@ -9,9 +9,8 @@ from types import MappingProxyType from typing import NamedTuple -from cgis.core.js_globals import specifier_fqn +from cgis.core.js_globals import JS_BUILTINS_ROOT, NODE_BUILTIN_MODULES, NODE_SCHEME, specifier_fqn from cgis.core.models import SELF_PREFIX, Node, NodeNamespace, NodeType -from cgis.resolver.js_builtins import JS_BUILTINS_ROOT, NODE_BUILTIN_MODULES, NODE_SCHEME _BUILTINS: frozenset[str] = frozenset(dir(builtins)) diff --git a/src/cgis/resolver/js_builtins.py b/src/cgis/resolver/js_builtins.py index 28322bfb..21fb4cb3 100644 --- a/src/cgis/resolver/js_builtins.py +++ b/src/cgis/resolver/js_builtins.py @@ -12,62 +12,7 @@ `Map.get` being mistaken for the JavaScript runtime. """ -from cgis.core.js_globals import JS_GLOBALS - -JS_BUILTINS_ROOT = "js_builtins" - -#: The `node:` scheme, which names a Node builtin and nothing else (#556). -NODE_SCHEME = "node:" - -#: Node's builtin modules importable without the `node:` scheme — Node's own -#: `module.builtinModules`, minus internals and the scheme-only ones (`node:test`, -#: `node:sqlite`). Matched on the specifier's root, so `fs/promises` counts (#556). -NODE_BUILTIN_MODULES: frozenset[str] = frozenset( - { - "assert", - "async_hooks", - "buffer", - "child_process", - "cluster", - "console", - "constants", - "crypto", - "dgram", - "diagnostics_channel", - "dns", - "domain", - "events", - "fs", - "http", - "http2", - "https", - "inspector", - "module", - "net", - "os", - "path", - "perf_hooks", - "process", - "punycode", - "querystring", - "readline", - "repl", - "stream", - "string_decoder", - "sys", - "timers", - "tls", - "trace_events", - "tty", - "url", - "util", - "v8", - "vm", - "wasi", - "worker_threads", - "zlib", - } -) +from cgis.core.js_globals import JS_BUILTINS_ROOT, JS_GLOBALS #: Extensions whose calls are JavaScript at runtime. Matches the extractors #: registered in `cgis.languages` for `.ts`/`.tsx`. diff --git a/tests/self_parsing/test_self_parse_ts.py b/tests/self_parsing/test_self_parse_ts.py index f61cbd5f..002ca9a4 100644 --- a/tests/self_parsing/test_self_parse_ts.py +++ b/tests/self_parsing/test_self_parse_ts.py @@ -9,9 +9,9 @@ import pytest +from cgis.core.js_globals import JS_BUILTINS_ROOT, JS_GLOBALS from cgis.core.models import Edge, EdgeType, Node, NodeNamespace, NodeType from cgis.languages.typescript.naming import file_path_to_module_fqn -from cgis.resolver.js_builtins import JS_BUILTINS_ROOT, JS_GLOBALS from cgis.storage.sqlite_store import SQLiteStore _TS_SRC = Path(__file__).parent.parent.parent / "ui" / "src" diff --git a/tests/unit/test_python_index.py b/tests/unit/test_python_index.py new file mode 100644 index 00000000..b4ccfcd6 --- /dev/null +++ b/tests/unit/test_python_index.py @@ -0,0 +1,91 @@ +"""The Python pack's index facts and classification (language-packs step 3). + +Values are those the pre-step-3 `IndexBuilder`/`classify_fqn` produced for the +same nodes; the equivalence over real graphs is the identity snapshot's job. +""" + +from types import MappingProxyType + +from cgis.core.index import ClassifyContext +from cgis.core.models import Node, NodeNamespace, NodeType +from cgis.languages.python.index import PythonIndexFacts, classify, index_facts + + +def _node( + fqn: str, + node_type: NodeType = NodeType.FUNCTION, + file_path: str | None = None, + metadata: dict[str, object] | None = None, +) -> Node: + return Node( + id=fqn, + type=node_type, + name=fqn.rsplit(".", 1)[-1], + file_path=file_path or f"{fqn.replace('.', '/')}.py", + start_line=1, + end_line=1, + metadata=metadata or {}, + ) + + +def _facts(nodes: list[Node], deps: list[str] | None = None) -> PythonIndexFacts: + by_id = {n.id: n for n in nodes} + return index_facts(MappingProxyType(by_id), {"python_dependencies": deps or []}) + + +def _ctx( + facts: PythonIndexFacts, nodes: list[Node], roots: set[str], source: str | None +) -> ClassifyContext: + return ClassifyContext( + source_file=source, + internal_roots=frozenset(roots) | facts.internal_roots, + nodes=MappingProxyType({n.id: n for n in nodes}), + facts=facts, + ) + + +def test_import_roots_split_into_external_and_first_party() -> None: + """#424: `app` reaches our nodes once stripped (twice: corroborated); `pydantic` never does.""" + file_node = _node( + "domains.views", + NodeType.FILE, + file_path="domains/views.py", + metadata={ + "import_map": { + "A": "app.domains.models.A", + "B": "app.domains.models.B", + "C": "pydantic.BaseModel", + } + }, + ) + nodes = [file_node, _node("domains.models.A"), _node("domains.models.B")] + facts = _facts(nodes) + assert facts.external_roots == frozenset({"pydantic"}) + assert "app" in facts.internal_roots + assert facts.layout_prefixes == frozenset({"app"}) + + +def test_src_and_lib_layouts_add_their_second_segment() -> None: + facts = _facts([_node("src.cgis.pipeline.run"), _node("lib.util.x")]) + assert {"cgis", "util"} <= facts.internal_roots + + +def test_only_colliding_dependencies_are_kept() -> None: + """#495: a declared dependency matters only where an internal root shares its name.""" + nodes = [_node("alembic.env.run_migrations"), _node("app.main")] + facts = _facts(nodes, deps=["alembic", "fastapi"]) + assert facts.dependency_roots == frozenset({"alembic"}) + assert facts.dependency_heads == frozenset({"alembic.env"}) + + +def test_classification_chain() -> None: + nodes = [_node("alembic.env.run_migrations"), _node("json.helpers.f")] + facts = _facts(nodes, deps=["alembic"]) + ctx = _ctx(facts, nodes, {"alembic", "json"}, "app/x.py") + assert classify("alembic.env.run_migrations", ctx) is NodeNamespace.INTERNAL + assert classify("alembic.op.add_column", ctx) is NodeNamespace.EXTERNAL # declared, not ours + assert classify("json.helpers.f", ctx) is NodeNamespace.INTERNAL # our package beats stdlib + assert classify("os.path.join", ctx) is NodeNamespace.STDLIB + assert classify("len", ctx) is NodeNamespace.STDLIB + assert classify("js_builtins.Math", ctx) is NodeNamespace.STDLIB + assert classify("nothing.known", ctx) is NodeNamespace.UNKNOWN From 388341838546b28db72f3ad834af27db8ed06f43 Mon Sep 17 00:00:00 2001 From: zaebee Date: Wed, 14 Oct 2026 12:12:55 +0000 Subject: [PATCH 3/6] feat(languages): the TypeScript pack's index facts and classification (not wired yet) TypeScriptIndexFacts carries the declared npm packages and each file's bare imports; package_namespace is resolver.indices.bare_import_namespace, moved. Co-Authored-By: Claude Opus 5.5 --- src/cgis/languages/typescript/index.py | 101 +++++++++++++++++++++++++ tests/unit/test_typescript_index.py | 64 ++++++++++++++++ 2 files changed, 165 insertions(+) create mode 100644 src/cgis/languages/typescript/index.py create mode 100644 tests/unit/test_typescript_index.py diff --git a/src/cgis/languages/typescript/index.py b/src/cgis/languages/typescript/index.py new file mode 100644 index 00000000..70451ff0 --- /dev/null +++ b/src/cgis/languages/typescript/index.py @@ -0,0 +1,101 @@ +"""The TypeScript pack's index facts and classification (spec: Index and classification). + +`package_namespace` moved from `cgis.resolver.indices.bare_import_namespace` in +language-packs step 3. +""" + +import os +from collections.abc import Mapping +from dataclasses import dataclass +from types import MappingProxyType +from typing import cast + +from cgis.core.index import ClassifyContext +from cgis.core.js_globals import ( + JS_BUILTINS_ROOT, + NODE_BUILTIN_MODULES, + NODE_SCHEME, + specifier_fqn, +) +from cgis.core.models import Node, NodeNamespace, NodeType + + +@dataclass(frozen=True) +class TypeScriptIndexFacts: + """Declared npm packages and each file's bare imports (#556).""" + + internal_roots: frozenset[str] + layout_prefixes: frozenset[str] + #: npm packages the manifests declare, as written, less our workspace packages. + npm_dependencies: frozenset[str] + #: normalized file_path -> the bare specifiers that file imports, as written. + file_bare_imports: Mapping[str, frozenset[str]] + + +def index_facts(nodes: Mapping[str, Node], state: Mapping[str, object]) -> TypeScriptIndexFacts: + """Read the bare imports the extractor recorded and the declared npm packages.""" + bare = { + os.path.normpath(n.file_path): frozenset(n.metadata["bare_imports"]) + for n in nodes.values() + if n.type == NodeType.FILE and n.metadata.get("bare_imports") + } + return TypeScriptIndexFacts( + internal_roots=frozenset(), + layout_prefixes=frozenset(), + npm_dependencies=frozenset(cast("list[str]", state.get("npm_dependencies", []))), + file_bare_imports=MappingProxyType(bare), + ) + + +def package_namespace( + fqn: str, source_file: str | None, facts: "TypeScriptIndexFacts" +) -> NodeNamespace | None: + """EXTERNAL or STDLIB when `fqn` lies under a package `source_file` imports, else None. + + Only TypeScript files record bare imports, so any other source — or none — + is None without a language check here. Two things must hold (#556). + + The file imports a bare specifier `fqn` sits under — so `react.render` on a + parameter named `react` in a file that never imports React stays + unresolved, the mistake #454 fixed for Python names. + And that specifier names a package: one a `package.json` declares, which + is EXTERNAL, or a Node builtin, which is STDLIB. An import of anything else + — an undeclared package, a broken alias — keeps its UNKNOWN, because that + is a lost edge `top_unresolved` should show. + + A declared package wins over a builtin of the same name: a browser app + that declares `buffer` bundles that package, not Node's. + + The specifier is matched to a package as written, `/` and all: dotted, + `lodash.debounce` (a package of its own) would pass for a subpath of a + declared `lodash`. When a target sits under several imported specifiers the + longest decides, since it is the one that names the target's module. + """ + imported = facts.file_bare_imports.get(source_file or "", frozenset()) + under = [ + s for s in imported if fqn == specifier_fqn(s) or fqn.startswith(f"{specifier_fqn(s)}.") + ] + if not under: + return None + specifier = max(under, key=len) + # Each `/`-prefix of the specifier is a candidate package name: `@x/a/b/c` + # tries `@x`, `@x/a`, … — a set lookup each, never a scan of the manifests. + parts = specifier.split("/") + if any("/".join(parts[:i]) in facts.npm_dependencies for i in range(1, len(parts) + 1)): + return NodeNamespace.EXTERNAL + if specifier.startswith(NODE_SCHEME) or specifier.split("/")[0] in NODE_BUILTIN_MODULES: + return NodeNamespace.STDLIB + return None + + +def classify(fqn: str, ctx: ClassifyContext) -> NodeNamespace: + """A package the file imports, then our own roots, then the JS builtins; else UNKNOWN (#454).""" + package = package_namespace(fqn, ctx.source_file, cast("TypeScriptIndexFacts", ctx.facts)) + if package is not None: + return package + root = fqn.split(".", maxsplit=1)[0] + if root in ctx.internal_roots: + return NodeNamespace.INTERNAL + if root == JS_BUILTINS_ROOT: + return NodeNamespace.STDLIB + return NodeNamespace.UNKNOWN diff --git a/tests/unit/test_typescript_index.py b/tests/unit/test_typescript_index.py new file mode 100644 index 00000000..706ed2c5 --- /dev/null +++ b/tests/unit/test_typescript_index.py @@ -0,0 +1,64 @@ +"""The TypeScript pack's index facts and classification (language-packs step 3).""" + +from types import MappingProxyType + +from cgis.core.index import ClassifyContext +from cgis.core.models import Node, NodeNamespace, NodeType +from cgis.languages.typescript.index import ( + TypeScriptIndexFacts, + classify, + index_facts, + package_namespace, +) + + +def _file(fqn: str, path: str, **metadata: object) -> Node: + return Node( + id=fqn, + type=NodeType.FILE, + name=path.rsplit("/", 1)[-1], + file_path=path, + start_line=1, + end_line=1, + language="typescript", + metadata=metadata, + ) + + +def _facts(nodes: list[Node], npm: list[str]) -> TypeScriptIndexFacts: + return index_facts(MappingProxyType({n.id: n for n in nodes}), {"npm_dependencies": npm}) + + +def test_only_typescript_files_carry_bare_imports() -> None: + nodes = [ + _file("app.h", "app/h.ts", bare_imports=["react", "node:fs"]), + _file("app.plain", "app/plain.ts"), + ] + facts = _facts(nodes, ["react"]) + assert facts.file_bare_imports == {"app/h.ts": frozenset({"react", "node:fs"})} + assert facts.internal_roots == frozenset() == facts.layout_prefixes + + +def test_a_package_the_file_imports_and_the_manifest_declares_is_external() -> None: + facts = _facts([_file("app.h", "app/h.ts", bare_imports=["react", "fs", "lodash"])], ["react"]) + assert package_namespace("react.useState", "app/h.ts", facts) is NodeNamespace.EXTERNAL + assert package_namespace("fs.readFile", "app/h.ts", facts) is NodeNamespace.STDLIB + assert package_namespace("lodash.map", "app/h.ts", facts) is None # undeclared: a lost edge + assert package_namespace("react.useState", "app/other.ts", facts) is None # not imported there + + +def test_classification_chain() -> None: + facts = _facts([_file("app.h", "app/h.ts", bare_imports=["react"])], ["react"]) + ctx = ClassifyContext( + source_file="app/h.ts", + internal_roots=frozenset({"app", "json"}), + nodes=MappingProxyType({}), + facts=facts, + ) + assert classify("react.useState", ctx) is NodeNamespace.EXTERNAL + assert classify("app.util", ctx) is NodeNamespace.INTERNAL + assert classify("js_builtins.Math", ctx) is NodeNamespace.STDLIB + assert ( + classify("json.parse", ctx) is NodeNamespace.INTERNAL + ) # an internal root, not Python's json + assert classify("os.path", ctx) is NodeNamespace.UNKNOWN # #454: Python names mean nothing here From d74b1d4dc2afc87ee061ea5b35243a85493641f2 Mon Sep 17 00:00:00 2001 From: zaebee Date: Wed, 14 Oct 2026 12:18:37 +0000 Subject: [PATCH 4/6] refactor(resolver): classification and its index facts come from the language packs SymbolIndex.classify_fqn keeps the two language-free rules and hands the rest to the source file's pack (no source -> Python, the spec's preserved quirk; a source no pack reads -> the old generic tail). IndexBuilder asks every pack for its index facts; SymbolIndex loses external_roots, dependency_roots/heads, npm_dependencies and file_bare_imports, now in SymbolIndex.facts by pack name. Resolution-only facts stay until step 4 (ruling C3). symbols._package_call reads the TypeScript pack's package_namespace directly: a TS rule inside the engine, recorded under a new forbidden-edge rule engine-packs with exactly that one edge allowlisted, for step 4 to empty (C5). The Python pack subtracts its own local roots from layout_prefixes, as the old builder did: subtracting the final internal roots would empty the set, since corroborated prefixes are first-party roots by construction. Annotation calibration re-centred 775/238 -> 841/258: main already held 814/252 after steps 1-2; this step adds 27/6, all in the signatures it adds or moves. Co-Authored-By: Claude Opus 5.5 --- src/cgis/core/extractor.py | 11 +- src/cgis/core/index.py | 2 +- src/cgis/languages/__init__.py | 20 +- src/cgis/languages/python/__init__.py | 13 + src/cgis/languages/python/index.py | 3 +- src/cgis/languages/typescript/__init__.py | 13 + src/cgis/resolver/indices.py | 249 +++--------------- src/cgis/resolver/symbols.py | 9 +- .../forbidden_edges_allowlist.json | 3 + .../test_annotation_calibration.py | 12 +- tests/self_parsing/test_architecture.py | 2 + tests/unit/test_languages.py | 27 +- tests/unit/test_resolver_indices.py | 8 +- 13 files changed, 149 insertions(+), 223 deletions(-) diff --git a/src/cgis/core/extractor.py b/src/cgis/core/extractor.py index 9bd30446..9b150116 100644 --- a/src/cgis/core/extractor.py +++ b/src/cgis/core/extractor.py @@ -9,7 +9,8 @@ from pathlib import Path from typing import Protocol, runtime_checkable -from cgis.core.models import Edge, Node +from cgis.core.index import ClassifyContext, IndexFacts +from cgis.core.models import Edge, Node, NodeNamespace from cgis.core.project import ProjectReader @@ -90,3 +91,11 @@ def extractor(self, extension: str, source_roots: list[str]) -> BaseExtractor: def project_reader(self, root: Path, extractors: Mapping[str, BaseExtractor]) -> ProjectReader: """A reader for this pack's configs around `root` (`extractors`: the walk's own).""" ... + + def index_facts(self, nodes: Mapping[str, Node], state: Mapping[str, object]) -> IndexFacts: + """This pack's facts over every node, given the merged project state (step 3, C2).""" + ... + + def classify(self, fqn: str, ctx: ClassifyContext) -> NodeNamespace: + """The namespace of `fqn` referenced from one of this pack's files: the whole chain (C1).""" + ... diff --git a/src/cgis/core/index.py b/src/cgis/core/index.py index 68a7916c..5545a4ac 100644 --- a/src/cgis/core/index.py +++ b/src/cgis/core/index.py @@ -22,7 +22,7 @@ def internal_roots(self) -> frozenset[str]: @property def layout_prefixes(self) -> frozenset[str]: - """Prefixes sources write that node ids do not carry, before internal roots are removed.""" + """Prefixes sources write that node ids do not carry (#494): stripped off a target.""" ... diff --git a/src/cgis/languages/__init__.py b/src/cgis/languages/__init__.py index a8b2f614..82ec4d6c 100644 --- a/src/cgis/languages/__init__.py +++ b/src/cgis/languages/__init__.py @@ -20,10 +20,11 @@ import re from collections.abc import Callable, Iterator, Mapping from dataclasses import dataclass -from functools import cache +from functools import cache, lru_cache from pathlib import Path, PurePosixPath from cgis.core.extractor import BaseExtractor, LanguagePack +from cgis.core.index import IndexFacts from cgis.core.models import Node from cgis.core.project import ProjectReader @@ -118,6 +119,23 @@ def project_readers(root: Path, extractors: Mapping[str, BaseExtractor]) -> list return [_load(m.name).project_reader(root, extractors) for m in MANIFESTS] +def index_facts(nodes: Mapping[str, Node], state: Mapping[str, object]) -> dict[str, IndexFacts]: + """Every pack's index facts, by pack name.""" + return {m.name: _load(m.name).index_facts(nodes, state) for m in MANIFESTS} + + +@lru_cache(maxsize=4096) +def classifying_pack(source_file: str | None) -> LanguagePack | None: + """The pack that classifies a reference from `source_file`; None when no pack reads it. + + No source means the Python pack: `classify_fqn(fqn)` has always applied the + Python reading, and removing that is a separate, measured change (spec, + Index and classification). Cached: classification is on the resolver's hot + path, and a path's suffix never changes. + """ + return _load("python") if source_file is None else pack_for_path(source_file) + + def skips(filename: str) -> bool: """Whether the walk never reads this file (`foo.test.ts`, `foo.spec.py`).""" return any(m.never_ingested.search(filename) for m in MANIFESTS) diff --git a/src/cgis/languages/python/__init__.py b/src/cgis/languages/python/__init__.py index b897d101..87d08dc0 100644 --- a/src/cgis/languages/python/__init__.py +++ b/src/cgis/languages/python/__init__.py @@ -5,8 +5,11 @@ from typing import override from cgis.core.extractor import BaseExtractor, LanguagePack +from cgis.core.index import ClassifyContext, IndexFacts +from cgis.core.models import Node, NodeNamespace from cgis.core.paths import is_test_path from cgis.core.project import ProjectReader +from cgis.languages.python import index from cgis.languages.python._syntax import PYTHON_LANG, file_path_to_module_fqn from cgis.languages.python.project import PythonProjectReader @@ -44,5 +47,15 @@ def project_reader(self, root: Path, extractors: Mapping[str, BaseExtractor]) -> """This pack's config reader for the project at `root`.""" return PythonProjectReader(root) + @override + def index_facts(self, nodes: Mapping[str, Node], state: Mapping[str, object]) -> IndexFacts: + """See `cgis.languages.python.index.index_facts`.""" + return index.index_facts(nodes, state) + + @override + def classify(self, fqn: str, ctx: ClassifyContext) -> NodeNamespace: + """See `cgis.languages.python.index.classify`.""" + return index.classify(fqn, ctx) + PACK = PythonPack() diff --git a/src/cgis/languages/python/index.py b/src/cgis/languages/python/index.py index 0a315f8b..8262b981 100644 --- a/src/cgis/languages/python/index.py +++ b/src/cgis/languages/python/index.py @@ -162,7 +162,8 @@ def index_facts(nodes: Mapping[str, Node], state: Mapping[str, object]) -> Pytho colliding = (local_roots | layout_roots) & declared return PythonIndexFacts( internal_roots=frozenset(layout_roots | first_party), - layout_prefixes=frozenset(corroborated), + # Our own local roots are never stripped off a target, whatever imports say. + layout_prefixes=frozenset(corroborated - local_roots - layout_roots), external_roots=frozenset(external), dependency_roots=frozenset(colliding), dependency_heads=_heads_under(colliding, node_ids), diff --git a/src/cgis/languages/typescript/__init__.py b/src/cgis/languages/typescript/__init__.py index 3ca22b0e..0176e007 100644 --- a/src/cgis/languages/typescript/__init__.py +++ b/src/cgis/languages/typescript/__init__.py @@ -5,8 +5,11 @@ from typing import override from cgis.core.extractor import BaseExtractor, LanguagePack +from cgis.core.index import ClassifyContext, IndexFacts +from cgis.core.models import Node, NodeNamespace from cgis.core.paths import is_test_path from cgis.core.project import ProjectReader +from cgis.languages.typescript import index from cgis.languages.typescript.naming import TYPESCRIPT_LANG, file_path_to_module_fqn from cgis.languages.typescript.project import TypeScriptProjectReader @@ -44,5 +47,15 @@ def project_reader(self, root: Path, extractors: Mapping[str, BaseExtractor]) -> """This pack's config reader for the project at `root`.""" return TypeScriptProjectReader(root, extractors) + @override + def index_facts(self, nodes: Mapping[str, Node], state: Mapping[str, object]) -> IndexFacts: + """See `cgis.languages.typescript.index.index_facts`.""" + return index.index_facts(nodes, state) + + @override + def classify(self, fqn: str, ctx: ClassifyContext) -> NodeNamespace: + """See `cgis.languages.typescript.index.classify`.""" + return index.classify(fqn, ctx) + PACK = TypeScriptPack() diff --git a/src/cgis/resolver/indices.py b/src/cgis/resolver/indices.py index bcc02f60..2fd00bfa 100644 --- a/src/cgis/resolver/indices.py +++ b/src/cgis/resolver/indices.py @@ -1,18 +1,17 @@ """Symbol indices for FQN resolution: built once by IndexBuilder, read-only after.""" -import builtins import os import posixpath -import sys from collections.abc import Iterable, Mapping from dataclasses import dataclass from types import MappingProxyType from typing import NamedTuple -from cgis.core.js_globals import JS_BUILTINS_ROOT, NODE_BUILTIN_MODULES, NODE_SCHEME, specifier_fqn +from cgis.core.index import ClassifyContext, IndexFacts +from cgis.core.js_globals import JS_BUILTINS_ROOT from cgis.core.models import SELF_PREFIX, Node, NodeNamespace, NodeType - -_BUILTINS: frozenset[str] = frozenset(dir(builtins)) +from cgis.languages import classifying_pack +from cgis.languages import index_facts as pack_index_facts #: Sources whose references read against Python's stdlib, builtins and import roots. _PYTHON_SUFFIXES: tuple[str, ...] = (".py",) @@ -78,8 +77,6 @@ class SymbolIndex: suffix_map: Mapping[str, list[str]] # top-level root segments of all internal nodes (for classify) internal_roots: frozenset[str] - # root segments of absolute imports (anything else is UNKNOWN) - external_roots: frozenset[str] # prefixes the sources write but the node ids do not carry, because the ingest # root was inside the package: `app.` when `cgis ingest app/`. Learned from the # import maps, never guessed — kept apart from `internal_roots` so a target can @@ -95,20 +92,10 @@ class SymbolIndex: # then wildcards longest prefix first — the order TypeScript picks one in. # A file uses the nearest project above it (#508). path_aliases: Mapping[str, tuple[PathAlias, ...]] = MappingProxyType({}) - # import roots of the dependencies the project's manifests declare, kept only - # where they collide with an internal root: `alembic` when `app/alembic/` - # exists and `alembic` is in pyproject.toml (#495). - dependency_roots: frozenset[str] = frozenset() - # `root.second` heads our own nodes have under those roots: `alembic.env`, - # `alembic.versions`. What tells our package apart from the library's. - dependency_heads: frozenset[str] = frozenset() - # npm packages the `package.json` files declare, as written (`@x/a`, not - # `@x.a`), less this repository's own workspace packages (#556). - npm_dependencies: frozenset[str] = frozenset() - # normalized file_path -> the bare (non-relative) specifiers a TypeScript file - # imports, as written: what licenses reading a name as a package's (#556). - file_bare_imports: Mapping[str, frozenset[str]] = MappingProxyType({}) - # normalized file_path -> {local name: package symbol FQN} for those imports: + # pack name -> that pack's index facts (language-packs step 3): Python's + # external and dependency roots, TypeScript's npm packages and bare imports. + facts: Mapping[str, IndexFacts] = MappingProxyType({}) + # normalized file_path -> {local name: package symbol FQN} for a TS file's imports: # `useState` -> `react.useState`, a namespace import `Icons` -> `lucide-react`. file_bare_bindings: Mapping[str, dict[str, str]] = MappingProxyType({}) @@ -122,8 +109,8 @@ def resolve_import_target(self, fqn: str, source_file: str | None = None) -> str Only a known layout prefix is stripped. Dropping arbitrary leading segments would resolve `pydantic.config` into a project that happens to - have a `config` module, which is the collision `_strips_to_a_node` keeps - out of root classification for the same reason. + have a `config` module, which is the collision the Python pack's + `_strips_to_a_node` keeps out of root classification for the same reason. `source_file` scopes the rule to the language that taught it: the prefixes come from Python import maps, so a TypeScript path alias `import … from @@ -132,8 +119,8 @@ def resolve_import_target(self, fqn: str, source_file: str | None = None) -> str Only a *root* segment is ever a layout prefix today, so a two-level one (`src.app.`) reconciles the symbol edge and not the module edge. Left as - is: `_build_external_roots` groups by root, and widening it needs the - collision guard below to be stronger first (#495). + is: the Python pack's `_split_import_roots` groups by root, and widening it + needs the collision guard below to be stronger first (#495). """ if fqn in self.nodes: return fqn @@ -284,6 +271,8 @@ def classify_fqn(self, fqn: str, source_file: str | None = None) -> NodeNamespac The check names Python rather than excluding TypeScript, so a language added later starts from UNKNOWN instead of inheriting Python's names. Without `source_file` the Python reading applies, as it always has. + + The language rules are the source's pack's (language-packs step 3). """ if fqn.startswith("."): return NodeNamespace.INTERNAL @@ -293,23 +282,20 @@ def classify_fqn(self, fqn: str, source_file: str | None = None) -> NodeNamespac # let unresolved_ratio *improve* as a codebase adopted more dependency # injection, so a green `validate` said nothing about the gap (#414). return NodeNamespace.UNKNOWN - package = bare_import_namespace(self, fqn, source_file) - if package is not None: - return package - root = fqn.split(".", maxsplit=1)[0] - if root in self.internal_roots: - if _is_declared_dependency(self, fqn, root, source_file): - return NodeNamespace.EXTERNAL - return NodeNamespace.INTERNAL - if root == JS_BUILTINS_ROOT: - return NodeNamespace.STDLIB - if source_file is not None and not source_file.endswith(_PYTHON_SUFFIXES): - return NodeNamespace.UNKNOWN - if root in sys.stdlib_module_names or root in _BUILTINS: - return NodeNamespace.STDLIB - if root in self.external_roots: - return NodeNamespace.EXTERNAL - return NodeNamespace.UNKNOWN + pack = classifying_pack(source_file) + if pack is None: + # A source no pack reads: the rules every language shared before step 3. + root = fqn.split(".", maxsplit=1)[0] + if root in self.internal_roots: + return NodeNamespace.INTERNAL + return NodeNamespace.STDLIB if root == JS_BUILTINS_ROOT else NodeNamespace.UNKNOWN + context = ClassifyContext( + source_file=source_file, + internal_roots=self.internal_roots, + nodes=self.nodes, + facts=self.facts[pack.name], + ) + return pack.classify(fqn, context) def is_variable_node(self, fqn: str) -> bool: """Return True when fqn names an existing VARIABLE node in the graph.""" @@ -324,119 +310,10 @@ def normalized_file_path(self, source_fqn: str, edge_file_path: str | None) -> s return os.path.normpath(edge_file_path) if edge_file_path else None -def bare_import_namespace( - index: SymbolIndex, fqn: str, source_file: str | None -) -> NodeNamespace | None: - """EXTERNAL or STDLIB when `fqn` lies under a package `source_file` imports, else None. - - Only TypeScript files record bare imports, so any other source — or none — - is None without a language check here. Two things must hold (#556). - - The file imports a bare specifier `fqn` sits under — so `react.render` on a - parameter named `react` in a file that never imports React stays - unresolved, the mistake #454 fixed for Python names. - And that specifier names a package: one a `package.json` declares, which - is EXTERNAL, or a Node builtin, which is STDLIB. An import of anything else - — an undeclared package, a broken alias — keeps its UNKNOWN, because that - is a lost edge `top_unresolved` should show. - - A declared package wins over a builtin of the same name: a browser app - that declares `buffer` bundles that package, not Node's. - - The specifier is matched to a package as written, `/` and all: dotted, - `lodash.debounce` (a package of its own) would pass for a subpath of a - declared `lodash`. When a target sits under several imported specifiers the - longest decides, since it is the one that names the target's module. - """ - imported = index.file_bare_imports.get(source_file or "", frozenset()) - under = [ - s for s in imported if fqn == specifier_fqn(s) or fqn.startswith(f"{specifier_fqn(s)}.") - ] - if not under: - return None - specifier = max(under, key=len) - # Each `/`-prefix of the specifier is a candidate package name: `@x/a/b/c` - # tries `@x`, `@x/a`, … — a set lookup each, never a scan of the manifests. - parts = specifier.split("/") - if any("/".join(parts[:i]) in index.npm_dependencies for i in range(1, len(parts) + 1)): - return NodeNamespace.EXTERNAL - if specifier.startswith(NODE_SCHEME) or specifier.split("/")[0] in NODE_BUILTIN_MODULES: - return NodeNamespace.STDLIB - return None - - -def _is_declared_dependency( - index: SymbolIndex, fqn: str, root: str, source_file: str | None -) -> bool: - """True when an internal-root `fqn` names a declared dependency, not our package. - - The root alone cannot say: `app/alembic/` makes `alembic` internal, while - `from alembic import op` is the library (#495). Our package is what it - holds, so `alembic.env.run_migrations` stays internal and - `alembic.op.add_column` — `op` being nothing of ours — is the dependency. - - The manifest is what licenses the second reading, never the graph alone: - a graph-only rule declared `scripts.ci.openapi_snapshot` third-party - because that module lives outside the ingest root, though it is ours and - `scripts` is no dependency. The roots come from Python manifests, so only - a Python source (or an unscoped call) reads them, as in #454. - """ - if root not in index.dependency_roots or fqn in index.nodes: - return False - if source_file is not None and not source_file.endswith(_PYTHON_SUFFIXES): - return False - return ".".join(fqn.split(".", maxsplit=2)[:2]) not in index.dependency_heads - - -#: How many of a root's imports must reach a node before it may be stripped off a -#: target (#498 review). One is not evidence: a third-party module that happens to -#: share a path with ours — `from boto3.utils.helpers import x` against our own -#: `utils/helpers.py` — made `boto3` first-party, and stripping it wired -#: `import boto3.config` into this project's `config.py` at full confidence. -#: Classification itself still accepts one, because there the cost of a wrong call -#: is a namespace label rather than a fabricated internal edge. -_LAYOUT_PREFIX_EVIDENCE = 2 - #: The head a relative import in the ingest root's own `__init__.py` carries (#319). _ROOT_PACKAGE = "__init__" -def _strips_to_a_node(imported_fqn: str, node_ids: set[str]) -> bool: - """True when dropping leading segments of `imported_fqn` reaches a real node. - - `app.domains.models.Thing` against a graph rooted at `domains` strips to - `domains.models.Thing`, which exists — so `app` is this project's own name, - not a package it depends on. - - The remainder must keep at least two segments. Stripping to a single one - matches any library whose submodule shares a name with a top-level module of - ours — `from pydantic import config` against a project that has `config.py` - strips to `config`, and the whole of pydantic would then classify INTERNAL, - breaking resolution for every symbol in it. Two segments cannot collide by - accident, and a genuine first-party prefix always has deeper imports than - that: on owner-api 3346 values resolve, none of them needing this. - """ - parts = imported_fqn.split(".") - return any(".".join(parts[i:]) in node_ids for i in range(1, len(parts) - 1)) - - -def _heads_under(roots: set[str], node_ids: Iterable[str]) -> frozenset[str]: - """The `root.second` heads of node ids under `roots`, read past a `src`/`lib` layout. - - The same two spellings `_add_node_to_suffix_map` makes a root internal by. - """ - if not roots: - return frozenset() - heads: set[str] = set() - for node_id in node_ids: - parts = node_id.split(".") - start = 1 if len(parts) > 2 and parts[0] in {"src", "lib"} else 0 - for offset in {0, start}: - if len(parts) > offset + 1 and parts[offset] in roots: - heads.add(f"{parts[offset]}.{parts[offset + 1]}") - return frozenset(heads) - - def _resolve_typescript_import( fqn: str, nodes: Mapping[str, Node], @@ -567,7 +444,6 @@ def build( file_variable_symbols: dict[tuple[str, str], list[str]] = {} file_imports: dict[str, dict[str, str]] = {} file_shadowed_globals: dict[str, frozenset[str]] = {} - file_bare_imports: dict[str, frozenset[str]] = {} file_bare_bindings: dict[str, dict[str, str]] = {} reexports: dict[str, dict[str, str]] = {} star_imports: dict[str, list[str]] = {} @@ -582,7 +458,6 @@ def build( self_types, reexports, star_imports, - file_bare_imports, file_bare_bindings, ) if node.type == NodeType.FILE and node.metadata.get("shadowed_globals"): @@ -608,10 +483,13 @@ def build( # "src.cgis.pipeline.X" → suffix "cgis.pipeline.X" also points to the node self._add_node_to_suffix_map(node.id, suffix_map, internal_roots) - colliding = internal_roots.intersection(dependency_roots) - external_roots, first_party, corroborated = self._build_external_roots( - file_imports, set(nodes_by_id) - ) + state = { + "python_dependencies": list(dependency_roots), + "npm_dependencies": list(npm_dependencies), + } + facts = pack_index_facts(MappingProxyType(nodes_by_id), state) + roots = frozenset(internal_roots.union(*(f.internal_roots for f in facts.values()))) + prefixes = frozenset[str]().union(*(f.layout_prefixes for f in facts.values())) self._expand_star_imports(star_imports, reexports, set(nodes_by_id)) # Read-only views, not copies: MappingProxyType wraps in O(1), so this @@ -628,19 +506,15 @@ def build( file_shadowed_globals=MappingProxyType(file_shadowed_globals), reexports=MappingProxyType(reexports), suffix_map=MappingProxyType(suffix_map), - internal_roots=frozenset(internal_roots | first_party), - external_roots=frozenset(external_roots), - layout_prefixes=frozenset(corroborated - internal_roots), + internal_roots=roots, + layout_prefixes=prefixes, + facts=MappingProxyType(facts), workspace_packages=MappingProxyType( dict(sorted((workspace_packages or {}).items(), key=lambda item: -len(item[0]))) ), path_aliases=MappingProxyType( {scope: _ordered_aliases(rules) for scope, rules in (tsconfig_paths or {}).items()} ), - dependency_roots=frozenset(colliding), - dependency_heads=_heads_under(colliding, nodes_by_id), - npm_dependencies=frozenset(npm_dependencies), - file_bare_imports=MappingProxyType(file_bare_imports), file_bare_bindings=MappingProxyType(file_bare_bindings), ) @@ -741,7 +615,6 @@ def _index_metadata( self_types: dict[str, dict[str, str]], reexports: dict[str, dict[str, str]], star_imports: dict[str, list[str]], - bare_imports: dict[str, frozenset[str]], bare_bindings: dict[str, dict[str, str]], ) -> None: """Read the maps the extractor attached to a FILE or CLASS node. @@ -756,7 +629,6 @@ def _index_metadata( file_imports[path] = node.metadata.get("import_map") or {} specifiers = node.metadata.get("bare_imports") if specifiers: - bare_imports[path] = frozenset(specifiers) bare_bindings[path] = dict(node.metadata.get("bare_bindings") or {}) forwarded = node.metadata.get("reexports") if forwarded: @@ -790,50 +662,7 @@ def _add_node_to_suffix_map( ) -> None: """Add node to suffix map and internal roots based on its ID.""" parts = node_id.split(".") - internal_roots.add(parts[0]) - if len(parts) > 1 and parts[0] in {"src", "lib"}: - internal_roots.add(parts[1]) + internal_roots.add(parts[0]) # `src`/`lib` second segments are the Python pack's for i in range(1, len(parts)): suffix = ".".join(parts[i:]) suffix_map.setdefault(suffix, []).append(node_id) - - def _build_external_roots( - self, file_imports: dict[str, dict[str, str]], node_ids: set[str] - ) -> tuple[set[str], set[str], set[str]]: - """Split import roots into genuinely external ones and first-party prefixes. - - Returns (external_roots, first_party_prefixes, corroborated_prefixes) — - the third being those with enough evidence to strip off a target, not only - to classify (see `_LAYOUT_PREFIX_EVIDENCE`). - - A root is a *first-party prefix* when some import under it reaches a real - node once its head is stripped. That happens whenever a project is - ingested at a subdirectory: `cgis ingest ownima-backend/app` roots its - nodes at `domains`, `api`, … while the code imports `from app.models - import X`, so `app` looks like a third-party package. Every `app.*` FQN - without a node then classified EXTERNAL, and the resolver minted a - boundary node for a class of ours that does not exist — 2 461 confident - edges to nothing on that codebase (#424). - - The discriminator is unambiguous in practice: measured there, `app` had - 3 346 of 4 885 import values reach a node while pydantic, sqlalchemy, - grpc, fastapi and httpx had zero each. It asks whether the *import value* - resolves, not whether some node's id ends with the root's name — the - latter would match the library `grpc` against a project's own - `api/dependencies/grpc/` package. - """ - imports_by_root: dict[str, set[str]] = {} - for import_map in file_imports.values(): - for val in import_map.values(): - if val and not val.startswith("."): - imports_by_root.setdefault(val.split(".", maxsplit=1)[0], set()).add(val) - - external: set[str] = set() - first_party: set[str] = set() - corroborated: set[str] = set() - for root, values in imports_by_root.items(): - resolving = sum(1 for v in values if _strips_to_a_node(v, node_ids)) - (first_party if resolving else external).add(root) - if resolving >= _LAYOUT_PREFIX_EVIDENCE: - corroborated.add(root) - return external, first_party, corroborated diff --git a/src/cgis/resolver/symbols.py b/src/cgis/resolver/symbols.py index 517f6682..16fa1b55 100644 --- a/src/cgis/resolver/symbols.py +++ b/src/cgis/resolver/symbols.py @@ -3,7 +3,8 @@ from collections.abc import Iterable, Mapping from cgis.core.models import RAW_CLASS_PREFIX, Edge, EdgeType, Node, NodeNamespace -from cgis.resolver.indices import IndexBuilder, SymbolIndex, bare_import_namespace +from cgis.languages.typescript.index import TypeScriptIndexFacts, package_namespace +from cgis.resolver.indices import IndexBuilder, SymbolIndex def _owning_class(index: SymbolIndex, source_fqn: str) -> str | None: @@ -40,7 +41,11 @@ def _package_call(index: SymbolIndex, name: str, file_path: str) -> str | None: if bound is None: return None candidate = f"{bound}.{rest}" if rest else bound - if bare_import_namespace(index, candidate, file_path) is None: + # A TypeScript rule inside the engine until step 4 makes it the TS pack's step (C5). + facts = index.facts.get("typescript") + if not isinstance(facts, TypeScriptIndexFacts): + return None + if package_namespace(candidate, file_path, facts) is None: return None return candidate diff --git a/tests/self_parsing/forbidden_edges_allowlist.json b/tests/self_parsing/forbidden_edges_allowlist.json index 3e056584..c39a6b7d 100644 --- a/tests/self_parsing/forbidden_edges_allowlist.json +++ b/tests/self_parsing/forbidden_edges_allowlist.json @@ -1,6 +1,9 @@ { "core-isolation": {}, "cross-pack": {}, + "engine-packs": { + "cgis.resolver.symbols -> cgis.languages.typescript.index": 4 + }, "pack-isolation": {}, "resolver-storage": { "cgis.resolver.uplift -> cgis.storage.sqlite_store": 8 diff --git a/tests/self_parsing/test_annotation_calibration.py b/tests/self_parsing/test_annotation_calibration.py index 01751c85..4a9c5ad8 100644 --- a/tests/self_parsing/test_annotation_calibration.py +++ b/tests/self_parsing/test_annotation_calibration.py @@ -50,8 +50,16 @@ # had drifted to 708+55 through ordinary growth — and #545 adds 12 / 4 (the # DISPATCHES pass's and the store's new signatures: Node, Edge, EdgeType, NodeType, # Mapping, SQLiteStore). Both re-centred on the branch measurement. -_EXPECTED_ANNOTATION_REFERENCES = 775 -_EXPECTED_NAME_REFERENCES = 238 +# +# Re-measured 2026-10-14 on the language-packs step-3 branch (fresh `cgis ingest src +# --source-root src`): origin/main 804a398 held 814 annotation / 252 name references — +# steps 1-2 (#597, #599) grew the annotation band to 775+39 through new pack modules — +# and step 3 adds 27 / 6, every one in a signature it adds or moves: the two packs' +# index modules (+6 each), their __init__s (+4 each), core/extractor.py (+4), the +# registry (+3), core/index.py (+2), less resolver/indices.py (-2). Both re-centred on +# the branch measurement. +_EXPECTED_ANNOTATION_REFERENCES = 841 +_EXPECTED_NAME_REFERENCES = 258 _TOLERANCE = 60 # ~10% of the annotation band: ordinary code churn, not a lost source _NAME_REF_TOLERANCE = 25 # the name-reference band is smaller, so its band is too diff --git a/tests/self_parsing/test_architecture.py b/tests/self_parsing/test_architecture.py index 00e2bb3b..cd05cf6c 100644 --- a/tests/self_parsing/test_architecture.py +++ b/tests/self_parsing/test_architecture.py @@ -242,6 +242,8 @@ class _Rule: _Rule("pack-isolation", ("cgis.languages",), ("cgis.storage", "cgis.query", "cgis.api", "cgis.guardian", "cgis.pipeline")), _Rule("resolver-storage", ("cgis.resolver",), ("cgis.storage",)), + _Rule("engine-packs", ("cgis.resolver",), ("cgis.languages",), + exempt_modules=frozenset({"cgis.languages"})), _Rule("surface-packs", ("cgis.query", "cgis.guardian", "cgis.cli", "cgis.api"), ("cgis.languages",), exempt_modules=frozenset({"cgis.languages"})), _Rule("surface-storage", ("cgis.cli", "cgis.api", "cgis.guardian"), ("cgis.storage",)), diff --git a/tests/unit/test_languages.py b/tests/unit/test_languages.py index 5ac50a63..19e79364 100644 --- a/tests/unit/test_languages.py +++ b/tests/unit/test_languages.py @@ -14,11 +14,12 @@ import pytest -from cgis.core.models import Node, NodeType +from cgis.core.models import Node, NodeNamespace, NodeType from cgis.core.paths import is_test_path from cgis.languages import ( MANIFESTS, Manifest, + classifying_pack, extractors, is_supported, pack_for_node, @@ -28,6 +29,7 @@ skips, ) from cgis.pipeline import IngestionPipeline +from cgis.resolver.indices import IndexBuilder TS_SOURCE = """ export function helper(value: number): number { @@ -313,3 +315,26 @@ def test_a_root_package_json_does_not_load_the_typescript_grammar(tmp_path: Path "package.json": '{"name": "tooling", "devDependencies": {"prettier": "3"}}', } assert _cli_ingest_loads(tmp_path, files) == ["tree_sitter_python"] + + +def test_no_source_classifies_as_python() -> None: + """Spec quirk, kept in the registry: classify_fqn(fqn) without a source reads it as Python.""" + pack = classifying_pack(None) + assert pack is not None + assert pack.name == "python" + index = IndexBuilder().build([]) + assert index.classify_fqn("os.path") is NodeNamespace.STDLIB + assert index.classify_fqn("os.path", "app/x.ts") is NodeNamespace.UNKNOWN + + +@pytest.mark.parametrize("source", ["web/a.js", "types/x.pyi", "Foo.PY"]) +def test_a_source_no_pack_reads_gets_the_generic_tail(source: str) -> None: + """Internal root → INTERNAL, the JS root → STDLIB, anything else UNKNOWN — as before.""" + assert classifying_pack(source) is None + node = Node( + id="app.f", type=NodeType.FUNCTION, name="f", file_path="app/f.py", start_line=1, end_line=1 + ) + index = IndexBuilder().build([node]) + assert index.classify_fqn("app.g", source) is NodeNamespace.INTERNAL + assert index.classify_fqn("js_builtins.Math", source) is NodeNamespace.STDLIB + assert index.classify_fqn("os.path", source) is NodeNamespace.UNKNOWN diff --git a/tests/unit/test_resolver_indices.py b/tests/unit/test_resolver_indices.py index 8c748fbb..32c05f0b 100644 --- a/tests/unit/test_resolver_indices.py +++ b/tests/unit/test_resolver_indices.py @@ -61,7 +61,7 @@ def test_build_indexes_file_import_map() -> None: ) index = IndexBuilder().build([file_node]) assert index.file_imports["mod.py"] == {"Dep": "other.Dep"} - assert index.external_roots == {"other"} + assert index.facts["python"].external_roots == {"other"} # --------------------------------------------------------------------------- @@ -210,7 +210,7 @@ def test_index_root_sets_are_frozen() -> None: index = IndexBuilder().build([_node("mod.fn")]) assert isinstance(index.internal_roots, frozenset) - assert isinstance(index.external_roots, frozenset) + assert isinstance(index.facts["python"].external_roots, frozenset) def test_index_still_reads_normally_through_the_views() -> None: @@ -451,12 +451,12 @@ def test_without_a_manifest_a_colliding_root_stays_internal() -> None: """No declared dependencies, no change: the pre-#495 reading is the fallback.""" index = _owner_api_index(dependency_roots=()) assert index.classify_fqn("alembic.op.add_column") is NodeNamespace.INTERNAL - assert index.dependency_roots == frozenset() + assert index.facts["python"].dependency_roots == frozenset() def test_only_colliding_dependencies_are_kept() -> None: """A dependency no internal root shares a name with is left to `external_roots`.""" - assert _owner_api_index().dependency_roots == frozenset({"alembic"}) + assert _owner_api_index().facts["python"].dependency_roots == frozenset({"alembic"}) def test_a_typescript_source_does_not_read_python_dependencies() -> None: From 44ad11cb08cca7585fc2d1716a644dbe3c9bb77c Mon Sep 17 00:00:00 2001 From: zaebee Date: Wed, 14 Oct 2026 12:37:21 +0000 Subject: [PATCH 5/6] docs: step 3 implementation notes; inventory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured on the branch (fresh ingest of src/, main 804a398 alongside): - extraction drift 0.250 -> 0.251, unresolved 0.334 -> 0.328 (lock 0.40/0.37) - resolution drift 0.385 -> 0.386, unresolved 0.433 -> 0.442 (lock 0.40/0.45; 71 nodes, down from 76 — the moved classification code left the domain) - owner-api ingest, median of three: main 22.25 s, branch 22.57 s (+1.5%) Review-path inventory gains core/index.py and the two packs' index modules. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 4 +++- docs/specs/2026-10-14-language-packs-design.md | 15 +++++++++++++++ tests/unit/review_path_inventory.txt | 3 +++ 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index f9a96a11..5abf7d20 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -75,7 +75,9 @@ IngestionPipeline (pipeline.py) (`parse(code, file_path) -> tuple[list[Node], list[Edge]]`; emit `CALLS` edges with `target="raw_call:"`, set `Node.language = ""`). 2. In its `__init__.py`, define `PACK`, subclassing `cgis.core.extractor.LanguagePack` (mark methods `@override`) - (`name`, `extensions`, `code_fence`, `is_test`, `scope_of`, `extractor`, `project_reader`). + (`name`, `extensions`, `code_fence`, `is_test`, `scope_of`, `extractor`, `project_reader`, + `index_facts`, `classify`). `classify` answers for references from the pack's own files and + runs the whole chain; a source with no pack gets the generic tail in `SymbolIndex.classify_fqn`. 3. Add one `Manifest` to `MANIFESTS` in `src/cgis/languages/__init__.py`, with a `load` function that imports the pack statically inside it (lazy, yet visible to the review-fingerprint walk). Nothing else names the pack: CLI, MCP server, guardian and scripts all go through the registry, and the diff --git a/docs/specs/2026-10-14-language-packs-design.md b/docs/specs/2026-10-14-language-packs-design.md index a81e1553..b7aea6cc 100644 --- a/docs/specs/2026-10-14-language-packs-design.md +++ b/docs/specs/2026-10-14-language-packs-design.md @@ -224,6 +224,21 @@ aliases. `import_names.py` becomes the collector that asks each pack. and 294, and indices.py:234). The registry keeps that: with no source, the Python pack classifies. Removing the quirk is a separate, measured change. + > **Implementation note (step 3):** `classify` returns a `NodeNamespace`, never + > `None`: today's order interleaves the languages (TS bare imports, internal + > roots with Python's declared-dependency rule, the JS root, Python stdlib and + > external roots), so each pack runs its whole chain (ruling C1). A source no + > pack reads gets the old generic tail in `SymbolIndex.classify_fqn`. + > `index_facts` takes the merged project state keyed by `ingest_state` key, not + > the pack's `ProjectFacts` object, since the resolver receives facts as values + > (C2). Resolution-only facts — `layout_prefixes`, star-import expansion, + > workspace packages, path aliases, bare bindings — stay on `SymbolIndex` until + > step 4 moves their consumers (C3). The outline's "`resolve_layout` gated by + > pack" was dropped: both run for every language today, and gating them changes + > TypeScript resolution (C4). One engine-to-pack edge remains, `_package_call` + > → the TS pack's `package_namespace`, allowlisted under the `engine-packs` + > rule for step 4 to empty (C5). The owner-api ingest moved +1.5% (median of three). + **Resolution.** The engine runs named steps in a fixed order. A pack contributes to them; it does not get free hook points: diff --git a/tests/unit/review_path_inventory.txt b/tests/unit/review_path_inventory.txt index 5dd857c2..7edb5352 100644 --- a/tests/unit/review_path_inventory.txt +++ b/tests/unit/review_path_inventory.txt @@ -2,6 +2,7 @@ src/cgis/__init__.py src/cgis/core/coverage.py src/cgis/core/extractor.py src/cgis/core/freshness.py +src/cgis/core/index.py src/cgis/core/js_globals.py src/cgis/core/models.py src/cgis/core/paths.py @@ -37,10 +38,12 @@ src/cgis/languages/python/_syntax.py src/cgis/languages/python/_types.py src/cgis/languages/python/dependencies.py src/cgis/languages/python/extractor.py +src/cgis/languages/python/index.py src/cgis/languages/python/project.py src/cgis/languages/typescript/__init__.py src/cgis/languages/typescript/_bindings.py src/cgis/languages/typescript/extractor.py +src/cgis/languages/typescript/index.py src/cgis/languages/typescript/naming.py src/cgis/languages/typescript/npm.py src/cgis/languages/typescript/project.py From e8b0fb82b13d7203f21b5df8bdfb189284fc83e8 Mon Sep 17 00:00:00 2001 From: zaebee Date: Wed, 14 Oct 2026 12:53:21 +0000 Subject: [PATCH 6/6] fix(resolver): SymbolIndex.facts is required, not an empty default Classification reads facts[pack]; with an empty default an index made without them raised a KeyError at the first classification, far from the cause. A guarded fallback (Gemini's suggestion) would be worse: a Python reference would silently take the generic tail. Required, the mistake fails where the index is made (final review minor, Gemini). Co-Authored-By: Claude Opus 5.5 --- src/cgis/resolver/indices.py | 7 ++++--- tests/unit/test_resolver_indices.py | 13 +++++++++++++ 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/src/cgis/resolver/indices.py b/src/cgis/resolver/indices.py index 2fd00bfa..c7439f46 100644 --- a/src/cgis/resolver/indices.py +++ b/src/cgis/resolver/indices.py @@ -77,6 +77,10 @@ class SymbolIndex: suffix_map: Mapping[str, list[str]] # top-level root segments of all internal nodes (for classify) internal_roots: frozenset[str] + # pack name -> that pack's index facts (language-packs step 3): Python's + # external and dependency roots, TypeScript's npm packages and bare imports. + # Required: classification reads it, so an index without it fails where made. + facts: Mapping[str, IndexFacts] # prefixes the sources write but the node ids do not carry, because the ingest # root was inside the package: `app.` when `cgis ingest app/`. Learned from the # import maps, never guessed — kept apart from `internal_roots` so a target can @@ -92,9 +96,6 @@ class SymbolIndex: # then wildcards longest prefix first — the order TypeScript picks one in. # A file uses the nearest project above it (#508). path_aliases: Mapping[str, tuple[PathAlias, ...]] = MappingProxyType({}) - # pack name -> that pack's index facts (language-packs step 3): Python's - # external and dependency roots, TypeScript's npm packages and bare imports. - facts: Mapping[str, IndexFacts] = MappingProxyType({}) # normalized file_path -> {local name: package symbol FQN} for a TS file's imports: # `useState` -> `react.useState`, a namespace import `Icons` -> `lucide-react`. file_bare_bindings: Mapping[str, dict[str, str]] = MappingProxyType({}) diff --git a/tests/unit/test_resolver_indices.py b/tests/unit/test_resolver_indices.py index 32c05f0b..1d72acba 100644 --- a/tests/unit/test_resolver_indices.py +++ b/tests/unit/test_resolver_indices.py @@ -1,5 +1,7 @@ """Unit tests for SymbolIndex and IndexBuilder.""" +import dataclasses + import pytest from cgis.core.models import Node, NodeNamespace, NodeType @@ -473,3 +475,14 @@ def test_a_src_layout_package_keeps_its_heads() -> None: ) assert index.classify_fqn("alembic.env.run") is NodeNamespace.INTERNAL assert index.classify_fqn("alembic.op.add_column") is NodeNamespace.EXTERNAL + + +def test_an_index_cannot_be_built_without_its_packs_facts() -> None: + """Classification reads `facts[pack]`: an index lacking them must fail where it is made. + + With an empty default the first classification would raise a KeyError far + from the cause — or, guarded, silently fall back and misread a Python name. + """ + facts = next(f for f in dataclasses.fields(SymbolIndex) if f.name == "facts") + assert facts.default is dataclasses.MISSING + assert facts.default_factory is dataclasses.MISSING