From e8b38a6d808e2061c1ab6223a47a10fd61984816 Mon Sep 17 00:00:00 2001 From: zaebee Date: Wed, 14 Oct 2026 07:21:24 +0000 Subject: [PATCH 1/4] docs(spec): language packs and a two-plane topology MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design for #146 and for adding languages: cgis is a build plane (pipeline → languages → resolver → storage) and a read plane (storage → query → surfaces) over a shared core, and the observe-only quotient cannot be enforced because one triad census averages the two (0.644 against pipeline_stage, 0.527 orchestrator, 0.492 layered_dag at b0d67f8). - A language becomes one package behind a LanguagePack contract (parser, project reader, classify, call rules) and one registry line; Go is the pilot that proves it. - Drift: one quotient per plane plus an enforced forbidden-edge list, replacing the single project-level binding. - Migration in eight PRs; steps 1–5 move code under a graph-identity test over a checked-in corpus and leave every graph's node and edge sets unchanged. Refs #146 Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-14-language-packs-design.md | 219 ++++++++++++++++++ 1 file changed, 219 insertions(+) create mode 100644 docs/specs/2026-10-14-language-packs-design.md diff --git a/docs/specs/2026-10-14-language-packs-design.md b/docs/specs/2026-10-14-language-packs-design.md new file mode 100644 index 00000000..61ec988c --- /dev/null +++ b/docs/specs/2026-10-14-language-packs-design.md @@ -0,0 +1,219 @@ +# Language packs and a two-plane topology — design + +Date: 2026-10-14. Status: proposed. Closes #146 on completion. + +## Why + +Two problems turned out to be one. + +**The domain-level drift gate cannot be enforced (#146).** Since #144, the +quotient graph (each cgis domain collapsed to a node) has been scored +observe-only against `pipeline_stage`. It measured 0.700 in June and 0.644 on +2026-10-14 (b0d67f8), with a tolerance of 0.15. No template fits: +`orchestrator` scores 0.527 and `layered_dag` 0.492. The reason is the shape, +not the template. Cgis is two different structures over one core, and a +single triad census averages them into neither: + +``` +core ← every package depends on it; it depends on nothing +storage → core +extractors → core +resolver → core, storage (8 edges) +pipeline → extractors, resolver, storage, import_names build plane +query → core, storage, extractors (3 edges) read plane +guardian → query, storage, extractors +cli, api → query, storage, pipeline, extractors, guardian +import_names → workspaces, tsconfig_paths, npm_/python_dependencies, storage +``` + +(Imports and calls between top-level packages, cgis at b0d67f8.) + +**Adding a language means touching the whole system.** Knowledge about each +language is spread over five shared modules. `resolver/indices.py` has about +65 Python-specific and 65 TS-specific references; `symbols.py`, `engine.py`, +`pipeline.py` and `import_names.py` have dozens more. Four ecosystem readers +sit at the package root (`tsconfig_paths`, `workspaces`, `npm_dependencies`, +`python_dependencies`). `extractors/registry.py` already says "adding a +language should touch this file and nothing else", but that is true only for +parsing. + +## Goals + +1. A language is one package, `cgis/languages//`, plus one line in a + registry. Adding Go, Rust, PHP or Astro does not edit the core, the + resolver or the pipeline. +2. Drift describes the real shape: one check per plane, plus forbidden edges. + An enforced domain-level gate replaces the observe-only quotient. +3. The leaking edges `resolver → storage` and `query → extractors` are gone. + +**Success:** the Go pilot (step 7) lands with a diff confined to +`languages/go/`, the registry line and tests. Both plane checks are enforced. +The forbidden-edge allowlist is empty. + +## Non-goals + +- Changing the graph format, FQN rules or MCP tool surface. The refactor is + internal, and steps 1–5 must leave every graph with identical node and edge sets. +- Astro (#227), PHP and Rust. They follow the contract once Go has proven it. +- Revising the drift formula (#554). +- A second storage backend, or new surfaces (approach C, rejected as YAGNI). + +## Architecture + +Two planes over a shared core: + +``` + core (graph model, FQN helpers, Node/Edge) + ▲ ▲ + ── build ── │ ── read ── │ + pipeline ─► languages/ ─► resolver ─► storage ─► query ─► cli | api | guardian + (parser + project (language-free + reader + rules) engine) +``` + +- **Build plane.** `pipeline` orchestrates: it asks the registry for each + file's pack, runs its extractor and project reader, hands nodes, edges and + project facts to the resolver, and persists the result. +- **Read plane.** `storage → query → surfaces`, layered. +- **Core.** Model types and FQN utilities. It imports nothing from cgis. + +### The `LanguagePack` contract + +```python +class LanguagePack(Protocol): + name: str # "python", "typescript", "go" + extensions: tuple[str, ...] # (".py",), (".ts", ".tsx"), (".go",) + code_fence: str + + def extractor(self, source_roots: list[str]) -> BaseExtractor: ... + def module_fqn(self, path: str, source_root: str | None) -> str: ... + def project_reader(self, root: Path) -> ProjectReader: ... + def classify(self, fqn: str, facts: ProjectFacts) -> NodeNamespace | None: ... + def call_rules(self) -> Sequence[CallRule]: ... +``` + +| Part | Replaces | Notes | +|---|---|---| +| `extensions`, `extractor`, `module_fqn`, `code_fence` | `extractors/registry.py` `Language` | `module_fqn` must stay the function the extractor uses for FILE ids (existing round-trip test). | +| `project_reader` | `import_names.py`, `workspaces.py`, `tsconfig_paths.py`, `npm_dependencies.py`, `python_dependencies.py` | It notes manifests during the walk and configs above the root (#555), and yields `ProjectFacts`. The facts are serialisable and recorded in the store under today's keys, so `ImportNames.differ_from` and freshness keep working on existing graphs. | +| `classify` | `classify_fqn` branches in `resolver/indices.py`, `resolver/js_builtins.py` | It returns STDLIB/EXTERNAL for what the pack recognises, or None to defer to the generic rules. | +| `call_rules` | language branches in `resolver/symbols.py` (`self.x`, `Annotated` aliases, npm package calls, …) | A `CallRule` declares a hook point (`before_import_map`, `after_import_map`, `on_receiver`). The engine runs generic steps and pack rules in a fixed order; with no rule the generic chain is unchanged. | + +**Registry.** `cgis/languages/__init__.py` lists the packs. It is the only +module outside `languages/` that names a language. `pipeline`, `query` and +`guardian` ask it "whose file is this?" and nothing else. + +**Rules for packs.** A pack does not import another pack. Shared TS logic that +Astro will need moves to a shared module, or the Astro pack declares it +extends the TS pack through the registry. A pack never touches `storage`: +anything beyond nodes reaches the resolver as `ProjectFacts`. + +## Drift gates + +Domains after migration (`docs/ontology/patterns.yaml`): + +| Domain | Template | Notes | +|---|---|---| +| `core` | `pure_utility` | new | +| `languages.python`, `languages.typescript`, `languages.go` | `layered_dag`, one shared tolerance | replaces `extraction` | +| `resolver` | `pipeline_stage` | unchanged (engine → symbols → indices) | +| `pipeline` | `orchestrator` | unchanged | +| `storage` | `pure_utility` | unchanged | +| `query` | `layered_dag` | unchanged | +| `cli`, `api`, `guardian` | hygiene / as today | unchanged | + +Project-level checks replace the single `project_level` quotient binding: + +1. **Build plane:** the quotient over `{pipeline, languages.*, resolver, + storage}` against `orchestrator`. +2. **Read plane:** the quotient over `{storage, query, guardian, cli, api}` + against `layered_dag`. +3. **Forbidden edges:** a new list in `patterns.yaml` of `from → to` prefix + pairs that no IMPORTS/IMPORTS_SYMBOL/CALLS edge may cross: + + ```yaml + forbidden_edges: + - {from: "cgis.languages.*", to: "cgis.languages.*", unless: same_pack} + - {from: "cgis.resolver", to: "cgis.storage"} + - {from: "cgis.query", to: "cgis.languages"} + - {from: "cgis.core", to: "cgis"} # core imports nothing + ``` + + A violation is reported as the edge, with file and line. An allowlist of + today's violations (`resolver → storage` ×8, `query → extractors` ×3) can + only shrink: a test fails if it grows, and also if it lists an edge that + is no longer there. + +**Why per plane.** A triad census averages everything in the quotient. The +build plane is a fan-out from `pipeline`; the read plane is layers. Each plane +is homogeneous, so a template can describe it. A single leaking edge is what +averaging hides, and check 3 catches it directly. + +**Enforcement.** Check 3 is enforced from step 0. Checks 1 and 2 run +observe-only through the refactor, and become enforced at step 6 with a +tolerance set from the measured value, as domain tolerances are today. The +old project-level binding is deleted then, and #146 closes. + +## Migration + +**Safety net, written first.** The input must stay fixed while the code +changes. A test that ingests cgis's own `src/` would compare a refactored +cgis reading refactored source, which proves nothing. + +- `tests/fixtures/graph_identity/` is a checked-in multi-language corpus: + Python packages with DI aliases, ABC overrides and src-layout; a TS + monorepo with `tsconfig` paths, workspaces, pnpm and npm dependencies; and + configs above a sub-directory root. +- `tests/unit/test_graph_identity.py` ingests the corpus and compares the + node and edge sets (ids, types, namespaces, targets, confidences) with a + recorded snapshot. The snapshot is regenerated only by a step that is + meant to change behaviour (step 7). Steps 1–5 must keep it unchanged. +- `scripts/graph_identity.py ` diffs two + ingests. Every PR in steps 1–5 runs it in its description on cgis `src/` + at a pinned sha (715cd9a, the A/B pin) and on owner-api @ 8de2245, with + `main`'s cgis and the branch's cgis over the same input. + +| Step | Change | Behaviour | +|---|---|---| +| 0. Measure and guard | Graph-identity corpus, snapshot test and diff script. Forbidden-edge check with the shrinking allowlist. Plane quotients measured on today's code and printed observe-only. | unchanged | +| 1. Skeleton | `languages/` with the protocol. `python` and `typescript` packs wrap today's extractors. `extractors/registry.py` becomes the pack registry. Language-specific extractor modules move under their pack. | unchanged | +| 2. Project readers | `python_dependencies` → `languages.python`; `npm_dependencies`, `workspaces`, `tsconfig_paths` → `languages.typescript`. `import_names.py` becomes a collector of per-pack `ProjectFacts`; store keys are kept. | unchanged | +| 3. Classification | stdlib/builtin/external knowledge → `classify` per pack; `indices.py` keeps the generic frame. | unchanged | +| 4. Call rules | Language steps in `symbols.py` → `call_rules`, with explicit hook points. | unchanged | +| 5. Leaking edges | `resolver → storage` and `query → extractors` removed; the allowlist is empty. | unchanged | +| 6. Gates | Plane checks enforced; old project-level binding deleted; #146 closed. | — | +| 7. Go pilot | `languages/go/`: tree-sitter-go extractor; `go.mod`/`go.work` reader (module path; required modules are EXTERNAL); `classify` (a standard-library import path has no dot in its first segment); a receiver rule for method calls. | new | + +Each step is one PR with the full gate, the independent reviewer and the +identity test. Steps 1–5 are mergeable one at a time. + +**Step 7 is the contract's exam.** Its acceptance is the diff: only +`languages/go/`, the registry line, the dependency on `tree-sitter-go` and +tests. If Go needs an edit anywhere else, the contract is missing something. +The contract gets fixed, and the change is made in the contract, not worked +around in Go. + +## Risks and open questions + +- **Template fit (#554).** It is unknown whether `orchestrator` and + `layered_dag` describe the planes well enough to enforce. Step 0 measures + them on today's code. If a plane is far from every template even after the + leak fixes, a decision is needed before step 6: a new template, or keeping + that plane observe-only and enforcing only forbidden edges. +- **Self-drift and god-object baselines move.** Moving modules changes domain + membership. The `extraction` domain becomes `languages.*`, and + `resolver/indices.py` shrinks. Each step re-measures; a tolerance may only + go down, or be set for a new domain with the measurement in the commit + message (the existing ratchet rule). +- **Hook points in `call_rules`** are the least certain part of the contract. + Today's language branches are interleaved with generic steps inside + `_resolve_local_type_call`, `resolve_global_call` and `resolve_self_call`. + Step 4 extracts them one at a time, and the identity test decides whether + the order is preserved. +- **Incremental ingest.** `ProjectFacts` must keep the `_CROSS_FILE_METADATA` + and `ImportNames.differ_from` semantics. A facts change must still trigger + the same rebuilds; the tests in `test_configs_above_root.py` and + `test_annotated_alias_receiver.py` cover this. +- **Performance.** A registry lookup per file and a rule dispatch per call + edge. Measure on the owner-api ingest at steps 1 and 4; the budget is no + more than 5% slower. From 8fbd256e1d69d5d411072038c55e1cf30ba0311d Mon Sep 17 00:00:00 2001 From: zaebee Date: Wed, 14 Oct 2026 07:36:18 +0000 Subject: [PATCH 2/4] =?UTF-8?q?docs(spec):=20revision=202=20=E2=80=94=20fo?= =?UTF-8?q?rbidden=20edges=20as=20the=20gate,=20scope=20per=20language,=20?= =?UTF-8?q?a=20fuller=20contract?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An independent design review (with measurements) and Gemini on #593: - Plane quotients cannot be the enforced gate: the read plane scores 0.676 against layered_dag (cli/api/guardian reach storage beside query), and the build plane against orchestrator reduces to "no leaf-to-leaf edge". The gate is now forbidden edges with shrinking allowlists, extending test_architecture.py; planes stay observe-only reports. - Go breaks file = namespace unit: the contract gains scope_of (file for Python/TS, package directory for Go) and is_test; same-file tiebreaks become same-scope. - The contract gains index_facts, classify with a context (source file, file metadata, facts), named resolution steps with type normalizers, a fallback with its own confidence and import reconciliation, and cross_file_metadata; ProjectReader/ProjectFacts and the same_pack matching are defined (Gemini). - resolver → storage is uplift.py (8 edges): uplift and refresh_dispatch move to a pipeline package. The registry is lazy (#506) and the one entry point for CLI, MCP server and scripts. - The identity net compares stored graphs with full node metadata, under two hash seeds. Refs #146 Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-14-language-packs-design.md | 447 ++++++++++++------ 1 file changed, 300 insertions(+), 147 deletions(-) diff --git a/docs/specs/2026-10-14-language-packs-design.md b/docs/specs/2026-10-14-language-packs-design.md index 61ec988c..08625809 100644 --- a/docs/specs/2026-10-14-language-packs-design.md +++ b/docs/specs/2026-10-14-language-packs-design.md @@ -1,6 +1,7 @@ -# Language packs and a two-plane topology — design +# Language packs and an enforced topology — design -Date: 2026-10-14. Status: proposed. Closes #146 on completion. +Date: 2026-10-14. Status: proposed (revision 2, after Gemini and an +independent design review). Closes #146 on completion. ## Why @@ -9,54 +10,73 @@ Two problems turned out to be one. **The domain-level drift gate cannot be enforced (#146).** Since #144, the quotient graph (each cgis domain collapsed to a node) has been scored observe-only against `pipeline_stage`. It measured 0.700 in June and 0.644 on -2026-10-14 (b0d67f8), with a tolerance of 0.15. No template fits: -`orchestrator` scores 0.527 and `layered_dag` 0.492. The reason is the shape, -not the template. Cgis is two different structures over one core, and a -single triad census averages them into neither: +2026-10-14 (b0d67f8), with a tolerance of 0.15. Measured on the same commit, +no template fits, whether the whole quotient or the planes inside it: + +| quotient | `pipeline_stage` | `orchestrator` | `layered_dag` | +|---|---|---|---| +| whole project | 0.644 | 0.527 | 0.492 | +| build plane `{pipeline, extractors, resolver, storage}` | — | 0.238 | — | +| read plane `{storage, query, guardian, cli, api}` | 0.677 | 0.677 | 0.676 | + +The build plane comes close to `orchestrator` once its one leak goes (0.018). +But with one hub and a few leaves, `orchestrator` only says "no leaf-to-leaf +edge", which a forbidden-edge rule says directly. The read plane is not +layered: `cli`, `api` and `guardian` reach `storage` directly beside `query` +(25, 27 and 13 edges). Every such triad is a layer-skipping triangle, which +`layered_dag` excludes by design. Both planes also fail hygiene on +`unresolved_ratio` (0.34 and 0.25 against a maximum of 0.2). A triad census +cannot be the enforced gate for this topology. + +The real dependencies (imports and calls between top-level packages): ``` core ← every package depends on it; it depends on nothing storage → core extractors → core -resolver → core, storage (8 edges) +resolver → core, storage (8 edges, all from resolver/uplift.py) pipeline → extractors, resolver, storage, import_names build plane -query → core, storage, extractors (3 edges) read plane +query → core, storage, extractors (3: render/mermaid.py → language_for) guardian → query, storage, extractors cli, api → query, storage, pipeline, extractors, guardian import_names → workspaces, tsconfig_paths, npm_/python_dependencies, storage ``` -(Imports and calls between top-level packages, cgis at b0d67f8.) - **Adding a language means touching the whole system.** Knowledge about each -language is spread over five shared modules. `resolver/indices.py` has about -65 Python-specific and 65 TS-specific references; `symbols.py`, `engine.py`, -`pipeline.py` and `import_names.py` have dozens more. Four ecosystem readers -sit at the package root (`tsconfig_paths`, `workspaces`, `npm_dependencies`, -`python_dependencies`). `extractors/registry.py` already says "adding a -language should touch this file and nothing else", but that is true only for -parsing. +language is spread over five shared modules: `resolver/indices.py` alone +holds about 65 Python-specific and 65 TS-specific references. Four ecosystem +readers sit at the package root. `api/mcp_server.py` and +`scripts/generate_health.py` list extractors by hand, so a new language would +be invisible to MCP ingest. FQNs assume one file = one namespace unit, which +Go breaks: a Go package is a directory. ## Goals -1. A language is one package, `cgis/languages//`, plus one line in a - registry. Adding Go, Rust, PHP or Astro does not edit the core, the - resolver or the pipeline. -2. Drift describes the real shape: one check per plane, plus forbidden edges. - An enforced domain-level gate replaces the observe-only quotient. -3. The leaking edges `resolver → storage` and `query → extractors` are gone. - -**Success:** the Go pilot (step 7) lands with a diff confined to -`languages/go/`, the registry line and tests. Both plane checks are enforced. -The forbidden-edge allowlist is empty. +1. A language is one package, `cgis/languages//`, plus one registry + entry. Adding Go does not edit the core, the resolver, the pipeline or a + surface. +2. The topology is enforced by **forbidden edges**, each with an allowlist of + today's violations that can only shrink. #146 closes on that gate. +3. Each language decides its namespace unit: file for Python and TS, package + directory for Go. + +**Success:** +- the Go pilot (step 7) lands with a diff confined to `languages/go/`, its + registry entry, the `tree-sitter-go` dependency and tests; +- the forbidden-edge gate is enforced in CI; +- the `resolver → storage` and `query → languages.` allowlists are + empty. ## Non-goals -- Changing the graph format, FQN rules or MCP tool surface. The refactor is - internal, and steps 1–5 must leave every graph with identical node and edge sets. -- Astro (#227), PHP and Rust. They follow the contract once Go has proven it. -- Revising the drift formula (#554). -- A second storage backend, or new surfaces (approach C, rejected as YAGNI). +- Changing the graph format, FQN rules or MCP tool surface for Python and TS. + Steps 1–5 must leave every stored graph identical (see Migration). +- Making the read plane layered. "Surfaces reach storage only through query" + is a rule with a shrinking allowlist (about 65 edges), paid down when those + files are touched anyway, not in one refactor. +- Enforcing plane quotients. They stay observe-only reports; see Drift gates. +- Astro (#227), PHP, Rust; revising the drift formula (#554); a second storage + backend. ## Architecture @@ -67,153 +87,286 @@ Two planes over a shared core: ▲ ▲ ── build ── │ ── read ── │ pipeline ─► languages/ ─► resolver ─► storage ─► query ─► cli | api | guardian - (parser + project (language-free - reader + rules) engine) + (parser, project (language-free + reader, rules) engine) ``` -- **Build plane.** `pipeline` orchestrates: it asks the registry for each - file's pack, runs its extractor and project reader, hands nodes, edges and - project facts to the resolver, and persists the result. -- **Read plane.** `storage → query → surfaces`, layered. -- **Core.** Model types and FQN utilities. It imports nothing from cgis. +- **Build plane.** `cgis/pipeline/` (today's `pipeline.py` becomes a package) + orchestrates: it asks the registry for each file's pack, runs its extractor + and project reader, hands nodes, edges and facts to the resolver, persists + the result, then runs the post-write passes: semantic uplift and + `refresh_dispatch`. Both move here from `resolver/` (step 5). They read and + write the store, which is what the resolver must not do. +- **Read plane.** `storage → query → surfaces`. +- **Core.** Model types and FQN utilities. It imports nothing else from cgis. + +### Registry + +`cgis/languages/__init__.py` maps a pack name to its module path and imports +a pack only on first use. Loading every grammar eagerly would undo #506. It +answers: + +- `pack_for_path(path) -> LanguagePack | None`, by extension; +- `pack_for_node(node) -> LanguagePack | None`, by `Node.language`, never by + re-sniffing a suffix; +- `extractors(source_roots)`, the mapping `IngestionPipeline` takes. + +The CLI, MCP server, health script and reviewer all use the registry; none +names an extractor class. Surfaces and `query` may import +`cgis.languages` (the registry) but not `cgis.languages.`. ### The `LanguagePack` contract ```python class LanguagePack(Protocol): - name: str # "python", "typescript", "go" - extensions: tuple[str, ...] # (".py",), (".ts", ".tsx"), (".go",) + name: str # "python", "typescript", "go" + extensions: tuple[str, ...] code_fence: str + # Files and namespaces + def is_test(self, path: str) -> bool: ... + def scope_of(self, path: str, source_root: str | None) -> str: ... def extractor(self, source_roots: list[str]) -> BaseExtractor: ... - def module_fqn(self, path: str, source_root: str | None) -> str: ... + + # Project facts def project_reader(self, root: Path) -> ProjectReader: ... - def classify(self, fqn: str, facts: ProjectFacts) -> NodeNamespace | None: ... - def call_rules(self) -> Sequence[CallRule]: ... + + # Index and classification + def index_facts(self, nodes: Iterable[Node], facts: ProjectFacts) -> IndexFacts: ... + def classify(self, fqn: str, ctx: ClassifyContext) -> NodeNamespace | None: ... + + # Resolution + def resolution(self) -> ResolutionRules: ... + + # Incremental ingest + cross_file_metadata: Mapping[NodeType, tuple[str, ...]] ``` -| Part | Replaces | Notes | -|---|---|---| -| `extensions`, `extractor`, `module_fqn`, `code_fence` | `extractors/registry.py` `Language` | `module_fqn` must stay the function the extractor uses for FILE ids (existing round-trip test). | -| `project_reader` | `import_names.py`, `workspaces.py`, `tsconfig_paths.py`, `npm_dependencies.py`, `python_dependencies.py` | It notes manifests during the walk and configs above the root (#555), and yields `ProjectFacts`. The facts are serialisable and recorded in the store under today's keys, so `ImportNames.differ_from` and freshness keep working on existing graphs. | -| `classify` | `classify_fqn` branches in `resolver/indices.py`, `resolver/js_builtins.py` | It returns STDLIB/EXTERNAL for what the pack recognises, or None to defer to the generic rules. | -| `call_rules` | language branches in `resolver/symbols.py` (`self.x`, `Annotated` aliases, npm package calls, …) | A `CallRule` declares a hook point (`before_import_map`, `after_import_map`, `on_receiver`). The engine runs generic steps and pack rules in a fixed order; with no rule the generic chain is unchanged. | +**Files and namespaces** + +- `is_test` replaces the Python-only `is_test_path` and the pipeline's + `_TEST_FILE_PATTERN` (pipeline.py:447). For Go it matches `_test.go`. +- `scope_of` returns the namespace unit a file's definitions live in: + - Python and TS return today's module FQN for the file, so nothing changes; + - Go returns the package directory, so `func (s *Server) Run()` in + `server.go` and `type Server` in `types.go` both live under + `.Server`. + + The extractor builds every id from it. The generic code that today assumes + file = unit uses the scope instead: + - the global-symbol tiebreak, "same file" → "same scope" (symbols.py:305); + - `_owning_module` (fingerprint.py:166); + - `_public_names_under` (indices.py:663); + - the guardian collector (collector.py:219). + + For Python and TS, scope and file coincide, so behaviour is unchanged. + +**Project facts** -**Registry.** `cgis/languages/__init__.py` lists the packs. It is the only -module outside `languages/` that names a language. `pipeline`, `query` and -`guardian` ask it "whose file is this?" and nothing else. +```python +class ProjectReader(Protocol): + def note(self, path: Path) -> bool: ... # a file the walk passed; True if taken + def note_above(self, directories: list[Path], bounded: bool) -> None: ... # #555 + def facts(self) -> ProjectFacts: ... + +class ProjectFacts(Protocol): # frozen, JSON-serialisable + def record(self, store: SQLiteStore) -> None: ... # under today's store keys + def differs_from(self, store: SQLiteStore) -> bool: ... # a change forces a rebuild +``` + +The Python pack's facts are today's dependency roots. The TS pack's are npm +dependencies, workspace packages (package.json and pnpm) and tsconfig path +aliases. `import_names.py` becomes the collector that asks each pack. + +**Index and classification** -**Rules for packs.** A pack does not import another pack. Shared TS logic that -Astro will need moves to a shared module, or the Astro pack declares it -extends the TS pack through the registry. A pack never touches `storage`: -anything beyond nodes reaches the resolver as `ProjectFacts`. +- `index_facts` holds what `IndexBuilder.build` derives from all nodes today + for one language: Python's `layout_prefixes`, `external_roots`, + `internal_roots` with src/lib stripping, `dependency_heads`, the + `__init__` root and star-import expansion (indices.py:402, 642, 671, + 795–840). TS's include the per-file `bare_imports`/`bare_bindings` + (indices.py:757). +- `classify` receives a `ClassifyContext`: the FQN's `source_file` (may be + None), that file's FILE metadata, the project facts and the index facts. + - **Preserved quirk:** today `classify_fqn(fqn)` called without a source + file applies the Python reading (indices.py:287; called at symbols.py:188 + and 294, and indices.py:234). The registry keeps that: with no source, the + Python pack classifies. Removing the quirk is a separate, measured change. + +**Resolution.** The engine runs named steps in a fixed order. A pack +contributes to them; it does not get free hook points: + +```python +class ResolutionRules(Protocol): + type_normalizers: Sequence[TypeNormalizer] # e.g. Annotated alias → T + steps: Sequence[tuple[str, CallStep]] # ("after:import_map", step) + fallback: Fallback | None # (name, file) -> (target, confidence) + reconcile_import: ImportReconciler | None # resolve_import_target's language half +``` + +The generic chain, with today's positions of language logic: + +1. `local_type`, with `type_normalizers` applied inside it, as + `_through_annotated_alias` is today (symbols.py:285), and in + `receiver_type_fqn` (symbols.py:63); +2. `import_map`; +3. pack steps positioned `after:import_map`, e.g. TS `_package_call`; +4. `global_symbols`, with the same-scope tiebreak; +5. `fallback`, e.g. JS builtins at confidence 0.8 (engine.py:172–176), not 1.0. + +`reconcile_import` takes the language-split half of `resolve_import_target` +(indices.py:141–146). Step 4 of the migration extracts these one at a time. + +**Incremental ingest.** `cross_file_metadata` lists, per node type, the +metadata keys other files resolve against. The pipeline's +`_CROSS_FILE_METADATA` (pipeline.py:32) becomes the union over packs: +`import_map`, `reexports`, `star_imports`, `self_types` and `annotated_type` +for Python, and so on per pack. + +**Pack rules** + +- A pack does not import another pack. Shared TS logic Astro needs moves to a + shared module, or Astro declares in the registry that it extends TS. +- A pack never touches `storage`: everything beyond nodes reaches the + resolver as `ProjectFacts` or `IndexFacts`. ## Drift gates -Domains after migration (`docs/ontology/patterns.yaml`): +**The enforced gate is forbidden edges**, implemented by extending +`tests/self_parsing/test_architecture.py`, which already enforces three import +boundaries (lines 21–23), rather than adding a parallel list to +`patterns.yaml`. It runs over the cgis self-graph's IMPORTS, IMPORTS_SYMBOL and +CALLS edges between INTERNAL nodes. -| Domain | Template | Notes | +| rule | today | after | |---|---|---| -| `core` | `pure_utility` | new | -| `languages.python`, `languages.typescript`, `languages.go` | `layered_dag`, one shared tolerance | replaces `extraction` | -| `resolver` | `pipeline_stage` | unchanged (engine → symbols → indices) | -| `pipeline` | `orchestrator` | unchanged | -| `storage` | `pure_utility` | unchanged | -| `query` | `layered_dag` | unchanged | -| `cli`, `api`, `guardian` | hygiene / as today | unchanged | - -Project-level checks replace the single `project_level` quotient binding: - -1. **Build plane:** the quotient over `{pipeline, languages.*, resolver, - storage}` against `orchestrator`. -2. **Read plane:** the quotient over `{storage, query, guardian, cli, api}` - against `layered_dag`. -3. **Forbidden edges:** a new list in `patterns.yaml` of `from → to` prefix - pairs that no IMPORTS/IMPORTS_SYMBOL/CALLS edge may cross: - - ```yaml - forbidden_edges: - - {from: "cgis.languages.*", to: "cgis.languages.*", unless: same_pack} - - {from: "cgis.resolver", to: "cgis.storage"} - - {from: "cgis.query", to: "cgis.languages"} - - {from: "cgis.core", to: "cgis"} # core imports nothing - ``` - - A violation is reported as the edge, with file and line. An allowlist of - today's violations (`resolver → storage` ×8, `query → extractors` ×3) can - only shrink: a test fails if it grows, and also if it lists an edge that - is no longer there. - -**Why per plane.** A triad census averages everything in the quotient. The -build plane is a fan-out from `pipeline`; the read plane is layers. Each plane -is homogeneous, so a template can describe it. A single leaking edge is what -averaging hides, and check 3 catches it directly. - -**Enforcement.** Check 3 is enforced from step 0. Checks 1 and 2 run -observe-only through the refactor, and become enforced at step 6 with a -tolerance set from the measured value, as domain tolerances are today. The -old project-level binding is deleted then, and #146 closes. +| `languages.` → `languages.`, a ≠ b | — | 0 | +| `languages.` → `storage`, `query`, `api`, `guardian`, `pipeline` | (extractors → storage etc. already 0) | 0 | +| `resolver` → `storage` | 8 (uplift.py) | 0 after step 5 | +| `query`, `guardian`, `cli`, `api` → `languages.` (registry allowed) | 3 (mermaid.py → `extractors.registry`) | 0 after step 1 | +| `cli`, `api`, `guardian` → `storage` (read through `query`) | about 65 | shrinking | +| `core` → any other cgis package | 0 (enforced today) | 0 | + +**Matching.** A rule names a source prefix and a target prefix. `` is +one dot-segment after `cgis.languages.`, taken from the edge's own source and +target. So `languages. → languages.` fires when the two segments +differ, and an edge inside one pack never fires. An edge whose target is the +registry module `cgis.languages` itself matches no `` rule. + +**Allowlist.** Each rule carries an allowlist of today's violations, keyed by +`(source module, target module)` with a count. The test fails when: + +- an edge appears that no entry covers; +- a count grows; +- an entry no longer matches any edge, so the list cannot rot. + +**Plane quotients** stay as observe-only reports in `test_drift.py`, printed +each run, for the record and for #554. The old `project_level` binding in +`patterns.yaml` is deleted at step 6, and #146 closes on the enforced +forbidden-edge gate. + +**Domains** (`patterns.yaml`). `extraction` becomes one domain per pack +(`languages.python`, `languages.typescript`, `languages.go`), each +`layered_dag` with one shared tolerance, so a loose Go pack shows before +review. `resolution` is re-measured after uplift leaves it. `core` gets +`pure_utility`. The other domains are unchanged. ## Migration **Safety net, written first.** The input must stay fixed while the code changes. A test that ingests cgis's own `src/` would compare a refactored -cgis reading refactored source, which proves nothing. - -- `tests/fixtures/graph_identity/` is a checked-in multi-language corpus: - Python packages with DI aliases, ABC overrides and src-layout; a TS - monorepo with `tsconfig` paths, workspaces, pnpm and npm dependencies; and - configs above a sub-directory root. -- `tests/unit/test_graph_identity.py` ingests the corpus and compares the - node and edge sets (ids, types, namespaces, targets, confidences) with a - recorded snapshot. The snapshot is regenerated only by a step that is - meant to change behaviour (step 7). Steps 1–5 must keep it unchanged. +cgis reading refactored source. + +- `tests/fixtures/graph_identity/` is a checked-in corpus that covers: + - Python: DI aliases, ABC overrides, star-import chains, `__init__`-rooted + relative imports, a `src/` and a `lib/` layout, and a dependency root + colliding with a package name (the alembic case); + - TS: a monorepo with `tsconfig` paths, package.json and pnpm workspaces, + npm dependencies, `.d.ts`, `node:` imports, shadowed globals; + - configs above a sub-directory root. +- `tests/unit/test_graph_identity.py` ingests the corpus into a store and + compares the **stored** graph, after uplift and `refresh_dispatch`, with a + recorded snapshot: + - nodes: id, type, namespace, file, lines, `language`, `is_test`, + `is_generated`, and the full `metadata` (import_map, local_types, + self_types, bare_*, annotated_type), which is exactly what steps 1–4 + relocate; + - edges: id, source, target, type, confidence, weight, context, file, + line; + - the `ingest_state` keys. + + It runs twice, under two `PYTHONHASHSEED` values, so set-iteration order + cannot pass by luck. The snapshot is regenerated only by a step meant to + change behaviour (step 7). - `scripts/graph_identity.py ` diffs two - ingests. Every PR in steps 1–5 runs it in its description on cgis `src/` - at a pinned sha (715cd9a, the A/B pin) and on owner-api @ 8de2245, with - `main`'s cgis and the branch's cgis over the same input. + ingests. Every PR in steps 1–5 reports it in its description, for cgis + `src/` at 715cd9a and owner-api @ 8de2245, with `main`'s cgis and the + branch's cgis over the same input. | Step | Change | Behaviour | |---|---|---| -| 0. Measure and guard | Graph-identity corpus, snapshot test and diff script. Forbidden-edge check with the shrinking allowlist. Plane quotients measured on today's code and printed observe-only. | unchanged | -| 1. Skeleton | `languages/` with the protocol. `python` and `typescript` packs wrap today's extractors. `extractors/registry.py` becomes the pack registry. Language-specific extractor modules move under their pack. | unchanged | -| 2. Project readers | `python_dependencies` → `languages.python`; `npm_dependencies`, `workspaces`, `tsconfig_paths` → `languages.typescript`. `import_names.py` becomes a collector of per-pack `ProjectFacts`; store keys are kept. | unchanged | -| 3. Classification | stdlib/builtin/external knowledge → `classify` per pack; `indices.py` keeps the generic frame. | unchanged | -| 4. Call rules | Language steps in `symbols.py` → `call_rules`, with explicit hook points. | unchanged | -| 5. Leaking edges | `resolver → storage` and `query → extractors` removed; the allowlist is empty. | unchanged | -| 6. Gates | Plane checks enforced; old project-level binding deleted; #146 closed. | — | -| 7. Go pilot | `languages/go/`: tree-sitter-go extractor; `go.mod`/`go.work` reader (module path; required modules are EXTERNAL); `classify` (a standard-library import path has no dot in its first segment); a receiver rule for method calls. | new | +| 0. Measure and guard | Identity corpus, snapshot test, diff script. Forbidden-edge rules in `test_architecture.py`, with today's allowlists. Plane quotients printed observe-only. | unchanged | +| 1. Skeleton and registry | `languages/` with the protocol; `python` and `typescript` packs wrap today's extractors and own `is_test` and `scope_of`. The lazy registry replaces `extractors/registry.py`. CLI, MCP server, health script, guardian and `mermaid.py` go through it. Moved modules are imported from their new paths, with test imports updated in the same PR (36 files) and no compatibility shims. The repo CLAUDE.md "adding an extractor" section is rewritten. The `extraction` domain in `patterns.yaml` is re-pointed at `cgis.languages`, with its tolerance re-measured. | unchanged | +| 2. Project readers | `ProjectReader`/`ProjectFacts` per pack; `import_names.py` collects. Store keys kept. | unchanged | +| 3. Index and classification | `index_facts` and `classify(fqn, ctx)` per pack; the no-source → Python rule kept in the registry. | unchanged | +| 4. Resolution steps | Named generic steps; type normalizers, positioned steps, fallback (with its 0.8) and import reconciliation per pack. `cross_file_metadata` per pack. | unchanged | +| 5. Build-plane passes | `pipeline.py` → `cgis/pipeline/`; `resolver/uplift.py` and `refresh_dispatch` move into it. `resolver → storage` allowlist empty; `resolution` domain re-measured. | unchanged | +| 6. Gates | Old `project_level` binding deleted; domains per pack; #146 closed. | — | +| 7. Go pilot | `languages/go/`, below. | new | Each step is one PR with the full gate, the independent reviewer and the -identity test. Steps 1–5 are mergeable one at a time. - -**Step 7 is the contract's exam.** Its acceptance is the diff: only -`languages/go/`, the registry line, the dependency on `tree-sitter-go` and -tests. If Go needs an edit anywhere else, the contract is missing something. -The contract gets fixed, and the change is made in the contract, not worked -around in Go. +identity report. Steps 1–5 are mergeable one at a time. Each step re-measures +the god-object and self-drift baselines. A tolerance may only go down, or be +set for a new domain with the measurement in the commit message. + +### Go pilot (step 7) + +- **Parser:** `tree-sitter-go`. +- **Scope and tests:** + - `scope_of` returns the package directory, relative to the module root; + - `is_test` matches `_test.go`. +- **Project facts from `go.mod`/`go.work`:** + - the module path; + - required modules, which are EXTERNAL; + - the replace directives that point inside the repository, which are + internal. +- **classify:** + 1. check the own module path first (`module myapp` has no dot either); + 2. then the pseudo-package `C` (cgo); + 3. then "no dot in the first path segment" means standard library. +- **Resolution:** + - a receiver normalizer (`*Server` → `Server`); + - package-qualified calls (`pkg.Run`) through the import map; + - unqualified calls through the same-scope tiebreak; + - struct embedding becomes EXTENDS, so the generic DISPATCHES pass covers + promoted methods. + +**Out of pilot scope**, recorded as follow-ups: + +- **Implicit interface satisfaction.** It needs a method-set pass, which + would add an optional `implementations()` contract element. +- **Generics beyond plain extraction.** +- **cgo beyond classification.** + +**Acceptance is the diff:** only `languages/go/`, the registry entry, the +`tree-sitter-go` dependency and tests. If Go needs an edit anywhere else, the +contract is missing something. The contract gets fixed, and the change is +made there, not worked around in Go. ## Risks and open questions -- **Template fit (#554).** It is unknown whether `orchestrator` and - `layered_dag` describe the planes well enough to enforce. Step 0 measures - them on today's code. If a plane is far from every template even after the - leak fixes, a decision is needed before step 6: a new template, or keeping - that plane observe-only and enforcing only forbidden edges. -- **Self-drift and god-object baselines move.** Moving modules changes domain - membership. The `extraction` domain becomes `languages.*`, and - `resolver/indices.py` shrinks. Each step re-measures; a tolerance may only - go down, or be set for a new domain with the measurement in the commit - message (the existing ratchet rule). -- **Hook points in `call_rules`** are the least certain part of the contract. - Today's language branches are interleaved with generic steps inside - `_resolve_local_type_call`, `resolve_global_call` and `resolve_self_call`. - Step 4 extracts them one at a time, and the identity test decides whether - the order is preserved. -- **Incremental ingest.** `ProjectFacts` must keep the `_CROSS_FILE_METADATA` - and `ImportNames.differ_from` semantics. A facts change must still trigger - the same rebuilds; the tests in `test_configs_above_root.py` and - `test_annotated_alias_receiver.py` cover this. -- **Performance.** A registry lookup per file and a rule dispatch per call - edge. Measure on the owner-api ingest at steps 1 and 4; the budget is no +- **Behaviour hidden in Python-only paths that Go also reaches.** For + example, `resolve_layout`/`map_to_node_fqn` run for every language today, + without a source-file gate (indices.py:229–236). Step 3 must gate them by + pack. The identity test proves Python and TS unchanged; the Go pilot's own + tests prove they no longer leak into Go. +- **Hook order.** Step 4 is the most delicate step. Extract one language + branch per commit, with the identity test after each. +- **Baselines move.** Domain membership changes at steps 1 and 5, and + `SymbolResolver` is at the god-object threshold. Steps follow the existing + ratchet rule. +- **Imports of moved modules.** MCP server, CLI, scripts, autodoc and 36 test + files. Step 1 updates them all; no shims. +- **Performance.** A lazy registry lookup per file and a step dispatch per + call edge. Measure the owner-api ingest at steps 1 and 4; the budget is no more than 5% slower. From 05e899f79c4d5de3edcf6584cc92ec550b60be38 Mon Sep 17 00:00:00 2001 From: zaebee Date: Wed, 14 Oct 2026 07:45:19 +0000 Subject: [PATCH 3/4] docs(spec): lazy registry manifests; Python/TS snapshot frozen through step 7; Go ontology conventions Gemini on #593: a ready mapping of every extractor, and a cross_file_metadata union read from pack modules, would both import every grammar and undo #506. The registry now holds lightweight per-pack manifests (name, module, extensions, cross_file_metadata) and builds an extractor on its pack's first file. Guardian on #593: the identity test runs in CI from step 0 and the Python/TS snapshot never changes, step 7 included (Go gets its own); the tolerance ratchet is the existing tolerances.lock test; Go emits raw_call: and passes the ontology-compliance tests. Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-10-14-language-packs-design.md | 40 ++++++++++++++----- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/docs/specs/2026-10-14-language-packs-design.md b/docs/specs/2026-10-14-language-packs-design.md index 08625809..3552f9b4 100644 --- a/docs/specs/2026-10-14-language-packs-design.md +++ b/docs/specs/2026-10-14-language-packs-design.md @@ -102,14 +102,22 @@ Two planes over a shared core: ### Registry -`cgis/languages/__init__.py` maps a pack name to its module path and imports -a pack only on first use. Loading every grammar eagerly would undo #506. It -answers: - -- `pack_for_path(path) -> LanguagePack | None`, by extension; +`cgis/languages/__init__.py` holds one lightweight **manifest** per pack: its +name, module path, extensions and `cross_file_metadata` keys. Everything the +pipeline needs before it meets a file is declared there, so nothing imports a +pack until a file of that language turns up. Loading every grammar eagerly +would undo #506 (Gemini on #593). It answers: + +- `pack_for_path(path) -> LanguagePack | None`, by extension; this imports + that one pack on first use; - `pack_for_node(node) -> LanguagePack | None`, by `Node.language`, never by re-sniffing a suffix; -- `extractors(source_roots)`, the mapping `IngestionPipeline` takes. +- `extractor_for(path, source_roots) -> BaseExtractor | None`, which + `IngestionPipeline` calls per file instead of taking a ready mapping of + every extractor; each pack's extractor is built once, when its first file + is met; +- `cross_file_metadata()`, the union of the manifests' keys, without + importing any pack. The CLI, MCP server, health script and reviewer all use the registry; none names an extractor class. Surfaces and `query` may import @@ -218,8 +226,10 @@ The generic chain, with today's positions of language logic: (indices.py:141–146). Step 4 of the migration extracts these one at a time. **Incremental ingest.** `cross_file_metadata` lists, per node type, the -metadata keys other files resolve against. The pipeline's -`_CROSS_FILE_METADATA` (pipeline.py:32) becomes the union over packs: +metadata keys other files resolve against. It is declared in the pack's +registry manifest, not read from the pack module, so collecting it imports +nothing. The pipeline's `_CROSS_FILE_METADATA` (pipeline.py:32) becomes the +union over the manifests: `import_map`, `reexports`, `star_imports`, `self_types` and `annotated_type` for Python, and so on per pack. @@ -296,8 +306,10 @@ cgis reading refactored source. - the `ingest_state` keys. It runs twice, under two `PYTHONHASHSEED` values, so set-iteration order - cannot pass by luck. The snapshot is regenerated only by a step meant to - change behaviour (step 7). + cannot pass by luck. It is an ordinary unit test, so CI runs it on every PR + from step 0 on. **The Python/TS snapshot never changes, step 7 included.** + Go gets a corpus and snapshot of its own, so the pilot cannot quietly move + the existing languages. - `scripts/graph_identity.py ` diffs two ingests. Every PR in steps 1–5 reports it in its description, for cgis `src/` at 715cd9a and owner-api @ 8de2245, with `main`'s cgis and the @@ -317,7 +329,9 @@ cgis reading refactored source. Each step is one PR with the full gate, the independent reviewer and the identity report. Steps 1–5 are mergeable one at a time. Each step re-measures the god-object and self-drift baselines. A tolerance may only go down, or be -set for a new domain with the measurement in the commit message. +set for a new domain with the measurement in the commit message. This is +already enforced: `docs/ontology/tolerances.lock` and +`test_tolerances_never_exceed_the_lock` (#151). ### Go pilot (step 7) @@ -334,6 +348,10 @@ set for a new domain with the measurement in the commit message. 1. check the own module path first (`module myapp` has no dot either); 2. then the pseudo-package `C` (cgo); 3. then "no dot in the first path segment" means standard library. +- **Ontology:** the Go extractor follows the same conventions as the others. + Unresolved calls are emitted as `raw_call:` for the resolver. Node + and edge types come from the existing enums, with no new `NodeType`. The + existing ontology-compliance tests run on the Go corpus. - **Resolution:** - a receiver normalizer (`*Server` → `Server`); - package-qualified calls (`pkg.Run`) through the import map; From aeba29cd594ec9827f1f8ce08a26c270dd5e974c Mon Sep 17 00:00:00 2001 From: zaebee Date: Wed, 14 Oct 2026 07:50:36 +0000 Subject: [PATCH 4/4] =?UTF-8?q?docs(plan):=20language=20packs=20=E2=80=94?= =?UTF-8?q?=20step=200=20in=20full,=20steps=201=E2=80=937=20outlined?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 0 (one PR): canonical graph dump/diff script, a checked-in Python/TS identity corpus with a snapshot compared under two hash seeds, the forbidden-edge gate with exact allowlists in test_architecture.py, and plane quotients reported observe-only. Steps 1–7 each get a detailed plan when the previous one has merged. Refs #146 Co-Authored-By: Claude Opus 5.5 --- docs/plans/2026-10-14-language-packs.md | 888 ++++++++++++++++++++++++ 1 file changed, 888 insertions(+) create mode 100644 docs/plans/2026-10-14-language-packs.md diff --git a/docs/plans/2026-10-14-language-packs.md b/docs/plans/2026-10-14-language-packs.md new file mode 100644 index 00000000..81ec7109 --- /dev/null +++ b/docs/plans/2026-10-14-language-packs.md @@ -0,0 +1,888 @@ +# Language Packs Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make a language one package behind a `LanguagePack` contract, and enforce cgis's topology through forbidden edges, without changing any Python or TS graph along the way. + +**Architecture:** Step 0, detailed below, lands the safety nets the refactor relies on: +- a checked-in multi-language corpus whose stored graph is snapshotted; +- a forbidden-edge gate with exact allowlists; +- observe-only plane quotients. + +Steps 1–7 move code under those nets, one PR each. Each step gets its own detailed plan when it starts; this document outlines them. + +**Tech Stack:** Python 3.12, tree-sitter, SQLite, pytest, ruff, mypy strict, interrogate. + +**Spec:** `docs/specs/2026-10-14-language-packs-design.md` (PR #593, revision 3). + +## Global Constraints + +- `make format && make lint && make type-check && make pytest && make doc-coverage` passes before every commit; gate with `&&`, never `;` or a pipe on the gating command. +- mypy strict on `src/`; docstring coverage ≥ 90% (interrogate); Sonar cognitive complexity ≤ 15 per function. +- Steps 1–5 leave every stored Python/TS graph's node and edge sets identical. The step-0 snapshot never changes before step 7, and its Python/TS part never changes at all. +- Drift tolerances only go down (`docs/ontology/tolerances.lock`, `test_tolerances_never_exceed_the_lock`). The god-object baseline (`test_god_object_baseline_not_exceeded`) is not raised to make a step pass. +- Test-count floor: raise `.github/test-count-baseline` in the same PR when the suite grows past it by 150 or more. +- Every PR gets an independent reviewer with execution tools before merge. +- Commits end with `Co-Authored-By: Claude Opus 5.5 `; PR bodies end with `🤖 Generated with [Claude Code](https://claude.com/claude-code)`. + +## Review Focus + +- A snapshot that embeds the temp directory the corpus was copied to must still compare equal on another machine. Every absolute path is normalised to `` (Task 1 test `test_paths_are_normalised`). +- Two runs of the same ingest must give byte-identical dumps, whatever the hash seed and file mtimes. `ingested_at` is dropped, and keys and rows are sorted (Task 3 runs the snapshot under two `PYTHONHASHSEED` values). +- An allowlist entry for an edge that has disappeared must fail, not linger (Task 4 `test_forbidden_edges_match_the_allowlist_exactly` compares for equality, not "≤"). +- Edges to virtual or unresolved nodes must not count as forbidden edges. Only INTERNAL targets whose FILE module is known are attributed (Task 4 `test_external_targets_are_never_attributed`). +- The configs-above-root case needs a `.git` marker that cannot be checked in. The snapshot command creates it in its temp copy (Task 2 corpus contract, Task 3). + +--- + +## Step 0 — Measure and guard (one PR) + +### Task 1: Canonical graph dump and diff + +**Files:** +- Create: `scripts/graph_identity.py` +- Test: `tests/unit/test_graph_identity_dump.py` + +**Interfaces:** +- Produces: + - `dump(db_path: Path, root: Path) -> dict[str, object]`: the canonical, normalised content of one stored graph; + - `diff(before: Mapping[str, object], after: Mapping[str, object]) -> list[str]`: human-readable differences, empty when equal; + - CLI `python scripts/graph_identity.py diff `, which exits 1 and prints the diff when the graphs differ. + +- [ ] **Step 1: Write the failing tests** + +```python +"""The canonical graph dump the identity net compares (spec: Migration, safety net).""" + +import json +import sqlite3 +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent.parent +sys.path.insert(0, str(REPO / "scripts")) + +import graph_identity as gi # noqa: E402 + +from cgis.extractors.registry import build_extractors # noqa: E402 +from cgis.pipeline import IngestionPipeline # noqa: E402 +from cgis.storage.sqlite_store import SQLiteStore # noqa: E402 + + +def _ingest(root: Path, db: Path) -> None: + with SQLiteStore(str(db)) as store: + IngestionPipeline(build_extractors([])).run(str(root), store=store) + + +def _repo(root: Path) -> Path: + (root / "pkg").mkdir(parents=True) + (root / "pkg" / "a.py").write_text("def f():\n g()\n\ndef g():\n pass\n") + return root + + +def test_dump_carries_nodes_edges_and_ingest_state(tmp_path: Path) -> None: + root = _repo(tmp_path / "r") + _ingest(root, tmp_path / "g.db") + out = gi.dump(tmp_path / "g.db", root) + + assert {n["id"] for n in out["nodes"]} >= {"pkg.a", "pkg.a.f", "pkg.a.g"} + call = next(e for e in out["edges"] if e["type"] == "CALLS") + assert (call["source"], call["target"]) == ("pkg.a.f", "pkg.a.g") + assert "ingested_at" not in out["ingest_state"] + assert set(out["nodes"][0]) >= {"metadata", "language", "is_test", "is_generated"} + + +def test_paths_are_normalised(tmp_path: Path) -> None: + root = _repo(tmp_path / "r") + _ingest(root, tmp_path / "g.db") + text = json.dumps(gi.dump(tmp_path / "g.db", root)) + + assert str(root) not in text + assert "" in text + + +def test_the_same_ingest_twice_dumps_identically(tmp_path: Path) -> None: + a, b = _repo(tmp_path / "a"), _repo(tmp_path / "b") + _ingest(a, tmp_path / "a.db") + _ingest(b, tmp_path / "b.db") + + assert gi.diff(gi.dump(tmp_path / "a.db", a), gi.dump(tmp_path / "b.db", b)) == [] + + +def test_diff_names_a_changed_edge(tmp_path: Path) -> None: + root = _repo(tmp_path / "r") + _ingest(root, tmp_path / "g.db") + before = gi.dump(tmp_path / "g.db", root) + with sqlite3.connect(tmp_path / "g.db") as conn: + conn.execute("UPDATE edges SET confidence = 0.5 WHERE type = 'CALLS'") + after = gi.dump(tmp_path / "g.db", root) + + lines = gi.diff(before, after) + assert lines, "a changed confidence must show" + assert all(line.startswith("edges ") for line in lines), "only edge rows changed" + assert any("pkg.a.f" in line and "0.5" in line for line in lines) +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `uv run pytest tests/unit/test_graph_identity_dump.py -q` +Expected: FAIL. `ModuleNotFoundError: No module named 'graph_identity'`. + +- [ ] **Step 3: Implement `scripts/graph_identity.py`** + +```python +"""Canonical dump and diff of a stored cgis graph — the language-packs identity net. + +Steps 1–5 of docs/specs/2026-10-14-language-packs-design.md move code without +changing behaviour. This is how that is checked: the same input ingested by two +cgis versions must dump identically. Absolute paths become `` and the +mtime-derived `ingested_at` is dropped, so a dump compares across machines and +temp directories. +""" + +import argparse +import json +import sqlite3 +import sys +from collections.abc import Mapping +from pathlib import Path + +_NODE_COLUMNS = ( + "id", "type", "name", "file_path", "start_line", "end_line", "language", + "namespace", "is_test", "is_generated", "metadata", +) # fmt: skip +_EDGE_COLUMNS = ( + "id", "source", "target", "type", "weight", "confidence", "context", + "file_path", "line_number", "type_only", +) # fmt: skip +_VOLATILE_STATE = frozenset({"ingested_at"}) + + +def _normalise(value: object, root: str) -> object: + """`value` with the corpus root replaced by ``, JSON text parsed to data.""" + if isinstance(value, str): + text = value.replace(root, "") + if text[:1] in "[{": + try: + return _normalise(json.loads(text), "\0") + except ValueError: + return text + return text + if isinstance(value, list): + return [_normalise(v, root) for v in value] + if isinstance(value, dict): + return {k: _normalise(v, root) for k, v in sorted(value.items())} + return value + + +def _rows(conn: sqlite3.Connection, table: str, columns: tuple[str, ...], root: str) -> list[dict[str, object]]: + """Every row of `table` as a normalised dict, sorted by id.""" + query = f"SELECT {', '.join(columns)} FROM {table} ORDER BY id" # table/columns are constants + return [ + {col: _normalise(val, root) for col, val in zip(columns, row, strict=True)} + for row in conn.execute(query) + ] + + +def dump(db_path: Path, root: Path) -> dict[str, object]: + """The canonical content of one stored graph, with `root` normalised away.""" + prefix = str(root.resolve()) + with sqlite3.connect(db_path) as conn: + state = { + key: _normalise(value, prefix) + for key, value in conn.execute("SELECT key, value FROM ingest_state ORDER BY key") + if key not in _VOLATILE_STATE + } + return { + "nodes": _rows(conn, "nodes", _NODE_COLUMNS, prefix), + "edges": _rows(conn, "edges", _EDGE_COLUMNS, prefix), + "ingest_state": state, + } + + +def diff(before: Mapping[str, object], after: Mapping[str, object]) -> list[str]: + """One line per row that differs between two dumps, keyed by table and id.""" + lines: list[str] = [] + for table in ("nodes", "edges"): + old = {row["id"]: row for row in before[table]} # type: ignore[index, union-attr] + new = {row["id"]: row for row in after[table]} # type: ignore[index, union-attr] + for key in sorted(old.keys() | new.keys()): + if old.get(key) != new.get(key): + lines.append(f"{table} {key}: {json.dumps(old.get(key))} → {json.dumps(new.get(key))}") + if before["ingest_state"] != after["ingest_state"]: + lines.append(f"ingest_state: {before['ingest_state']} → {after['ingest_state']}") + return lines + + +def main(argv: list[str] | None = None) -> int: + """`diff `: exit 1 and print when the graphs differ.""" + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + sub = parser.add_subparsers(dest="command", required=True) + d = sub.add_parser("diff") + for name in ("db_a", "root_a", "db_b", "root_b"): + d.add_argument(name, type=Path) + args = parser.parse_args(argv) + lines = diff(dump(args.db_a, args.root_a), dump(args.db_b, args.root_b)) + print("\n".join(lines) if lines else "identical") + return 1 if lines else 0 + + +if __name__ == "__main__": + sys.exit(main()) +``` + +- [ ] **Step 4: Run the tests** + +Run: `uv run pytest tests/unit/test_graph_identity_dump.py -q` +Expected: 4 passed. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/graph_identity.py tests/unit/test_graph_identity_dump.py +git commit -m "feat(bench): canonical dump and diff of a stored graph — the language-packs identity net + +Co-Authored-By: Claude Opus 5.5 " +``` + +### Task 2: The identity corpus + +**Files:** +- Create: the tree under `tests/fixtures/graph_identity/corpus/`, listed below +- Test: none of its own; Task 3 ingests it. + +**Interfaces:** +- Produces: a corpus with two ingest roots that Task 3 snapshots: + - `corpus/` (whole); + - `corpus/ts/apps/web/src` (a sub-directory root, which exercises configs above the root). + + The snapshot command adds `.git` at the copy's root. + +- [ ] **Step 1: Write the Python half** + +`corpus/py/pyproject.toml`: +```toml +[project] +name = "shop" +dependencies = ["fastapi>=0.110", "alembic>=1.13"] +``` + +`corpus/py/src/shop/__init__.py`: +```python +from .service import * # noqa: F403 — star-import chain under test +``` + +`corpus/py/src/shop/repo.py`: +```python +from abc import ABC, abstractmethod + + +class Repo(ABC): + @abstractmethod + def save(self, item: str) -> None: ... + + +class SqlRepo(Repo): + def save(self, item: str) -> None: + print(item) +``` + +`corpus/py/src/shop/deps.py`: +```python +from typing import Annotated + +from fastapi import Depends + +from .repo import Repo, SqlRepo + + +def get_repo() -> Repo: + return SqlRepo() + + +RepoDep = Annotated[Repo, Depends(get_repo)] +``` + +`corpus/py/src/shop/service.py`: +```python +from .deps import RepoDep +from .repo import Repo + +__all__ = ["OrderService", "place"] + + +class OrderService: + def __init__(self, repo: RepoDep) -> None: + self.repo = repo + + def place(self, item: str) -> None: + self.repo.save(item) + + +def place(repo: Repo, item: str) -> None: + repo.save(item) +``` + +`corpus/py/src/alembic/env.py` (a local package named like a declared dependency, the collision case): +```python +def run_migrations() -> None: + pass +``` + +`corpus/py/lib/legacy/tool.py`: +```python +from shop import place + + +def main() -> None: + place(None, "x") +``` + +`corpus/py/tests/test_service.py`: +```python +from shop.service import place + + +def test_place() -> None: + place(None, "x") +``` + +- [ ] **Step 2: Write the TypeScript half** + +`corpus/ts/package.json`: +```json +{"name": "mono", "private": true, "workspaces": ["packages/*"], "dependencies": {"react": "^19.0.0"}} +``` + +`corpus/ts/pnpm-workspace.yaml`: +```yaml +packages: + - "apps/*" + - "!**/fixtures/**" +``` + +`corpus/ts/tsconfig.json`: +```json +{"compilerOptions": {"baseUrl": ".", "paths": {"@app/*": ["apps/web/src/*"]}}} +``` + +`corpus/ts/packages/lib/package.json`: +```json +{"name": "@x/lib", "version": "1.0.0"} +``` + +`corpus/ts/packages/lib/index.ts`: +```ts +export function libFn(): number { + return 1; +} +``` + +`corpus/ts/apps/web/package.json`: +```json +{"name": "web", "dependencies": {"@x/lib": "^1.0.0", "react": "^19.0.0", "lodash.debounce": "^4.0.8"}} +``` + +`corpus/ts/apps/web/src/util.ts`: +```ts +export function helper(n: number): number { + return n + 1; +} +``` + +`corpus/ts/apps/web/src/types.d.ts`: +```ts +export interface Shape { + area(): number; +} +``` + +`corpus/ts/apps/web/src/main.ts`: +```ts +import { randomUUID } from "node:crypto"; +import debounce from "lodash.debounce"; +import { useState } from "react"; +import { libFn } from "@x/lib"; +import { helper } from "@app/util"; + +function setTimeout(fn: () => void): void { + fn(); +} + +export function main(): number { + useState(0); + randomUUID(); + setTimeout(() => debounce(() => helper(libFn()), 10)); + return libFn(); +} +``` + +- [ ] **Step 3: Check the corpus ingests cleanly under today's cgis** + +Run: `uv run cgis ingest tests/fixtures/graph_identity/corpus -o /tmp/claude-1001/corpus-check.db` +Expected: "✅ Success!" and no traceback. Then check for each case: + +- `@x.lib` is UNKNOWN; +- `react` is EXTERNAL; +- `node:crypto` is STDLIB; +- `shop.repo.Repo.save → shop.repo.SqlRepo.save` is a DISPATCHES edge; +- `shop.service.OrderService.place` CALLS `shop.repo.Repo.save` through the `RepoDep` alias. + +```bash +sqlite3 /tmp/claude-1001/corpus-check.db "select id, namespace from nodes where id in ('@x.lib','react','node:crypto')" +sqlite3 /tmp/claude-1001/corpus-check.db "select source, target, type from edges where type in ('DISPATCHES','CALLS') and source like 'shop.%'" +``` + +If a case does not appear, adjust that file in the corpus until it does, so that every case listed in the spec is present in the graph. Record which ones needed adjusting in the commit message. + +- [ ] **Step 4: Commit** + +```bash +git add tests/fixtures/graph_identity/corpus +git commit -m "test(fixtures): the language-packs identity corpus — Python DI/ABC/star/layouts, TS monorepo/pnpm/tsconfig/node: + +Co-Authored-By: Claude Opus 5.5 " +``` + +### Task 3: Snapshot test under two hash seeds + +**Files:** +- Modify: `scripts/graph_identity.py` (add the `snapshot` subcommand) +- Create: `tests/fixtures/graph_identity/snapshot.json` (generated) +- Test: `tests/unit/test_graph_identity.py` + +**Interfaces:** +- Consumes: `dump`, `diff` (Task 1); the corpus (Task 2). +- Produces: CLI `python scripts/graph_identity.py snapshot [--out FILE]`. It copies the corpus to a temp directory, adds `.git`, and ingests two roots: `whole` = the copy, `sub` = `ts/apps/web/src`. It writes `{"whole": dump, "sub": dump}` as sorted, indented JSON to FILE, or to stdout. + +- [ ] **Step 1: Write the failing test** + +```python +"""The Python/TS graph a fixed corpus produces must not change (spec: Migration). + +Regenerate only in a step meant to change behaviour, with the reason in the +commit message: + uv run python scripts/graph_identity.py snapshot tests/fixtures/graph_identity/corpus \ + --out tests/fixtures/graph_identity/snapshot.json +""" + +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parent.parent.parent +CORPUS = REPO / "tests" / "fixtures" / "graph_identity" / "corpus" +SNAPSHOT = REPO / "tests" / "fixtures" / "graph_identity" / "snapshot.json" + + +def _snapshot(seed: str) -> dict[str, object]: + env = {**os.environ, "PYTHONHASHSEED": seed} + out = subprocess.run( + [sys.executable, str(REPO / "scripts" / "graph_identity.py"), "snapshot", str(CORPUS)], + env=env, capture_output=True, text=True, check=True, + ) + return json.loads(out.stdout) + + +@pytest.mark.parametrize("seed", ["0", "1"]) +def test_the_corpus_graph_matches_the_snapshot(seed: str) -> None: + sys.path.insert(0, str(REPO / "scripts")) + import graph_identity as gi + + recorded = json.loads(SNAPSHOT.read_text(encoding="utf-8")) + actual = _snapshot(seed) + for root in ("whole", "sub"): + lines = gi.diff(recorded[root], actual[root]) + assert not lines, f"{root} graph changed (PYTHONHASHSEED={seed}):\n" + "\n".join(lines[:40]) +``` + +- [ ] **Step 2: Run it to see it fail** + +Run: `uv run pytest tests/unit/test_graph_identity.py -q` +Expected: FAIL. Either `snapshot` is an invalid choice for the argparse subcommand, or `snapshot.json` is not found. + +- [ ] **Step 3: Add the `snapshot` subcommand** + +In `scripts/graph_identity.py`, add: + +```python +import shutil +import tempfile + +from cgis.extractors.registry import build_extractors +from cgis.pipeline import IngestionPipeline +from cgis.storage.sqlite_store import SQLiteStore + +_SUB_ROOT = Path("ts/apps/web/src") + + +def _ingest(root: Path, db: Path) -> None: + """One full ingest of `root` into `db`, as `cgis ingest` does it.""" + with SQLiteStore(str(db)) as store: + IngestionPipeline(build_extractors([])).run(str(root), store=store) + + +def snapshot(corpus: Path) -> dict[str, object]: + """Both corpus roots, ingested from a temp copy that has a `.git` marker.""" + with tempfile.TemporaryDirectory() as tmp: + copy = Path(tmp) / "corpus" + shutil.copytree(corpus, copy) + (copy / ".git").mkdir() + dumps: dict[str, object] = {} + for name, root in (("whole", copy), ("sub", copy / _SUB_ROOT)): + db = Path(tmp) / f"{name}.db" + _ingest(root, db) + dumps[name] = dump(db, copy) + return dumps +``` + +In `main`, register the subcommand and handle it: + +```python + s = sub.add_parser("snapshot") + s.add_argument("corpus", type=Path) + s.add_argument("--out", type=Path) + ... + if args.command == "snapshot": + text = json.dumps(snapshot(args.corpus), indent=1, sort_keys=True) + if args.out: + args.out.write_text(text + "\n", encoding="utf-8") + else: + print(text) + return 0 +``` + +`dump(db, copy)` normalises against the copy root for both roots, so the `sub` dump's paths read `/ts/apps/web/src/...` as well. + +- [ ] **Step 4: Generate the snapshot and run the test** + +```bash +uv run python scripts/graph_identity.py snapshot tests/fixtures/graph_identity/corpus --out tests/fixtures/graph_identity/snapshot.json +uv run pytest tests/unit/test_graph_identity.py -q +``` + +Expected: 2 passed. Then make sure the test bites. Change one corpus line (rename `helper` to `helper2` in `util.ts`), run the test and see it FAIL with a diff naming `helper2`, then revert the corpus line and confirm with `git diff --stat tests/fixtures` that only `snapshot.json` is new. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/graph_identity.py tests/unit/test_graph_identity.py tests/fixtures/graph_identity/snapshot.json +git commit -m "test: the Python/TS identity snapshot, compared under two hash seeds + +Co-Authored-By: Claude Opus 5.5 " +``` + +### Task 4: Forbidden-edge gate with exact allowlists + +**Files:** +- Modify: `tests/self_parsing/test_architecture.py` (append "Invariant 7") +- Create: `tests/self_parsing/forbidden_edges_allowlist.json` (generated, then reviewed) + +**Interfaces:** +- Consumes: the `root_graph_data` fixture (`tests/self_parsing/conftest.py`), the self-graph of `src/` with `cgis.`-prefixed ids. +- Produces: the `RULES` table and `forbidden_edges(nodes, edges) -> dict[str, dict[str, int]]` (rule → `"src module -> dst module"` → count). Later steps add rules or shrink the allowlist; they never add entries. + +- [ ] **Step 1: Write the failing tests** + +Add to the import block at the top of `tests/self_parsing/test_architecture.py` (ruff E402 forbids imports further down), and replace the existing `from cgis.core.models import Edge, EdgeType, Node`: + +```python +import json +from dataclasses import dataclass, field +from pathlib import Path + +from cgis.core.models import Edge, EdgeType, Node, NodeNamespace, NodeType +``` + +Then append to the end of the file: + +```python +# --------------------------------------------------------------------------- +# Invariant 7 — Forbidden edges (language-packs spec, Drift gates) +# --------------------------------------------------------------------------- + +_ALLOWLIST = Path(__file__).parent / "forbidden_edges_allowlist.json" +_RULE_EDGES = frozenset({EdgeType.IMPORTS, EdgeType.IMPORTS_SYMBOL, EdgeType.CALLS}) + + +@dataclass(frozen=True) +class _Rule: + """No edge from a module under `sources` to one under `targets`, bar the exemptions.""" + + name: str + sources: tuple[str, ...] + targets: tuple[str, ...] + exempt_modules: frozenset[str] = field(default_factory=frozenset) # exact module ids + exempt_prefixes: tuple[str, ...] = () + cross_pack: bool = False # fire only when the `cgis.languages.` segments differ + + +RULES: tuple[_Rule, ...] = ( + _Rule("cross-pack", ("cgis.languages",), ("cgis.languages",), + exempt_modules=frozenset({"cgis.languages"}), cross_pack=True), + _Rule("pack-isolation", ("cgis.languages", "cgis.extractors"), + ("cgis.storage", "cgis.query", "cgis.api", "cgis.guardian", "cgis.pipeline")), + _Rule("resolver-storage", ("cgis.resolver",), ("cgis.storage",)), + _Rule("surface-packs", ("cgis.query", "cgis.guardian", "cgis.cli", "cgis.api"), + ("cgis.languages", "cgis.extractors"), + exempt_modules=frozenset({"cgis.languages", "cgis.extractors.registry"})), + _Rule("surface-storage", ("cgis.cli", "cgis.api", "cgis.guardian"), ("cgis.storage",)), + _Rule("core-isolation", ("cgis.core",), ("cgis",), exempt_prefixes=("cgis.core",)), +) # fmt: skip + + +def _under(module: str, prefix: str) -> bool: + return module == prefix or module.startswith(prefix + ".") + + +def _pack(module: str) -> str | None: + parts = module.split(".") + return parts[2] if len(parts) > 2 and parts[:2] == ["cgis", "languages"] else None + + +def _fires(rule: _Rule, src: str, dst: str) -> bool: + if not any(_under(src, p) for p in rule.sources) or not any(_under(dst, p) for p in rule.targets): + return False + if dst in rule.exempt_modules or any(_under(dst, p) for p in rule.exempt_prefixes): + return False + return not rule.cross_pack or (_pack(src) is not None and _pack(src) != _pack(dst)) + + +def forbidden_edges(nodes: list[Node], edges: list[Edge]) -> dict[str, dict[str, int]]: + """Rule → "src module -> dst module" → edge count, over INTERNAL targets only.""" + module_of_file = {n.file_path: n.id for n in nodes if n.type == NodeType.FILE} + by_id = {n.id: n for n in nodes} + found: dict[str, dict[str, int]] = {rule.name: {} for rule in RULES} + for e in edges: + src, dst = by_id.get(e.source), by_id.get(e.target) + if e.type not in _RULE_EDGES or src is None or dst is None: + continue + if dst.namespace != NodeNamespace.INTERNAL: + continue + a, b = module_of_file.get(src.file_path), module_of_file.get(dst.file_path) + if a is None or b is None or a == b: + continue + for rule in RULES: + if _fires(rule, a, b): + key = f"{a} -> {b}" + found[rule.name][key] = found[rule.name].get(key, 0) + 1 + return found + + +def test_forbidden_edges_match_the_allowlist_exactly( + root_graph_data: tuple[SQLiteStore, list[Node], list[Edge]], +) -> None: + """Exact equality: a new edge fails, a grown count fails, a vanished entry fails. + + When a refactor removes violations, lower or delete the entries in the same PR. + Never add one: a new forbidden edge is the regression this gate exists for. + """ + _, nodes, edges = root_graph_data + actual = forbidden_edges(nodes, edges) + allowed = json.loads(_ALLOWLIST.read_text(encoding="utf-8")) + assert actual == allowed, ( + "Forbidden edges differ from forbidden_edges_allowlist.json. Actual:\n" + + json.dumps(actual, indent=1, sort_keys=True) + ) + + +def test_external_targets_are_never_attributed() -> None: + """A call into a stdlib or unresolved node is not an edge between our modules.""" + file_a = Node(id="cgis.resolver.x", type=NodeType.FILE, name="x", file_path="cgis/resolver/x.py", + start_line=1, end_line=1) + fn = Node(id="cgis.resolver.x.f", type=NodeType.FUNCTION, name="f", file_path="cgis/resolver/x.py", + start_line=1, end_line=1) + ext = Node(id="cgis.storage.fake", type=NodeType.FUNCTION, name="fake", file_path="", + start_line=0, end_line=0, namespace=NodeNamespace.UNKNOWN) + edge = Edge(id="e", source=fn.id, target=ext.id, type=EdgeType.CALLS) + assert forbidden_edges([file_a, fn, ext], [edge])["resolver-storage"] == {} + + +def test_cross_pack_fires_only_between_different_packs() -> None: + rule = RULES[0] + assert _fires(rule, "cgis.languages.go.extractor", "cgis.languages.python.indices") + assert not _fires(rule, "cgis.languages.go.extractor", "cgis.languages.go.rules") + assert not _fires(rule, "cgis.languages.go.extractor", "cgis.languages") +``` + +- [ ] **Step 2: Run them to see the gate fail** + +Run: `uv run pytest tests/self_parsing/test_architecture.py -q -k "forbidden or external or cross_pack"` +Expected: `test_forbidden_edges_match_the_allowlist_exactly` FAILS with `FileNotFoundError` for `forbidden_edges_allowlist.json`. The two synthetic tests pass. + +- [ ] **Step 3: Generate the allowlist and review it by hand** + +```bash +uv run python - <<'EOF' +import json, sys +sys.path.insert(0, "tests/self_parsing") +from test_architecture import forbidden_edges +from cgis.pipeline import IngestionPipeline +from cgis.extractors.python_extractor import PythonExtractor +from cgis.storage.sqlite_store import SQLiteStore +with SQLiteStore("/tmp/claude-1001/fe.db") as s: + nodes, _, edges = IngestionPipeline({".py": PythonExtractor()}).run("src", store=s) + with open("tests/self_parsing/forbidden_edges_allowlist.json", "w") as fh: + fh.write(json.dumps(forbidden_edges(nodes, edges), indent=1, sort_keys=True) + "\n") +EOF +``` + +Review the file against the spec's table. Expected contents: +- `resolver-storage`: only `cgis.resolver.uplift -> cgis.storage.sqlite_store` (8 edges at b0d67f8); +- `surface-packs`: `cgis.query.render.mermaid` must not appear, because it goes through the exempt registry. Direct extractor-class imports from `cgis.cli`, `cgis.api.mcp_server` and `cgis.guardian.*` may appear; +- `surface-storage`: `cgis.cli`, `cgis.api.*` and `cgis.guardian.*` → `cgis.storage.*`, about 65 edges in total; +- `cross-pack`, `pack-isolation` and `core-isolation`: empty. + +If any other pair appears, it is either a real finding or a rule bug. Stop and decide before committing. + +The script builds the graph exactly as the `root_graph_data` fixture does: `pipeline.run` on `src/`, and the resolved edges it returns. The allowlist then matches the test's own input. + +- [ ] **Step 4: Run the gate** + +Run: `uv run pytest tests/self_parsing/test_architecture.py -q` +Expected: all pass. Then make sure it bites: append `from cgis.storage.sqlite_store import SQLiteStore` to `src/cgis/core/paths.py` and run again. It must FAIL, naming `cgis.core.paths -> cgis.storage.sqlite_store` under `core-isolation`. Revert, and confirm with `git diff src/` showing nothing. + +- [ ] **Step 5: Commit** + +```bash +git add tests/self_parsing/test_architecture.py tests/self_parsing/forbidden_edges_allowlist.json +git commit -m "test(architecture): forbidden-edge gate with exact allowlists (language-packs spec) + +Co-Authored-By: Claude Opus 5.5 " +``` + +### Task 5: Plane quotients, observe-only + +**Files:** +- Modify: `tests/self_parsing/test_drift.py` + +**Interfaces:** +- Consumes: `build_quotient`, `quotient_call_tally` (`cgis/query/drift/quotient.py`), `FingerprintExtractor.from_graph`, `DriftScorer.score`, `DriftScorer.load_project_level`, `_selected_domains` (already in `test_drift.py`). +- Produces: a printed report per plane; nothing enforces it. + +- [ ] **Step 1: Write the test** + +Append to `tests/self_parsing/test_drift.py`: + +```python +import dataclasses + +_PLANES = { + "build": (("pipeline", "extraction", "resolution", "storage"), "orchestrator"), + "read": (("storage", "query", "guardian", "cli", "api"), "layered_dag"), +} + + +def test_plane_quotients_observed_not_enforced( + root_graph_data: tuple[SQLiteStore, list[Node], list[Edge]], +) -> None: + """Score each plane's quotient against its template; print, never fail (spec: Drift gates). + + The enforced topology gate is the forbidden-edge test in test_architecture.py. + These numbers are kept for the record and for #554. + """ + store, _, _ = root_graph_data + scorer = DriftScorer(_PATTERNS) + binding = scorer.load_project_level()[0] + by_name = {d.name: d for d in _selected_domains(scorer, graph="python")} + nodes, edges = store.get_all_nodes(), store.get_all_edges() + for plane, (names, template) in _PLANES.items(): + domains = [by_name[n] for n in names] + qnodes, qedges = build_quotient(nodes, edges, domains) + extractor = FingerprintExtractor.from_graph( + qnodes, qedges, quotient_call_tally(nodes, edges, domains) + ) + fingerprint = extractor.extract(binding.fqn_prefix) + assert fingerprint.node_count == len(domains), f"{plane}: a domain matched nothing" + report = scorer.score(fingerprint, dataclasses.replace(binding, expected_pattern=template)) + print(f"\n[plane observe-only] {plane} vs {template}: drift={report.drift_score:.3f}") +``` + +- [ ] **Step 2: Run it** + +Run: `uv run pytest tests/self_parsing/test_drift.py -q -s -k plane` +Expected: 1 passed. It prints `build vs orchestrator: drift=0.238` and `read vs layered_dag: drift=0.676`, ±0.01 of the spec's b0d67f8 figures. If either number is far off, the plane membership is wrong: check the domain names in `patterns.yaml` before going on. + +`node_count` is the quotient's node count on the fingerprint (`cgis/query/drift/fingerprint.py`). The assertion exists so a misspelled domain fails loudly instead of scoring an empty plane. + +- [ ] **Step 3: Full gate and commit** + +```bash +make format && make lint && make type-check && make pytest && make doc-coverage && \ +git add tests/self_parsing/test_drift.py && \ +git commit -m "test(drift): plane quotients reported observe-only (language-packs spec) + +Co-Authored-By: Claude Opus 5.5 " +``` + +### Step 0 PR + +- [ ] Push the branch, open the PR ("test: language-packs step 0 — identity net, forbidden-edge gate, plane reports"), and run the independent reviewer. Ask it to mutation-check the gate and the snapshot. Merge on approval. + +--- + +## Steps 1–7 — outline (each gets its own detailed plan when started) + +Each step below is one PR. Its plan is written when the previous step has merged, against the code as it then is. Every step's PR description carries the `graph_identity.py diff` result for cgis `src/` at 715cd9a and owner-api @ 8de2245, with `main`'s cgis and the branch's cgis over the same input. + +**Step 1 — Skeleton and registry.** +- `cgis/languages/__init__.py` (lazy registry, per-pack manifests), `languages/python/`, `languages/typescript/`. +- Extractor modules move under their pack. +- `scope_of` and `is_test` per pack; the pipeline's `_TEST_FILE_PATTERN` is replaced. +- CLI, MCP server, `scripts/generate_health.py`, guardian and `mermaid.py` go through the registry. +- Test imports are updated (36 files), with no shims. +- The `extraction` domain in `patterns.yaml` is re-pointed, and its tolerance re-measured. +- CLAUDE.md is updated. + +Acceptance: +- the snapshot is unchanged; +- the `surface-packs` allowlist is empty; +- an ingest of a Python-only repo imports no TS grammar (test by checking `sys.modules`). + +**Step 2 — Project readers.** +- `ProjectReader`/`ProjectFacts` per pack. +- `import_names.py` collects them; the store keys are unchanged. + +Acceptance: snapshot unchanged, including `ingest_state`; the `test_configs_above_root.py` tests are green, unmodified. + +**Step 3 — Index and classification.** +- `index_facts` and `classify(fqn, ctx)` per pack. +- The no-source → Python rule stays in the registry. +- `resolve_layout`/`map_to_node_fqn` are gated by pack. + +Acceptance: snapshot unchanged. + +**Step 4 — Resolution steps.** +- Named generic steps, plus per-pack type normalizers, positioned steps, fallback (JS builtins at 0.8) and import reconciliation. +- `cross_file_metadata` per pack manifest. +- One language branch is extracted per commit, with the snapshot test run after each. + +Acceptance: +- snapshot unchanged; +- owner-api ingest no more than 5% slower than `main`. + +**Step 5 — Build-plane passes.** +- `pipeline.py` becomes `cgis/pipeline/`. +- `resolver/uplift.py` and `refresh_dispatch` move into it. +- The `resolution` domain is re-measured. + +Acceptance: the `resolver-storage` allowlist is empty, and the snapshot is unchanged. + +**Step 6 — Gates.** +- The `project_level` binding is deleted from `patterns.yaml`, along with its test in `test_drift.py`. +- `languages.` domains with one shared tolerance; `core` gets `pure_utility`. +- #146 is closed with the forbidden-edge gate as its answer. + +**Step 7 — Go pilot.** +- `languages/go/` and its registry manifest. +- The `tree-sitter-go` dependency. +- A Go corpus with its own snapshot. + +Acceptance: +- `git diff --stat main` touches only `src/cgis/languages/go/`, `src/cgis/languages/__init__.py` (one manifest), `pyproject.toml`/`uv.lock` and Go tests and fixtures; +- the Python/TS snapshot is unchanged; +- the ontology-compliance tests pass on the Go corpus.