diff --git a/README.md b/README.md index 21b37bc..60e9e26 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,7 @@ claude mcp add -s user -t stdio codebase-intelligence -- npx -y codebase-intelli - **18 CLI commands** for architecture analysis, dependency impact, improvement opportunities, dead code detection, search, CI rules, and agent setup - **Machine-readable JSON output** (`--json`) for automation and CI pipelines - **Auto-cached index** in `.codebase-intelligence/` for fast repeat queries +- **Cache migration facts** in JSON (`cacheDir`, `legacyCacheDir`, `migrated`, `gitignoreUpdated`, `warnings[]`) - **11 architectural metrics** — PageRank, betweenness, coupling, cohesion, tension, churn, complexity, blast radius, dead exports, test coverage, escape velocity - **Symbol-level analysis** — callers/callees, symbol importance, impact blast radius - **BM25 search** — ranked keyword search across files and symbols @@ -123,6 +124,8 @@ codebase-intelligence [options] The scanner always excludes common generated and agent-workspace directories such as `.codebase-intelligence/`, legacy `.code-visualizer/`, `.next/`, `dist/`, `coverage/`, `.worktrees/`, and `.claude/worktrees/`. +Analysis commands with `--json` and `init --json` include a top-level `cache` object with the canonical cache path, legacy cache path, migration status, `.gitignore` update status, and non-fatal cache warnings. + For full command details, see [docs/cli-reference.md](docs/cli-reference.md). ## Agent Adoption diff --git a/docs/cli-reference.md b/docs/cli-reference.md index f14edeb..0c288eb 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -199,7 +199,7 @@ The global skill is never installed unless `--skill` is passed. codebase-intelligence init [path] [--agents ] [--all] [--skill] [--gitignore] [--yes] [--json] ``` -**Output:** per-file actions (created / updated / unchanged) and skill install status. +**Output:** per-file actions (created / updated / unchanged), optional `.gitignore` action, skill install status, and cache facts in JSON mode. ## Flags @@ -234,6 +234,18 @@ codebase-intelligence init [path] [--agents ] [--all] [--skill] [--gitigno **Auto-caching:** First CLI invocation parses the codebase and saves the index to `.codebase-intelligence/`. Subsequent commands use the cache only when `git HEAD`, dirty/untracked file contents under the analyzed path, the CLI version, and parser cache settings match. Legacy `.code-visualizer/` is migrated when `.codebase-intelligence/` is absent. Add `.codebase-intelligence/` to `.gitignore` manually or run `codebase-intelligence init --gitignore`. +**Cache JSON facts:** Analysis commands with `--json` and `init --json` include a top-level `cache` object: + +```json +{ + "cacheDir": "/repo/.codebase-intelligence", + "legacyCacheDir": "/repo/.code-visualizer", + "migrated": false, + "gitignoreUpdated": false, + "warnings": [] +} +``` + **Default scanner excludes:** The parser always skips `.git`, `node_modules`, `.codebase-intelligence`, legacy `.code-visualizer`, `.next`, `dist`, `coverage`, `.turbo`, `.cache`, `.worktrees`, and `.claude/worktrees`, even if the target repo has no matching `.gitignore` entry. **Large repo mode:** Repos above 1500 TypeScript files use a lightweight AST parser by default to avoid TypeScript program OOM. File/import/export/dependency metrics remain available; type-resolved call graph details are reduced. Set `CBI_FULL_PROGRAM_FILE_LIMIT=` to tune the cutoff. diff --git a/docs/data-model.md b/docs/data-model.md index 1dcfd06..f0a8e36 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -116,3 +116,18 @@ CodebaseGraph { } } ``` + +## CLI Cache Facts + +Analysis commands with `--json` and `init --json` include this top-level `cache` +object. Paths are absolute. + +```typescript +CacheFacts { + cacheDir: string // Canonical .codebase-intelligence/ directory + legacyCacheDir: string // Legacy .code-visualizer/ directory + migrated: boolean // true when legacy-only cache moved this run + gitignoreUpdated: boolean // true when init --gitignore wrote/updated .gitignore + warnings: string[] // Non-fatal legacy/cache state warnings +} +``` diff --git a/llms-full.txt b/llms-full.txt index 3c47f4e..a1cb1e4 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -75,6 +75,7 @@ analyzeGraph(builtGraph, parsedFiles) - **Dead export detection**: Cross-references parsed exports against edge symbol lists. May miss `import *` or re-exports. - **Graceful degradation**: Non-git dirs get churn=0, no-test codebases get coverage=false. Never crashes. - **Auto-caching**: CLI commands always cache the graph index to `.codebase-intelligence/`. MCP mode requires `--index` to persist. Legacy `.code-visualizer/` is migrated when canonical cache is absent. +- **Cache JSON facts**: Analysis commands with `--json` and `init --json` include top-level `cache`: `cacheDir`, `legacyCacheDir`, `migrated`, `gitignoreUpdated`, `warnings[]`. --- @@ -426,6 +427,7 @@ Gate modes: all, new-only. Returns pass/warn/fail verdict, findings, and summary ## Global Behavior - **Auto-caching**: First run parses and saves index to `.codebase-intelligence/`. Subsequent runs use cache only when HEAD, dirty/untracked file contents under the analyzed path, CLI version, and parser cache settings match. Legacy `.code-visualizer/` is migrated when canonical cache is absent. +- **Cache JSON facts**: Analysis commands with `--json` and `init --json` include top-level `cache`: `cacheDir`, `legacyCacheDir`, `migrated`, `gitignoreUpdated`, `warnings[]`. - **Default scanner excludes**: `.git`, `node_modules`, `.codebase-intelligence`, legacy `.code-visualizer`, `.next`, `dist`, `coverage`, `.turbo`, `.cache`, `.worktrees`, and `.claude/worktrees`. - **Monorepo imports**: Root tsconfig path aliases and local package.json package names resolve to source files before graph construction. - **Large repo mode**: Above 1500 TypeScript files, use AST-only extraction to avoid TypeScript program OOM. Override with `CBI_FULL_PROGRAM_FILE_LIMIT`. diff --git a/llms.txt b/llms.txt index 800327d..b328bf6 100644 --- a/llms.txt +++ b/llms.txt @@ -43,6 +43,7 @@ codebase-intelligence init . # make AI agents use CI - CLI analysis commands cache to `.codebase-intelligence/`; legacy `.code-visualizer/` is migration input only. Run `codebase-intelligence init --gitignore` to ignore the canonical cache directory. - `overview --json` includes `analysis.mode`, `analysis.callGraphPrecision`, and `analysis.fullProgramFileLimit`. +- Analysis commands with `--json` and `init --json` include `cache`: `cacheDir`, `legacyCacheDir`, `migrated`, `gitignoreUpdated`, `warnings[]`. - Repos above the full-program file limit use AST-only extraction; dependency metrics remain available, call graph precision becomes syntax-only. ## Optional diff --git a/package.json b/package.json index 443adb2..f5da5e6 100644 --- a/package.json +++ b/package.json @@ -12,7 +12,7 @@ "build": "tsc -p tsconfig.build.json", "dev": "tsx src/cli.ts", "start": "node dist/cli.js", - "test": "pnpm build && vitest run --pool threads --no-file-parallelism", + "test": "pnpm build && vitest run --no-cache --pool threads --no-file-parallelism --maxWorkers=1", "test:coverage": "pnpm build && vitest run --coverage --pool threads --no-file-parallelism --exclude 'tests/*.e2e.test.ts'", "verify:cli-real": "pnpm build && node tools/verify-cli-real-codebases.mjs", "verify:cli-real:heavy": "pnpm build && CBI_REAL_PROFILE=heavy node tools/verify-cli-real-codebases.mjs", diff --git a/roadmap.md b/roadmap.md index b16f123..f2c24b1 100644 --- a/roadmap.md +++ b/roadmap.md @@ -3,7 +3,7 @@ > Future work only. Deterministic, graph-native codebase intelligence for TypeScript & JavaScript. > Read-first, agent-native, architecture-aware. No invented findings: every claim is graph-backed evidence a human or agent can inspect. -**Last updated:** 2026-06-30 +**Last updated:** 2026-07-01 ## Direction @@ -60,7 +60,7 @@ Guiding constraints for all future work: | Item | Dev-ready contract | |---|---| -| Cache migration | All cache states are covered by tests: only legacy, only canonical, both same signature, both different signature, neither. Scanner excludes both folders. Writes go only to `.codebase-intelligence/`. `init --gitignore` is idempotent. | +| Cache migration | All cache states are covered by tests: only legacy, only canonical, both same signature, both different signature, neither. Scanner excludes both folders. Writes go only to `.codebase-intelligence/`. `init --gitignore` is idempotent. JSON surfaces expose `cacheDir`, `legacyCacheDir`, `migrated`, `gitignoreUpdated`, and `warnings[]`. | | Docs/help alignment | README, CLI help, `docs/cli-reference.md`, `docs/mcp-tools.md`, `docs/data-model.md`, `llms.txt`, and `llms-full.txt` describe `.codebase-intelligence/` as canonical and `.code-visualizer/` as legacy migration input. | | Test-runner determinism | `pnpm test` exits 0 locally after assertions pass; the Vitest worker `onTaskUpdate` timeout is gone or the runner configuration is made deterministic. | | Targeted framework fixes | Any remaining framework/config false positive has a minimal fixture proving the bug and a focused fix. No broad plugin framework lands in `2.5.0`. | @@ -144,7 +144,7 @@ Purpose: prove the CLI, MCP server, library outputs, docs, CI behavior, and agen | Chain | User story | Required actions | |---|---|---| -| CH-P0-01 cache migration | As US-MIGRATOR, I can upgrade without losing or duplicating cache state. | Create fixture with only `.code-visualizer/` -> run real CLI command -> assert `.codebase-intelligence/` exists -> assert legacy handling warning -> rerun command -> assert canonical cache is used -> run `--status` -> run `--clean`. | +| CH-P0-01 cache migration | As US-MIGRATOR, I can upgrade without losing or duplicating cache state. | Create fixture with only `.code-visualizer/` -> run real CLI command -> assert `.codebase-intelligence/` exists -> assert JSON `cache.migrated` -> rerun command -> assert canonical cache is used -> run `--status` -> run `--clean`. | | CH-P0-02 new cache gitignore | As US-CLI-HUMAN, I can initialize ignore rules safely. | Run init/gitignore command in temp repo -> assert `.gitignore` gets `.codebase-intelligence/` once -> rerun -> assert idempotent -> assert legacy folder is documented as migration input only. | | CH-P0-03 docs/help alignment | As US-CLI-AGENT, I can learn the canonical cache path from every surface. | Build CLI -> capture `--help` and command help -> scan README/docs/LLM docs -> assert `.codebase-intelligence/` canonical wording -> assert `.code-visualizer/` appears only in legacy migration context. | | CH-P0-04 deterministic test runner | As US-MAINTAINER, I can trust release gates. | Run `pnpm test` -> assert exit 0 -> assert no Vitest `onTaskUpdate` timeout -> rerun focused long E2E suite -> assert no open-handle/worker timeout. | @@ -204,7 +204,7 @@ Purpose: prove the CLI, MCP server, library outputs, docs, CI behavior, and agen Rename the legacy index/cache folder from `.code-visualizer/` to `.codebase-intelligence/` without breaking existing users. -**Status:** Partially shipped in the P0 canary PR. +**Status:** Shipped across P0 canary PRs. **Shipped:** @@ -215,10 +215,7 @@ Rename the legacy index/cache folder from `.code-visualizer/` to `.codebase-inte - Add `init --gitignore` to append `.codebase-intelligence/` idempotently. - Add `--clean` behavior that removes canonical and legacy cache directories. - Update docs, README, CLI help, tests, and fixtures to mention legacy auto-migration. - -**Remaining:** - -- Expose migration facts in JSON: `cacheDir`, `legacyCacheDir`, `migrated`, `gitignoreUpdated`, `warnings[]`. +- Expose migration facts in JSON on analysis commands and `init --json`: `cacheDir`, `legacyCacheDir`, `migrated`, `gitignoreUpdated`, `warnings[]`. ### Docs + Help Alignment @@ -243,6 +240,7 @@ Fix the repeated Vitest runner timeout so release gates are deterministic. - Switch `pnpm test` to `--pool threads --no-file-parallelism`. - Keep `429` assertions passing and make `pnpm test` exit 0 locally. - Bound non-blocking CI coverage collection so coverage hangs cannot block the release gate. +- Bound `pnpm test` to one Vitest worker and disabled Vitest duration cache so host-level memory pressure and stale file ordering do not terminate the release gate. ### Targeted Framework Entry Fixes diff --git a/src/cli.ts b/src/cli.ts index e87c10e..6ba1b96 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -24,7 +24,13 @@ import { startMcpServer } from "./mcp/index.js"; import { setIndexedHead, setRoot } from "./server/graph-store.js"; import { exportGraph, importGraph } from "./persistence/index.js"; import { getCacheKey } from "./persistence/cache-key.js"; -import { cleanIndexDirectories, prepareIndexDirectory, type IndexDirectoryResolution } from "./persistence/index-dir.js"; +import { + cleanIndexDirectories, + getCacheFacts, + getCacheFactsForTarget, + prepareIndexDirectory, + type IndexDirectoryResolution, +} from "./persistence/index-dir.js"; import { computeOverview, computeFileContext, @@ -58,10 +64,12 @@ import { promptSelection } from "./install/prompt.js"; import { runCheck, exitCodeFor } from "./rules/check.js"; import { formatResult, formatSummaryLine } from "./rules/format.js"; import { ConfigError } from "./config/index.js"; -import type { CodebaseGraph, OutputFormat } from "./types/index.js"; +import type { CacheFacts, CodebaseGraph, OutputFormat } from "./types/index.js"; // ── Helpers ───────────────────────────────────────────────── +let activeCacheFacts: CacheFacts | null = null; + function reportIndexMigration(resolution: IndexDirectoryResolution): void { if (resolution.migration === "migrated-legacy") { progress(`Migrated legacy index ${resolution.legacyDir} to ${resolution.canonicalDir}`); @@ -92,8 +100,13 @@ function output(data: string): void { process.stdout.write(`${data}\n`); } +function isJsonObject(data: unknown): data is Record { + return typeof data === "object" && data !== null && !Array.isArray(data); +} + function outputJson(data: unknown): void { - process.stdout.write(`${JSON.stringify(data, null, 2)}\n`); + const payload = activeCacheFacts && isJsonObject(data) ? { ...data, cache: activeCacheFacts } : data; + process.stdout.write(`${JSON.stringify(payload, null, 2)}\n`); } /** Load (or parse+cache) the codebase graph for a target path. */ @@ -107,6 +120,7 @@ function loadGraph(targetPath: string, force = false): { graph: CodebaseGraph; h const indexResolution = prepareIndexDirectory(targetPath); reportIndexMigration(indexResolution); + activeCacheFacts = getCacheFacts(indexResolution); const indexDir = indexResolution.activeDir; const headHash = getHeadHash(targetPath); const cacheKey = getCacheKey(targetPath, { headHash, cliVersion: pkg.version }); @@ -1073,7 +1087,12 @@ program const skillResult = installSkill ? installGlobalSkill() : undefined; if (options.json) { - outputJson({ repoFiles: repoResults, gitignore: gitignoreResult ?? null, skill: skillResult ?? null }); + outputJson({ + repoFiles: repoResults, + gitignore: gitignoreResult ?? null, + skill: skillResult ?? null, + cache: getCacheFactsForTarget(resolved, gitignoreResult?.action === "created" || gitignoreResult?.action === "updated"), + }); return; } @@ -1220,6 +1239,7 @@ async function runMcpMode(targetPath: string, options: McpOptions): Promise { try { const overview = run(["overview", repo, "--json", "--force"]); expect(overview.status).toBe(0); + const overviewJson = JSON.parse(overview.stdout) as JsonWithCache; + expect(overviewJson.cache).toEqual({ + cacheDir: path.join(repo, CANONICAL_INDEX_DIR_NAME), + legacyCacheDir: path.join(repo, LEGACY_INDEX_DIR_NAME), + migrated: false, + gitignoreUpdated: false, + warnings: [], + }); expect(fs.existsSync(path.join(repo, CANONICAL_INDEX_DIR_NAME, "graph.json"))).toBe(true); expect(fs.existsSync(path.join(repo, LEGACY_INDEX_DIR_NAME))).toBe(false); @@ -47,6 +67,7 @@ describe("cache CLI lifecycle", () => { const clean = run([repo, "--clean"]); expect(clean.status).toBe(0); expect(clean.stderr).toContain(CANONICAL_INDEX_DIR_NAME); + expect(clean.stderr).toContain(LEGACY_INDEX_DIR_NAME); expect(fs.existsSync(path.join(repo, CANONICAL_INDEX_DIR_NAME))).toBe(false); expect(fs.existsSync(path.join(repo, LEGACY_INDEX_DIR_NAME))).toBe(false); } finally { @@ -65,6 +86,14 @@ describe("cache CLI lifecycle", () => { const overview = run(["overview", repo, "--json", "--force"]); expect(overview.status).toBe(0); expect(overview.stderr).toContain("Migrated legacy index"); + const parsed = JSON.parse(overview.stdout) as JsonWithCache; + expect(parsed.cache).toEqual({ + cacheDir: canonical, + legacyCacheDir: legacy, + migrated: true, + gitignoreUpdated: false, + warnings: [], + }); expect(fs.existsSync(legacy)).toBe(false); expect(fs.readFileSync(path.join(canonical, "marker.txt"), "utf-8")).toBe("legacy\n"); expect(fs.existsSync(path.join(canonical, "graph.json"))).toBe(true); @@ -72,4 +101,27 @@ describe("cache CLI lifecycle", () => { fs.rmSync(repo, { recursive: true, force: true }); } }); + + it("warns in JSON when canonical and legacy cache directories both exist", () => { + const repo = fixtureRepo(); + try { + const canonical = path.join(repo, CANONICAL_INDEX_DIR_NAME); + const legacy = path.join(repo, LEGACY_INDEX_DIR_NAME); + fs.mkdirSync(canonical); + fs.mkdirSync(legacy); + + const overview = run(["overview", repo, "--json", "--force"]); + expect(overview.status).toBe(0); + const parsed = JSON.parse(overview.stdout) as JsonWithCache; + expect(parsed.cache).toEqual({ + cacheDir: canonical, + legacyCacheDir: legacy, + migrated: false, + gitignoreUpdated: false, + warnings: [`Legacy cache directory exists but ${canonical} is active: ${legacy}`], + }); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); }); diff --git a/tests/cli-init.e2e.test.ts b/tests/cli-init.e2e.test.ts index 5596c87..3c4432f 100644 --- a/tests/cli-init.e2e.test.ts +++ b/tests/cli-init.e2e.test.ts @@ -92,10 +92,14 @@ describe("init lifecycle (e2e)", () => { const parsed = JSON.parse(stdout) as { repoFiles: { path: string; action: string }[]; + cache: { cacheDir: string; legacyCacheDir: string; migrated: boolean; gitignoreUpdated: boolean; warnings: string[] }; skill: unknown; }; expect(parsed.repoFiles.map((r) => r.path).sort()).toEqual(["AGENTS.md", "CLAUDE.md"]); expect(parsed.repoFiles.every((r) => r.action === "created")).toBe(true); + expect(parsed.cache.gitignoreUpdated).toBe(false); + expect(parsed.cache.cacheDir).toBe(path.join(repo, ".codebase-intelligence")); + expect(parsed.cache.legacyCacheDir).toBe(path.join(repo, ".code-visualizer")); expect(parsed.skill).toBeNull(); expect(read("AGENTS.md")).toContain(DEFAULT_MARKERS.start); @@ -189,6 +193,26 @@ describe("init lifecycle (e2e)", () => { expect(read(".gitignore")).toBe(".codebase-intelligence/\n"); }); + it("--json --gitignore emits cache facts with gitignoreUpdated", () => { + const { status, stdout } = run(["init", "--agents", "", "--gitignore", "--json", repo], home); + expect(status).toBe(0); + + const parsed = JSON.parse(stdout) as { + repoFiles: unknown[]; + gitignore: { path: string; action: string }; + cache: { cacheDir: string; legacyCacheDir: string; migrated: boolean; gitignoreUpdated: boolean; warnings: string[] }; + }; + expect(parsed.repoFiles).toEqual([]); + expect(parsed.gitignore).toEqual({ path: ".gitignore", action: "created" }); + expect(parsed.cache).toEqual({ + cacheDir: path.join(repo, ".codebase-intelligence"), + legacyCacheDir: path.join(repo, ".code-visualizer"), + migrated: false, + gitignoreUpdated: true, + warnings: [], + }); + }); + it("exits 1 when the target path does not exist", () => { const missing = path.join(os.tmpdir(), "ci-init-missing-zzz-do-not-create"); const { status, stderr } = run(["init", missing], home); diff --git a/tests/index-directory.test.ts b/tests/index-directory.test.ts index 84eae10..dbe319a 100644 --- a/tests/index-directory.test.ts +++ b/tests/index-directory.test.ts @@ -3,7 +3,7 @@ import os from "os"; import path from "path"; import { describe, expect, it } from "vitest"; import { CANONICAL_INDEX_DIR_NAME, LEGACY_INDEX_DIR_NAME } from "../src/persistence/cache-key.js"; -import { cleanIndexDirectories, prepareIndexDirectory } from "../src/persistence/index-dir.js"; +import { cleanIndexDirectories, getCacheFacts, prepareIndexDirectory } from "../src/persistence/index-dir.js"; function tempDir(): string { return fs.mkdtempSync(path.join(os.tmpdir(), "ci-index-dir-")); @@ -48,6 +48,13 @@ describe("index directory migration", () => { const result = prepareIndexDirectory(dir); expect(result.activeDir).toBe(canonical); expect(result.migration).toBe("migrated-legacy"); + expect(getCacheFacts(result)).toEqual({ + cacheDir: canonical, + legacyCacheDir: legacy, + migrated: true, + gitignoreUpdated: false, + warnings: [], + }); expect(fs.existsSync(legacy)).toBe(false); expect(fs.readFileSync(path.join(canonical, "marker.txt"), "utf-8")).toBe("legacy\n"); } finally { @@ -70,6 +77,9 @@ describe("index directory migration", () => { const result = prepareIndexDirectory(dir); expect(result.activeDir).toBe(canonical); expect(result.migration).toBe("ignored-legacy"); + expect(getCacheFacts(result).warnings).toEqual([ + `Legacy cache directory exists but ${canonical} is active: ${legacy}`, + ]); expect(fs.readFileSync(path.join(canonical, "marker.txt"), "utf-8")).toBe("canonical\n"); expect(fs.readFileSync(path.join(legacy, "marker.txt"), "utf-8")).toBe("legacy\n"); } finally { @@ -90,6 +100,9 @@ describe("index directory migration", () => { const result = prepareIndexDirectory(dir); expect(result.activeDir).toBe(canonical); expect(result.migration).toBe("ignored-legacy"); + expect(getCacheFacts(result).warnings).toEqual([ + `Legacy cache directory exists but ${canonical} is active: ${legacy}`, + ]); expect(JSON.parse(fs.readFileSync(path.join(canonical, "meta.json"), "utf-8"))).toEqual({ cacheKey: "canonical" }); expect(JSON.parse(fs.readFileSync(path.join(legacy, "meta.json"), "utf-8"))).toEqual({ cacheKey: "legacy" }); } finally { diff --git a/tools/verify-cli-real-codebases.mjs b/tools/verify-cli-real-codebases.mjs index b8b9238..b8c0d57 100644 --- a/tools/verify-cli-real-codebases.mjs +++ b/tools/verify-cli-real-codebases.mjs @@ -275,7 +275,7 @@ function assertNoBannedPaths(value, location) { } } -function findBannedPath(value) { +function findBannedPath(value, keyPath = []) { if (typeof value === "string") { const normalized = value.replaceAll("\\", "/"); if (normalized.includes("package.json:")) return ""; @@ -289,15 +289,16 @@ function findBannedPath(value) { if (Array.isArray(value)) { for (const item of value) { - const result = findBannedPath(item); + const result = findBannedPath(item, keyPath); if (result) return result; } return ""; } if (value && typeof value === "object") { - for (const item of Object.values(value)) { - const result = findBannedPath(item); + for (const [key, item] of Object.entries(value)) { + if (keyPath.length === 0 && key === "cache") continue; + const result = findBannedPath(item, [...keyPath, key]); if (result) return result; } }