Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
123 lines (97 loc) · 6.87 KB

ARCHITECTURE.md

File metadata and controls

123 lines (97 loc) · 6.87 KB
Raw

Architecture

codebase-memory-zig is a hybrid MCP server: Zig owns the process, the stdio JSON-RPC framing, and a hard filesystem-security bound; the vendored C engine executes every tool with full upstream parity. This document describes how the pieces fit together and why the fork exists.

What it is

A local, single-binary code-structure intelligence MCP server — tree-sitter AST + LSP-style symbol/call-graph over a repository, exposed as stdio JSON-RPC MCP tools (index_repository, search_graph, query_graph, trace_path, get_architecture, detect_changes, …). serverInfo.name stays codebase-memory-mcp and the 15-tool surface is byte-for-byte identical to upstream, so existing MCP clients need no change.

It is a general-purpose tool; the core is kept generic and is not coupled to any specific downstream stack.

Why this fork

Two defects in the upstream prebuilt binary cannot be fixed from outside it, so the engine was forked to fix them at the source.

  1. Unbounded dependency-resolution scan (the CPU-DoS). On an index_repository call the C engine resolves external dependency sources by walking far outside the target repo — in the worst case a full-disk find / -path */registry/src/* crate-resolution scan that runs for hours, orphans when the caller moves on, and stacks across concurrent agents until the host starves. Upstream exposes no scan-root clamp (CBM_ALLOWED_ROOT only refuses an out-of-root target; it does not bound the walk). The fork's hard invariant: every filesystem walk is rooted at the repo tree plus explicitly declared dependency directories, and never touches / or $HOME.

  2. Build / portability pain. The upstream static ELF is ~266 MB, needs glibc 2.38 (vs. the glibc-2.36 environments it commonly runs in), and SIGSEGVs when mmap'd off a Windows bind mount. Zig's cross-compilation (x86_64-linux-gnu.2.36, glibc-2.35, and x86_64-windows) removes all of that from a single zig build.

The hybrid split

The engine's core is inherently C — vendored SQLite, ~200 tree-sitter grammars, xxHash / lz4 / zstd, and a C++ preprocessor — and is neither feasible nor desirable to rewrite. So the responsibilities divide cleanly:

  • Zig owns the process entry point, stdio JSON-RPC framing, argument parsing, and the security bound. See src/main_hybrid.zig.
  • Zig's build system cross-compiles the vendored C engine to a static library (zig build engine), which is the build-portability fix.
  • The hybrid binary delegates tool execution to the vendored C via cbm_mcp_server_handle — one JSON-RPC line in, one response line out. No tool handler is reimplemented, which is what preserves exact upstream behavior.

Pure-Zig code also provides safe file discovery and the WalkRoots types (src/discover/, src/foundation/), and a native MCP tool-surface scaffold (src/mcp/, src/main.zig) whose 15-tool table is mechanically extracted from the upstream source.

The WalkRoots bound (the CPU-DoS fix)

Every directory walk in the C engine — discovery and the dependency-resolution passes (pass_pkgmap, path_alias, pass_cross_repo, pass_envscan, …) — ultimately calls cbm_opendir (defined in reference/upstream-c/src/foundation/compat_fs.c). That makes it the single point at which a filesystem-walk bound can be installed without touching the vendored source or reimplementing any handler.

The build compiles compat_fs.c with -Dcbm_opendir=cbm_opendir_real, renaming its definition. src/cshim/walk_guard.c then defines the real cbm_opendir: it canonicalizes the path, checks it against the set of allowed roots, and either delegates to cbm_opendir_real or refuses (EACCES) with a log line. Because every other translation unit still calls the plain cbm_opendir symbol, the linker routes all of them through the guard.

Allowed roots are declared at startup with repeatable --walk-root=<dir> flags, plus the engine's own trusted data directories (the SQLite store and its resolved cache dir, added automatically). A walk that escapes to /, $HOME, or CARGO_HOME matches none of the declared roots and is refused. Only directory iteration is bounded; subprocess spawn (cbm_popen, for git/curl) is out of scope for this guard.

Build targets

zig build hybrid       Linux MCP server, glibc 2.36 deploy target
zig build hybrid-test  Linux MCP server, glibc 2.35 (for WSL smoke tests)
zig build hybrid-win   Windows MCP server (.exe)
zig build engine       cross-compile the vendored C engine to a static lib
zig build test         Zig unit + parity tests

Vendored reference (behavioral source of truth)

reference/upstream-c/ is a clean snapshot of DeusData/codebase-memory-mcp @ e678b2b (MIT). It is read-only: fixes are applied via build flags and new files under src/, never by editing the vendored tree, so the snapshot stays a faithful oracle for behavior parity.

Module map

Where the engine's logic lives in the vendored tree, roughly in dependency order:

Path Role
src/foundation/ arena/slab allocation, compat_fs (opendir/popen wrappers), platform, logging — the walk primitives live here
src/discover/ file discovery, .gitignore / .cbmignore, language detection — where the walk root + ignore predicate are decided
src/mcp/ JSON-RPC tool surface, tool schemas, autoindex file-count guard — the wire contract
src/pipeline/ extract → registry → resolve passes; the dependency-resolution passes are the ones the WalkRoots bound constrains
internal/cbm/lsp/ LSP-style cross-repo + manifest resolution
src/semantic/, src/simhash/, src/graph_buffer/, src/store/ AST profiling, hashing, in-memory graph, persistence
src/cypher/, src/traces/, src/git/ query engine, traces, git context
src/ui/, graph-ui/ 3D graph HTTP UI (optional; typically run as a separate host service)

Design principles

  1. Bounded I/O is non-negotiable. No filesystem walk may escape the declared roots. There is no code path that falls back to /, $HOME, CARGO_HOME, or a global package cache without an explicit, operator-supplied root.
  2. Behavior parity with the vendored C. The MCP tool contract is identical to upstream; parity is verified against the vendored C on fixtures (see docs/PORTING-PLAN.md).
  3. Cross-compilation is a feature. Static Linux (glibc-pinned) and Windows binaries come from one zig build, with no 266 MB bind-mount hazard.
  4. Local and dependency-light. stdio JSON-RPC only — no network and no API keys on the trust path, which suits sensitive-code environments.

MIT-licensed; upstream copyright (c) 2025 DeusData is preserved, and the vendored snapshot keeps its own LICENSE / THIRD_PARTY notices. See docs/PORTING-PLAN.md for the phased C → Zig plan and the parity-test approach.