feat(mem): a central allocation core, the memory migrated onto it, and results spilled to disk under budget

[DeusData]

Memory in this project is allocated through one core, with a linter that keeps new code on it and phase attribution in the index pipeline. This increment migrates the memory that matters onto it, proves where an index's memory actually goes, removes the waste that proof exposed — and makes the memory budget a promise: over budget, extraction results are parked on disk and the run completes at the floor instead of aborting. The Linux kernel indexes on the default budget again (it aborted on v0.10.9-rc), and indexes on a 16 GB budget through the spill path.

Every number is from a real index on an M5 Pro (48 GB) with CBM_MEM_PHASES=1, release build.

What the proof said

The 2026-07 research blamed the graph buffer for the peak. Measured, the graph buffer is 3% of the memory (0.55 GB peak on the Go corpus). The per-file extraction results are the memory:

Go corpus (21,875 files, 345 MB of source)
written into the per-file result arenas	14,804 MB
reachable from the results (records + strings + retained source)	~3,400 MB
arena capacity charged (block doubling headroom)	28,817 MB

The rest is temporaries — cbm_node_text copies at 496 call sites, per-node QN strings, abandoned generations of every growable array — that an arena cannot free and that the result kept alive until resolve ended, because the result owned the arena. On the kernel, resolve then grew the results 8.9 → 13.3 GB used / 17.2 GB capacity because the C walk allocated its working set in the result arena and stored arena pointers into the shared registry. After that, 7 GB of the kernel's post-resolve memory had no class at all. Git history is not a factor (streamed git log, freed per commit).

What changed

Attribution. Graph buffer (87 raw sites → 0), arena blocks, the tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the hash table (Verstable, one class per table via cbm_ht_create_in) and CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table, dyn_array join the table. Two cross-boundary frees of buffer-owned strings (pass_importance, pass_complexity) now go through cbm_gbuf_node_set_properties_json. Linter baseline 4086 → 3916.

Result compaction (internal/cbm/result_compact.c). At the end of a file's extraction, everything reachable from its result is measured, copied into one exact-size arena (strings interned by content within the file), and the working arena is handed back. Later appends restart growth at the default block (cbm_arena_init_exact, arena grow_size). The stale duplicate internal/cbm/arena.h — same include guard as the foundation header, winning or losing by include order — is a shim.

Record slimming. CBMCall 320 → 72 B (the eight inline argument slots, 256 B and mostly empty, are allocated on first capture); CBMUsage 48 → 40 B.

The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One lifecycle for every language's cross-file resolve: scratch arena → per-file overlay registry chained to the sealed shared base → the walk runs against the overlay → resolved calls appended into the result → scratch destroyed. type_registry gains chain-aware iterators (every yielded index belongs to it->reg) and copy-on-write refinement (cbm_registry_func/type_for_update); the C, C#, Go and Python walks adopt them. An ASan heap-use-after-free in lsp_resolution_probe forced this: the C walk stored per-call arena objects into the shared registry, and a kernel run with a naive scratch arena lost 1,045 edges to garbage reads. It also ends unsynchronized writes into the shared registry from parallel resolve workers.

The 7 GB with no class, found and cleaned. gbuf_index tables and keys were 2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five secondary indexes they never query (cbm_gbuf_new_worker builds none), and 18M per-key index arrays started at 8 slots (cbm_da_push_min starts them at 2); edge property JSON is 81% {} (interned). The semantic pass allocated a 512-token stride per function up front (packed tokens with offsets). Edge dedup keys are 128-bit hashes instead of sprintf'd strings. Each worker keeps one reusable working arena between files (cbm_work_arena_take/give: rewind, not free), and every phase mark returns freed pages to the OS.

Budget metric. cbm_mem_over_budget compares the charged footprint — max(phys_footprint, mimalloc commit) on macOS — not resident_size: after extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged (purged pages stay resident until the OS wants them), and phys_footprint under-reports after MADV_FREE_REUSABLE cycles.

Spill / admission control (internal/cbm/result_spill.c, the extract gate in pass_parallel.c). At 15/16 of the budget — or over it, or with CBM_MEM_SPILL=1 — the gate latches spill mode: every compacted result is parked on disk as header + block in one append-only file per writer under <cache>/spill/, the results already cached are swept out by every worker, and registry build, def collection, surface rows, resolve and the infra-route passes load a result only for the moment they read it (relocated by base delta). The abort path stays closed while a sweep runs or a cached result remains; it fires only at the floor (graph + registries + in-flight files). A retained parse tree is dropped with the parked result and re-parsed on load; a result that owns sub-results stays in memory.

Every consumer of the result cache reads a slot through cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked slot is NULL in the array. The transient loads found what had been borrowed from results all along: a def label that was the result's own pointer (SIGBUS in the Go cross registry builder), two JVM helpers that passed their input through, the infra-route passes indexing the array (39 → 0 __route__infra__ nodes, INFRA_MAPS gone) and their dedup table borrowing sr->value keys (the insert spun forever on freed memory), and a latch that was visible before the store was open (a peer found nothing to park and failed the run while its neighbour wrote 9 GB). The arena rewind reuse branch had defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it now reuses only blocks that exist. The incremental and probe routes still index the cache array themselves and therefore never spill (ctx->spill_allowed stays false there) — follow-up.

The semantic pass under a budget (pass_semantic_edges.c). The pass held every function's tokens, the corpus's per-document token ids and the vectors being stored at once — 4.9 GB on the kernel on top of the floor, and nothing sized it to the headroom. Now, when the charged footprint plus that transient would cross the budget, phases 2–4 run per batch of functions (tokenize → count into the corpus → free; finalize once; tokenize again → vectorize → store → free), pages returned to the OS after every batch. Tokenization is deterministic, so both passes see the same tokens and the graph does not change; the run pays with a second tokenize. CBM_SEM_BATCH=<n> forces the batch size. The corpus frees its per-document token ids at the end of finalize, and mem.semantic.step lines (CBM_MEM_PHASES=1) attribute the pass's memory per sub-phase.

Cross-allocator free, found and closed. The semantic corpus and tokenizer now allocate through the core (54 raw sites → 0), and that is not attribution only: cbm_sem_tokenize handed out libc strdup blocks that the pass freed with cbm_free — mi_free in the production build — since the pass moved onto the core. No libc-backed test build can see that. The core now refuses a block it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set, fatally; every proof run in this PR ran with it armed and reports zero foreign blocks.

Accounting without contention. The core charged every allocation and free to shared per-class atomics; with 18 workers on 18 cores every block bounced the same three cache lines, and the same-machine bench against the shipped v0.10.8 showed it as CPU time rising faster than wall time for the same graph (Kotlin 38 → 121 s CPU, +58% wall; TypeScript 210 → 596 s, +67%; Go 177 → 312 s). The deltas are thread-local now and reach the shared counters once per 256 KB or 512 blocks of change per thread, when a parallel-for worker ends, and whenever a reader looks. Phase marks read after the join and are exact; a class peak is a diagnostic and can be low by threads × 256 KB. Result: Kotlin 10.9 s (the residual +22% against v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s (v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged.

SQLite on a heap of its own. The same bench put the kernel's write step at 132 s where v0.10.8 needed 9.7 s, all of it in one coverage-publish statement. A stack sample named the cost: mimalloc's free-page search and its periodic heap collect walking the graph's pages — millions of them once the graph lives on the core — for every statement-journal chunk SQLite allocates. SQLite now allocates from a mimalloc heap of its own per thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes), and the periodic collect is set to the maximum mimalloc allows, since the explicit release at phase marks and spill sweeps is where memory goes back. Coverage publish 132 → 9.4 s; profiled kernel 376 → 248 s (v0.10.8: 296 s).

Formatting without a process lock. On macOS every vsnprintf takes the process locale under an unfair lock (localeconv_l inside __vfprintf); eighteen workers formatting type names on the C# corpus collapsed into it — the 10 MB JIT test files took 68 s each instead of under a second — and cbm_arena_sprintf paid it twice per string (size pass, write pass). Each thread now formats with a C locale object of its own (vsnprintf_l), and the common short string is formatted once into a stack buffer.

Per-file budgets that cover the whole file. The 5 s per-file budget covered the parse only; the LSP walks after it had none. C# JIT stress files (a 23 MB single expression among them) parsed inside the budget — the old wall-clock parse timeout used to drop them, which hid the rest — and then held a worker for 346 s each in the usage walk (tree-sitter's ts_node_parent descends from the root: quadratic on a deep tree), and the cross-file resolve ran on the same trees. Three rules now, one site for every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a parse that used more than half the budget disqualifies the file from the LSP walks (lsp_skipped); the unified walk checks its thread CPU time every 1024 nodes against six budgets and stops there, keeping what it found (walk_truncated, implies lsp_skipped); a file whose parse plus walk spent the budget is skipped by the LSP walks as well. Each decision is logged with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction 355 → 39 s; the whole index 463 → 128 s wall and 27 → 11 GB peak against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the 15 MB System.Runtime.Intrinsics reference file walks to the end and, with four generic-nesting JIT regression tests, only skips the LSP walk under the parse-plus-walk rule.

The crash behind the C# bench. dotnet/runtime is 42,555 files, and its resolve phase died with SIGBUS on every run of this branch. A libc-backed ASan build of the server named it: heap-use-after-free in c_adl_resolve, reading a type name that another worker had allocated in its per-file scratch arena and freed at the end of its file. The C++ class walk refines a method's return type when the declaration is more specific than the pre-registered one (NAMED → pointer/reference/template), and it did that by casting the chained lookup result to non-const and writing a scratch-arena signature into it — into the sealed shared base when the entry lived there. The overlay contract had exactly one bypass, and it was this cast; every lookup returns const, so a grep for the cast is the audit. The refinement now goes through cbm_registry_func_for_update: copy-on-write into the overlay, the base untouched. Test: clsp_method_return_refinement_is_copy_on_write — RED on the cast, GREEN on the accessor. The fixed ASan build then indexed the whole corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker exit 0, no report.

Backing. The production build backs the core with mimalloc explicitly (never the global override, which macOS's two-level namespace forbids); the test build keeps libc so the sanitizers see every block — the split the tree-sitter/SQLite bindings already use.

Proof

	before	after
Go peak RSS (all cbm processes)	16.7 GB	4.0–5.2 GB
Go result arenas used / capacity	14,804 / 28,817 MB	856 / 856 MB
Go wall	55–59 s	52–60 s
Go, CBM_MEM_SPILL=1	—	1.7–2.3 GB peak; 21,875 results parked (950 MB on disk); graph identical (39 infra routes, 158 INFRA_MAPS, 21,875 surface rows)
Go, CBM_MEM_BUDGET_MB=1200	aborts	latched at 1,204 MB, first sweep → 768 MB, completes at 1.99 GB peak
Kernel, default budget (24,576 MB)	aborts (mem.budget.exceeded)	completes, no spill
kernel extraction	35.2 GB RSS	16.5 GB RSS / 5.6 GB footprint / 15.3 GB committed; results 8.9 GB = capacity
kernel resolve, tracked	24.6 GB	19.1 GB
kernel peak RSS	27.3 GB	20.1 GB
Kernel, CBM_MEM_BUDGET_MB=16000	aborts	completes through spill: latched at 17.2 GB charged, 89,731 results parked (9.3 GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6 GB (the semantic-pass plateau, 13.3 GB RSS, is graph-derived and outside what spill can move); nodes identical, edges 15,773,598
Kernel, CBM_MEM_BUDGET_MB=15000, before the semantic batching	aborts	completes; every phase mark under 15,000 MB by every metric (max RSS 13.9 GB, footprint 13.7 GB, commit 11.5 GB), but the process RSS high-water mark reached 16.9 GB inside the semantic pass (class peak 4.9 GB against 0.8 GB live)
Kernel, CBM_MEM_BUDGET_MB=15000, with the anticipatory latch	aborts	completes; spill mode enters at 14,062 MB (near_budget); the charged footprint never exceeds 11.8 GB at any mark; the process RSS high-water mark is 15.1 GB in extraction and 15.3 GB during the final write-out (1–2%: the in-flight window and the dump transient); nodes identical, zero foreign blocks
Kernel, CBM_MEM_BUDGET_MB=15000, with the semantic batching only	aborts	completes; the charged footprint stays under 15,000 MB through the whole semantic pass (14,018 → 14,381 MB across its sub-phases, 6 batches of 125,644 functions); every phase mark under budget by every metric (max footprint 12.6 GB, max commit 11.5 GB); the process RSS high-water mark, 15.65 GB, is set in extraction at the moment the gate latches (15,002 MB charged), before the first sweep lands — a 4% overshoot of in-flight work; nodes identical, zero foreign blocks
kernel graph	8,529,729 nodes / 15,772,562 edges	8,529,729 nodes / 15,772,669 edges

What the budget holds today. The extraction gate acts on the charged metric and the sweeps bring the run to its floor; the semantic pass sizes its token phases to the headroom. What no gate can move is the floor itself — the graph, the registries, and the semantic pass's retained working set (the per-function records with their quantized codes, the corpus entries, the tf-idf arrays: ~5 GB on the kernel) — so a budget below floor-plus-retained is met at the phase marks and overshot inside the semantic pass by the difference. On the kernel at 15 GB that no longer happens: the semantic pass's retained set (2.5 GB of per-function records, ~1 GB of corpus, ~1 GB of tf-idf arrays) plus the 10.6 GB floor fits once its transient is batched and its pages are returned per batch. The latch moment itself is anticipated: spill mode enters at 15/16 of the budget, because the gate observes the crossing per file pull and the 18 workers' in-flight files would otherwise carry the charge ~4% past it before the first sweep lands; what remains is 1–2% of in-flight work and the final write-out.

Nodes are identical to the release bench; the edge delta is inside the ±400 run-to-run band the kernel shows between two runs of identical code. On Go, two identical no-spill runs differ by 21 gRPC Route nodes / 40 HANDLES edges (a Route node is minted for whichever of several .proto copies a worker reaches first) — a pre-existing nondeterminism with its own item; the spill runs sit inside that band.

Bench against the shipped v0.10.8 (same machine, same driver)

corpus	wall s (v0.10.8 → final)	peak RSS GB	CPU s	nodes	edges	CALLS
perl	5.7 → 5.3	0.09 → 0.05	5.9 → 4.7	2,351 → 2,348	5,254 → 5,124	1,981 → 1,978
php	6.3 → 5.8	0.66 → 0.20	12.7 → 11.6	13,590 → 16,561 (+21.9%)	56,365 → 57,182	8,468 → 10,484 (+23.8%)
rust	6.8 → 6.3	1.15 → 0.67	19.2 → 15.6	18,404 → 18,405	97,423 → 97,479	26,402 → 26,033
c	5.9 → 5.7	0.16 → 0.19*	7.8 → 8.6	38,623 =	135,644 =	36,849 =
kotlin	8.9 → 9.4	1.55 → 0.96	38.2 → 40.8	25,083 → 33,494 (+33.5%)	263,711 → 203,562 (−22.8%)	29,901 → 33,518 (+12.1%)
django	10.9 → 10.2	3.13 → 1.12	40.8 → 35.7	55,371 → 55,377	343,613 → 293,715 (−14.5%)	86,086 → 61,672 (−28.4%)
go	52.6 → 42.9 (−18%)	17.17 → 3.93 (−77%)	176.9 → 170.0	292,994 → 334,488 (+14.2%)	2,002,252 → 1,795,311 (−10.3%)	314,660 → 321,441 (+2.2%)
java	79.8 → 73.8 (−8%)	27.64 → 11.00 (−60%)	644.6 → 543.1	693,250 → 693,129	5,633,006 → 5,523,596 (−1.9%)	1,552,246 → 1,542,428 (−0.6%)
csharp	463.3 → 128.1 (−72%)	27.03 → 11.15 (−59%)	1,596.9 → 1,208.6	1,224,682 → 1,224,981	5,848,553 → 5,794,310 (−0.9%)	1,036,932 → 1,030,918 (−0.6%)
typescript	33.4 → 33.6	12.00 → 2.49 (−79%)	210.3 → 188.5	291,791 → 291,605	865,796 → 835,285 (−3.5%)	83,558 → 83,688
linux (kernel)	294.8 → 233.3 (−21%)	27.51 → 25.08 (−9%)	2,442.4 → 1,403.0 (−43%)	8,529,900 → 8,529,729	15,985,598 → 15,772,630 (−1.3%)	1,971,880 → 1,970,114

* the chained run read 0.78 GB for C once; two standalone reruns read 0.19 and 0.24 GB (a sequencing artifact of the kill-then-start between corpora, not the binary).

No corpus is slower beyond noise. C# −72% wall is the walk budget, the locale-free formatter and the use-after-free fix that let the run complete at all; the kernel −21% wall / −43% CPU is the batched accounting and the SQLite heap. Memory is −55…−80% on the four large LSP corpora; the kernel at 36 GB never spills and its floor is the graph. Graph deltas: Every graph delta against v0.10.8 is main's work between the release and this branch's base (339b3f4), not this PR: a build of the base commit run through the same driver reproduces the final graphs exactly on django (55,377 / 293,715 / 61,672) and kotlin (33,494 / 203,562 / 33,518), within the pre-existing gRPC Route jitter on go (30 nodes, 27 edges, CALLS identical), and on C# with the same node count; this PR's own C# difference against the base is 5 CALLS (the five truncated JIT stress files) plus one ijwhost swap and 563 edges out of 5.79 M, and the base takes 448 s / 29.7 GB for it.

Budget checks, final binary. C# at the default budget: identical to the 36 GB run (128 s, 11.2 GB). Kernel at 15,000 MB: spill latches at 14,062 MB (near_budget), 89,731 results parked (9.3 GB on disk), charged footprint at every mark ≤ 7.1 GB, mimalloc commit ≤ 11.5 GB, RSS at marks ≤ 15.5 GB (the final write-out), process RSS high-water 18.8 GB inside extraction — and, measured by the budget metric itself (peak_charged_mb, new on every mem.phase line), a charged high-water of 16,771 MB: the latch fires at 14,061 MB and the in-flight working set of the 18 workers carries the charge 12% past the budget before the sweeps land (the rest of the RSS figure is pages already purged to the OS and not yet reclaimed). At the default budget (24,576 MB) the charged high-water is 22,272 MB, under budget, no latch; RSS high-water 29.5 GB, 7 GB of it purged-not-reclaimed pages. Go at 1,200 MB: completes at 1.71 GB summed peak.

Verification

• make -f Makefile.cbm lint-ci green on the final tree (cppcheck, clang-format, NOLINT whitelist, memory-core linter: 3862 raw sites across 85 files, none grew).
• build/c/test-runner extraction parallel pipeline: 699 passed, 0 failed (the deterministic Kotlin/Rust failures of an earlier build were a mis-nested LSP dispatch, fixed). c_lsp lsp_resolution_probe: 850 passed. Revert check: clsp_method_return_refinement_is_copy_on_write fails on the cast (base signature pointer changed), passes on the accessor.
• ASan/UBSan server build (libc-backed, -O1 -g) over the whole dotnet/runtime corpus: worker exit 0, no report (before the fix: heap-use-after-free in c_adl_resolve on the first resolve).
• scripts/test.sh (the full macOS leg) on the final tree: 8049 passed, 0 failed, 10 skipped (141 suites, 18 jobs); every contract test and the security-strings allow-list test passed. Linux (Colima) and Windows (VM) legs not run for this increment by decision (macOS-only result set); the Windows VM is currently down (utmctl start fails) — recorded, not bypassed.

Diagnostics that made this measurable stay in, behind CBM_MEM_PHASES=1: the extract.arenas and extract.census lines, the CBM_MEM_RELEASE=1 release-to-OS probe, and peak_charged_mb on every mem.phase line — the high-water mark of the budget metric itself (cbm_mem_peak_charged), next to the RSS peak that counts pages already purged but not yet reclaimed.

Supersedes #2201.