daemon: initial index of this repository is OOM-killed under a 6 GB memory cap

[ScriptedAlchemy]

Found while measuring #2018.

Repro: perf+hotpath build of dbf7f13, isolated HOME, XDG_RUNTIME_DIR=/run/user/1000 systemd-run --user --scope -p MemoryMax=6G -p MemorySwapMax=1G -- tracedecay daemon run, then tracedecay init <clone of tracedecay @dbf7f13a25> (6,521 files).

Observed (twice; second run with TRACEDECAY_INDEX_WORKERS=8):

• daemon log: daemon_resident_memory_over_budget observed_bytes=5994946560 limit_bytes=6442450944 (first run, ~35 s in), observed_bytes=5607731200 (second run, text artifact bulk_commit phase);
• journal: run-u42567.scope: Failed with result 'oom-kill' / run-u42571.scope: Failed with result 'oom-kill' a few seconds later.

The daemon sees the cgroup limit and reports the high watermark, but the growth it admits before that point still exceeds the cap, so a first index of a 6.5k-file repo cannot complete inside the 6 GB scope agents are required to use. A 3.3k-file corpus (rsbuild) indexes fine under the same cap (RSS ~1.1 GB once the clone index was ready). Possibly related to #2050 (long-run growth), but this is the initial build.

[ScriptedAlchemy]

Measurement on master at 92f4b97 (with #2209): built CLI production,hotpath (debug, mimalloc), isolated daemon under MemoryMax=6G MemorySwapMax=1G, tracedecay init on a full clone of this repo (6,537 tracked files, 5,683 captured, 93,913,662 source bytes).

The daemon is OOM-killed about 30 s in, before graph publication and before any text serving (anon-rss:6258728kB file-rss:193748kB, CONSTRAINT_MEMCG). Hotpath gauges sampled every 3 s, with RSS+swap:

t	RSS+swap	phase (Hotpath gauges)
12 s	1.47 GB	extraction, 36 of 48 requested workers (code_index_workers_memory_safe 36, code_index_workers_reservation_bytes 4,831,838,208)
17 s	5.29 GB	extraction; runtime_core.resident.used_bytes 4,925,751,870 over 5,683 per-file reservations
22 s	5.98 GB	extraction done (6,011 parses, 1,351,616 unresolved refs, 242,569 nodes, 236,903 edges)
25 s	6.62 GB	code_index.seal.resolve started (815,237 unresolved references, 36 workers), then OOM

So the peak comes from holding every file's extraction artifacts at once (about 55x the source bytes, dominated by the per-token clone streams, 48-byte ConservativeCloneTokenV1 for both the conservative and rename streams, and per-token subtoken strings), plus seal resolution on top. Bounding it needs the seal to stream each file's segment to disk as it is extracted, keep only what cross-file resolution reads (symbols, unresolved references, imports), and hold clone streams in a compact encoded form at extraction. Graph publication afterwards also needs the decoded generation (2.98 GB on this repo) released between batches. That is a seal-pipeline restructure, not a budget change; still open.

[ScriptedAlchemy]

State after #2237 (compact clone token streams, merged as 0bece17), measured on a clone of this repository at f1576d0: perf+hotpath debug CLI, isolated profile, one capped systemd-run scope, cgroup RSS+swap sampled every 2 s.

Seal step.

• An in-process counting allocator puts the published generation at 2.96 GB live (6,095 files). The generation's own retained_bytes() went from 3.15 GB to 1.22 GB; clone bodies went from 2.16 GB to 226 MB.
• The seal-stage transients measured over that generation are small: edge evidence with cross-file resolution 143 MB, validate_uncached 192 MB, encode_partitioned_sealed 401 MB, import evidence 37 MB. Batching resolution would not move the peak.
• Under 6 GB the daemon now gets through extraction (4.61 GB) and seal to queryable (4.75 GB at 27 s).

What still OOMs the 6 GB run (at 88 s, text-artifact phase, 7.49 GB RSS+swap):

1. Re-decode during the text-artifact build. The build sheds the decoded generation to get headroom. Meanwhile a full generation decode starts (code_index.generation.decode.active 1 from 44 s to 71 s, and code_index.seal.reference_candidate_lookup calls double, because restore re-resolves). Nothing admits that decode through the resident-memory authority (DaemonCodeIndexPublicationStoreV1::load_active_shared / decode_active_generation). Under 12 GB the decode feeds the graph publication, which then waits for the text build (decode at 74–109 s, publish_start at 137 s).
2. The code-graph manifest is built whole. build_published_code_graph_manifest_checked on this corpus holds 703,741 entities and 1,081,941 relations. It retains 2.15 GB and peaks at 2.70 GB above the 2.94 GB live generation, before the sealed-store build adds its own bytes (1.27 GB canonical). Under 12 GB the graph phase goes from 8.55 to 13.41 GB. graph-db already accepts rows one at a time (IncrementalCompactStoreBuilder::push_node / push_edge), so the manifest could stream per file batch from the sealed segments with only the resolved edges resident. The canonical entity order and the manifest digest are the constraints that change needs to preserve.
3. Idle retention after current. The 12 GB run reached current at 676 s and then sat at 10.4 GB process RSS for 50 minutes, over budget. The resident owners account for about 5 GB of that: sealed store 1.27 GB, catalog of 639,801 entities, decoded generation. Related to daemon: every touched project keeps its full decoded generation, graph engine and attribution resident until exit #2106.

Per-phase RSS+swap (Hotpath daemon.process.resident_bytes maximum in parentheses):

run	extraction	seal/resolve	queryable	text artifact	graph publication	end
master, 6 GB, 36 workers	5.93 GB	6.49 GB	–	–	–	OOM 16 s
#2237, 6 GB, 36 workers	4.61 GB	6.51 GB	4.75 GB at 27 s	2.68 → 7.49 GB	–	OOM 88 s (5.46 GB)
master, 12 GB, 4 workers	5.45 GB	6.38 GB	6.04 GB at 67 s	1.85 → 9.48 GB	9.15 → 13.82 GB	OOM 364 s (13.02 GB)
#2237, 12 GB, 48 workers	–	6.40 GB	5.14 GB at 36 s	2.77 → 8.80 GB	8.55 → 13.41 GB	fresh 676 s (12.88 GB)

[ScriptedAlchemy]

State after #2254 (re-decode admission, merged as 07e4a8a). Same setup as before: a clone of this repository, perf+hotpath CLI, systemd-run --user --scope -p MemoryMax=6G -p MemorySwapMax=1G, default workers, cgroup RSS+swap every 2 s. Hotpath daemon.process.resident_bytes peaked at 4.68 GB.

t	RSS+swap	phase
8 s	3.99 GB	extraction
16 s	6.56 GB	seal (survives on swap)
30 s	4.80 GB	queryable
38–254 s	2.56–3.18 GB	text-artifact build; the graph seat's decode is refused (2,692,353,615 bytes needed, 1,847,351,705 available below the 5.80 GB watermark)
261 s	3.42 GB	text artifact complete (954 pages); decode admitted
283 s	6.06 GB	decode done (+2.64 GB against its 2.69 GB charge)
295 s	OOM	code-graph publication

The remaining wall is the code-graph publication, and streaming the manifest out of the decoded generation cannot clear it.

• By the time the decode is admitted, it alone consumes the headroom: 3.42 GB base plus 2.64 GB decode is 6.06 GB against the 6.44 GB kill line.
• The one-piece manifest adds 2.15 GB retained on top, peaking at 2.70 GB above the generation. The sealed-store builder adds its own compact store, 1.27 GB canonical on this repository.
• Streaming rows out of the resident decode removes only the manifest; it still needs about 3.4 + 2.7 + 1.3 = 7.4 GB.

What fits is publishing the graph without the whole decode. Rows are built per file batch from the sealed segments, and each batch is released after its rows are written. The constraints that design has to meet:

1. Global order. The sealed store (push_manifest_rows) writes entities, then relations, in global identity order and resolves each relation endpoint to its entity's index. The manifest digest is defined over that same order. Per-file batches therefore need an external sort (sorted runs spilled per batch, then a k-way merge into IncrementalCompactStoreBuilder), with only the sorted entity identities resident for endpoint lookup (about 700k identities).
2. Cross-file edges. They are derived at restore, never persisted, so the batch pass has to keep resident only what resolution reads: symbol identities, unresolved references and imports (about 600 MB on this repository per the extraction breakdown), not chunks or clone streams.
3. A streaming row source in graph-db. It needs to compute the expected recovered digest and the manifest digest from the merged stream, so that the sealed container and digests stay byte-identical to the one-piece build, and to keep perf(code-index): make the seal the build's one durable point #2186's temp-name and one-syncfs contract.

At 6 GB that path would peak at about 3.4 GB base + 0.6 GB resolution inputs + sort buffers + 1.3 GB builder, roughly 5.5 GB.

Two further notes:

• The 16 s seal spike (6.56 GB) is the other near-miss. The seal-stage working sets measured in-process are small (resolution 143 MB, validation 192 MB, segment encode 401 MB), so it is allocator growth across the 36 index workers. It needs its own measurement before it can be called fixed.
• Idle retention over budget (10.4 GB idle at 12 GB) belongs to daemon: every touched project keeps its full decoded generation, graph engine and attribution resident until exit #2106.

Regression check: the #2118 binary on this clone, at 12 GB with 4 workers, saturates the cap exactly as current master does (memory.peak = 12 GiB, swap 1 GiB full, 1,259 memory.max events). It survives and settles at 11.37 GB. On the 768-file corpus #2118 peaks at 4.09 GB and f1576d0 at 3.97 GB. There is no regressing merge; the master OOM at 12 GB is the same cap, with the survival coming down to chance.

[ScriptedAlchemy]

#2271 (merged as beefa81) builds code-graph rows from the sealed segments one file window at a time. There is no whole-generation decode and no one-piece manifest any more. The published graph, recovered digest, and sealed container sections are byte-identical to master on the 768-file corpus and on rsbuild.

The 6 GB acceptance is still not met. On a clone of this repository under MemoryMax=6G, the daemon now gets past the text build: the graph build is refused and parked until the text reservation is released. It is then OOM-killed at about 289 s in the sealed compact-store build.

The new wall

• In-process on this repository's generation (858,397 entities / 1,415,305 relations), the row build itself peaks under 1 GB live (VmHWM 1.44 GB).
• grafeo's IncrementalCompactStoreBuilder reaches VmHWM 5.39 GB, 3.50 GB of it before finish. It keeps every pushed Value until finish has encoded all the columns.
• The old path paid the same cost, on top of the decoded generation and the manifest.
• Candidate next slices:
  • Encode and drop each column as finish goes, in the grafeo fork.
  • Intern the repeated namespace, projection, kind, and endpoint strings that the builder holds per row.

Seal spike (reported, not bounded)

• At about 16 s the seal rises from 5.27 to 6.75 GB within one second, holds for about 3 s, and falls back.
• At the same width the live transients are small: encode 0.35 GB, resolution about 0.1 GB, validation 0.19 GB.
• So the spike is allocator growth across the worker heaps, not live data. Bounding it needs a mimalloc heap measurement first.

The PR body has the per-phase RSS+swap tables and the Hotpath peaks.

[ScriptedAlchemy]

The grafeo builder wall is gone: ScriptedAlchemy/grafeo#5 (merged 2027d1093c) plus #2282 (merged 8090d37d66). The builder now spools pushed values to an anonymous file and writes the container one column at a time. The store bytes and recovered digest match the unbounded build on j768 and rsbuild, and retrieval is 80/80 identical.

• Builder memory, on the perf(graph-db): store each code edge as one relation row #2277 base:
  • The sealed store phases peak at 3.49 GB anon (push) and 4.46 GB (write), where master reaches 5.50 GB in finish.
  • Before perf(graph-db): store each code edge as one relation row #2277, master's finish hit 7.87 GB anon.
  • In-process, the 30k/45k fixture build fell from 89,678,855 to 21,528,303 bytes.
• 6 GB acceptance: still not met. At MemoryMax=6G every graph-publish phase now stays at or below 4.06 GB anon, and the graph reaches ready at about 411 s. The scope is then OOM-killed at about 426 s.
• The next wall is post-publish serving:
  • The serving decode is admitted with a 2.8 GB estimate but grows anon from about 3.5 to 6.2 GB.
  • Activation then maps the 1.2–1.3 GB sealed container and warms the catalog.
  • At 12 GB, activation peaks at 10.2 GB anon.
• Seal spike: anon rises 4.84 → 6.39 GB in one second, while file pages stay flat. mimalloc at the spike shows peak commit 7.6 GiB against a peak RSS of 6.5 GiB, and 55.3K of 65.2K in-use pages abandoned by exited threads (326 threads created, 75 live). That is thread-heap retention, not one allocation site, so it isn't bounded here.

The #2282 body has the per-phase tables.

[ScriptedAlchemy]

#2295 (merged 450a55fb4d) fixes the thread churn and the allocator release. The 6 GB acceptance is still not met.

• Threads: an index of this repository created 1,075 threads. There were two spawners: gix status (one worker per core on every call, 547 threads) and sealed-graph verification (one scoped thread per proof chunk, 349 threads). It now creates 189.
• Allocator release: the release behind the pressure reclaimer and the publication boundaries called glibc malloc_trim, which is a no-op for the shipped mimalloc heap. It now collects every persistent pool worker's heap: 438 MB returned at publish.
• Correction to my last report: in mimalloc v3, full pages are abandoned by design. The "55.3K abandoned pages" were mostly live threads' pages. With churn fixed, the spike's commit (7.7 → 7.8 GiB) and RSS (6.5 → 6.6 GiB) don't move, so the spike is live seal data.
• Resident at publish: a jemalloc heap profile shows 1.83 GB live against 3.65 GB anon. Nothing from extract, seal, the row spill, or verify is retained.
• The wall is the serving decode:
  • It retains 2.8 GB, which matches its charge.
  • It peaks 4.5 GB above its start: the 1.7 GB transient comes from evidence expansion, authority restore, and corpus validation.
  • It then grows to 3.39 GB with test attribution.
  • At 6 GB, master completes 1 of 2 runs and is OOM-killed in the other. With perf(daemon): reuse worker threads and release mimalloc heaps #2295, 4 of 4 runs park typed without an OOM kill and never admit the decode: available 1.9–2.6 GB, with 0.77 GB of clean mapped container pages counted as resident.
• Next slice options:
  • Serve the LSP census and graph reads from the sealed manifest and mapped graph store, so only search and diagnostics need decoded chunks and symbols.
  • Or bound the decode's post-window transient and exclude clean file-backed pages from the admission measure.

[ScriptedAlchemy]

#2332 (merged as 60eec1f) makes the first index at MemoryMax=6G reach ready (fresh, graph serving) without the serving decode and without an OOM kill. The acceptance is still not fully met: 1 of 4 final-build runs parked 435 s at the sealed graph build's admission before a retry fit.

• Admission reads RssAnon + RssShmem instead of VmRSS. At the post-text-build retry this took the observation from 3.32–3.47 GB (perf(daemon): reuse worker threads and release mimalloc heaps #2295) to 2.93 GB, so 2.86 GB is available against the 2.80 GB graph-build charge.
• Serving: a publication seats its graph head on the text owner (verified-head recovery), and an empty decoded seat is terminal. Project-open owners, the LSP census and feedback identity no longer demand the whole decode.
• Runs: at 6 GB, ready in 322, 344, 330 and 724 s. The 724 s run is the one with the park. There was no oom_kill, and the whole-run peak was 6.56–6.83 GB, during extract and seal.
• 12 GB: time to ready is within noise (master 243–310 s, this PR 280–288 s). Peak drops from 10.95–11.38 GB to 7.13–7.17 GB.
• Why perf(daemon): reuse worker threads and release mimalloc heaps #2295 parked where master completed: at the retry, perf(daemon): reuse worker threads and release mimalloc heaps #2295's anon was 3.00–3.06 GB against pre-perf(daemon): reuse worker threads and release mimalloc heaps #2295's 2.55–2.60 GB, so the VmRSS view left 2.4–2.6 GB against the 2.82 GB charge. Pre-perf(daemon): reuse worker threads and release mimalloc heaps #2295 admitted the build, then its uncharged serving decode went from 2.55 to 6.2 GB anon and OOM-killed the scope at 553 s.

Still open:

1. The sealed graph build is charged the decode's retained 2.80 GB. It measures 2.1–2.4 GB of growth, so the headroom margin is 30–120 MB short in the unlucky run. It needs an honest pre-run graph-build charge, or the graph build run before the text build while the base is about 2.4 GB.
2. The decode transient is not bounded. A re-decode is now charged the measured peak (5.19 GB peak growth when doctor demands it at 12 GB).
3. Lazy per-partition decode is not implemented for symbol_search, qualified_name, signature_search, source_metadata, facets, timeline, diagnostics, doctor and branch publication.

The #2332 body has the tables.