perf(graph-db): stream the sealed store build one column at a time

[ScriptedAlchemy]

Refs #2123. The sealed store build no longer holds every pushed value until finish. It spools values to disk and writes the container one column at a time, so the build's peak is the row topology plus one column.

This does not close #2123. At 6 GB the daemon now builds, verifies, and publishes the graph of this repository, and the graph reaches ready. The daemon is then OOM-killed in the serving decode and activation that follow the publish. Those phases are measured below.

Where grafeo lives

• grafeo-* 0.5.42 come from a git pin on ScriptedAlchemy/grafeo, our fork of GrafeoDB/grafeo; this account has admin on it.
• The pin sits on the branch stacked for the direct compact build (feat(compact): direct compact container build from a row stream grafeo#4, perf/compact-direct-build).
• .cargo/config.toml replaces that git source with pnpm's vendored copy, per build: manage npm packages and Cargo sources with pnpm #2091.

What changed in grafeo (ScriptedAlchemy/grafeo#5, merged as 2027d1093c)

• Push: IncrementalCompactStoreBuilder no longer keeps Vec<Value> columns. Each column appends only its present values to a byte spool, as bincode (rows skipped, value) records.
  • spooling_to(file) moves each full 32 KiB column buffer into a caller-owned file.
  • Absent rows decode as Value::Null, exactly what the null-padded column held.
• Write: write_section decodes, encodes, writes, and drops one column at a time. A relationship table's CSR is built only while that table is written.
• One encoder: the CompactStore section encoder is now one SectionStream that both CompactStoreSection and the builder drive. A streamed section is the finished store's section by construction, and a test compares the bytes.
• Container write: GrafeoDB::write_compact_container takes the builder and streams it; the CompactStore is never built on this path.

What changed here

• Cargo pin bumped to 2027d1093c.
• SealedCompactRows spools to tempfile::tempfile_in(staging), an anonymous file the OS removes even after SIGKILL. It hands the builder straight to write_compact_container. The separate finish step and its Hotpath span are gone.
• direct_build_holds_one_column_not_every_value (graph-db lib, per-thread counting allocator):
  • The direct sealed build of a 30,000-entity, 45,000-relation generation must stay within 30,000,000 bytes above the manifest, and only the container may remain in staging.
  • This branch: 21,528,303 bytes.
  • origin/master: 89,678,855 bytes, so it fails there.

Builder peak, daemon (clone of this repository, perf+hotpath CLI, one capped scope)

Anon comes from the scope's memory.stat. Phase windows come from Hotpath span timings. Both columns are on the same base (#2277), under a 12 GB cap:

phase	master anon peak	this PR anon peak
sealed store: push rows	4.78 GB	3.49 GB
sealed store: finish (all columns)	5.50 GB (Hotpath resident 4.92)	n/a (removed)
sealed store: write container	4.60 GB	4.46 GB (Hotpath resident 3.72)

Before #2277 halved the code-edge rows, the same comparison was 7.87 GB anon in master's finish (Hotpath resident 7.81 GB) against 4.24 GB in this PR's container write.

Per-phase RSS+swap, this PR on #2277, 6 GB cap

phase	window	rss+swap peak	anon peak	Hotpath resident
extract + code-index seal	6–36 s	6.64 GB	6.38 GB	4.52 GB
graph rows: resolve	262–271 s	5.64 GB	4.85 GB	2.71 GB
graph rows: emit	271–293 s	4.43 GB	3.65 GB	3.21 GB
graph rows: merge	294–302 s	3.55 GB	2.72 GB	3.21 GB
sealed store: push rows	303–316 s	3.54 GB	2.71 GB	2.80 GB
sealed store: write container	316–329 s	4.50 GB	3.67 GB	2.80 GB
sealed store: verify (reopen + row proof)	329–345 s	5.48 GB	4.06 GB	4.45 GB
serving decode	348–370 s	7.31 GB	6.22 GB	6.41 GB

• The graph reports ready at 411 s while the project is still warming. The scope is OOM-killed at about 426 s.
• Under a 12 GB cap the same binary reaches fresh with the graph ready: 319 s, peak 10.90 GB, Hotpath resident peak 9.33 GB (in activation).
  • Master on the same base: 221 s, peak 10.99 GB.
  • The 100 s gap is text-artifact finalization (query.artifact.finalization.canonical_layout, +95 s), which doesn't touch graph-db.

The next wall is not the builder. After the graph publish, the serving decode is admitted with a 2.8 GB estimate but grows anon from about 3.5 to 6.2 GB. Activation then maps the sealed container, 1.2–1.3 GB of file pages, and warms the catalog.

Identical store bytes and results

Same method as #2271:

1. Index with master.
2. Delete that profile's tracedecay.grafeo, its WAL, and tracedecay.sealed/.
3. Restart with this PR's binary. It rebuilds from the sealed segments and must bind the digest master journaled.
4. Compare the container past its timestamped 12,288-byte headers.

corpus	master sections sha256	this PR, same profile	recovered digest	rows
j768	2aeddd58b843331b7439…	2aeddd58b843331b7439…	sha256:12ca34bd5ee1f229…	99,075 / 98,305
rsbuild	005411bf59d269645e51…	005411bf59d269645e51…	sha256:0ebf2ee223be14b6…	15,545 / 19,104

The #2185 query set (search/context/callers/callees on 20 symbols), warm:

corpus	master vs this PR, fresh profiles	master vs this PR's same-profile rebuild
j768	80/80 identical	80/80 identical
rsbuild	80/80 identical	80/80 identical

Seal-time allocator spike (reported, not bounded)

• What the scope shows: at about 18 s anon rises from 4.84 to 6.39 GB within one second, while file pages stay at 0.07 GB. It is heap, not page cache.
• What mimalloc shows: stats captured by SIGTERM right after the spike (MIMALLOC_SHOW_STATS=1, daemon stopped at 15.8 s with anon at 6.7–6.96 GB):
  • peak commit 7.6 GiB against a peak RSS of 6.5 GiB;
  • 3.7 GiB purged over the run;
  • 326 threads created, 75 still alive;
  • in-use pages peaked at 65.2K, of which 55.3K were abandoned, meaning owned by threads that had already exited.
• Reading: the spike is retained pages from exited threads plus commit that hasn't been purged yet. It is not one allocation site: the code-index pools are persistent, so the churn comes from other short-lived threads.
• Why not bounded: without allocation-site attribution no single pattern explains it, so it isn't bounded here.

Tests and suites

• SIGKILL durability (daemon_suite::sealed_generation_crash_test, against the built CLI): sigkill_after_the_seal_restarts_on_the_identical_sealed_generation and sigkill_during_the_seal_keeps_the_prior_generation_published, 2 passed.
• tracedecay-graph-db: lib 144 (1 ignored), graph_db_suite 153 (2 ignored).
• tracedecay-code-index: lib 258, code_index_suite 171, resident_accounting 2.
• tracedecay-query: lib 263 (1 ignored), search_quality_suite 71, canonical_execution_equivalence 5, retrieval_contract_spine 2.
• tracedecay-store-runtime: lib 121 (1 ignored).
• tracedecay-code-index-runtime: lib 516 when run alone. The full suite on a host at load 45–114 fails a varying set of deadline and git-identity tests, and the base commit fails them too. Filed as test(code-index-runtime): lib tests fail under host load #2281; each passes alone.
• grafeo: grafeo-core --features compact-store compact tests 509 passed; compact_container_write 5 passed.
• Lint: cargo clippy ... -D warnings on the touched crates and cargo fmt --all -- --check are clean.

[changeset-bot]

⚠️ No Changeset found

Latest commit: a374ac5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-27T01:50:11.743572Z	a374ac5	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a374ac5fc4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Remove the diagnostic eprintln from the test

Every successful graph-db unit-test run executes this ad hoc diagnostic, even though the assertion immediately below already reports the measured peak when the budget fails. Remove it rather than merging investigative stderr scaffolding, as the repository requires Hotpath/tracing for maintained observability and explicitly prohibits merged eprintln! scaffolds.

AGENTS.md reference: AGENTS.md:L151-L160

Useful? React with 👍 / 👎.