code-index: a one-line edit re-encodes the whole lexical text artifact #2906

Activity

1. ScriptedAlchemy commented on Oct 2, 2026

   

   ScriptedAlchemy

   on Oct 2, 2026

   OwnerAuthor

   More actions

   Measurement and design notes from the incremental-text-2906 lane. The first slice landed in #2943, merge dcbf5e8: the text build now reads each sealed segment once. On a cold build that took segment decodes from 26,632 to 5,881, identity-restored bytes from 5.1 GB to 1.0 GB, and the text source scan from 99.5 s to 73.2 s. The edit-path findings below are still open.

   What decodes the whole parent on a one-line edit: the incremental build itself, not text projection, catalog or serving. reconcile_now_with_capture → reserve_worker_memory → load_active_shared, then CodeIndexProductionOwnerV1::build_and_publish → materialize_increment. The child generation is an in-memory value built by Arc-sharing every parent file:

   • staged_generation / aggregate_from_parent build whole-corpus chunk and symbol rosters.
   • edge_evidence_over_parent concatenates every file's edges and scans the parent's edges and unresolved calls.
   • The Rust and TypeScript module indexes used by resolution are built over all files.
   • The sealed evidence (PersistedCrossFileEdgesV1, lineage) names endpoints by position in the whole-corpus symbol roster. Even decoding the evidence needs every file's symbols.

   Size of that decode on a full clone at 730026c: 5,881 files, 2.53 GB retained, 3.5–3.9 GB peak growth.

   • Chunks: 1,038 MB across 439k chunks. About 408 MB is per-chunk identity and revision strings, 287 MB subtokens (7.6M), 205 MB exact terms, 138 MB text.
   • File pages: 854 MB, including clone bodies at 215 MB and unresolved references at 202 MB.
   • Symbols: 164 MB.
   • Generation-level edges, unresolved calls and lineage: about 470 MB.

   No single field dominates, so field trimming cannot make the decode proportional.

   Why the 6 GiB edit stalls:

   • The serving catalog and engine are not the cause. Warming the catalog before the edit, so the refusal could yield it, still left RSS at 5.92 GB with only decoded_generation resident, and the edit was refused.
   • In a second run the retry was refused permanently while memory was nominal: "needs 3,918,462,976 resident bytes; 3,866,012,057 are available below the 5,798,205,849-byte admission watermark". With nothing left to release, that park never clears. Master's 852 s run passed only because its earlier decode measured 3.53 GB.

   What a proportional build needs (each item is a format or authority change):

   1. Sealed evidence keyed per file and by occurrence id instead of roster position. Cross-file edges and unresolved calls grouped by source file. Generation format bump with typed rebuild.
   2. Persisted, per-file-updatable resolution indexes: name to defining occurrences, name to referencing occurrences, and module/import rows.
   3. A sparse incremental build over the sealed parent that carries unchanged file segments by descriptor and decodes only changed files plus their resolution closure. Statistics and coverage become reversible aggregates. The materialize_increment arc-share path is deleted. Large changes take the cold build.
   4. Then the segmented lexical artifact (slice 2) and the incremental catalog (slice 3) can consume the per-file deltas.
2. ScriptedAlchemy

   added a commit that references this issue on Oct 2, 2026

   perf(code-index)!: seal cross-file evidence per file (#2951)

   ...

   455d41d
3. ScriptedAlchemy commented on Oct 2, 2026

   

   ScriptedAlchemy

   on Oct 2, 2026

   OwnerAuthor

   More actions

   State of the sparse-build lane after step 1. Step 1 has landed. Steps 2 and 3 are not started, and the reasons are below.

   Step 1 landed: #2951, merge 455d41d

   Each file now has its own content-addressed sealed evidence segment: its name-resolved edges, its unbound call sites, and its symbols' lineage. Rows name the file's own symbols and references by their content-fixed positions, and edge targets by target path plus symbol identity. The whole-corpus roster is gone from the format, so one file's evidence decodes against that file and the snapshot alone, and a carried file keeps its segment bytes. The format moved to revision 19 with a typed rebuild; a revision-18 profile logged sealed generation format revision 18 predates this build; the generation will be rebuilt from source and rebuilt cold once. A differential on the full clone compared a cold build on master with a cold build on the branch: 41 callers/callees/file_dependents/find_exact answers, 0 different.

   Workload: full clone at dcbf5e8, --profile perf --features production,hotpath, 6 GiB scope, restart, then a one-line edit. Two runs on each side:

   	master	step 1
   edit → current	195 s, 204 s	264 s; 820 s (parked ~11 min on the 6 GiB refusal, then built)
   generation evidence pages written	454	1 (+1 file evidence segment)
   parent evidence decode (stream + expand)	3.35 s	1.31 s
   decode peak growth	3.29–3.37 GB	3.19–3.53 GB
   segment_decode calls / text pages written	11,764 / 7,735	11,764 / 7,735
   build.and_publish (of which sealed_encode.evidence)	37.3 s (33.7 s)	36.7 s (32.1 s)

   The baseline's refusal didn't reproduce in either master run here. It did reproduce once on the branch, because the build still decodes the whole parent.

   Why steps 2 and 3 stop here

   Step 2 cannot land on its own. The in-memory build holds every parent file, so nothing in production would read persisted name→definition, name→reference, or module/import rows before the sparse build exists. A standalone step 2 would be a write-only authority, which AGENTS.md rules out. It has to land with step 3, which is its only reader.

   Step 3 is a cross-crate cutover. Each item below decodes or rebuilds the whole parent today. The sparse build has to replace all of them in one slice before materialize_increment can be deleted:

   1. Admission decodes the parent first. reconcile.rs::reserve_worker_memory calls load_active_shared, which is a full decode (9–13 s, 3.2–3.5 GB peak), before the build starts. git_tree_capture.rs (650, 748), ignored_dependencies.rs:206, and registry/mount.rs:2328 also load the decoded generation. Publication would have to hand them a sealed-only child, or each one would decode the child right after it publishes.

   2. Publication takes an in-memory generation. build_and_publish → publish_atomically(Arc<CodeIndexPublishedGenerationV1>) → encode_partitioned_sealed_with_parent. A sealed increment needs its own publish that writes the child manifest from parent descriptors plus the changed segments.

   3. Projection request and receipt rows are still keyed by whole-corpus chunk-roster positions (projection_rows.rs, PersistedProjectionRequestRefV1::new(request, &chunk_roster)). That needs another per-file or by-id format change.

   4. Code graph pages. for_each_sealed_code_graph_page rebuilds and digests all 5,882 pages on every edit, plus a full emit_persisted_code_graph_page per page for page_build_footprint. This is ~32 s of the 37 s build. A sparse plan has to carry unaffected page descriptors and rewrite only:

      • the edited files' pages;
      • the pages of files whose sites moved;
      • the pages of files with edges into an edited file. Those edges' to_occurrence and target_files change, because the occurrence ids embed the content digest.

      That last group needs a target→source reverse index, part of step 2.

   5. Resolution for moved sites needs the whole corpus.

      • ChangedSitesV1::new scans every file's references and runs forward_through_imports over every file's imports.
      • resolve_selected_cross_file_references builds by_simple_name, RustFileIndexV1, TypeScriptModuleIndexV1, and ModuleImportIndexV1 over all files.

      These are the step 2 indexes. The resolvers in helpers.rs, module_resolution.rs, and typescript_resolution.rs read files[candidate_index] directly, so they would need an index-backed file view.

   6. Statistics and coverage are recomputed from every file (CodeIndexGenerationStatisticsV1::from_generation_parts, coverage_summary). They need to become reversible per-file aggregates: parent, minus replaced files, plus new files.

   7. Lineage. Files edited in the parent generation carry non-trivial lineage rows. In the child they become all-unchanged, so their evidence segments have to be decoded and rewritten. That set is bounded by the previous edit, but it needs each file's symbol count. Today the count comes only from the decoded segment, not from the descriptor.

   8. The current time is dominated by the text rebuild. The text artifact still rewrites all 7,735 pages and the catalog rescans every page (scheduler_wake 138 s, catalog.build 77 s per edit). "Current/ready in a small fraction of cold time" cannot be met until the segmented lexical artifact and incremental catalog (items 4 of the design comment) land too, even with items 1–7 done.

   Lane stopped here; no daemon, worktree, or scratch data left behind. Filed #2952, a timing race in a_read_past_its_budget_answers_the_measured_remaining_rewarm that I hit while testing.
4. ScriptedAlchemy

   added a commit that references this issue on Oct 2, 2026

   perf(code-index)!: seal one-file edits over the sealed parent (#2985)

   ...

   88e5b13
5. ScriptedAlchemy commented on Oct 2, 2026

   

   ScriptedAlchemy

   on Oct 2, 2026

   OwnerAuthor

   More actions

   State after #2985 (merged as 88e5b13): steps 2 and 3 are landed. The incremental lexical artifact and the incremental catalog are not started, so this issue stays open.

   Landed in #2985. A one-file edit seals its successor over the sealed parent and never decodes it.

   • The new sealed resolution index holds name→definitions and name→references pages plus import-alias rows. Each page is bound by its digest. The sealed format is now revision 20; a revision-19 store takes the typed rebuild.
   • A body edit re-resolves only the names it changes. Files whose sealed edges land in the edited file are re-pointed.
   • Daemon admission, capture, reconcile, freshness, and mount read the sealed manifest.
   • The whole-parent in-memory incremental path is deleted.

   Measured on this repo at 455d41d (5885 files, 7740 pages, perf+hotpath CLI, MemoryMax=6G):

   metric	master	after #2985
   hub edit (tracedecay-domain/src/lib.rs) → current	816 s (about 679 s parked; admission refused twice)	212 s
   leaf edit → current	n/a	251 s
   code_index.reconcile.pass	38 s plus parking	6.75 s avg
   whole-generation decodes on the edit	2 (peak growth 3.51 GB)	0
   files decoded for resolution, two edits	n/a	181 (1929 before name precision)

   On 60 sampled queries (find_exact_symbol, callers, callees, file_dependents), the sparse-built generation answers identically to a cold build of the same tree.

   Where edit → current goes now. These costs are not part of step 2 or 3:

   1. Lexical text artifact, about 150–200 s per edit (next PR). Every edit rebuilds the whole artifact:

      • query.artifact.batch.scheduler_wake is about 220 s per edit;
      • code_index.restore.file_admit admits all 5885 files and restore.segment_decode makes 5885+ calls per edit;
      • all 7740 pages are recommitted.

      This stage also drives RSS to 5.5–5.98 GB (mem=over_budget) under the 6 GiB cap. So even with the code index fixed, a one-line edit still runs close to refusal.

      Per fix(memory): retain cgroup pressure after allocator trim #2851, fix(code-index): resume lexical text from the parent artifact #2437 was withdrawn because the parent-carry delta (artifact/delta.rs) sealed incomplete source receipts under the current revision. The plan for the next PR:

      • bump to revision 31;
      • carry unchanged blocks and postings by content digest, and re-encode only the changed pages;
      • make the source receipt cover every page, carried or re-encoded;
      • require the sealed digest to equal a cold build of the same generation, as an equality test plus the runtime journey.

   2. Catalog, about 45 s per code_graph.catalog.build call; 2–3 calls per edit. This is the incremental catalog (reversible aggregates).

   3. Smaller items seen while measuring:

      • code_index.graph.build_layered_rows takes about 12.9 s per edit, but emit and finish take under 1.6 s of that. The rest is open, plan, or admission.
      • In a cold seal, the per-file code-graph page loop (code_index.sealed_encode.graph_pages, newly measured) runs serially.
      • Each sparse edit decodes 185 MB of resolution-index pages (code_index.sparse.index_bytes_decoded). A path→dependents index would remove most of that read.

   The lexical artifact and catalog follow-ups are not started; they are the remaining work on this issue.
6. ScriptedAlchemy commented on Oct 3, 2026

   

   ScriptedAlchemy

   on Oct 3, 2026

   OwnerAuthor

   More actions

   State after #3004 (merged as 1386c99): PR B, the incremental lexical artifact, is landed. PR C (incremental catalog) and the smaller items are not started, so this issue stays open.

   Landed in #3004. An edit's text artifact is built from the nearest published ancestor's artifact:

   • only the changed files are decoded and re-encoded;
   • rows, postings, dictionary and clone rows are patched in place at the cold positions;
   • the cold build's own digest walk, layout rewrite and receipt seal it;
   • the sealed file is byte-identical to a cold build of the same generation.

   Format revision 31 makes every stored page value position-free. Revision 30 takes the typed rebuild. More than an eighth of the files changed, a different file roster, or no published ancestor builds cold.

   Measured on this repo at 88e5b13 (5,891 files, perf+hotpath CLI, one daemon under MemoryMax=6G, the same echo '// probe line' >> crates/tracedecay-domain/src/lib.rs && tracedecay sync workload):

   metric	before	after #3004
   one-line edit → current	146 s	48 s
   peak RSS during the edit	5.37–5.40 GB	3.34 GB
   text pages encoded	7,752	11
   code_index.restore.file_admit / segment_decode calls	5,891 / 5,945	1 / 55
   query.artifact.batch.scheduler_wake	137.8 s	25.0 s
   cold build	275 s	227 s (typed rebuild from revision 30)

   After the edit, I cold-built the same tree in a fresh profile with the same binary:

   • both profiles sealed the same artifact file (SHA-256 482f3834…, 461,713,408 bytes);
   • a 30-query sample (10 search, 10 phrase search, 10 grep) returned identical results, 367 rows.

   Where the 48 s goes now:

   1. Code-index reconcile: 6.5 s.
   2. Text carry and verification: 24 s.
      • carry 11.9 s: the n-gram pass, which still decodes every document set, is 7.0 s; term 2.2 s; clones 2.0 s;
      • digest walk 8.4 s;
      • layout rewrite 3.5 s;
      • clone census 2.2 s.
   3. Graph publication: about 14 s (code_index.graph.build_layered_rows). This is the "12.9 s where <1.6 s is real work" item.
   4. Background, after current: the catalog, 46 s over 2 code_graph.catalog.build calls. This is PR C.

   Remaining work on this issue:

   1. PR C, incremental catalog (reversible aggregates for degree counts and file dependencies). Its acceptance counter, catalog pages rebuilt, is still the full catalog per edit.
   2. The layered-graph publication (about 14 s per edit, mostly open, plan and admission).
   3. The file→dependents index for the 185 MB of resolution-index reads per edit, and serial cold graph-page sealing.
   4. Carry follow-ups:
      • skip the n-gram document-set decode for sets that do not intersect the replaced ranges (needs per-set bounds);
      • the digest walk is shared with cold builds;
      • a parent page subdivided under memory refusal is carried with its cut, so that page can differ from a cold build that did not subdivide.
7. devin-ai-integration commented on Oct 4, 2026

   

   devin-ai-integrationbot

   on Oct 4, 2026 – with Devin.ai Integration

   Contributor

   More actions

   Devin is picking up the remaining roster-change gap here: a checkout that adds or removes files still builds the lexical text artifact cold because changed_files_since returns None on any roster difference. Plan: align parent/successor rosters by path and let the parent-artifact carry delete removed files' pages and stage added files' pages at their cold positions, with a byte-identical-to-cold regression test.

   Written by Devin
8. ScriptedAlchemy

   closed this as completedin #3073on Oct 5, 2026
9. spa5k

   added a commit that references this issue on Oct 8, 2026

   fix(lexical): carry text artifact across added and removed files

   ...

   3e85bd7