Skip to content

perf(memory): share index connections and batch code-index writes - #262

Merged
kumanday merged 1 commit into
mainfrom
perf/memory-code-index
Nov 9, 2026
Merged

kumanday merged 1 commit into
mainfrom
perf/memory-code-index

Conversation

@kumanday

@kumanday kumanday commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR cuts the DuckDB and Git overhead in memory code indexing and in code-intelligence persistence. Assertions and persisted results are unchanged.

Index runs (index_code_repository_at_checked)

  • Two write connections per run. The first, before parsing, discards leftover staged rows, reads the previous snapshot, and records the running state. The second persists, compares, and promotes. Before, every helper opened its own connection and migrated it. No connection is held while files are parsed, so other processes can still open the index then. The old helper signatures stay as thin wrappers; the ones only tests call are #[cfg(test)].
  • Snapshot membership read once per phase. CurrentCodeSnapshotRows replaces current_code_snapshot_file_matches, which opened a read-only connection for every file. The rows are read once before parsing and once after staged rows are written. Rows with NULL key columns are left out, as the old SQL equality checks left them out.
  • Git. Blob sizes come from git ls-tree -l instead of one git cat-file -s per file. Contents stream through one long-lived git cat-file --batch process (GitBlobBatch) instead of one git cat-file blob per file. A malformed or missing response fails the read and discards the process.
  • Promotion. Stale marking for changed and edge-refresh paths runs as one UPDATE ... path IN (temp table) per table instead of five per path. Marking a row stale takes it out of freshness = 'current', so the counts equal the per-path counts even when a path repeats.

Document persistence (persist_code_intel_documents_with_freshness_on)

  • Multi-row upserts. CodeRowUpserts buffers rows per table and writes them with multi-row INSERT OR REPLACE statements of 256 rows. A statement can't upsert one key twice, so only the last row per primary key is written, which is what writing each row in turn leaves behind.
  • Stale marking for the whole batch. When stale_existing is set and every document has a distinct path, staleness is marked with one UPDATE ... FROM per table before any rows are written. Symbol and edge ids include the path, so this gives the same rows and counts as staling each document just before its rows. If paths repeat, as in the dedupe-repo fixture, the original per-document order is kept and rows are written one at a time.
  • Existing-row checks for staged batches. The "current row exists" checks for symbols and edges become one query per table against a temporary table of candidate ids. Staged rows never create current rows, so this matches the per-row checks when nothing is staled.

Migration (migrate_index)

  • Run once per fingerprint. A completed migration records a fingerprint in the new memory_migrations table. The fingerprint covers the migration SQL, the per-table Rust steps (through MEMORY_MIGRATION_RUST_STEPS_REVISION), the table list, and the crate version. Later write opens skip the migration while the fingerprint matches. A fresh or recreated store, a new release, or a changed migration migrates once.
  • Why skipping the NULL fixups is safe. Current writers never insert the NULLs those fixups repair: the issues columns are SET NOT NULL, and the code tables are always written with values.
  • New test. migration_runs_once_per_fingerprint covers the skip and the re-run.

Docs: docs/memory.md describes the migration fingerprint, and docs/code-intelligence.md describes the index-run write path.

Evidence

Local runs on macOS, three runs each, compared against main with #261 merged:

Test Before After
opensymphony_memory::code_graph_tests::target_branch_index_bootstraps_and_advances_immutable_snapshots 6.2 s 3.7–3.8 s
gateway::gateway_serves_code_graph_contract_endpoints 3.6 s 3.2–3.3 s
gateway::gateway_code_graph_atlas_caps_and_dedupes_large_repositories 6.9 s 4.2–4.4 s

The "before" runs had a load average of 5 and the "after" runs a load average of 23, so the gains shown are a lower bound.

A profile of the target-branch test before the migration fingerprint showed migrate_index at about 30% of samples. In the atlas test, INSERT OR REPLACE statements (each planned as a MERGE) were 83% of persistence time.

Checks:

  • cargo nextest run --workspace --no-fail-fast: 2,267 of 2,269 passed. The two failures are the known macOS-only opensymphony_cli::migration /var symlink tests.
  • Memory, gateway, and code-graph tests: 470 passed.
  • cargo fmt --all --check and cargo clippy-dev are clean.

CI timings will be added once this PR's rust-test run finishes.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-08T23:14:58.687241Z 6a15454 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Base automatically changed from perf/workspace-checkout-verification to main November 9, 2026 01:14
- Open the index for writing twice per index run, once before parsing
  and once to persist and promote, instead of once per helper call.
- Read the current snapshot once per phase instead of opening the index
  per file to compare membership.
- Take blob sizes from ls-tree -l and stream contents through one
  git cat-file --batch process instead of two Git processes per file.
- Write code rows with multi-row upserts that keep the last row per
  primary key, and look up existing current symbol and edge ids once per
  staged batch. Stale marking for distinct paths and for promoted paths
  runs as one statement per table.
- Record a migration fingerprint and skip the migration on later opens
  while it matches.
@kumanday
kumanday force-pushed the perf/memory-code-index branch from 6a15454 to dd551cc Compare November 9, 2026 01:15
@kumanday
kumanday merged commit eec821a into main Nov 9, 2026
7 checks passed
@kumanday
kumanday deleted the perf/memory-code-index branch November 9, 2026 01:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant