Repository navigation
perf(memory): share index connections and batch code-index writes - #262
Merged
Merged
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
- Open the index for writing twice per index run, once before parsing and once to persist and promote, instead of once per helper call. - Read the current snapshot once per phase instead of opening the index per file to compare membership. - Take blob sizes from ls-tree -l and stream contents through one git cat-file --batch process instead of two Git processes per file. - Write code rows with multi-row upserts that keep the last row per primary key, and look up existing current symbol and edge ids once per staged batch. Stale marking for distinct paths and for promoted paths runs as one statement per table. - Record a migration fingerprint and skip the migration on later opens while it matches.
kumanday
force-pushed
the
perf/memory-code-index
branch
from
November 9, 2026 01:15
6a15454 to
dd551cc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR cuts the DuckDB and Git overhead in memory code indexing and in code-intelligence persistence. Assertions and persisted results are unchanged.
Index runs (
index_code_repository_at_checked)runningstate. The second persists, compares, and promotes. Before, every helper opened its own connection and migrated it. No connection is held while files are parsed, so other processes can still open the index then. The old helper signatures stay as thin wrappers; the ones only tests call are#[cfg(test)].CurrentCodeSnapshotRowsreplacescurrent_code_snapshot_file_matches, which opened a read-only connection for every file. The rows are read once before parsing and once after staged rows are written. Rows with NULL key columns are left out, as the old SQL equality checks left them out.git ls-tree -linstead of onegit cat-file -sper file. Contents stream through one long-livedgit cat-file --batchprocess (GitBlobBatch) instead of onegit cat-file blobper file. A malformed or missing response fails the read and discards the process.UPDATE ... path IN (temp table)per table instead of five per path. Marking a row stale takes it out offreshness = 'current', so the counts equal the per-path counts even when a path repeats.Document persistence (
persist_code_intel_documents_with_freshness_on)CodeRowUpsertsbuffers rows per table and writes them with multi-rowINSERT OR REPLACEstatements of 256 rows. A statement can't upsert one key twice, so only the last row per primary key is written, which is what writing each row in turn leaves behind.stale_existingis set and every document has a distinct path, staleness is marked with oneUPDATE ... FROMper table before any rows are written. Symbol and edge ids include the path, so this gives the same rows and counts as staling each document just before its rows. If paths repeat, as in thededupe-repofixture, the original per-document order is kept and rows are written one at a time.Migration (
migrate_index)memory_migrationstable. The fingerprint covers the migration SQL, the per-table Rust steps (throughMEMORY_MIGRATION_RUST_STEPS_REVISION), the table list, and the crate version. Later write opens skip the migration while the fingerprint matches. A fresh or recreated store, a new release, or a changed migration migrates once.issuescolumns areSET NOT NULL, and the code tables are always written with values.migration_runs_once_per_fingerprintcovers the skip and the re-run.Docs:
docs/memory.mddescribes the migration fingerprint, anddocs/code-intelligence.mddescribes the index-run write path.Evidence
Local runs on macOS, three runs each, compared against main with #261 merged:
opensymphony_memory::code_graph_tests::target_branch_index_bootstraps_and_advances_immutable_snapshotsgateway::gateway_serves_code_graph_contract_endpointsgateway::gateway_code_graph_atlas_caps_and_dedupes_large_repositoriesThe "before" runs had a load average of 5 and the "after" runs a load average of 23, so the gains shown are a lower bound.
A profile of the target-branch test before the migration fingerprint showed
migrate_indexat about 30% of samples. In the atlas test,INSERT OR REPLACEstatements (each planned as a MERGE) were 83% of persistence time.Checks:
cargo nextest run --workspace --no-fail-fast: 2,267 of 2,269 passed. The two failures are the known macOS-onlyopensymphony_cli::migration/varsymlink tests.cargo fmt --all --checkandcargo clippy-devare clean.CI timings will be added once this PR's
rust-testrun finishes.