perf(code-index): build code-graph rows from sealed segments - #2271
Conversation
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1193449c0c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if entry | ||
| .file_name() | ||
| .to_str() | ||
| .is_some_and(|name| name.starts_with(ROW_SPILL_PREFIX) && !name.starts_with(&own)) |
There was a problem hiding this comment.
When a daemon is restarted with a PID previously used by a process that was killed during a spill, this filter treats the abandoned .rows-<pid>-* directories as belonging to the new process and never removes them. Since NEXT_SPILL also restarts at zero, the next graph publication can collide with .rows-<pid>-0 and fail at directory creation, while potentially large stale spills remain on disk; use an instance-unique owner token or otherwise prove liveness instead of PID alone.
AGENTS.md reference: AGENTS.md:L41-L46
Useful? React with 👍 / 👎.
| let entities = self | ||
| .entities()? | ||
| .map(|row| row.map(|(value, _)| value)) | ||
| .collect::<Result<Vec<_>, _>>()?; |
There was a problem hiding this comment.
When direct sealing is unavailable—such as with TRACEDECAY_GRAPH_SEALED_STORE=off, or during a staging repair—this collection can deserialize the entire corpus-sized spill without invoking check; cancellation or deadline expiry is only observed later by new_checked, after all entities and relations have been loaded. For multi-GB graphs that defeats cancellation and can exhaust memory before returning the typed interruption, so the spill should be read in checked windows or probe between rows.
Useful? React with 👍 / 👎.
1 participant
Refs #2123. Third slice. Code-graph publication now builds its rows from the sealed segments on disk, one window of files at a time. It no longer decodes the whole generation or builds the one-piece manifest. The published graph, its recovered digest, and its sealed container sections are byte-identical to what the old path produced.
This does not close #2123. At 6 GB the daemon now reaches the sealed compact-store build and is OOM-killed there. That build peaks at 5.39 GB on this repository (measured below), not the 1.3 GB estimated on the issue.
Design as built
graph-db: a streaming row source (
generation/spill.rs)GraphGenerationRowSpilltakes rows in batches, in any order. Each row is validated the way the manifest constructor validates it and stored as its canonical JSON (the exact bytes the recovered digest hashes).finishdoes a k-way merge of the runs into one sorted entity file and one sorted relation file:SpilledGraphGenerationstreams the merged rows back. Its sealed build (push_spilled_rows) pushes entities and then relations in identity order. Each relation endpoint resolves by binary search over the resident identities, the same node idspush_manifest_rowsassigns.GraphGenerationRows(a manifest or spilled rows) replacesOption<Arc<GraphGenerationManifest>>onpublish_verifiedandprepare_verified_publication. It also carries the journaled sealed replay and the supplied-rows binding check.GraphGenerationManifestProvidernow hydrates a sealed generation into a spill the registry creates. The registry creates it under the store's*.sealed/.rows-<pid>-<n>directory.code-index: per-file-batch rows (
graph_projection/builder.rs,production/graph_inputs.rs,SealedGenerationFileWindowsV1)resolve_cross_file_references,unresolved_typescript_import_calls, and receiver-site rules.emit_code_graph_rows) serves both the per-window pass and the whole-set hermetic build, so the two row sets are unioned the same way.build_published_code_graph_manifest_checked, the generation's graph-manifest memo, and the provider's decoded-generation offers (offer, hydration retention, pressure reclaimer).store-runtime / scheduler
publish_verified_snapshotno longer takes a decoded generation. It classifies against the journal first. Rows are built from the runtime's sealed replay binding only when a publish arm needs them, before the replay-pool lock is taken. A retry or twin publisher of a landed generation recovers the head without reading a segment.admit_sealed_graph_build, charged the measured bytes of the active generation). While the text build holds its reservation, the graph build is refused and parked on the sameResidentMemorypark the decode used.syncfsat seal. The sealed graph container still builds under.staging-*and installs by rename.Measured
Fixture, fails on the old path. Test
a_sealed_graph_build_holds_windows_not_the_decoded_generation(resident_accounting, counting allocator, 300-file generation, one worker):origin/master's equivalent (decode_partitioned_sealedplusbuild_published_code_graph_manifest_checked, same fixture and width) peaks at 38,827,295 bytes for the same 4,201 entities and 5,100 relations, so it fails the budget.This repository's sealed generation, in-process (counting allocator, glibc test binary, perf profile):
finish)The last row is the new wall. grafeo's
IncrementalCompactStoreBuilderkeeps every pushedValueuntilfinishhas encoded all columns. Both paths pay it; the old one paid it on top of a resident decode and manifest.Daemon journeys. Clone of this repository at 3ce6d2d, perf+hotpath CLI, isolated profile, one
systemd-run --user --scope -p MemoryMax=… -p MemorySwapMax=1Gscope, cgroup RSS+swap every 250 ms (per-second maximum).For comparison from #2123: master at 12 GB with 4 workers was OOM-killed at 364 s, and #2237 at 12 GB reached fresh at 676 s.
Identical results
The #2185 80-response fixture:
search,context,callers, andcalleeson 20 symbols. Timestamps, request ids, cursors, handles,wall_micros, and the profile-keyed generation id are normalized.root) graph lanes answering "interactive catalog is warming", the post-activation warm race perf(graph-db): store unique keys as binary namespace-id keys #2257 also saw.rooton both warm daemons: 4/4 identical.Direct digest comparison of the graph store files. The same profile, the same code generation id, rebuilt by this PR from master's journal:
tracedecay.grafeo, its WAL, andtracedecay.sealed/.The container header's first 12,288 bytes carry the wall-clock write time. Everything after them is compared:
3cfa82adc754e7240279…3cfa82adc754e7240279…sha256:8d35f215208e4d85…both438005fdf983473e022d…438005fdf983473e022d…sha256:8123d63f73cad052…bothAfter the republish, the query set is 80/80 (rsbuild) and 79/80 (j768, the same
rootcontext warm race) identical to master's answers.Seal spike (report only, not bounded here)
seal.reference_candidate_lookup426,850 calls,seal.qualified_path_walk18.3M) and the first 2,006sealed_encode.filecalls.Tests
graph_db_suite):spilled_rows_seal_the_same_generation_as_their_manifest: shuffled batches with repeats give the same digest, journaled replay, receipt, and container sections as the manifest.spilled_rows_refuse_conflicting_repeats_and_dangling_endpointsstore_open_sweeps_row_spills_abandoned_by_another_processan_interrupted_graph_build_fails_typed_and_the_next_build_completesPartitionedSealV1segments, including the serial/parallel width test that forces different window sizes. The store-runtime provider and publication tests now use spills.daemon_suite::sealed_generation_crash_test, run against the built CLI):sigkill_after_the_seal_restarts_on_the_identical_sealed_generationandsigkill_during_the_seal_keeps_the_prior_generation_published, 2 passed.Suites
Run on this branch rebased onto bf0ddaf. The journeys above ran on the pre-rebase tip.
tracedecay-code-index: lib 258,code_index_suite169,resident_accounting2tracedecay-code-index-runtime: lib 516 (2 ignored)tracedecay-graph-db: lib 143 (1 ignored),graph_db_suite153 (2 ignored)tracedecay-query: lib 263 (1 ignored),search_quality_suite71,canonical_execution_equivalence5,retrieval_contract_spine2tracedecay-store-runtime: lib 121 (1 ignored)tracedecaylib tests (ported daemon activation and store-runtime tests) compile.cargo clippy -p tracedecay-graph-db -p tracedecay-code-index -p tracedecay-code-index-runtime -p tracedecay-store-runtime --all-targets --features tracedecay-graph-db/test-helpers,tracedecay-code-index/test-helpers -- -D warningsandcargo fmt --all -- --checkare clean.