perf: dep_graph: serve edge lists in place from the retained file bytes [-0.64% incr-unchanged]#59
Open
xmakro wants to merge 2 commits into
Open
Conversation
Compaction was triggered only by a fixed generation cap: after eight carried generations the file is rewritten fresh, whatever its actual state. Track the live record bytes exactly in the footer, as an O(changed) delta like the other counts, and rewrite once the record region exceeds twice its live bytes. High-churn graphs compact as often as before, while low-churn graphs carry for up to sixteen generations instead of eight, avoiding the periodic compaction spikes the fixed cap forced on them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decoding used to copy every edge byte out of the file into a freshly allocated buffer, the largest allocation and copy of the load. The file stays mapped for the whole session anyway (the carry copies its record region forward), so serve the edge lists directly from the retained bytes: the on-disk varint encoding was already the in-memory representation, and each node's edge header now records a position in the file instead of a position in the copied buffer. Decoding a record shrinks to reading its fixed header and skipping over the edges. The fixed-size overread at the end of an edge list stays in bounds because the footer always follows the records. On Windows, where the mapping must not outlive the load because the save renames over the mapped file, the bytes are copied out once instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #44 (append-only save via wholesale carry of the record region). Split out of that PR to be measurable on its own; independent of the skip-edge-collection sibling PR #58.
Decoding used to copy every edge byte out of the file into a freshly allocated buffer, the largest allocation and copy of the load. The file stays mapped for the whole session anyway (the carry copies its record region forward), so serve the edge lists directly from the retained bytes: the on-disk varint encoding was already the in-memory representation, and each node's edge header now records a position in the file instead of a position in the copied buffer. Decoding a record shrinks to reading its fixed header and skipping over the edges. The fixed-size overread at the end of an edge list stays in bounds because the footer always follows the records. On Windows, where the mapping must not outlive the load because the save renames over the mapped file, the bytes are copied out once instead.
Local rustc-perf (primary crates, instructions:u), this change alone against the #44 branch:
27 of 60 cells improved by at least 0.25%, 0 regressed.
Validation: 20 generations each of incr-unchanged, of alternating real edits, and of alternating edits under
-Zincremental-verify-ich, all crossing the compaction boundary: every generation clean.