feat(log): use ahash on live overlays#250
Merged
arkpar merged 2 commits intoparitytech:masterfrom May 11, 2026
Merged
Conversation
The three LogOverlay maps (index, value, ref_count) are keyed by u64 chunk-position indices, not user input — they originate from the internal value-table free list or from the high bits of blake2(column_key) (the column-level cryptographic key hash that sits above this layer). The default std HashMap hasher (SipHash-1-3, per-process random seed) adds a second DoS-resistant layer as defence-in-depth. ahash::RandomState provides the same property — randomly seeded per process, so an attacker cannot precompute collisions — at ~3× lower CPU on short keys, using AES-NI rounds where available. Both are non-cryptographic in the formal sense; this is a "well-studied DoS hash" vs "newer DoS hash" trade. On the warm-cache Db::get benchmark (8 reader threads, multi-column substrate-shape workload) this gives +15-22 % concurrent reader qps with no on-disk format change, no semantics change, and 36/36 lib tests passing. The local LogWriter scratch maps keep their existing hashers — they're single-threaded and short-lived. Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Tomasz Bartos <tomasz.bartos@shielded.io>
14 tasks
Member
|
Look good, thank you for the PR |
Member
|
Published as v0.5.5 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The three LogOverlay maps (index, value, ref_count) are keyed by u64 chunk-position indices, not user input — they originate from the internal value-table free list or from the high bits of blake2(column_key) (the column-level cryptographic key hash that sits above this layer).
The default std HashMap hasher (SipHash-1-3, per-process random seed) adds a second DoS-resistant layer as defence-in-depth. ahash::RandomState provides the same property — randomly seeded per process, so an attacker cannot precompute collisions — at ~3× lower CPU on short keys, using AES-NI rounds where available. Both are non-cryptographic in the formal sense; this is a "well-studied DoS hash" vs "newer DoS hash" trade.
On the warm-cache Db::get benchmark (8 reader threads, multi-column substrate-shape workload) this gives +15-22 % concurrent reader qps with no on-disk format change, no semantics change, and 36/36 lib tests passing.
The local LogWriter scratch maps keep their existing hashers — they're single-threaded and short-lived.
Benchmarking
We made an experiment with live network sync with following results:
What this PR does
File:
src/log.rsstd::collections::hash_map::RandomState(asStdBuildHasher) andstd::hash::BuildHasher.LogWriterScratchBuildHasher = StdBuildHasherfor writer-local index and ref-count scratch maps, with a short doc note.ahashmap builder alias toOverlayHasher(ahash::RandomState).IndexLogOverlay,ValueLogOverlay, andRefCountLogOverlaygeneric overS: BuildHasher + Default, defaulting toOverlayHasher(unchanged forLogOverlays).IndexLogOverlay<LogWriterScratchBuildHasher>andRefCountLogOverlay<LogWriterScratchBuildHasher>insideLogChangeandFlushedLog.ValueLogOverlayLocalstruct withpub type ValueLogOverlayLocal = ValueLogOverlay<BuildIdHash>.