Skip to content

feat(log): use ahash on live overlays#250

Merged
arkpar merged 2 commits intoparitytech:masterfrom
midnightntwrk:feat/ahash
May 11, 2026
Merged

feat(log): use ahash on live overlays#250
arkpar merged 2 commits intoparitytech:masterfrom
midnightntwrk:feat/ahash

Conversation

@Klapeyron
Copy link
Copy Markdown
Contributor

Background

The three LogOverlay maps (index, value, ref_count) are keyed by u64 chunk-position indices, not user input — they originate from the internal value-table free list or from the high bits of blake2(column_key) (the column-level cryptographic key hash that sits above this layer).

The default std HashMap hasher (SipHash-1-3, per-process random seed) adds a second DoS-resistant layer as defence-in-depth. ahash::RandomState provides the same property — randomly seeded per process, so an attacker cannot precompute collisions — at ~3× lower CPU on short keys, using AES-NI rounds where available. Both are non-cryptographic in the formal sense; this is a "well-studied DoS hash" vs "newer DoS hash" trade.

On the warm-cache Db::get benchmark (8 reader threads, multi-column substrate-shape workload) this gives +15-22 % concurrent reader qps with no on-disk format change, no semantics change, and 36/36 lib tests passing.

The local LogWriter scratch maps keep their existing hashers — they're single-threaded and short-lived.

Benchmarking

We made an experiment with live network sync with following results:

Validator	Start (UTC)	Finish (UTC, extrap.)	Duration	Avg rate
validator-04	16:41:51	18:03:38	1h 21m 47s	34.1 bps
validator-05	16:41:52	17:51:33	1h 09m 40s	40.0 bps
val-05 finished 12m 06s earlier than val-04.
sync_progress

What this PR does

File: src/log.rs

  • Import std::collections::hash_map::RandomState (as StdBuildHasher) and std::hash::BuildHasher.
  • Add LogWriterScratchBuildHasher = StdBuildHasher for writer-local index and ref-count scratch maps, with a short doc note.
  • Rename the ahash map builder alias to OverlayHasher (ahash::RandomState).
  • Make IndexLogOverlay, ValueLogOverlay, and RefCountLogOverlay generic over S: BuildHasher + Default, defaulting to OverlayHasher (unchanged for LogOverlays).
  • Use IndexLogOverlay<LogWriterScratchBuildHasher> and RefCountLogOverlay<LogWriterScratchBuildHasher> inside LogChange and FlushedLog.
  • Replace the standalone ValueLogOverlayLocal struct with pub type ValueLogOverlayLocal = ValueLogOverlay<BuildIdHash>.
  • Extend the module comments so they match the background above and describe which hasher is used where (live overlays vs scratch maps).

gilescope and others added 2 commits May 8, 2026 08:44
  The three LogOverlay maps (index, value, ref_count) are keyed by u64
  chunk-position indices, not user input — they originate from the
  internal value-table free list or from the high bits of
  blake2(column_key) (the column-level cryptographic key hash that sits
  above this layer).

  The default std HashMap hasher (SipHash-1-3, per-process random seed)
  adds a second DoS-resistant layer as defence-in-depth. ahash::RandomState
  provides the same property — randomly seeded per process, so an attacker
  cannot precompute collisions — at ~3× lower CPU on short keys, using
  AES-NI rounds where available. Both are non-cryptographic in the formal
  sense; this is a "well-studied DoS hash" vs "newer DoS hash" trade.

  On the warm-cache Db::get benchmark (8 reader threads, multi-column
  substrate-shape workload) this gives +15-22 % concurrent reader qps
  with no on-disk format change, no semantics change, and 36/36 lib
  tests passing.

  The local LogWriter scratch maps keep their existing hashers — they're
  single-threaded and short-lived.

Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Tomasz Bartos <tomasz.bartos@shielded.io>
@arkpar
Copy link
Copy Markdown
Member

arkpar commented May 11, 2026

Look good, thank you for the PR

@arkpar arkpar merged commit 95d3790 into paritytech:master May 11, 2026
9 of 11 checks passed
@arkpar
Copy link
Copy Markdown
Member

arkpar commented May 11, 2026

Published as v0.5.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants