registry: hot-path performance — memoized sig verification, lock-free telemetry, chunked reaper, single-write framing#100
Open
TeoSlayer wants to merge 1 commit into
Open
registry: hot-path performance — memoized sig verification, lock-free telemetry, chunked reaper, single-write framing#100TeoSlayer wants to merge 1 commit into
TeoSlayer wants to merge 1 commit into
Conversation
…k-free telemetry, chunked reaper, single-write framing Production profiling on the live registry (207K conns, reconnect storm) showed ~30% of CPU in repeat ed25519 verifications of deterministic challenges, ~26% in per-message write syscalls, and reader pile-ups behind the reaper's full-map sweep under the write lock. - authz.VerifyCached: sharded memoization of successful (pubkey, message, sig) verifications; ed25519 is deterministic and registry challenges are static per node/request shape, so identical tuples always re-verify identically. Failures are never cached. Wired into VerifyNodeSignature and the binary heartbeat/resolve paths. - handleMessage: drop the s.mu wrap around walStore.IsStandby (wal has its own lock); per-network request counters use TryRLock and skip a telemetry tick instead of queueing behind writers. - ReapStaleNodes: collect+sort+scan under RLock, write-lock only to delete confirmed-stale nodes (re-checked under the lock), Save() signalled outside the lock. Common no-reap sweep takes no write lock. - accept.writeMessage: single buffered write per frame instead of prefix+body (halves write syscalls). Deployed 2026-07-24: storm-phase CPU dropped from ~1000% to ~240% at equivalent connection counts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| var lenBuf [4]byte | ||
| binary.BigEndian.PutUint32(lenBuf[:], uint32(len(body))) | ||
| frame := make([]byte, 4+len(body)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Production profiling on the live registry during today's fleet-wide outage (207K conns, reconnect storm) showed:
crypto/ed25519.Verify— re-verifying deterministic challenges (heartbeat:%d,resolve:%d:%d) whose signatures are identical on every repeat (ed25519 is deterministic)writeMessage= 2 writes/frame)ReapStaleNodes, which collected + sorted the full 240K-node key set under the write lock on every sweep — even when nothing was stales.mu.RLockper message for a standby flag that already has its own lock, plus per-network telemetry counters queueing behind writersWhat
authz.VerifyCached: sharded, bounded memoization of successful(pubkey, message, sig)verifications (sha256 key, 64 shards × 8192, reset-on-full). Identical tuple ⇒ identical verify result, so this is semantically equivalent; failures are never cached. Wired intoVerifyNodeSignature(all JSON paths) and the binary heartbeat + binary resolve paths.handleMessage:IsStandby()no longer wrapped ins.mu; per-network request counters useTryRLockand skip a telemetry tick under writer contention instead of blocking the request.ReapStaleNodes: snapshot + sort + chunk-scan under RLock; write lock only when confirmed-stale nodes exist (staleness re-checked under the lock);Save()signalled outside. The common no-reap sweep now takes no write lock at all.accept.writeMessage: one write per frame (len-prefix + body in a single buffer).Production results (deployed to pilot-rendezvous-new 2026-07-24)
Reconnect-storm phase at equivalent conn counts (~100–200K):
Tests
TestVerifyCached(valid→cached→valid, tampered sig rejected on repeat, wrong key/message not confused by cache, failure not cached) + shard-cap testgo test -race ./authz/... ./directory/... ./accept/...green; fullgo test ./...green (20 pkgs);go vetclean🤖 Generated with Claude Code