fix(syncing): cut scheduler+discovery CPU regression#575
Open
juligasa wants to merge 2 commits into
Open
Conversation
Two compounding sources of post-Apr-29 CPU/GC growth:
- The scheduler rewrite walked s.tasks linearly under s.mu on every
timer tick (preempt/cleanup/wake) and allocated a context.WithCancel
per dispatch attempt — including saturated ones that immediately
discarded it. This commit:
* indexes in-progress tasks separately: ascending heap by
hotDeadline for ephemeral hot, map by key for subscriptions, so
preempt/cleanup/wake decisions are O(log n)/O(1) instead of O(n);
* defers context.WithCancel and Progress allocation until after
the capacity check, so saturated peeks don't allocate;
* moves time.Timer.Reset out of s.mu in the run loop and drops
the redundant lock/unlock at executeTask entry by passing
taskCtx and progress in via the dispatch closure.
- DiscoverObjectWithProgress scanned the whole peers table and parsed
every multiaddr just to filter on Connectedness. Replaced with
host.Network().Peers() for the connected fast path; the DB query
now runs only when no peers are connected, ORDER BY updated_at DESC
LIMIT, so the multiaddr parse cost is bounded. The disconnected
fallback uses peerstore.RecentlyConnectedAddrTTL since dials may
run seconds later — TempAddrTTL was short enough that addresses
could expire before we got to dial.
Adds a Discovery scheduler section to /debug/network exposing queue/
in-progress sizes plus cumulative subscription/hot preempt counters
(atomic), so we can verify whether preemption fires in production
without a metrics scraper. Wired through initHTTP as a function arg
rather than as setter-installed Node state, since the snapshot
provider is set once at startup.
revive's context-as-argument rule wants context.Context as the first non-receiver parameter. Reorder executeTask(task, taskCtx, prog) → executeTask(taskCtx, task, prog).
5eee41d to
020aec8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A 12-min pprof comparison on
mainshowed 10.5× user CPU, 7.4× heap allocs, 28× GC cycles vs the same daemon 12 minutes earlier. Two compounding sources, both confirmed against current source:81bdc70c). The two-tier priority heap + per-task preemption walkeds.taskslinearly unders.muevery timer tick (preempt / cleanup / boundedWake), and allocatedcontext.WithCancel+Progressper peeked task — including ones immediately discarded under saturation.DiscoverObjectWithProgressscanned the wholepeerstable and parsed every multiaddr just to filter onConnectedness, keeping at most 20 fallback peers and discarding the rest.Changes
hotDeadline) and subscriptions (map by key) so preempt/cleanup/wake are O(log n)/O(1) instead of O(n) overs.tasks.markInProgress/markFinishedkeep the indexes consistent across dispatch, executeTask cleanup, preemption, and heartbeat-expiry cancel paths.markFinishedis idempotent so popped-then-cancelled tasks don't double-remove.context.WithCancelandProgressallocation moved past the capacity check — saturated peeks no longer allocate.time.Timer.Resetmoved out ofs.muin the run loop.executeTaskno longer takess.muto readrunCtx/progress— passed via dispatch closure.host.Network().Peers()directly. The bounded DB query (ORDER BY updated_at DESC LIMIT ?) only runs when no peer is currently connected, so the per-row multiaddr parse is bounded too.peerstore.RecentlyConnectedAddrTTLinstead ofTempAddrTTL— dials may run many seconds later as syncs queue up.atomic.Uint64). The snapshot provider is plumbed throughinitHTTPas a function arg — no setter-installed Node state — since it's set exactly once at startup.Test plan
go test -race -count=1 ./backend/hmnet/...cleanscheduler_test.gotests pass (subscription preemption, hot-vs-hot preemption, heartbeat cleanup, hot-tier-beats-cold, retouch promotion, heap-index correctness)http_debug_network_test.gocases for the Scheduler section render/debug/network, confirm the new section renders and the preempt counters update under load.ai/shows heap-alloc rate ≤ 1.2× baseline and GC count ≤ 1.5× baseline (was 28×)Notes
AddrInfoFromStringsCachedLRU helper and versioned scheduler/discovery benchmarks. Both dropped after review: