Feat/parallel area offload - #2790
Closed
reggiedroid wants to merge 8 commits into
Closed
Conversation
Skyrim is effectively single-threaded and so was the server: one Node
thread drives ScampServer::Tick, pumping every packet and running every
handler in sequence. Movement is the dominant packet by volume and its
cost is not linear in player count -- each update is relayed to everyone
who can see the sender, so N players in one area cost ~N^2 relay
decisions per tick. That quadratic term is what makes crowded areas
choppy while the host machine sits mostly idle.
This adds an opt-in framework that spreads that work across the cores the
host actually has. Per tick the server now:
1. ingests packets on the main thread as before, but flattens each
update into a plain-data snapshot instead of relaying immediately;
2. partitions actors into area clusters that provably cannot influence
one another this tick, then slices each cluster into work units;
3. processes each unit on a worker thread (validate, cull, build the
outbound send list);
4. joins on the main thread, applying world state and handing packets
to the network in a fixed, reproducible order.
Steps 1 and 4 stay serial because they touch MpActor, WorldState, the
Papyrus VM, RakNet and V8. Workers read only the immutable snapshot and
write only their own output buffer, so the parallel phase needs no locks.
Cluster safety follows from the game's own visibility rule: Grid.h uses a
3x3 chunk stencil (reach 1) and movement validation caps one update just
under a chunk (reach 1), so an actor's influence spans at most 2 chunks.
Clusters are connected components under "same worldOrCell, Chebyshev
chunk distance <= S", and S is clamped up to 3 so every possible
recipient lands in the sender's own cluster.
Clusters alone are not enough. On a realistic population one hub holds
~70% of a tick's relay work at every size from 40 to 300 players, which
caps a cluster-per-task scheme at ~1.4x regardless of core count -- the
framework would help least exactly where the server hurts most. The unit
of scheduling is therefore a shard: a contiguous slice of one cluster's
members. Each sender's work depends only on the snapshot, so any split is
safe, and modelled speedup rises to ~4x/8x/15x on 4/8/16 cores.
Determinism is preserved: clusters are ordered by lowest chunk
coordinate, members by submission index, shards are contiguous slices of
that order, and the join walks work units by index. The same inputs
produce the same sequence of world writes on 2 cores or 32.
Under measured overload, distant relays inside a hot cluster are spaced
across several ticks; players close to each other are never throttled.
Off by default. With parallelism.enabled absent or false the original
code path runs unchanged.
Two behavioural differences when enabled, both documented:
- movement relays batch to the end of the tick rather than being
emitted mid-ingest (unobservable in play, visible in packet captures,
and tests must tick before asserting);
- a world/cell form id from an unloaded plugin is rejected with a
teleport correction instead of throwing out of the packet handler.
Tests: unit/Parallel{ThreadPool,Partitioner,Offload,Config}Test.cpp cover
the pool, the partitioner's safety property, the dispatcher and the
config. unit/PartOne_MovementParallelTest.cpp drives the real PartOne,
packet parser and send target with the offload enabled and asserts the
same messages reach the same users as the inline path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Removed the redundant SIMDJSON parser instantiation inside each deserializer. - Passed the pre-parsed `simdjson::dom::element` directly from the factory down to the specific message deserializers. - This eliminates heavy string allocations and parsing overhead per message, substantially speeding up ingest on legacy JSON clients.
…pper pass-through
…est management
The speedups previously claimed for the parallel area offload were modelled
from a relay-edge cost model, never measured. They were wrong. This adds a
benchmark, corrects the record, and adds the optimisation that actually pays.
unit/ParallelBenchmark.cpp times the inline path against the offloaded one
with every player in a single chunk, over the binary wire format a real
client uses, counting full ingest plus PartOne::Tick so both paths cover the
same unit of work. Hidden behind Catch2's "[.]" tag so ctest ignores it:
players 25 50 100 150 250 400
speedup 0.25x 0.43x 0.66x 0.81x 0.88x 1.10x
Break-even is near 300-400 players, so minActorsToOffload now defaults to 300
instead of 24. At the old default the feature was a regression on every
realistic server, since SkyMP's own maxPlayers default is 100.
The parallel phase itself is fine -- at 400 players it retires 966us of task
work in 96us of wall clock. The ceiling is the join, which emits 160k relays
serially for 787us of an 1118us tick even with a send target that does
nothing. Parallelising decisions cannot help that. Sending fewer can.
So this adds distance-based interest management, on by default and
independent of load. Recipients within interestFullRateUnits (2048, about
half a chunk) always get every update; beyond that the rate steps down to a
half, a third, a quarter, capped by maxInterestSkipTicks. At 60Hz a distant
player still receives ~15 updates a second. Measured on 400 players spread
across a chunk:
inline (baseline) 160000 relays 1233us
offload only 160000 relays 1048us
offload + interest management 119866 relays 890us (1.39x)
It stacks with adaptiveThrottling by taking the stronger factor rather than
multiplying, so an overloaded server never starves a distant player past
whichever cap is more conservative. Phases are derived from a hash of the
(sender, recipient) pair, so traffic spreads across the window instead of
bursting, and the suite asserts every pair still transmits exactly once per
window: reduced rate, never silence.
Also in this change:
- MessageSerializer::Deserialize no longer walks the whole deserializer table
for JSON, re-parsing the document once per candidate. It reads the type
once and dispatches, reuses a thread_local simdjson parser instead of
constructing one per message, and only materializes the message text when
trace logging is actually enabled. Measured 3.4-6.3x cheaper on the legacy
JSON path (addresses the TODO(skyrim-multiplayer#2257) that lived there). Production clients
send binary, so this only helps legacy ones, but it is a hot path either
way.
- Repaired six [ParallelOffload] tests left asserting zero relays after the
snapshot rework: they built target vectors that were never handed to the
dispatcher. Recipients now come from SetPotentialTargets, so the tests wire
that up, and the crowded-area case asserts the full 60x60 fan-out rather
than a stale count.
- Deserialize takes the parsed element by pointer to a named local rather
than the address of an rvalue-qualified accessor's result.
Full suite green: 480346 assertions in 310 cases, 12/12 ctest.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Written against the benchmark rather than the earlier cost model, so the numbers here match what unit/ParallelBenchmark.cpp prints. Records the break-even population, the interest-management figures, and the limitations that still stand -- notably that the offload is a regression below ~300 players and that nothing here has been load-tested with real clients yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Names the effort (SkyMP: MOP - Multiplayer Optimization Project) and states what it does, what it measures, and what it does not claim. Kept as its own file rather than rewriting README.md, which belongs to the parent project. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Author
|
Closing in favor of #2789 which contains the squashed commit and full PR description. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.