[core][actor scalability] Remove the GCS resource view reset that causes PG retry storms at scale

[Yicheng-Lu-llll]

Background

Currently, and at least for now, Ray uses distributed scheduling: every raylet and the GCS has its own resource view for making scheduling decisions.

Say a request is sent to node A. Node A's raylet decides to schedule the request on node B. Node A first optimistically deducts the resources in its local view, then replies to the submitter with the info that we chose node B instead. The submitter then forwards the request to node B, and if we are lucky, node B accepts it.

It is also possible that node B rejects the request. The rejection goes back to the submitter, schedules it again. node A's earlier deduction for node B is now wrong and we do not add back resources, we rely on Ray syncer to eventually correct everyone's resource view.

The Ray syncer works as follows: every raylet sends its latest resource view to the GCS, and the GCS fans it out to every node.

The issue, and why the reset exists

Now, you might already see the issue. When node A's raylet decides that the request needs to be placed on node B, it deducts node B's resources in its local view and sends the request back to the owner. Node A does not track this request any more. It is now up to the next hop to guarantee delivery of the request. It may not land on node B, and we rely on the Ray syncer to eventually correct that.

But there is another risk: the request itself could get dropped or ignored. For example, what if the submitter dies right when it receives the request back from node A? Or what if the submitter decides not to forward the request to node B at all, which can happen with Ray's "lease reuse" feature(see https://github.com/ray-project/ray/pull/28054/changes/da74e7ea355825e8869e9211a0efa22255e5f4de#diff-09f46509f86d054afbfa[…]9705ce7f4d67fc3fa1ad4R359)?

In all of these cases, there is no resource change on any other node, so the Ray syncer never fires, but node A has already deducted resources in its local view for node B. That means the view leaks resources.

We don't have a perfect solution for this, so we decided: fine, every 3 seconds, reset every locally modified node entry in the resource view back to the last view received from the syncer.

This works well because normally the Ray syncer is fast and the propagation time is well under 3s.

But this is no longer true when we scale, for example to 10k nodes. Specifically, the GCS's local resource view is used for PG scheduling today. At the 10k level (see test here: #64446), scheduling the PGs takes ~227s, mostly because the Ray syncer is very slow at that scale, so the resource view gets reset every 3 seconds while we are scheduling PGs. This cause wrong resource view and retry storms:

• At init, all raylets report an initial resource view snapshot to the GCS. Of course, all resources are available.
• The GCS starts PG scheduling, and everything is fine for the first 3s.
• At 3s, the syncer is too slow, so the raylets that received PGs can't send the correct resource view back in time. The 3s reset then resets the resource view to the init snapshot, meaning all resources look available again.
• Our topology aware scheduling checks every rack one by one in a deterministic order and chooses the first available rack.
• That means, because of this reset, the GCS isn't really scheduling at all, it just keeps sending requests to the same already occupied rack until the syncer finally catches up.

This is super bad. And if we think about it more deeply, GCS scheduling always tries to schedule the request after doing the allocation, and that will remain true. So there is no situation where we deduct the resources but the request gets lost and no allocation happens. Therefore the GCS local resource view does not need the reset at all.

this PR only turns it off in the GCS, raylets keep it, because the submitter death and lease reuse cases above are real there.

Perf

The performance win is large: PG scheduling at the 10k level went from ~227s (see benchmark scripts here:#64446) down to (7.9 / 9.6 / 9.2 across three runs) and retries 1086 -> 0.

Honest Caveat

Although we fixed the reset issue, the Ray syncer is still very slow. For PG scheduling plus actor scheduling within PGs, the biggest bottleneck now becomes the following chain: PG scheduling gets much faster → every node reports its resource change at the same time → the Ray syncer comes under heavy pressure → actors sent to the head raylet then wait a very long time for the PG bundle info to propagate to the head raylet.

[gemini-code-assist]

Code Review

This pull request introduces a reset_remote_node_views parameter to ClusterResourceManager and ClusterResourceScheduler to control whether remote node views are periodically reset, disabling it for the GCS server. The reviewer suggested moving the RAY_CHECK for periodical_runner_ below the reset_remote_node_views check in ClusterResourceManager's constructor, which would allow passing a nullptr for the periodical runner when resetting is disabled.

[Sparks0219]

do we necessarily need an extra field for this? Why can't I just see that the periodical_runner is nullptr and early return using that

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit f4fe146. Configure here.}

[Yicheng-Lu-llll]

Ran all related core release tests. All LGTM: https://buildkite.com/ray-project/release/builds/103444/canvas