feat: R3 gym notq router replay by zyzhou5 · Pull Request #2915 · NVIDIA-NeMo/RL

zyzhou5 · 2026-06-24T19:11:53Z

What does this PR do ?

Adds no-TQ async Gym path support for R3 router replay.

When policy.router_replay.enabled=true, async Gym rollouts now request routed expert indices from Gym, validate that they are present, and convert them into Nemo-RL message logs. The routed expert indices are sliced across prompt and generation tokens for each Gym model call so the existing R3 training path can consume them.

This also updates the async OpenAI-compatible vLLM endpoint to return routed expert indices in chat response messages when requested.

copy-pr-bot · 2026-06-24T19:11:56Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

zyzhou5 · 2026-06-24T19:20:05Z

/ok to test 763fb26

zyzhou5 · 2026-06-24T20:09:20Z

/ok to test c289b97

zyzhou5 · 2026-06-24T21:35:04Z

/ok to test df70c93

zyzhou5 · 2026-06-24T22:50:22Z

/ok to test f594c2d

ZhiyuLi-Nvidia

Just 2 nit.
LGTM.

zyzhou5 · 2026-06-25T19:27:26Z

/ok to test 3879443

zyzhou5 · 2026-06-26T20:00:18Z

/ok to test f15074c

github-actions · 2026-06-26T20:00:58Z

✅ Submodule Fast-Forward Check Results

Check based on commit: f15074c (PR #2915 from r3-gym-notq-router-replay)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

zyzhou5 · 2026-06-26T20:06:19Z

/ok to test 338f8fc

github-actions · 2026-06-26T20:07:36Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 338f8fc (PR #2915 from r3-gym-notq-router-replay)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

zyzhou5 · 2026-06-26T21:02:29Z

/ok to test 20f7389

github-actions · 2026-06-26T21:03:19Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 20f7389 (PR #2915 from r3-gym-notq-router-replay)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

zyzhou5 · 2026-06-26T22:14:31Z

/ok to test fe5f434

github-actions · 2026-06-26T22:15:22Z

✅ Submodule Fast-Forward Check Results

Check based on commit: fe5f434 (PR #2915 from r3-gym-notq-router-replay)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2026-06-26T22:38:15Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 2ff1ffc (PR #2915 from r3-gym-notq-router-replay)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

zyzhou5 · 2026-06-26T22:39:17Z

/ok to test 2ff1ffc

Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>

zyzhou5 · 2026-06-29T17:01:07Z

/ok to test c25a0bf

zyzhou5 requested review from a team as code owners June 24, 2026 19:11

zyzhou5 added the CI:L1 Run doctests, unit tests, and functional tests label Jun 24, 2026

zyzhou5 changed the title ~~R3 gym notq router replay~~ feat: R3 gym notq router replay Jun 24, 2026

copy-pr-bot Bot temporarily deployed to public June 24, 2026 19:20 Inactive

copy-pr-bot Bot temporarily deployed to public June 24, 2026 19:26 Inactive

zyzhou5 force-pushed the r3-gym-notq-router-replay branch from 763fb26 to c289b97 Compare June 24, 2026 20:08

copy-pr-bot Bot temporarily deployed to public June 24, 2026 20:09 Inactive

copy-pr-bot Bot temporarily deployed to public June 24, 2026 20:10 Inactive

copy-pr-bot Bot temporarily deployed to test June 24, 2026 20:12 Inactive

copy-pr-bot Bot temporarily deployed to public June 24, 2026 20:13 Inactive

copy-pr-bot Bot temporarily deployed to public June 24, 2026 21:35 Inactive

copy-pr-bot Bot temporarily deployed to public June 24, 2026 21:39 Inactive

zyzhou5 force-pushed the r3-gym-notq-router-replay branch from df70c93 to f594c2d Compare June 24, 2026 22:49

copy-pr-bot Bot temporarily deployed to public June 24, 2026 22:50 Inactive

copy-pr-bot Bot temporarily deployed to public June 24, 2026 22:51 Inactive

copy-pr-bot Bot temporarily deployed to test June 24, 2026 22:53 Inactive

copy-pr-bot Bot temporarily deployed to public June 25, 2026 17:26 Inactive

copy-pr-bot Bot temporarily deployed to test June 25, 2026 17:27 Inactive

copy-pr-bot Bot temporarily deployed to public June 25, 2026 17:29 Inactive

ZhiyuLi-Nvidia reviewed Jun 25, 2026

View reviewed changes

Comment thread nemo_rl/models/generation/vllm/vllm_worker_async.py

ZhiyuLi-Nvidia reviewed Jun 25, 2026

View reviewed changes

Comment thread nemo_rl/models/generation/vllm/utils.py

ZhiyuLi-Nvidia previously approved these changes Jun 25, 2026

View reviewed changes

zyzhou5 dismissed ZhiyuLi-Nvidia’s stale review via 3879443 June 25, 2026 19:27

zyzhou5 force-pushed the r3-gym-notq-router-replay branch from 9c00c8d to 3879443 Compare June 25, 2026 19:27

zyzhou5 added the r0.7.0 label Jun 26, 2026

zyzhou5 requested a review from a team as a code owner June 26, 2026 20:00

zyzhou5 force-pushed the r3-gym-notq-router-replay branch from f15074c to 338f8fc Compare June 26, 2026 20:06

zyzhou5 added 4 commits June 29, 2026 09:47

feat: support R3 router replay for async Gym rollouts

6bfbf0b

Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>

test: cover async vLLM routed expert chat attachment

87b3c9e

Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>

fix: preserve R3 routes in async vLLM chat JSON

1e39dd6

Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>

refactor: move async chat R3 hook into mixin

c25a0bf

Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>

zyzhou5 added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L1 Run doctests, unit tests, and functional tests labels Jun 29, 2026

zyzhou5 force-pushed the r3-gym-notq-router-replay branch from 2ff1ffc to c25a0bf Compare June 29, 2026 17:00

Uh oh!

Conversation

zyzhou5 commented Jun 24, 2026

What does this PR do ?

Uh oh!

copy-pr-bot Bot commented Jun 24, 2026

Uh oh!

zyzhou5 commented Jun 24, 2026

Uh oh!

zyzhou5 commented Jun 24, 2026

Uh oh!

zyzhou5 commented Jun 24, 2026

Uh oh!

zyzhou5 commented Jun 24, 2026

Uh oh!

Uh oh!

Uh oh!

ZhiyuLi-Nvidia left a comment

Choose a reason for hiding this comment

Uh oh!

zyzhou5 commented Jun 25, 2026

Uh oh!

zyzhou5 commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

zyzhou5 commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

zyzhou5 commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

zyzhou5 commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Jun 26, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

zyzhou5 commented Jun 26, 2026

Uh oh!

zyzhou5 commented Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants