Repository navigation
Conversation
… Qwen3.5 EP dispatcher TpuQwen35RoutedExperts.forward calls get_proxy_mode() in its chunk-snap warning branch when the token count is not divisible by num_token_chunks (e.g. ep_token_chunks=12 with 2^k tokens). Under torch.compile (per-layer compile on the regular trainer) Dynamo cannot trace torch._C._get_dispatch_mode(_TorchDispatchModeKey.PROXY) and raises NotImplementedError: UserDefinedObjectVariable(_TorchDispatchModeKey). Guard both eager-only checks (that warning and the debug branch in _ragged_all_to_all_impl) with torch.compiler.is_compiling() first.
…ompile The dropless ragged_all_to_all path (EP > top_k) passes the dispatch size tensors again as the combine sizes (comb_recv_sizes is disp_send_sizes), so _RaggedAllToAll.apply receives the same tensor object twice. With grad enabled Dynamo raises torch._dynamo.exc.Unsupported: autograd.Function.apply: duplicate tensor input (seen on Qwen3.5-397B EP16 update_actor; forward-only log-prob was fine). Clone repeated size/offset vectors (tiny int32) before apply.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Targets #201 (
feat/tpu-qwen35-cp-on-ep).Bug
On Qwen3.5-397B-A17B RL (verl + verl-hardware-plugin, per-layer
torch.compile(backend='tpu'), EP8,ep_token_chunks=12), step 1 crashes incompute_log_prob:The token count (a power of two) isn't divisible by 12, so
TpuQwen35RoutedExperts.forwardenters its chunk-snap warning branch. That branch callsget_proxy_mode(), which Dynamo can't trace.Fix
Check
torch.compiler.is_compiling()beforeget_proxy_mode()in that warning, and in the eager-only debug branch of_ragged_all_to_all_impl. Both branches are for eager-mode diagnostics and are skipped under tracing anyway. No behavior change in eager mode.Workaround until merged: choose
ep_token_chunksthat divides the per-device token count (e.g. 8).