Skip to content

fix(tpu): skip get_proxy_mode() checks while torch.compile traces the Qwen3.5 EP dispatcher - #210

Draft
jialei777 wants to merge 2 commits into
feat/tpu-qwen35-cp-on-epfrom
jialeic/fix-dispatcher-proxy-mode-under-compile
Draft

jialei777 wants to merge 2 commits into
feat/tpu-qwen35-cp-on-epfrom
jialeic/fix-dispatcher-proxy-mode-under-compile

Conversation

@jialei777

Copy link
Copy Markdown
Contributor

Targets #201 (feat/tpu-qwen35-cp-on-ep).

Bug

On Qwen3.5-397B-A17B RL (verl + verl-hardware-plugin, per-layer torch.compile(backend='tpu'), EP8, ep_token_chunks=12), step 1 crashes in compute_log_prob:

File ".../experiments/tpu/qwen3_5/token_dispatcher.py", line 1010, in forward
    get_proxy_mode() is None
File ".../torch/fx/experimental/proxy_tensor.py", line 3084, in get_proxy_mode
    mode = torch._C._get_dispatch_mode(torch._C._TorchDispatchModeKey.PROXY)
NotImplementedError: UserDefinedObjectVariable(_TorchDispatchModeKey)

The token count (a power of two) isn't divisible by 12, so TpuQwen35RoutedExperts.forward enters its chunk-snap warning branch. That branch calls get_proxy_mode(), which Dynamo can't trace.

Fix

Check torch.compiler.is_compiling() before get_proxy_mode() in that warning, and in the eager-only debug branch of _ragged_all_to_all_impl. Both branches are for eager-mode diagnostics and are skipped under tracing anyway. No behavior change in eager mode.

Workaround until merged: choose ep_token_chunks that divides the per-device token count (e.g. 8).

… Qwen3.5 EP dispatcher

TpuQwen35RoutedExperts.forward calls get_proxy_mode() in its chunk-snap warning branch when the
token count is not divisible by num_token_chunks (e.g. ep_token_chunks=12 with 2^k tokens). Under
torch.compile (per-layer compile on the regular trainer) Dynamo cannot trace
torch._C._get_dispatch_mode(_TorchDispatchModeKey.PROXY) and raises
NotImplementedError: UserDefinedObjectVariable(_TorchDispatchModeKey). Guard both eager-only
checks (that warning and the debug branch in _ragged_all_to_all_impl) with
torch.compiler.is_compiling() first.
…ompile

The dropless ragged_all_to_all path (EP > top_k) passes the dispatch size tensors again as the
combine sizes (comb_recv_sizes is disp_send_sizes), so _RaggedAllToAll.apply receives the same
tensor object twice. With grad enabled Dynamo raises
torch._dynamo.exc.Unsupported: autograd.Function.apply: duplicate tensor input
(seen on Qwen3.5-397B EP16 update_actor; forward-only log-prob was fine). Clone repeated
size/offset vectors (tiny int32) before apply.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant