Skip to content

Support TP-sharded matrices in DistributedMuon - #4051

Closed
weifengpy wants to merge 1 commit into
weifeng/distributed-muon-standalonefrom
weifeng/distributed-muon-tp-shard1
Closed

Support TP-sharded matrices in DistributedMuon#4051
weifengpy wants to merge 1 commit into
weifeng/distributed-muon-standalonefrom
weifeng/distributed-muon-tp-shard1

Conversation

@weifengpy

@weifengpy weifengpy commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

Stacked on #4034.

Summary

  • extend Add standalone distributed Muon optimizer #4034 Owned() compute to rank-2 matrices stored on a named 2D mesh with exact Shard(0), Shard(1) placements
  • use the exact 1D redistribution mesh supplied by BucketSpec; the generic Add standalone distributed Muon optimizer #4034 runtime derives uneven rectangular storage blocks from each DTensor mesh coordinate
  • reconstruct each full matrix on its assigned owner, run Muon once, and return update blocks through the reverse all-to-all
  • preserve the original parameter, gradient, and momentum DTensor placements

The extension is intentionally limited to uniquely tiled Shard(0), Shard(1) storage. TP-replicated Shard(0), Replicate() storage and _StridedShard redistribution remain out of scope.

Test plan

  • four-GPU parity test with uneven 5x7 and 6x5 matrices on a named 2x2 FSDP/TP mesh
  • verifies one forward and one reverse all-to-all for the bucket, matches plain Muon numerically, and preserves parameter/momentum placements (1 passed)

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 1, 2026
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-standalone branch from f1276f2 to 8dcadb4 Compare August 2, 2026 02:18
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-tp-shard1 branch from bd08427 to 2dc8b90 Compare August 2, 2026 02:18
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-standalone branch from 8dcadb4 to 740421b Compare August 2, 2026 17:43
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-tp-shard1 branch from 2dc8b90 to 5735fef Compare August 2, 2026 17:43
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-standalone branch from 740421b to bd0726b Compare August 3, 2026 01:22
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-tp-shard1 branch from 5735fef to ee259fc Compare August 3, 2026 01:22
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-standalone branch from bd0726b to fe03766 Compare August 3, 2026 02:06
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-tp-shard1 branch from ee259fc to c194c06 Compare August 3, 2026 02:06
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-standalone branch from fe03766 to 0a2d554 Compare August 3, 2026 18:28
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-tp-shard1 branch from c194c06 to 325bb10 Compare August 3, 2026 18:28
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-standalone branch from 0a2d554 to 0e7805d Compare August 3, 2026 18:31
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-tp-shard1 branch from 325bb10 to 159d476 Compare August 3, 2026 18:31
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-standalone branch from 0e7805d to 8b5068d Compare August 3, 2026 20:57
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-tp-shard1 branch from 159d476 to 084f8c6 Compare August 3, 2026 20:57
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-standalone branch from 8b5068d to 4c62cea Compare August 3, 2026 21:14
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-tp-shard1 branch 2 times, most recently from 0fb38e1 to 74cf3e6 Compare August 3, 2026 21:24
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-standalone branch 2 times, most recently from 19645aa to 3787193 Compare August 3, 2026 21:30
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-tp-shard1 branch from 74cf3e6 to fe008f0 Compare August 3, 2026 21:30
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-standalone branch from 3787193 to 93f15a5 Compare August 3, 2026 21:47
@weifengpy
weifengpy force-pushed the weifeng/distributed-muon-tp-shard1 branch from fe008f0 to f0ff6c6 Compare August 3, 2026 21:47
@weifengpy weifengpy closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/8gpu CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant