Support TP-sharded matrices in DistributedMuon - #4051
Closed
weifengpy wants to merge 1 commit into
Closed
Conversation
weifengpy
force-pushed
the
weifeng/distributed-muon-standalone
branch
from
August 2, 2026 02:18
f1276f2 to
8dcadb4
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-tp-shard1
branch
from
August 2, 2026 02:18
bd08427 to
2dc8b90
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-standalone
branch
from
August 2, 2026 17:43
8dcadb4 to
740421b
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-tp-shard1
branch
from
August 2, 2026 17:43
2dc8b90 to
5735fef
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-standalone
branch
from
August 3, 2026 01:22
740421b to
bd0726b
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-tp-shard1
branch
from
August 3, 2026 01:22
5735fef to
ee259fc
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-standalone
branch
from
August 3, 2026 02:06
bd0726b to
fe03766
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-tp-shard1
branch
from
August 3, 2026 02:06
ee259fc to
c194c06
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-standalone
branch
from
August 3, 2026 18:28
fe03766 to
0a2d554
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-tp-shard1
branch
from
August 3, 2026 18:28
c194c06 to
325bb10
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-standalone
branch
from
August 3, 2026 18:31
0a2d554 to
0e7805d
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-tp-shard1
branch
from
August 3, 2026 18:31
325bb10 to
159d476
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-standalone
branch
from
August 3, 2026 20:57
0e7805d to
8b5068d
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-tp-shard1
branch
from
August 3, 2026 20:57
159d476 to
084f8c6
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-standalone
branch
from
August 3, 2026 21:14
8b5068d to
4c62cea
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-tp-shard1
branch
2 times, most recently
from
August 3, 2026 21:24
0fb38e1 to
74cf3e6
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-standalone
branch
2 times, most recently
from
August 3, 2026 21:30
19645aa to
3787193
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-tp-shard1
branch
from
August 3, 2026 21:30
74cf3e6 to
fe008f0
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-standalone
branch
from
August 3, 2026 21:47
3787193 to
93f15a5
Compare
weifengpy
force-pushed
the
weifeng/distributed-muon-tp-shard1
branch
from
August 3, 2026 21:47
fe008f0 to
f0ff6c6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
Stacked on #4034.
Summary
Owned()compute to rank-2 matrices stored on a named 2D mesh with exactShard(0), Shard(1)placementsBucketSpec; the generic Add standalone distributed Muon optimizer #4034 runtime derives uneven rectangular storage blocks from each DTensor mesh coordinateThe extension is intentionally limited to uniquely tiled
Shard(0), Shard(1)storage. TP-replicatedShard(0), Replicate()storage and_StridedShardredistribution remain out of scope.Test plan
5x7and6x5matrices on a named2x2FSDP/TP mesh1 passed)