[ROCm][CI] Skip cp_spmd_types on ROCm pending upstream spmd_types fix

[rishisinhanj]

cp_spmd_types deadlocks on ROCm and is the only failing test in the "8 GPU Feature Tests" suite — the other 32 of 33 spmd tests pass. It blocks re-enabling the ROCm leg in #4039.

Root cause is upstream in spmd_types, written up in #4064: its redistribution collectives are issued as raw blocking calls, so with async_op=False the NCCL kernel lands on the caller's stream unfenced. When cp_degree == world_size the CP axis resolves to default_pg, and those collectives then race FSDP's on the same communicator.

This skips that one test on ROCm for now as a temporary measure. It is not a fix, and the comment points at #4064 so it can be removed once that lands.

ROCm keeps CP coverage at cp=2 through fsdp+cp, hsdp+cp_without_dp_shard, hsdp+cp_with_dp_shard, fsdp+tp+cp and validation_tp_cp_pp. What it loses is cp=4 and the CP-alone topology, which is the configuration that deadlocks.

[pytorch-bot]

Unknown label ciflow/rocm.
Currently recognized labels are

• ciflow/8gpu
• ciflow/h100.8
• ciflow/rl

[pytorch-bot]

The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:

• ciflow/rocm
• ciflow/8gpu

Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows.

[pytorch-bot]

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: argument command: invalid choice: 'add' (choose from 'merge', 'revert', 'rebase', 'label', 'drci', 'lint', 'fix-lint', 'apply-lint', 'cherry-pick')

usage: @pytorchbot [-h]
                   
                   {merge,revert,rebase,label,drci,lint,fix-lint,apply-lint,cherry-pick}
                   ...

Try @pytorchbot --help for more info.

[tianyu-l]

what's the value add of keeping rocm tests in torchtitan vs. keep them in your own setup?

[yuankaichen-amd]

@tianyu-l The failure was caused by the introduction of spmd. I guess it was merged once it passed cuda CI.

Ideally we should make rocm CI block PRs that break it. What do you think?

[tianyu-l]

Ideally we should make rocm CI block PRs that break it. What do you think?

If cuda CI passes, and rocm CI fails, is there anything meaningful that developers could do?

[yuankaichen-amd]

Ideally we should make rocm CI block PRs that break it. What do you think?

If cuda CI passes, and rocm CI fails, is there anything meaningful that developers could do?

Same as if PR fails cuda CI?

We have dedicated engineers to watch and maintain torchtitan's rocm CI. On the other hand, we welcome community developers reach out to us when they discover issues like this, before CI breakage if possible.

We'd definitely love rocm CI to stay in torchtitan so that the public can see rocm support is there (and we will put in our best efforts to keep it green!). Let us know if there is anything we can do to make the rocm support better.

[tianyu-l]

Same as if PR fails cuda CI?

I think it's quite different. E.g. our team don't develop with rocm. Whenever some submit a PR, it usually already works with cuda. The point is that we should slow down development just because rocm CI is not green (just like how this PR helps).

Note that this is NOT saying that we don't care about rocm, or we think titan x rocm doesn't bring value.