Skip to content

Fix HIP 2^32 grid-overflow in batch_index_select_dim0 forward kernels (#6053) - #6053

Open
cx-yin wants to merge 1 commit into
pytorch:mainfrom
cx-yin:export-D113306267
Open

Fix HIP 2^32 grid-overflow in batch_index_select_dim0 forward kernels (#6053)#6053
cx-yin wants to merge 1 commit into
pytorch:mainfrom
cx-yin:export-D113306267

Conversation

@cx-yin

@cx-yin cx-yin commented Jul 23, 2026

Copy link
Copy Markdown

Summary:

X-link: https://github.com/facebookresearch/FBGEMM/pull/2953

The batch_index_select_dim0 forward kernel crashes on AMD/ROCm for large workloads. MAST job fire-changbin-train-UFM-10EBF_full_length-4d036 failed with:
[batch_index_select_dim0_codegen_forward_kernel] [grid dim 10773753 x 1 x 1] [block dim 64 x 8 x 1]:
Total number of threads 5516161536 is greater than the HIP limit of 2^32

Mirror the D94944619 cap-plus-grid-stride pattern for the index-select forward:

  1. Launch sites (embedding_forward_split_template.cu): wrap both the small- and main-kernel grids in utils::cuda::cap_grid_dim_x(..., BlockCapPolicy::OverflowOnly), gated on is_index_select.
  2. Kernels (embedding_forward_split_kernel_template.cu, ..._nobag_small_template.cu): add an #ifdef USE_ROCM grid-stride loop for the is_index_select path; convert t >= T -> break (t is monotonic in b_t) and L_start >= total_L -> continue.

Also complete the grid-x cap for permute_multi_embs_kernel (permute_multi_embedding_ops.cu), which shares this ROCm CI test suite. The prior cap (PR #6040) passed only kMaxThreads as the per-grid-x thread count, ignoring the non-grid-strided batch_id axis (gridDim.y * gridDim.z). For large permute_size with batch_size > 1 the launch still exceeded 2^32 (test_permute_multi_embedding_large_grid: grid 262145 x 32 x 1, block 64 x 16 x 1 -> 8.59e9 threads). Fold grid_dim_y * grid_dim_z into the cap's per-grid-x thread count, matching the sparse_index_add.cu / jagged_softmax precedent. This unblocks the ROCm CI test suite.

Reviewed By: q10

Differential Revision: D113306267

@meta-cla meta-cla Bot added the cla signed label Jul 23, 2026
@meta-codesync

meta-codesync Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@cx-yin has exported this pull request. If you are a Meta employee, you can view the originating Diff in D113306267.

@meta-codesync meta-codesync Bot changed the title Fix HIP 2^32 grid-overflow in batch_index_select_dim0 forward kernels Fix HIP 2^32 grid-overflow in batch_index_select_dim0 forward kernels (#6053) Jul 24, 2026
@cx-yin
cx-yin force-pushed the export-D113306267 branch from c8d0c7b to 0cb1032 Compare July 24, 2026 00:19
cx-yin pushed a commit to cx-yin/FBGEMM-1 that referenced this pull request Jul 24, 2026
…pytorch#6053)

Summary:

X-link: facebookresearch/FBGEMM#2953

The batch_index_select_dim0 forward kernel crashes on AMD/ROCm for large workloads. MAST job fire-changbin-train-UFM-10EBF_full_length-4d036 failed with:
[batch_index_select_dim0_codegen_forward_kernel] [grid dim 10773753 x 1 x 1] [block dim 64 x 8 x 1]:
Total number of threads 5516161536 is greater than the HIP limit of 2^32

Mirror the D94944619 cap-plus-grid-stride pattern for the index-select forward:
1. Launch sites (embedding_forward_split_template.cu): wrap both the small- and main-kernel grids in utils::cuda::cap_grid_dim_x(..., BlockCapPolicy::OverflowOnly), gated on is_index_select.
2. Kernels (embedding_forward_split_kernel_template.cu, ..._nobag_small_template.cu): add an #ifdef USE_ROCM grid-stride loop for the is_index_select path; convert t >= T → break (t is monotonic in b_t) and L_start >= total_L → continue.

Reviewed By: q10

Differential Revision: D113306267
cx-yin pushed a commit to cx-yin/FBGEMM-1 that referenced this pull request Jul 24, 2026
…pytorch#6053)

Summary:

X-link: facebookresearch/FBGEMM#2953

The batch_index_select_dim0 forward kernel crashes on AMD/ROCm for large workloads. MAST job fire-changbin-train-UFM-10EBF_full_length-4d036 failed with:
[batch_index_select_dim0_codegen_forward_kernel] [grid dim 10773753 x 1 x 1] [block dim 64 x 8 x 1]:
Total number of threads 5516161536 is greater than the HIP limit of 2^32

Mirror the D94944619 cap-plus-grid-stride pattern for the index-select forward:
1. Launch sites (embedding_forward_split_template.cu): wrap both the small- and main-kernel grids in utils::cuda::cap_grid_dim_x(..., BlockCapPolicy::OverflowOnly), gated on is_index_select.
2. Kernels (embedding_forward_split_kernel_template.cu, ..._nobag_small_template.cu): add an #ifdef USE_ROCM grid-stride loop for the is_index_select path; convert t >= T → break (t is monotonic in b_t) and L_start >= total_L → continue.

Reviewed By: q10

Differential Revision: D113306267
@cx-yin
cx-yin force-pushed the export-D113306267 branch from 0cb1032 to 60bc71c Compare July 24, 2026 00:21
…pytorch#6053)

Summary:

X-link: facebookresearch/FBGEMM#2953

The batch_index_select_dim0 forward kernel crashes on AMD/ROCm for large workloads. MAST job fire-changbin-train-UFM-10EBF_full_length-4d036 failed with:
[batch_index_select_dim0_codegen_forward_kernel] [grid dim 10773753 x 1 x 1] [block dim 64 x 8 x 1]:
Total number of threads 5516161536 is greater than the HIP limit of 2^32

Mirror the D94944619 cap-plus-grid-stride pattern for the index-select forward:
1. Launch sites (embedding_forward_split_template.cu): wrap both the small- and main-kernel grids in utils::cuda::cap_grid_dim_x(..., BlockCapPolicy::OverflowOnly), gated on is_index_select.
2. Kernels (embedding_forward_split_kernel_template.cu, ..._nobag_small_template.cu): add an #ifdef USE_ROCM grid-stride loop for the is_index_select path; convert t >= T -> break (t is monotonic in b_t) and L_start >= total_L -> continue.

Also complete the grid-x cap for permute_multi_embs_kernel (permute_multi_embedding_ops.cu), which shares this ROCm CI test suite. The prior cap (PR pytorch#6040) passed only kMaxThreads as the per-grid-x thread count, ignoring the non-grid-strided batch_id axis (gridDim.y * gridDim.z). For large permute_size with batch_size > 1 the launch still exceeded 2^32 (test_permute_multi_embedding_large_grid: grid 262145 x 32 x 1, block 64 x 16 x 1 -> 8.59e9 threads). Fold grid_dim_y * grid_dim_z into the cap's per-grid-x thread count, matching the sparse_index_add.cu / jagged_softmax precedent. This unblocks the ROCm CI test suite.

Reviewed By: q10

Differential Revision: D113306267
@cx-yin
cx-yin force-pushed the export-D113306267 branch from 60bc71c to 569aa30 Compare July 24, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants