Problem
tests/test_matmul.py::TestNativeMatmulOpBatchInvariance::test_batch_grad_invariance intermittently fails in GPU CI, blocking unrelated kernel PRs.
Why This Is Incorrect
The two executions are mathematically equivalent:
full batch:
M = B × S
per-row batch:
M = S
However, native PyTorch GEMM does not guarantee bitwise-identical floating-point results when matrix shapes, tiling, backend kernel selection, or accumulation order differ.
Even with CPU execution limited to one thread, splitting the batch changes the GEMM M dimension and can change floating-point reduction order.
The test inputs are CPU tensors, so the failure depends on the CI pod host CPU and BLAS/GEMM dispatch rather than the requested GPU architecture.
Impact
- GPU CI can fail on unrelated PRs before their relevant tests are evaluated.
Problem
tests/test_matmul.py::TestNativeMatmulOpBatchInvariance::test_batch_grad_invarianceintermittently fails in GPU CI, blocking unrelated kernel PRs.Why This Is Incorrect
The two executions are mathematically equivalent:
full batch:
M = B × S
per-row batch:
M = S
However, native PyTorch GEMM does not guarantee bitwise-identical floating-point results when matrix shapes, tiling, backend kernel selection, or accumulation order differ.
Even with CPU execution limited to one thread, splitting the batch changes the GEMM M dimension and can change floating-point reduction order.
The test inputs are CPU tensors, so the failure depends on the CI pod host CPU and BLAS/GEMM dispatch rather than the requested GPU architecture.
Impact