[Pallas] Sample TPU bridge shapes equally-spaced (cap kl_div / rms_norm to feasible shapes) - #3284
Open
norx1991 wants to merge 1 commit into
Open
[Pallas] Sample TPU bridge shapes equally-spaced (cap kl_div / rms_norm to feasible shapes)#3284norx1991 wants to merge 1 commit into
norx1991 wants to merge 1 commit into
Conversation
norx1991
force-pushed
the
yifeixu/tpu-bridge-equally-spaced
branch
from
August 4, 2026 23:15
c7b7513 to
78cca00
Compare
norx1991
force-pushed
the
yifeixu/tpu-bridge-equally-spaced
branch
from
August 5, 2026 21:45
78cca00 to
c1e777d
Compare
norx1991
marked this pull request as ready for review
August 5, 2026 21:46
AmesingFlank
approved these changes
Aug 5, 2026
…rm to feasible shapes) The bridge ran --num-inputs 2 first-k, so the dashboard showed only the two smallest shapes per kernel. Sample 6 shapes equally-spaced across each operator's range so small/mid/large are represented, matching the GPU nightly. Two kernels take the feasible low end via first-k instead: - kl_div: at fp32 its reduction reaches V=131072, which OOMs TPU vmem and autotunes for hours (first-k 4, V<=32768). - rms_norm / rms_norm-bwd: their largest shape H=32768 sits ~66KB over the 32MB scoped-vmem ceiling and helion's autotuner bails (no working config found), so keep H<=16384 (first-k 5) until the autotuner handles it.
norx1991
force-pushed
the
yifeixu/tpu-bridge-equally-spaced
branch
from
August 5, 2026 22:20
c1e777d to
ac5e172
Compare
Contributor
Author
|
Welford is having some timeout issue in a trial dispatch. Need to hold on this before figuring it out. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The TPU bridge sampled only
--num-inputs 2with the default first-k mode, so the dashboard showed just the two smallest shapes per kernel. This samples 6 shapesequally-spacedacross each operator's range (matching the GPU nightly's--input-sample-mode equally-spaced-k), so small/mid/large are all represented.Two kernels take the feasible low end via
first-kinstead of spanning to the top, because their largest sampled shape isn't runnable on TPU:kl_div: at fp32 its reduction dim reaches V=131072, which OOMs TPU vmem and autotunes for hours →first-k 4(V≤32768).rms_norm/rms_norm-bwd: their largest shape (H=32768) sits ~66 KB over the 32 MB scoped-vmem ceiling, and helion's autotuner bails with "No working config found" →first-k 5(H≤16384).Validated via manual TPU-bridge dispatch: with these caps all bridged kernels run under the 6h job cap.
The
rms_normH=32768 cap is temporary. Root cause is a helion autotuner limitation (the autotune search abandons when its initial configs all miss the scoped-vmem ceiling by a hair, rather than finding a config that fits); a separate PR adds anautotune_baseline_fnfor rms_norm (removes the earlier "Default config failed while computing baseline" crash, a prerequisite), and the autotuner search-robustness fix is tracked separately. Once that lands, the H≤16384 cap can be lifted.