Skip to content

[Pallas] Sample TPU bridge shapes equally-spaced (cap kl_div / rms_norm to feasible shapes) - #3284

Open
norx1991 wants to merge 1 commit into
mainfrom
yifeixu/tpu-bridge-equally-spaced
Open

[Pallas] Sample TPU bridge shapes equally-spaced (cap kl_div / rms_norm to feasible shapes)#3284
norx1991 wants to merge 1 commit into
mainfrom
yifeixu/tpu-bridge-equally-spaced

Conversation

@norx1991

@norx1991 norx1991 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

The TPU bridge sampled only --num-inputs 2 with the default first-k mode, so the dashboard showed just the two smallest shapes per kernel. This samples 6 shapes equally-spaced across each operator's range (matching the GPU nightly's --input-sample-mode equally-spaced-k), so small/mid/large are all represented.

Two kernels take the feasible low end via first-k instead of spanning to the top, because their largest sampled shape isn't runnable on TPU:

  • kl_div: at fp32 its reduction dim reaches V=131072, which OOMs TPU vmem and autotunes for hours → first-k 4 (V≤32768).
  • rms_norm / rms_norm-bwd: their largest shape (H=32768) sits ~66 KB over the 32 MB scoped-vmem ceiling, and helion's autotuner bails with "No working config found" → first-k 5 (H≤16384).

Validated via manual TPU-bridge dispatch: with these caps all bridged kernels run under the 6h job cap.

The rms_norm H=32768 cap is temporary. Root cause is a helion autotuner limitation (the autotune search abandons when its initial configs all miss the scoped-vmem ceiling by a hair, rather than finding a config that fits); a separate PR adds an autotune_baseline_fn for rms_norm (removes the earlier "Default config failed while computing baseline" crash, a prerequisite), and the autotuner search-robustness fix is tracked separately. Once that lands, the H≤16384 cap can be lifted.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 4, 2026
@norx1991
norx1991 force-pushed the yifeixu/tpu-bridge-equally-spaced branch from c7b7513 to 78cca00 Compare August 4, 2026 23:15
@norx1991 norx1991 changed the title [Pallas] Sample TPU bridge shapes equally-spaced (kl_div capped to feasible small-V) [Pallas] Sample TPU bridge shapes equally-spaced (cap kl_div / rms_norm to feasible shapes) Aug 4, 2026
@norx1991
norx1991 force-pushed the yifeixu/tpu-bridge-equally-spaced branch from 78cca00 to c1e777d Compare August 5, 2026 21:45
@norx1991
norx1991 marked this pull request as ready for review August 5, 2026 21:46
@norx1991
norx1991 requested review from AmesingFlank and ethche August 5, 2026 21:58
…rm to feasible shapes)

The bridge ran --num-inputs 2 first-k, so the dashboard showed only the two
smallest shapes per kernel. Sample 6 shapes equally-spaced across each
operator's range so small/mid/large are represented, matching the GPU nightly.

Two kernels take the feasible low end via first-k instead:
- kl_div: at fp32 its reduction reaches V=131072, which OOMs TPU vmem and
  autotunes for hours (first-k 4, V<=32768).
- rms_norm / rms_norm-bwd: their largest shape H=32768 sits ~66KB over the 32MB
  scoped-vmem ceiling and helion's autotuner bails (no working config found), so
  keep H<=16384 (first-k 5) until the autotuner handles it.
@norx1991
norx1991 force-pushed the yifeixu/tpu-bridge-equally-spaced branch from c1e777d to ac5e172 Compare August 5, 2026 22:20
@norx1991

norx1991 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Welford is having some timeout issue in a trial dispatch. Need to hold on this before figuring it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants