[examples] Add a variable-length KDA forward - #3289
Draft
tarinduj wants to merge 1 commit into
Draft
Conversation
FLA, vLLM and FlashKDA all take a variable-length batch the same way: one tensor of all the tokens plus `cu_seqlens` marking the boundaries. The engine had no way to express it, so a ragged batch had to be padded to its longest sequence. `helion_chunk_kda` now takes `cu_seqlens` and runs the batch as it is. Forward only: FlashKDA implements no backward, and vLLM is an inference engine. - **`cu_seqlens`** switches `helion_chunk_kda` to a varlen batch: inputs token-major `[1, T_total, H, D]`, state `[N, H, D, DV]` one per sequence, and `state_v_first` for the `[N, H, DV, D]` vLLM and FlashKDA hold. - **Five varlen kernels**, one per stage of the dense `diag_anchored` pipeline. - **`chunk_fwd_h_delta_varlen_helion`** runs the serial state pass as one program per `(head, sequence)`, looping over that sequence's chunks `chunk_offsets[n] : chunk_offsets[n + 1]`. The state is initialized inside that loop, so each sequence starts from its own `h0` and writes its own final state. - **`naive_recurrent_reference`** takes `cu_seqlens` and resets its state at each boundary. - **`l2norm_fwd_helion`** tiles one axis over a `[N, D]` view instead of three over `[B, T, H, D]`. `test_linear_kda_varlen` runs the varlen path through the same harness as `test_linear_kda`. A `kda_varlen` variant in `benchmarks/run_linattn.py` benchmarks it on FlashKDA's sequence partitions rather than the dense grid, and is forward-only, so it emits no `-bwd` dashboard row. `%FLA = 100 * fla_ms / helion_ms` on H100, autotuned with `HELION_AUTOTUNE_EFFORT=full`. **Higher is faster than FLA.** Shapes (`<partition>_T*_H*_D*` encodes the sequence lengths, total tokens, heads, head dim), from FlashKDA's `benchmarks/bench_fwd.py`: - `fixed_T8192_H96_D128`, one sequence: `[8192]` - `fixed_T8192_H64_D128`, one sequence: `[8192]` - `ragged_T8192_H96_D128`, six sequences: `[1300, 547, 2048, 963, 271, 3063]` - `ragged_T8192_H64_D128`, six sequences: `[1300, 547, 2048, 963, 271, 3063]` - `uniform_T8192_H96_D128`, eight sequences: `[1024] * 8` - `uniform_T8192_H64_D128`, eight sequences: `[1024] * 8` Forward: | variant | fixed_T8192_H96_D128 | fixed_T8192_H64_D128 | ragged_T8192_H96_D128 | ragged_T8192_H64_D128 | uniform_T8192_H96_D128 | uniform_T8192_H64_D128 | |---|---|---|---|---|---|---| | kda_varlen | 101% | 90% | 95% | 89% | 99% | 90% | stack-info: PR: #3289, branch: tarinduj/stack/3
tarinduj
force-pushed
the
tarinduj/stack/3
branch
from
August 5, 2026 22:34
a804a31 to
7e39307
Compare
tarinduj
force-pushed
the
tarinduj/stack/2
branch
from
August 5, 2026 22:34
bddd4bf to
7f77070
Compare
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PRs:
[examples] Add a variable-length KDA forward
FLA, vLLM and FlashKDA all take a variable-length batch the same way: one tensor of all the tokens plus
cu_seqlensmarking the boundaries. The engine had no way to express it, so a ragged batch had to be padded to its longest sequence.helion_chunk_kdanow takescu_seqlensand runs the batch as it is. Forward only: FlashKDA implements no backward, and vLLM is an inference engine.cu_seqlensswitcheshelion_chunk_kdato a varlen batch: inputs token-major[1, T_total, H, D], state[N, H, D, DV]one per sequence, andstate_v_firstfor the[N, H, DV, D]vLLM and FlashKDA hold.diag_anchoredpipeline.chunk_fwd_h_delta_varlen_helionruns the serial state pass as one program per(head, sequence), looping over that sequence's chunkschunk_offsets[n] : chunk_offsets[n + 1]. The state is initialized inside that loop, so each sequence starts from its ownh0and writes its own final state.naive_recurrent_referencetakescu_seqlensand resets its state at each boundary.l2norm_fwd_heliontiles one axis over a[N, D]view instead of three over[B, T, H, D].test_linear_kda_varlenruns the varlen path through the same harness astest_linear_kda. Akda_varlenvariant inbenchmarks/run_linattn.pybenchmarks it on FlashKDA's sequence partitions rather than the dense grid, and is forward-only, so it emits no-bwddashboard row.%FLA = 100 * fla_ms / helion_mson H100, autotuned withHELION_AUTOTUNE_EFFORT=full. Higher is faster than FLA.Shapes (
<partition>_T*_H*_D*encodes the sequence lengths, total tokens, heads, head dim), from FlashKDA'sbenchmarks/bench_fwd.py:fixed_T8192_H96_D128, one sequence:[8192]fixed_T8192_H64_D128, one sequence:[8192]ragged_T8192_H96_D128, six sequences:[1300, 547, 2048, 963, 271, 3063]ragged_T8192_H64_D128, six sequences:[1300, 547, 2048, 963, 271, 3063]uniform_T8192_H96_D128, eight sequences:[1024] * 8uniform_T8192_H64_D128, eight sequences:[1024] * 8Forward: