Skip to content

fix: handle Ascend paged flash attention layout - #984

Merged
voltjia merged 1 commit into
masterfrom
fix/ascend-paged-flash-attn
Sep 10, 2026
Merged

voltjia merged 1 commit into
masterfrom
fix/ascend-paged-flash-attn

Conversation

@baominghelly

@baominghelly baominghelly commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Convert paged K/V ACL descriptors from the public BnBsND layout to the CANN-required BnBsH layout in src/native/ascend/ops/flash_attn_varlen_func/kernel.h.
  • Keep query and dense K/V sequence lengths as cumulative endpoints while passing paged K/V lengths as per-batch lengths.
  • Add FP16/BF16, causal/non-causal, GQA, multi-block, and reordered-block-table coverage in tests/test_flash_attn_varlen_func.py.

Motivation

Ascend paged prefill could produce corrupted output when InfiniLM passed a nontrivial block table. CANN PagedAttention expects the cache layout and length convention used here, while InfiniOps publicly preserves its existing paged tensor contract. No issue is currently linked to this change.

Type of Change

  • feat — new feature / new operator / new platform
  • fix — bug fix
  • perf — performance improvement (no behavioral change)
  • refactor — code restructuring without behavior change
  • test — adding or fixing tests only
  • docs — documentation only
  • build / ci — build system or CI configuration
  • chore — tooling, formatting, or other non-code changes
  • Breaking change (requires a ! in the Conventional Commits prefix or a BREAKING CHANGE: footer)

Platforms Affected

  • CPU (WITH_CPU)
  • NVIDIA (WITH_NVIDIA)
  • Iluvatar (WITH_ILUVATAR)
  • MetaX (WITH_METAX)
  • Cambricon (WITH_CAMBRICON)
  • Moore (WITH_MOORE)
  • Ascend (WITH_ASCEND)
  • PyTorch C++ bindings (WITH_TORCH)
  • Build system / CMake / CI
  • Python bindings / user-facing API

Smoke Test Result

Ascend build/install:
cmake --build /workspace/build/InfiniOps --target infiniops ops -j16
cmake --install /workspace/build/InfiniOps

Ascend affected test:
27 passed, 9 skipped in 2.22s

Command:
PYTHONPATH=/workspace/prefix python -c 'import sys,types; m=types.ModuleType("infini"); m.__path__=["/workspace/prefix"]; sys.modules["infini"]=m; import pytest; raise SystemExit(pytest.main(["tests/test_flash_attn_varlen_func.py","--devices","npu","-q"]))'

Test Results on Supported Platforms

Platform Affected Build / Smoke Result Full Result / Notes
NVIDIA No N/A - not affected N/A - not affected
Iluvatar No N/A - not affected N/A - not affected
MetaX No N/A - not affected N/A - not affected
Cambricon No N/A - not affected N/A - not affected
Moore No N/A - not affected N/A - not affected
Ascend Yes build and affected tests passed 27 passed, 9 skipped; eager ChatGLM3 paged flash-attn inference produced coherent output
Full `pytest` output (optional)
.....s.....s.....s.....s.....s.s.sss                                     [100%]
27 passed, 9 skipped, 14 warnings in 2.22s

Benchmark / Performance Impact

N/A

Notes for Reviewers

  • The InfiniOps public tensor remains [block_num, block_size, kv_heads, head_dim]; only the ACL descriptor flattens the last two dimensions without copying data.
  • Dense K/V still uses cumulative endpoints. Paged K/V uses each batch's actual length, matching CANN PagedAttention.
  • Graph-mode corruption is tracked separately in InfiniLM and is intentionally outside this operator-layout fix.
  • Final e2e log: /workspace/model-matrix-route-a-20260910/final-pr-smoke-chatglm.log on the Ascend 910C test container.

@baominghelly
baominghelly marked this pull request as ready for review September 10, 2026 07:44
@baominghelly
baominghelly requested review from a team and voltjia September 10, 2026 07:44
@voltjia
voltjia merged commit a612093 into master Sep 10, 2026
12 of 20 checks passed
@voltjia
voltjia deleted the fix/ascend-paged-flash-attn branch September 10, 2026 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants