Skip to content

Add debugger and CANN profiler support for the Ascend backend#786

Open
ChijinZ wants to merge 1 commit into
flagos-ai:triton_v3.5.xfrom
ChijinZ:pr/ascend-debugger-profiler
Open

Add debugger and CANN profiler support for the Ascend backend#786
ChijinZ wants to merge 1 commit into
flagos-ai:triton_v3.5.xfrom
ChijinZ:pr/ascend-debugger-profiler

Conversation

@ChijinZ

@ChijinZ ChijinZ commented Jul 13, 2026

Copy link
Copy Markdown

This PR adds integrated debugger and CANN profiler support for the Ascend backend.

It introduces compiler-level instrumentation, runtime data collection, debugger report generation, and CANN-backed Proton profiling. It also adds
automatic profiler instrumentation so Triton kernels can expose internal execution details in the generated timeline without requiring manual
instrumentation in user kernels.

Debugger

  • Add debugger instrumentation and metadata MLIR passes.
  • Add runtime buffers and asynchronous export support for Ascend.
  • Add decoding and report generation for:
    • summary records
    • memory accesses
    • full-value references
    • statement-level execution records
  • Add Python APIs for debugger configuration and kernel launch integration.
  • Support level 1 and level 2 data collection.
  • Export text reports, JSON reports, operation logs, raw records, and optional NumPy artifacts.
  • Add two curated regression samples:
    • pointwise abs
    • reduction softmax

Profiler

  • Add a CANN vendor adapter to Proton.
  • Support CANN runtime, AICore, bandwidth, MSTX, and internal kernel timeline data.
  • Add Proton artifacts for metadata, vendor associations, and Chrome-compatible timelines.
  • Add automatic Triton IR instrumentation for profiler sessions.
  • Preserve and restore existing Triton launch hooks when profiler sessions start and stop.
  • Support explicit CANN device selection through device_id.
  • Add host-timing fallback when vendor runtime metrics are unavailable.

Instrumentation Buffer

The profiler IR record buffer can be configured with:

export PROTON_IR_RECORD_BUFFER_MB=<size-in-MB>

The default size is 32 MB.

The configured capacity is included in the compilation instrumentation mode, so kernels compiled with different buffer capacities use separate cache
entries.

## Build Integration

- Add debugger libraries and Python bindings to the FlagTree build.
- Integrate debugger passes into Triton and Ascend compiler pipelines.
- Integrate the CANN profiler into Proton.
- Keep debugger support configurable through FLAGTREE_ENABLE_DEBUGGER.
- Exclude generated kernels, runtime reports, caches, and local build dependencies from Git.

## Validation

The changes were validated in an Ascend CANN container.

- Ascend wheel build and installation passed.
- Ruff, YAPF, clang-format, AST, whitespace, and repository hygiene checks passed.
- Python tests: 47 passed, 2 skipped.
- Debugger lit tests: 10 passed.
- Debugger C++ tests: 45 passed.
- Real debugger abs example passed on Ascend NPU.
- Real profiler tiny_mlp example passed on Ascend NPU.
- The generated MLP timeline contained 37,500 events, including internal records for both linear kernels and the ReLU kernel.

## Example Outputs

The profiler produces:

profile.hatchet
profile.timeline.json
profile.meta.json
profile.vendor.json

The debugger produces timestamped text and JSON reports with optional operation logs, raw records, and full-value artifacts.

@ChijinZ ChijinZ requested a review from zhzhcookie as a code owner July 13, 2026 14:07
@CLAassistant

CLAassistant commented Jul 13, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants