Commit 30bc705
committed
ggml-cann: implement quantized MUL_MAT_ID for MoE models
Implement ggml_cann_mul_mat_id_quant function to support quantized matrix
multiplication for Mixture of Experts (MoE) architectures on CANN backend.
Key features:
- Support Q4_0 and Q8_0 quantized weight formats
- Use IndexSelect to dynamically route expert-specific weights based on indices
- Leverage WeightQuantBatchMatmulV2 for efficient quantized computation
- Handle automatic F16 type conversion for hardware compatibility
- Support both per-expert and broadcast input modes
Implementation details:
- Extract expert weights and scales using CANN IndexSelect operation
- Process each batch and expert combination independently
- Create proper tensor views with correct stride for matmul operations
- Automatic input/output type casting to/from F16 as needed
Code quality improvements:
- Clear variable naming (n_batches, n_experts, batch_idx, expert_idx)
- Reusable lambda function for F16 buffer preparation
- Simplified array initialization and memory layout calculations
- Comprehensive inline documentation
Testing: All 297 test cases passed for supported types (F32, F16, Q4_0, Q8_0)
across various configurations (different n_mats, n_used, batch parameters).1 parent b1a5754 commit 30bc705
1 file changed
Lines changed: 174 additions & 243 deletions
0 commit comments