Skip to content

Commit 30bc705

Browse files
committed
ggml-cann: implement quantized MUL_MAT_ID for MoE models
Implement ggml_cann_mul_mat_id_quant function to support quantized matrix multiplication for Mixture of Experts (MoE) architectures on CANN backend. Key features: - Support Q4_0 and Q8_0 quantized weight formats - Use IndexSelect to dynamically route expert-specific weights based on indices - Leverage WeightQuantBatchMatmulV2 for efficient quantized computation - Handle automatic F16 type conversion for hardware compatibility - Support both per-expert and broadcast input modes Implementation details: - Extract expert weights and scales using CANN IndexSelect operation - Process each batch and expert combination independently - Create proper tensor views with correct stride for matmul operations - Automatic input/output type casting to/from F16 as needed Code quality improvements: - Clear variable naming (n_batches, n_experts, batch_idx, expert_idx) - Reusable lambda function for F16 buffer preparation - Simplified array initialization and memory layout calculations - Comprehensive inline documentation Testing: All 297 test cases passed for supported types (F32, F16, Q4_0, Q8_0) across various configurations (different n_mats, n_used, batch parameters).
1 parent b1a5754 commit 30bc705

1 file changed

Lines changed: 174 additions & 243 deletions

File tree

0 commit comments

Comments
 (0)