Conversation
7c36e2f to
077867b
Compare
BREAKING CHANGE: legacy InfiniCore runtime and inference APIs are removed in favor of InfiniRT, InfiniOps, and the migrated InfiniLM runtime.
Carry segmented P2P graph replay, graph-aware tensor initializers, runtime compatibility helpers, and long-context chunking into the InfiniLM-owned runtime. Preserve the stronger migrated graph ownership, cancellation, and allocation lease semantics while reconciling current InfiniLM attention behavior.
9439ea6 to
9d64e14
Compare
| size_t num_blocks_per_layer = config.num_blocks(); | ||
| size_t block_size = config.block_size(); | ||
|
|
||
| infinicore::Shape kv_shape; |
There was a problem hiding this comment.
这里相当于paged attention也使用了对flash attention友好的kv cache排布，paged attention算子是否有相关修改？
There was a problem hiding this comment.
这里不该修改的。
可能是之前的InfiniLM中cache创建，attn计算，cache更新，这三个行为的组织方式不好。
ai看着容易混淆了。
There was a problem hiding this comment.
后续可以尝试调整到一起去。
static，paged, flash三个attn计算，在自己命名空间里，都各自对应了一份cache创建，attn计算，cache更新的函数。
| @@ -97,8 +97,8 @@ std::tuple<infinicore::Tensor, infinicore::Tensor> FlashAttentionImpl::do_kv_cac | |||
| auto k_cache_layer = kv_cache->narrow({{0, 0, 1}})->squeeze(0); | |||
| auto v_cache_layer = kv_cache->narrow({{0, 1, 1}})->squeeze(0); | |||
| infinicore::op::paged_caching_( | |||
| k_cache_layer->permute({0, 2, 1, 3}), // permute to BHSD for paged_caching_ | |||
There was a problem hiding this comment.
这里是修改了paged caching算子所用的kv cache排布么？
There was a problem hiding this comment.
那这样修改后flash attention还能说话么
| {1}, infinicore::DataType::kInt32, | ||
| infinicore::Device{infinicore::Device::Type::kCpu}); | ||
| *reinterpret_cast<int32_t *>(last_token_shift_cpu->data()) = -1; | ||
| last_token_shift_ = last_token_shift_cpu->to(device); | ||
| } |
There was a problem hiding this comment.
is_last_pp_stage为true时，才初始化last_token_shift_ 。
那is_last_pp_stage为false时，下面也会使用，能保证各个平台last_token_shift_ 的默认值时0么？
应该为is_last_pp_stage为false时也执行，把last_token_shift_ 置0.
| end_offsets, last_token_shift_); | ||
| auto packed_hidden = hidden_states->view( | ||
| {hidden_states->size(1), hidden_states->size(2)}); | ||
| lm_head_input = infinicore::op::embedding( |
There was a problem hiding this comment.
infinicore::op::embedding的作用是把一个token id转为一个tensor向量。
修改后的pr，通过last_token_positions调用infinicore::op::embedding筛选hidden_states。这样做超出了infinicore::op::embedding算子的能力范围。反而没有之前的 infinicore::op::select_last_token_hidden_好。
这里为什么要修改。
There was a problem hiding this comment.
但貌似也行。
但感觉 infinicore::op::embedding出现在这里怪怪的。
如果没有比较合适的理由，还是建议用之前的 infinicore::op::select_last_token_hidden_。
| ->view({batch_size, seq_len, num_heads_ * value_head_dim}); // [bs, seq_len, n_q_head * value_head_dim] | ||
| } | ||
|
|
||
| infinicore::Tensor StaticAttentionImpl::forward_graph_( |
There was a problem hiding this comment.
这里的 StaticAttentionImpl::forward_graph_没有必要加。
之前的版本中， static attn 本身不支持graph。感觉为了强行支持，却调用了paged attention, 不太合适。
| @@ -40,14 +42,19 @@ infinicore::Tensor PagedAttentionImpl::forward(const AttentionLayer &layer, | |||
| const size_t value_head_dim = value->size(value->ndim() - 1); | |||
| infinicore::Tensor attn_output = infinicore::Tensor::empty({seq_len, num_heads_, value_head_dim}, query->dtype(), query->device()); | |||
| if (is_prefill) { | |||
There was a problem hiding this comment.
为什么把 infinicore::op::paged_attention_prefill_修改为infinicore::op::mha_varlen_算子。
paged_attention_prefill_不再使用了么
| // 3. Project down | ||
| // GateUpParallelLinear produces the packed [gate, up] layout expected here. | ||
| auto gate_up = gate_up_proj_->forward(hidden_states_mutable); | ||
| auto intermediate = infinicore::op::silu_and_mul(gate_up); |
There was a problem hiding this comment.
这里将 infinicore::op::swiglu修改为了 infinicore::op::silu_and_mul， 那 infinicore::op::silu_and_mul是各个平台都支持么？
为什么要换，是 infinicore::op::silu_and_mul算子性能更好么
There was a problem hiding this comment.
这里将 infinicore::op::swiglu修改为了 infinicore::op::silu_and_mul， 那 infinicore::op::silu_and_mul是各个平台都支持么？ 为什么要换，是 infinicore::op::silu_and_mul算子性能更好么
开源框架好像主流是用silu and mul
| } | ||
|
|
||
| return infinicore::op::linear_w4a16_awq(input_contiguous->contiguous(), qweight, scales, qzeros, bias_opt); | ||
| throw std::runtime_error( |
| infinicore::op::distributed::allreduce_( | ||
| output, output, INFINICCL_SUM, communicator); | ||
| output, output, infinicclSum, communicator); | ||
| if (has_bias) { |
There was a problem hiding this comment.
这里为什么要把has_bias单独拆出来，做一次add。 有的量化算子是可以将gemm+add一起算的。
| @@ -27,13 +27,6 @@ class NoneQuantization : public BaseQuantization { | |||
| bool has_bias, | |||
| float alpha = 1.0f) const override; | |||
|
|
|||
| infinicore::Tensor forward_allreduce( | |||
There was a problem hiding this comment.
为什么把forward_allreduce函数删除了？
那allreduce通信算子在哪里被调用呢
There was a problem hiding this comment.
原来父类BaseQuantization有forward_allreduce默认实现。
| @@ -45,34 +48,6 @@ infinicore::Tensor NoneQuantization::forward( | |||
| return infinicore::op::linear(input_contiguous->contiguous(), weight->contiguous(), bias_opt, alpha); | |||
| } | |||
|
|
|||
| infinicore::Tensor NoneQuantization::forward_allreduce( | |||
There was a problem hiding this comment.
NoneQuantization::forward函数gemm后不调用 allreduce.
NoneQuantization::forward_allreduce函数gemm后 调用 allreduce
| const cache::PagedKVCacheConfig &paged_config, | ||
| const config::ModelConfig *model_config, | ||
| bool has_mamba_state) { | ||
| if (!env_flag_enabled(kBaichuanFixedPrefillGraphEnv) || has_mamba_state) { |
There was a problem hiding this comment.
fixed_prefill感觉是一个新的功能了，不应该出现在这个pr中。
而且fixed_prefill应该有配合的调度吧，但这个pr中没有看到。
baichuan模型的依赖has_mamba_state？baichuan模型的跟has_mamba_state有关系么？
baichuan_fixed_prefill_graph的代码逻辑会被执行到么，能测试过么
| : GraphCompiler(model, barrier) { | ||
| } | ||
|
|
||
| void StaticBatchingCompiler::compile() { |
There was a problem hiding this comment.
static就别强行支持graph了。 修改后虽然叫static，但执行的确实paged的代码
|
|
||
| std::optional<size_t> first_past_sequence_length; | ||
| std::optional<size_t> first_total_sequence_length; | ||
| if (snapshot_static_sequence_lengths) { |
There was a problem hiding this comment.
如果static不考虑支持graph的话first_past_sequence_length，first_total_sequence_length这两个参数可以移除么。
而且在static计算函数 中，first_past_sequence_length和first_total_sequence_length使用用法也是to到cpu上，获得size_t类型的变量。
那为什么不在static计算函数中，直接从past_sequence_length和total_sequence_length的gputensor中to到cpu上获取呢？
| /// Maximum total sequence length in the current batch. | ||
| size_t max_sequence_length{0}; | ||
| /// Element 0 of past_sequence_lengths, snapshotted for static eager attention. | ||
| std::optional<size_t> first_past_sequence_length; |
There was a problem hiding this comment.
first_past_sequence_length这两个变量觉得没必要
| quantization::KVQuantAlgo kv_quant_scheme; | ||
| }; | ||
|
|
||
| const ReviewedPagedGraphProfile kBaichuanFixedPrefillProfile{ |
There was a problem hiding this comment.
"internlm3 "chatglm ""baichuan"这三个模型印象中都是llama结构。
放在通用框架里，要支持应该都支持，不支持的话都不支持。
| [](const auto &block) { | ||
| return !block->in_use; | ||
| }); | ||
| if (free_block != cls.free_blocks.end()) { |
There was a problem hiding this comment.
这里的逻辑感觉以前版本的效率更高些：先判断if (!cls.free_blocks.empty())，然后从后面往前找空闲的block。
修改后：从前往后找空闲的block, 然后还要调用复杂度是O(n)的erase函数。
这里修改的原因是什么
|
|
||
| auto block = it->second; | ||
| if (in_use) { | ||
| for (auto &cls : size_classes_) { |
There was a problem hiding this comment.
这里for循环里面又是remove，又是erase的。但效率是O(n)
如果用以前的写法，可以不用加这段代码。
| protected: | ||
| using run_schema = void (*)(void *); | ||
| using cleanup_schema = void (*)(void **); | ||
| void *planned_meta_; |
There was a problem hiding this comment.
planned_meta_、runner_ 和 deleter_ 都没有初始化。派生 Graph 算子在完成赋值前可能抛出异常，例如设备校验、dispatcher 查找或 plan 创建失败。此时 DispatchableGraphOperator 的析构函数仍会执行，并可能用不确定的 deleter_ 调用 deleter_(&planned_meta_)。
最小复现使用了一个在 plan 建立前抛异常的派生算子。进程在 DispatchableGraphOperator::~DispatchableGraphOperator() 中崩溃，GDB 显示崩溃点是 graph.cc:155 通过 deleter_ 的无效间接调用。
建议将这些字段默认初始化，并让 plan 获取过程具备异常安全保证。至少应保证：只有 plan 元数据和对应的清理函数都成功取得后，析构路径才允许调用它们。
|
|
||
| namespace infinicore { | ||
|
|
||
| class Memory { |
There was a problem hiding this comment.
Memory 的默认复制可能导致同一块内存被释放两次
Memory 直接拥有 data_ 和 deleter_，但复制构造和复制赋值仍由编译器隐式生成。执行 auto copy = original 后，两个对象持有相同指针和相同 deleter；二者析构时 deleter 会被调用两次。复制赋值还可能直接丢弃目标对象原有资源而不释放。
使用计数 deleter 的最小复现显示，同一块内存的释放回调被调用了两次。连接真实分配器时，这会表现为 double free，或同一块内存被重复归还给内存池。
建议显式定义所有权语义：禁用复制；移动要么禁用，要么实现为带正确清理的转移。需要共享分配时继续使用 shared_ptr<Memory>。
| auto &cache_vector = caches_[static_cast<size_t>(device_type)]; | ||
|
|
||
| if (cache_vector.size() <= device_index) { | ||
| cache_vector.resize(device_index + 1, BaseCache(capacity_, destructor_)); |
There was a problem hiding this comment.
LRUCache 所在 vector 扩容后，复制来的迭代器仍指向已销毁的 list
LRUCache 在 map_ 中保存 std::list 迭代器。隐式生成的复制操作只会复制这些迭代器，不会为新 cache 重建指向其自身 list 的迭代器。由于类提供了用户定义析构函数，隐式移动构造也不会生成。
OpCache::getCache 使用 cache_vector.resize(device_index + 1, BaseCache(...))。如果扩容时复制了已有且非空的 cache，随后原 cache 被销毁，之后的 get / touch 就会使用悬空迭代器。用 ASan 复现同样的扩容路径，会在 LRUCache::touch 中报告 heap-use-after-free。
建议显式定义复制和移动的所有权规则，或避免复制活跃 cache；引入具有稳定所有权的间接层也可以。如果确实需要支持复制，副本必须重建所有 map 迭代器，并明确析构回调如何处理其持有的 Value。
| } | ||
|
|
||
| /// Record a new op trace entry. | ||
| void write(const OpTraceEntry &entry) { |
There was a problem hiding this comment.
读写协议仍是 C++ data race
注释承诺 getRecentEntries() 可以从其他线程安全地取得快照。write_pos_ 上的 acquire/release 能让已发布的写入可见，但无法阻止生产者绕回环形缓冲区，并在读者复制非原子 OpTraceEntry 时覆盖同一个槽位。
在一生产者、一读者的场景下，TSan 报告 write() 第 81 行与 getRecentEntries() 第 98 行之间存在数据竞争。“允许读到半写入 entry”并不能消除未定义行为。
建议用 mutex 保护，改用正确的 seqlock / snapshot 设计，或让槽位所有权协议本身无竞争。注释文档也应与实际并发契约保持一致。
| if (entry.ndim >= 2) { | ||
| // Heuristic: for attention-like ops, look at the sequence dimension | ||
| // Typically shape = [batch, seq_len, ...] or [batch, heads, seq_len, ...] | ||
| uint32_t seq_dim = (entry.ndim >= 3) ? entry.shape[2] : entry.shape[1]; |
There was a problem hiding this comment.
MhaKVCache 的 head 数被解释成序列长度
当前逻辑对所有维度大于等于三的 attention trace 都取 shape[2] 作为序列长度。MhaKVCache 文档写明输出布局是 [batch_size, seqlen_q, num_heads, head_size]，因此 shape[2] 通常是 head 数量。
对于形状为 [1, 1, 32, 128] 的单 token decode trace，检测器读到 32 而不是 1，于是把 phase 判成 prefill。这会选择错误的优化目标。
建议按操作类型和 layout 提取序列轴，或直接在 OpTraceEntry 中记录标准化后的 sequence length。
| // 绑定 out-of-place 接口: output = acos(input) | ||
| m.def("acos", | ||
| &op::acos, | ||
| py::arg("input"), | ||
| R"doc(Computes the inverse cosine (arccosine) of each element of input. | ||
|
|
||
| Returns a new tensor with the arccosine of the elements of input. | ||
| The range of the result is [0, pi].)doc"); | ||
|
|
||
| // 绑定 in-place 接口: acos_(output, input) |
| // 绑定函数接口: output = adaptive_avg_pool1d(input, output_size) | ||
| m.def("adaptive_avg_pool1d", | ||
| &op::adaptive_avg_pool1d, | ||
| py::arg("input"), | ||
| py::arg("output_size"), | ||
| R"doc(Applies a 1D adaptive average pooling over an input signal composed of several input planes. |
|
|
||
| #include <pybind11/pybind11.h> | ||
|
|
||
| #include "infinicore/ops/all.hpp" |
There was a problem hiding this comment.
include 了不存在的 public header
|
|
||
| #include <pybind11/pybind11.h> | ||
|
|
||
| #include "infinicore/ops/equal.hpp" |
There was a problem hiding this comment.
include 了不存在的 public header
| #pragma once | ||
| #include <pybind11/pybind11.h> | ||
|
|
||
| #include "infinicore/ops/logical_and.hpp" |
There was a problem hiding this comment.
include 了不存在的 public header
|
|
||
| inline void bind_addbmm(py::module &m) { | ||
| // ----------------------------------------------------------- | ||
| // 1. Out-of-place 接口: output = addbmm(...) |
There was a problem hiding this comment.
这些中文注释最好都改成英文吧
| 文件 | 行号 |
|---|---|
| acos.hpp (line 3) | 3, 11, 20 |
| adaptive_avg_pool1d.hpp (line 12) | 12 |
| addbmm.hpp (line 12) | 12, 39, 42, 43 |
| addcmul.hpp (line 12) | 12, 32 |
| affine_grid.hpp (line 12) | 12, 17 |
| binary_cross_entropy_with_logits.hpp (line 11) | 11, 44 |
| cdist.hpp (line 12) | 12, 29 |
| flipud.hpp (line 11) | 11, 23 |
| float_power.hpp (line 32) | 32, 37 |
| floor.hpp (line 11) | 11, 17 |
| huber_loss.hpp (line 30) | 30, 35 |
| hypot.hpp (line 11) | 11, 18 |
| kthvalue.hpp (line 11) | 11, 27 |
| ldexp.hpp (line 19) | 19, 31 |
| lerp.hpp (line 12) | 12, 19, 22, 33, 42, 45, 54 |
| log_softmax.hpp (line 11) | 11, 23 |
| logcumsumexp.hpp (line 11) | 11, 20 |
| logical_and.hpp (line 10) | 10, 17, 18 |
| logical_not.hpp (line 18) | 18 |
| multi_margin_loss.hpp (line 22) | 22, 44 |
| scatter.hpp (line 12) | 12, 14, 15, 20, 26, 34, 36, 41, 48 |
| smooth_l1_loss.hpp (line 11) | 11 |
| topk.hpp (line 4) | 4 |
| triplet_margin_loss.hpp (line 11) | 11, 35 |
| unfold.hpp (line 13) | 13, 40 |
| upsample_bilinear.hpp (line 12) | 12, 26 |
| upsample_nearest.hpp (line 12) | 12, 24 |
| vander.hpp (line 28) | 28 |
| bind_add(m); | ||
| bind_cat(m); | ||
| bind_causal_softmax(m); | ||
| bind_embedding(m); | ||
| bind_linear(m); | ||
| bind_matmul(m); | ||
| bind_random_sample(m); | ||
| bind_rms_norm(m); | ||
| bind_rope(m); | ||
| bind_silu(m); | ||
| bind_silu_and_mul(m); |
There was a problem hiding this comment.
咱们这里只注册了11个算子，还有130个没有注册，这130个bind是必要的不？
| return value._underlying | ||
|
|
||
|
|
||
| def add(input, other, *, out=None): |
There was a problem hiding this comment.
native ops.hpp 注册了 rope 和 RoPEAlgo，但 python/infinicore/ops/__init__.py没有 rope() wrapper，nn.functional 也没有导出
There was a problem hiding this comment.
现在的infinicore好像没有这个类型
| .value("FLOAT16", DataType::kFloat16) | ||
| .value("BFLOAT16", DataType::kBFloat16) | ||
| .value("FLOAT32", DataType::kFloat32) | ||
| .value("FLOAT64", DataType::kFloat64); |
There was a problem hiding this comment.
这个地方要不要加一个bool，下面的测试里面有 infinicore.bool
| git clone --recurse-submodules https://github.com/InfiniTensor/InfiniLM.git | ||
| ``` | ||
|
|
||
| From InfiniLM, build the NVIDIA dependency stack pinned by the InfiniCore |
There was a problem hiding this comment.
默认依赖栈构建缺少声明的 FlashAttention / FlashInfer provider 前置条件
默认 scripts/configs/infiniops_ops.json 选择所有 FlashAttention 实现，并为 top_k_top_p_sampling_from_logits 选择 slot 16。按当前 InfiniOps 的 linked operator 元数据，解析这些算子要求 Python 环境中存在 flash-attn 以及 flashinfer-jit-cache>=0.6.7,<0.7。
README 目前从 clone 直接进入 build_infini_stack.py --test，pyproject.toml 也没有列出这些 provider。在干净的依赖栈构建中，InfiniRT 测试通过，但 InfiniOps configure 阶段停止，报错为：
error: Python distribution package 'flash-attn' required by ... flash_attn.yaml is not installed
建议文档写清 provider 的精确版本，并在构建前预检；也可以提供不需要外部 provider 的默认配置。同时应说明哪些 provider 属于构建或运行依赖，哪些库会随 InfiniLM 一起打包。
| from typing import Dict, List, Mapping, Optional, Sequence | ||
|
|
||
| PROJECT_ROOT = Path(__file__).resolve().parents[1] | ||
| SUPPORTED_BACKENDS = ("nvidia", "iluvatar", "hygon") |
| target("_infinicore") | ||
| add_packages("pybind11") | ||
| set_default(false) | ||
| add_rules("python.module", { soabi = true }) |
There was a problem hiding this comment.
请确认 wheel 是否有意依赖系统可见的 libpython
我检查了当前构建生成的 _infinicore 和 _infinilm 扩展，它们的 ELF DT_NEEDED 都包含 libpython3.13.so.1.0，但 wheel 中没有打包该库。
在独立目录安装 wheel，并清空 LD_LIBRARY_PATH 后，源码目录外执行 import infinicore 会失败：
ImportError: libpython3.13.so.1.0: cannot open shared object file
将当前 conda Python 的 lib 目录加入动态库搜索路径后可以正常导入。
请确认这是预期的部署要求，还是应调整 python.module 的链接方式，避免扩展直接依赖特定路径下的 libpython。如果这是有意设计，也建议在构建或安装文档中明确要求目标环境提供匹配版本的 libpython，并说明 wheel 的 Python/平台限制。
| py::arg("input"), | ||
| py::arg("vec1"), | ||
| py::arg("vec2"), | ||
| py::arg("alpha"), |
There was a problem hiding this comment.
addr 的关键字参数名颠倒了 C++ 的 beta / alpha 顺序
C++ 声明是 addr(..., float beta = 1.0f, float alpha = 1.0f) 和 addr_(..., float beta, float alpha)。这里 binding 把传给 C++ beta 的参数命名为 alpha，又把传给 C++ alpha 的参数命名为 beta。使用关键字参数时两个系数的语义被颠倒；仅看 Python 签名，位置调用也很容易写反。
如果这个 binding 计划启用，建议让关键字顺序与公开声明一致，并补充关键字参数和默认值的契约测试。
|
|
||
| inline void bind_scaled_mm_i8(py::module &m) { | ||
| m.def("scaled_mm_i8", | ||
| &op::scaled_mm_i8, |
There was a problem hiding this comment.
scaled_mm_i8 binding 引用了未声明的 API
公开头文件只声明了 scaled_mm_i8_，参数是五个 tensor 加一个 optional bias。这个未注册 binding 取的是 op::scaled_mm_i8 的地址，但该函数没有声明，因此按当前代码无法启用。注册的 in-place overload 也只暴露四个参数且没有 bias，与公开声明的六个参数不一致。
建议在 API 存在前移除该 binding，或补齐匹配的公开 out-of-place/in-place 声明和 binding。量化路径目前受配置门控，但迁移源码中不应保留一旦打开就无法编译的 binding。
| Run the migrated stack-builder unit tests with: | ||
|
|
||
| ```shell | ||
| python -m unittest test/scripts/test_build_infini_stack.py -v |
There was a problem hiding this comment.
直接指定 stack-builder unittest 文件的方式在 Python 3.13 下存在解析歧义
python -m unittest test/scripts/test_build_infini_stack.py -v 可能把 test 解析成标准库的 test 包，然后报 ModuleNotFoundError: No module named 'test.scripts'。改用测试发现机制是稳定的：
python -m unittest discover -s test/scripts -p 'test_build_infini_stack.py' -v
建议把文档命令改为测试发现形式。在本次审查环境中，该形式下 24 个测试全部通过。
| cum_seqlens_q, | ||
| cum_seqlens_k_tensor, | ||
| block_tables, | ||
| static_cast<int>(q->size(0)), |
There was a problem hiding this comment.
在 packed varlen 格式下，q 的 shape 通常是 [total_q_tokens, num_heads, head_dim]，其中 total_q_而不是单个 request 的最tokens 是所有 request 的 query token 总数，大 query 长度。
FlashAttention 的 varlen 接口中，max_seqlen_q 的语义是单个序列的最大 query 长度，用于kernel 调度等。而传 q->size(0) 会把这个值放大到总 token 数，导致调度器认为最大序列极长，可能选择不合适的 kernel 配置，workspace 被过度分配等情况。虽然由于有 cum_seqlens_q 做 masking，最终计算结果仍然正确，但性能和显存效率可能会下降
| cum_seqlens_k_tensor, | ||
| block_tables, | ||
| static_cast<int>(q->size(0)), | ||
| static_cast<int>(max_seqlen_k), |
There was a problem hiding this comment.
如果当前实现用 block table 容量（例如 max_num_blocks_per_seq * block_size）当作 max_seqlen_k，那它只是一个保守上界，不是真实最大 KV 长度。这通常不会导致正确性问题，但会让 kernel 的循环上界、workspace 分配偏大，可能会影响性能
| Tensor block_tables, Tensor kv_lens, Tensor cum_seqlens_q, | ||
| std::optional<Tensor> alibi_slopes, float scale) { | ||
|
|
||
| auto out = Tensor::empty(q->shape(), q->dtype(), q->device()); |
There was a problem hiding this comment.
输出形状的最后一维应该来自 V，而不是 Q。
q->shape() 的最后一维是 head_dim_qk，但 attention 输出的最后一维是由 V 决定的 head_dim_v，这样设置 out 的 shape 似乎有些不准确
|
|
||
| namespace infinicore::op { | ||
|
|
||
| INFINICORE_GRAPH_OP_DISPATCHERS_IMPL(MoeFusedDense); |
There was a problem hiding this comment.
这里声明了 MoeFusedDense 的 dispatcher，但仓库中没有找到对应的后端注册。调用 moe_fused_dense 时会在 INFINICORE_GRAPH_OP_RECORD_OR_RUN 阶段因找不到实现而失败
| "Cannot copy from tensor with different dtype. Src: " + src->info() + " Dst: " + this->info()); | ||
| } | ||
| if (this->device() == src->device()) { | ||
| if (this->device().type() == Device::Type::kCpu) { |
There was a problem hiding this comment.
这里的cpu上的copy为什么不想以前那样走 op::rearrange了？
| } | ||
|
|
||
| switch (tensor->dtype()) { | ||
| case DataType::kInt8: // 3 |
There was a problem hiding this comment.
这里把bool类型的case打印移除了。那么bool类型不能打印么。
6 participants
Summary
mainat80bb09ecebc9aabf198b9b866a89456bca1df946.Current head:
5de9a0283ec66ed30006372c535a75983407cdfb.The branch contains 43 commits on top of current
mainand remains mergeable.Related to InfiniTensor/InfiniCore#1373.
Migration and Runtime Changes
MhaKVCachework in host segments while the remaining operators run in device-graph segments; TP paged decode no longer falls back to a wholly eager engine.beta=0plus broadcast Add, including correct row/column-parallel placement and pre-transposed weights.infinicclCommInitRank. This avoids heap corruption caused by destroying a short-lived InfiniCore Runtime in each worker thread.Current Upstream Stack
55cfe5e6761c4ebb5e8eb77b65301479a7b6032c0cdbb16967e15f2e055dea1ec9641617bf3b6cf670e50081f181d8a3c4f9a6226c8524a648ed90c4c2d76051ca21774f0b515dfd8afe59537e200ac2e8ccc0cb23ca5b1b1d63be29ba61ea031de72807f74ce8033ce601dbb0bbf31277755d033693768c#506 uses AllGather, Send, and Recv, so #57/#58/#59 remain required. #69 is an independent
master-based prerequisite; validation combined itsMARCH_TYPE=310behavior with the API stack.Newly Validated Moore Capability
InfiniOps #819/#962 provide the missing paged prefill/decode attention closure. InfiniCCL #69 passes the actual MUSA architecture into MCCL so S5000's existing BF16 collective support is visible.
The selected formal matrix passed 13/13 commands:
Every row reported segmented graph execution with
host_segments > 0; none used whole-engine eager fallback. D14 and D17 additionally prove TP4/TP2 communicator setup and BF16 AllReduce. D14 completed in 191.391s and D17 in 74.904s after the two single-purpose fixes.The formal operator smoke also passed D64/D128 paged prefill/decode coverage:
4 passed, 8 deselected.All selected Moore commands are greedy/default sampling. Non-greedy sampling remains gated because the Moore manifest does not yet include a supported
top_k_top_p_sampling_from_logitsimplementation.Preserved NVIDIA Validation
CUDA_LAUNCH_BLOCKING.Gates Intentionally Retained
sliding_window, so a short prompt is not sufficient semantic validation.Verification
git diff --check: passed.git range-diff.Type of Change
Landing Order
master.masterbefore updating the final component pin.This PR remains draft until the upstream component PRs and final pins land, but it is ready for code review against the dependency order above.