【训练营】小模型训练支持 - #229
Open
AAekko wants to merge 1 commit into
Open
【训练营】小模型训练支持#229AAekko wants to merge 1 commit into
AAekko wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
InfiniTrain 原有 MNIST MLP 示例无法覆盖卷积训练。本 PR 增加 FP32 Conv2d、ReLU、Flatten 的 CPU/CUDA Forward/Backward,并将 Demo 升级为两层 CNN,完整接入数据加载、CrossEntropyLoss、Autograd、SGD、评估与 checkpoint。
MNIST 入口进一步复用框架 NCCL/DistributedDataParallel,支持单机一进程一卡、初始参数广播、确定性分片、bucket/逐参数梯度平均、全局加权指标和 rank 0 保存。保留相同大小的各卡训练尾批,评估集完整覆盖;训练结束校验所有卡参数完全一致。CPU/单卡默认行为保留。修复首次 GPU 数量查询的初始化,以及 NCCL_ROOT 未实际控制链接库的问题。
所有新增或修改源码均补齐直接标准库头文件依赖,并已使用仓库
.clang-format与 CI 指定的 clang-format 16.0.6 格式化。最终 29 个变更 C/C++/CUDA 文件通过--dry-run --Werror --style=file,git diff --cached --check通过。验证:
复现命令见
docs/mnist_ddp.md。双 GPU CTest 由BUILD_MNIST_DDP_TESTS=ON显式启用;默认关闭以避免影响单 GPU 常规测试。本 Demo 的 DDP 范围为单机 FP32、普通 SGD。PR 只包含功能代码、CMake、代表性测试、Demo 与文档。PyTorch 对齐代码、训练数据、大体积日志和 checkpoint 均在 PR 外随报告单独准备。