Repository navigation
Conversation
Three of the five benchmark exporters did not produce the MLperf Tiny reference model. Checked against the int8 reference graphs the repos already ship in TrainDeeploy (Tests/Models/MLPerf/). DS-CNN. The KWS export defaulted to the "xs" PULP variant (16 channels, 25x10 MFCC, 2,732 params) where the reference is DS-CNN-S (64 channels, 49x10, ~24.9K). The registry already had a DSCNN-S entry, but its stem used valid padding and emitted a 20x4 map against the reference's 25x5, because the reference stem is Conv2D(64,(10,4),strides=2,padding='same'). Adds stem_padding="same" (symmetric padding computed to reproduce Keras' output size, so no separate Pad node): 23,180 params, 2.657M MACs, which is the published figure. Autoencoder. The AD export was a 128-dim toy (128->64->32->64->..., 29,120 params). The reference is 640 -> 128x4 -> 8 -> 128x4 -> 640 with BatchNorm on every hidden layer, ~268K params; the pre-existing "mlperf" variant (128->[128,128,128]->128) was not it either. Adds MLPerfADAutoencoder behind variant "mlperf_ad" and an Autoencoder-AD registry entry: 267,928 params, 264,192 MACs, 10 Gemm. BatchNorm1d over an (N,C) tensor is undefined at N=1 -- zero variance, and PyTorch refuses outright -- but on-device training runs at batch 1. FrozenStatsBatchNorm1d keeps the layers and their learnable affine parameters and normalizes with randomized running statistics instead, so the training graph carries the reference's 9 BatchNormInternal and 9 BatchNormalizationGrad nodes and its parameter count. MobileNetV1. The MLperf Tiny VWW reference is MobileNetV1-0.25, not the MobileNetV2-0.35 the registry described as such. There was no V1 exporter at all, so the mobilenetv1_train fixture deployed in TrainDeeploy could not be regenerated from this repo, although README.md and scripts/make_single_step.py both use "--model MobileNetV1" as an example. Adds the model and its exporter, with module names matching that fixture's parameters so regenerated graphs stay drop-in: 213,586 params, 7.490M MACs. Also corrects two registry descriptions that misstated what the entry builds: DSCNN-S claimed ~270K params (it is ~23K), and MobileNetV2-VWW claimed to be the MLperf VWW reference. Verified with onnxruntime-training 1.19.2 under the pinned versions; regenerating ResNet-8 reproduces the committed fixture node for node. tests/: 154 passed, 4 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012vb2GVs3qbiyLV1KkV2baN
cct_1 hard-coded mlp_ratio=2, so every CCT exported through cct_test had an FFN hidden size of 2x the embedding, while the official CCT-2 (and the TrainDeeploy CCT fixtures) use 1. The FFN LoRA adapters were also only reachable by uncommenting LinearwLora in transformers.py. Both are config keys now, with defaults that keep existing exports unchanged: - mlp_ratio (default 2) is forwarded cct_test -> cct_1 -> _cct. - lora_ffn (default False) puts LinearwLora on linear1/linear2 when use_lora is set, which is how the CCT_LoRA_R1 fixture was exported. With mlp_ratio=1, img 32, emb 128, 2 tokenizer convs, the re-exported CCT-2 training graph matches the TrainDeeploy fixture node for node (244 nodes, same op histogram). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AttentionwLora stores A as (dim, r) so the forward needs no Transpose, but kaiming_uniform_ reads fan-in from size(1), which for that layout is r. The bound came out sqrt(dim/r) = 5.7x too large at dim 128, r 4. The init is now applied to the (r, dim) transposed view. B starts at zero, as in the LoRA paper, and the export pipeline's randomize_initializers then replaces it with a Kaiming draw whose fan-in is shape[0] = r (std 0.7 at r = 4). Together with the oversized A that put the adapter output ~50x above the base weights and CCT's attention logits at +-600 with near-ties -- a reference no fp32 device reproduces to 1e-3. lora_b_init_std (default 0, the paper's init) lets a test export draw B from N(0, std) instead; 0.02 matches the base weights and brings the attention output back to the scale of the unadapted model (max 0.2). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Exporter changes needed for the CCT fine-tuning experiments in TrainDeeploy (runwangdl/TrainDeeploy#80). Stacked on #24.
1.
mlp_ratioandlora_ffnbecome config keyscct_1hard-codedmlp_ratio=2. The official CCT-2, and the TrainDeeploy CCT fixtures, use 1.LinearwLoraonlinear1/linear2) were only reachable by uncommenting code intransformers.py.mlp_ratio=2,lora_ffn=False), so existing exports are bit-identical.mlp_ratio=1, img 32, emb 128 and 2 tokenizer convs, the re-exported CCT-2 training graph matches the TrainDeeploy fixture node for node: 244 nodes, same op histogram, and a 35-node optimizer graph.2. LoRA init
AttentionwLorastores A as(dim, r)so the forward needs no Transpose.kaiming_uniform_reads fan-in fromsize(1), which for that layout isr, so the bound wassqrt(dim/r)= 5.7× too large. The init is now applied to the(r, dim)transposed view.randomize_initializersthen draws it with fan-inshape[0] = r, i.e. std 0.7 at r = 4.lora_b_init_std(default 0, the paper's init) draws B from N(0, std) for test exports. At 0.02 the attention output is back at the unadapted model's scale (max |out| 0.2 instead of ~1000).Verification
max|attention out|for CCT-2 LoRA: 1087.9 → 0.2.Export driver and recipe:
AI_AGENT/scripts/cct_export.py,AI_AGENT/Deeploy_Training/cct_advanced_strategies.md.🤖 Generated with Claude Code