[runwangdl]

Supersedes #71 and #72, which were the same work split into a stack. Each of them was red on its own — not because anything was broken in it, but because #71 restores a check that #72 then satisfies, so the middle of the stack could never be green. Merged here as one change, and CI is green.

The dead check

CompareLossesOnCluster() opened with

if (pi_core_id() != 0)
  return;

but it is dispatched through pi_cluster_send_task_to_cl() without a fork, so it runs exactly once, on the cluster controller. On GAP9 that core reports pi_core_id() == 8 (0..7 are the workers), so the guard fired every time. The comparison loop never executed, *err_count kept its initial 0, and every GAP9 training test printed Errors: 0 out of N no matter what the network computed.

Instrumented, ResNet-8 / L2 single-buffer:

probe	before	after
times entered	1	1
pi_core_id()	8	8
a->n read	no (sentinel intact)	yes, 4
[loss i] lines	0	4
reported	Errors: 0 out of 4 → PASSED	Errors: 4 out of 4 → FAILED

The losses that run actually produced: 2.302585, 1.435176e+28, 1.435176e+28, 1.435055e+28 against references 1.884626, 2.120300, 2.083486, 2.225912. 2.302585 is ln(10) — the first step was still at the random initialisation, and it overflowed from there.

Siracusa carries the same line but its cluster master is core 0, so its check has been live all along. That is why the same models pass there, and it is what made the rest of this diagnosable.

What the check then found

Initial weights were never loaded with --defaultMemLevel L2

The bake was keyed on the platform name:

_emit_init_weights = "GAP9" not in _platform_name

The intent was that GAP9 hex-loads every L3-resident weight at boot, so baking them again is redundant. True in L3. In L2 the weights live in L2, no hex file is emitted, and suppressing the bake also emits TRAINING_SKIP_INITWEIGHT_COPY, so the harness skips its copy too. All three supply paths off at once; every trainable weight stays zero.

Zero weights give all-equal logits, so the loss sits at exactly ln(num_classes):

model	classes	expected	measured
DS-CNN	12	ln(12) = 2.484907	2.484907
ResNet-8	10	ln(10) = 2.302585	2.302585

Bit-exact, at every --l1. The decision is now per weight and keyed on where the weight actually lives: L3-resident ones keep the hex path and get a NULL slot the harness skips; the rest are staged through L3 as hex and copied in by the existing LoadTestDataL3() + l3_aware_copy. Staging rather than baking matters — ResNet-8 on-chip needs 1,350,236 B of arena and its weights are 312 KB, which does not fit in 1.5 MB of L2_shared at all.

Six models forced a wrong num_data_inputs

"num_data_inputs": 1 on CCT, CCT_LoRA_R1, CCT_QLORA_FT, SleepConViT, TSDR and MCUNet. The flag exists for one documented case — "required when inputs.npz has only one mini-batch" — which holds for none of them; every one carries mb1, mb2, mb3.

The value is wrong too. These graphs take two inputs that change per mini-batch (CCT: input [1,3,32,32] and labels [1]). Declaring one shifts the harness's weight indexing by a slot in the optimizer's copy-in and copy-out. Step 0 is unaffected, which is why the first loss stayed bit-exact while the later ones collapsed.

Single-variable A/B on CCT-2, L3 single-buffer, l1=122000:

auto-detected (2)      Errors: 0 out of 4   diffs <= 1e-4
--num-data-inputs 1    Errors: 3 out of 4   2.301697 / 2.302683 / 2.302585

the failing values bit-identical to CI's. Siracusa never sets the override, which is again why it was unaffected.

pi_l1_malloc failures were silent

The SDK prints Allocation failed for allocator 2 and continues. The generated code then computes every tile address from a NULL base, so the failure appears much later as an unrelated Invalid access — or not at all, if the wild reads land on mapped memory and the run just produces wrong numbers. Both GAP9 L1 allocation templates now abort at the point of failure with the requested size.

Known failures, recorded rather than hidden

Three tests still fail. They never passed — they printed Errors: 0 because nothing was compared — so xfail(strict=True) records a pre-existing state rather than accepting a regression. They still compile, simulate and compare; strict=True means an unexpected PASS fails the run, so no entry can rot here. All five entries sit in one table in conftest.py with the evidence attached.

CCT_QLORA_FT (L2 + L3) and CCT_LoRA_R1 (L3) — the fixtures are wrong, not the deployment. The two ship byte-identical inputs.npz and outputs.npz (md5 2d0875bc../a01d0bba..) although their graphs differ: 387 nodes with no quantisation against 389 with 14 Dequant. The quantised model is checked against references generated from the float one. Measured error is 0.12–0.50; the 5e-2 tolerance already in the config was an earlier attempt to absorb exactly this. The fix is to regenerate the references per graph, which needs ORT and belongs in its own PR.

ResNet-8 with PromoteTensorsToL2 (SB + DB) — promotion shifts the FP32 accumulation order. The step-0 forward alone is off by 8.7e-4 against a 1e-3 tolerance where plain L3 is bit-exact. Reproduced 3/3 with the same 3-of-4 count, so it is deterministic rather than flaky. MobileNetV1 and CCT-2 take the same path and stay within 1e-4, which makes it look specific rather than inherent. It is off the critical path: ResNet-8's fastest verified configuration is on-chip L2 at 116.2 ms, faster than promote and numerically clean.

Measured

GAP9, tiled, single-buffer, four mini-batches, serial (CI's own markers):

group	before	after
L3 single-buffer	3 passed / 6 failed	7 passed / 2 failed
L2 single-buffer	2 passed / 2 failed	3 passed / 1 failed

and the two "passes" in the L2 group beforehand were themselves vacuous.

Recovered: DS-CNN (L2), ResNet-8 on-chip (L2), CCT-2, SleepConViT, TSDR, MCUNet. Unchanged and bit-identical: Autoencoder, ResNet-8 and MobileNetV1 in L3, SimpleMLP in L2.

Cycle counts, memory footprints and power numbers measured before this are unaffected — the number of instructions executed does not depend on whether the loss is right — but no correctness claim could rest on Errors: 0 until now.

🤖 Generated with Claude Code

https://claude.ai/code/session_012vb2GVs3qbiyLV1KkV2baN