Builds of the same code calibrate to two different weight sets #598

What's wrong

Pull request builds of the same data code calibrate to two distinct sets of weights. The targets are identical between them, and the difference is about five times the run-to-run dropout noise.

Evidence

Five pull request builds ran on one data code state: main from b34a4ea to 2b9c1c2 changed only CI and docs. They were docs-only, changelog-only and CI-only PRs, all TESTING builds at 32 epochs, on the same runner image (ubuntu24/20261004.327). The national rows of each run's final-epoch constituency_calibration_log.csv (epoch 30), compared pairwise as per-target change in |estimate / target − 1|:

• The national share of targets within 10% stays within 0.646–0.655 in every build.
• Per target, the two groups differ most on CGT bands, the benefit cap totals and the UC payment distribution.
• Every constituency target value is identical across the five builds. Every national target value is identical too, except the Scottish council tax bands that Scottish council tax targets change with whether a download succeeds #596 explains. That difference doesn't follow the groups: run 38009046475 has group A's Scottish targets and group B's fit.
• Neither runner region nor image follows the groups: region is centralus and eastus in both.

So something in the data build itself, before calibration, takes one of two states. The dropout noise is the much smaller within-group spread, which #507 seeds.

Why it matters

The W3 no-worsening gate compares a pull request's build with its base's, under a matched protocol. A same-code difference this large in either direction swamps the changes it is meant to attribute: 0.21 on key OBR and DWP targets. It has to be found and removed, or measured and allowed for, before a per-target comparison can gate.

Next step

Diff the pre-calibration datasets of two builds from different groups, as aggregates by variable. The divergent step probably shows up as an imputed variable whose totals take two values. Candidates are any imputation or download with unseeded randomness or order dependence.