EXPERIMENT (do not merge): A/B the A5 daily sweep against ptoas op fusion

[lwDavid]

Draft, not for merge. This branch exists only to run one experiment on CI hardware. Every change is marked TEMPORARY in-line and is meant to be reverted.

The question

Does ptoas v0.54's A5 tile fusion — on by default on a5 since v0.49 — account for the V4-Pro failures in the A5 daily sweep?

Local evidence this is testing

Measured on a clean-room tree with zero source modifications to pypto or pypto-lib; the two arms differed only in the ptoas command line:

arm	V4-Pro single-card
fusion on (stock v0.54)	12/27
fusion off	27/27

The mechanism looked like fusion sinking a producer below its consumer while the UB allocator coalesced the chain onto one offset — a stale read plus a WAW clobber. Enabling the same flags on the v0.48 binary reproduced v0.54 byte-for-byte across a frozen 473-kernel corpus, which is what pins it to the flag rather than to the release. Reported upstream as PTOAS#1111.

This PR asks CI hardware to confirm or refute that, since everything above was measured on one host.

What the branch does

change	why
daily_ci.yml gains a push: trigger scoped to this branch	collect the sweep without waiting for the 21:00 UTC schedule
model-tests-sim and model-tests-a2a3 skipped on this branch	the A/B needs only the A5 V4-Pro sweep; no reason to hold the arm64 CPU and 910B runners for an hour each
every job in ci.yml and docs.yml skipped on this branch	nothing here is exercised by them
model-tests-a5 becomes a 2-arm matrix (fusion: on / off)	the actual experiment
the off arm points PTOAS_ROOT at a generated wrapper dir	appends --enable-op-fusion=false --enable-shape-inference=false to every ptoas call
each arm gets its own build-namespace	see below — this one is not cosmetic
summary table renders the second arm as its own column	side-by-side result

Why a wrapper and not a source change. pypto resolves the assembler as $PTOAS_ROOT/ptoas, falling back to $PTOAS_ROOT/bin/ptoas (python/pypto/backend/_ptoas_locate.py). A directory holding one executable named ptoas is therefore enough to interpose. Both arms then run byte-identical pypto and pypto-lib sources, so the only difference in the experiment is the assembler command line. The shim probes the two ptoas layouts in the same order pypto does, since <root>/ptoas is a launcher up to v0.50 and a package directory from v0.51.

Why per-arm build-namespace. setup-ci-job keys the pypto source tree as pypto-src-<namespace>-<runner>, and its own documentation warns that two jobs sharing one tree will reset --hard / clean -ffdx / rm -rf build under the other's in-flight configure. Both arms can land on the same npu-a5 runner, so sharing a5-daily would have produced exactly the intermittent corruption that would discredit the result. (I hit this class of failure locally: a parallel 4-device sweep sharing one build tree produced 9 spurious device faults that all vanished on a serial re-run.)

Caveats, stated up front

• The off arm disables shape inference as well as op fusion, because that is the configuration measured locally. So it demonstrates "these two flags together recover the sweep", not "op fusion alone is responsible". Isolating them is a follow-up once the coarse result is confirmed on CI hardware.
• model-tests-a5 keeps continue-on-error: true, so neither arm turns the workflow red. The deliverable is the comparison table, not a red/green verdict.
• If the A5 runner is still unregistered, both arms stay queued and this yields nothing — that is a known precondition, noted in the job's own comment.

Expected outcome

The a5 (DeepSeek V4-Pro) summary table should show two columns whose pass counts differ substantially. If they do not differ, the local finding does not generalise to CI hardware and PTOAS#1111 needs re-scoping — which is equally worth knowing.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}