[Raymondlol]

Explains the A100 fp32 deviation reported in 0.1.0 and 0.2.0 (1.3e-2 max rel RMS vs the RTX 5090 at highest), and removes advice that assumed it was GPU precision.

Cause

Not GPU numerics. The official code builds its grid↔mesh graphs with NumPy on the host CPU from float32 lat/lon, and NumPy's SIMD code path depends on the host. Modal's A100-40GB hosts are AVX2-only; the RTX 5090 desktop and Modal's A100-80GB and T4 hosts have AVX-512. The two build graphs that differ in 13 of 1.6M grid→mesh edges and 517 mesh→grid triangles, which is enough for 1.3e-2 in the most sensitive field (median 6e-4). It happens with or without faster-weathernext.

Evidence (docs/validation.md §4, A100 note; docs/results/a100_host_graph/), one 0.25° step at highest:

• One card, one environment variable: an A100-80GB (AVX-512 host) with NumPy forced onto its AVX2 code builds exactly the 40GB host's graph (same hash), moves by 1.32e-2 from its own default run, and then matches the A100-40GB at 1.69e-5.
• Graph swap: the RTX 5090 on the 40GB host's graph differs from the RTX 5090 on its own graph by the same 1.32e-2 (same worst fields), and vice versa on the A100-40GB. With the graph swapped the two cards are 1.3e-4 apart; the rest comes from the host-built edge features.
• On the 40GB card, autotuning off, NVIDIA_TF32_OVERRIDE=0 and Triton GEMMs off all give the same 1.32e-2.
• The earlier "1 of 6 runs, an autotune pick" reading was wrong: Modal had served the "A100-40GB" repeat requests on 80GB machines, so every 40GB run deviated and every 80GB run matched.

Changes

• cloud/canary_run.py: records graph hashes and NumPy's SIMD targets, --edges runs on another machine's graph, --steps 0 skips the rollout.
• New scripts/graph_edges.py (counts and saves the graphs on the CPU), cloud/modal_a100_tf32.py, cloud/modal_graph_hosts.py (the probes).
• README FAQ, --no-autotune help, AGENTS.md: drop the "use --no-autotune for strict fp32 on A100" advice; the FAQ says why results can differ between machines.
• docs/validation.md §4 and Limitations; CHANGELOG under Unreleased.

No change to patch.py or pallas_attn.py. CPU tests: 32 passed, 10 skipped.

🤖 Generated with Claude Code