Skip to content

Strict W3OUTG and POISON in every run, the resident restart pair, W3SDS4 on vector lanes - #201

Merged
h0ffmann merged 10 commits into
mainfrom
goal/abbb0456-8241-strict-outg-poison
Oct 19, 2026
Merged

h0ffmann merged 10 commits into
mainfrom
goal/abbb0456-8241-strict-outg-poison

Conversation

@h0ffmann

@h0ffmann h0ffmann commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Before. After #191, a passing Kokkos run did not prove that W3OUTG ran on Kokkos. For the fields the translation leaves out (WBT, SKW), W3KOKKOS_OUTG ran the Fortran W3OUTG without telling strict mode. No script set the missed-sync detector WW_KOKKOS_VA_POISON. Under WW3GPU_KOKKOS=all, the matrix set strict mode, so every entry whose switch file a part does not cover stopped in W3INIT. The resident restart had never been regtested, and scripts/restart_pair.sh had never run: it read the Kokkos-parts line from the wrong file and exited 1 without a message. W3SDS4, 24.8 % of the one-thread batched W3SRCE in the S0 profile, ran one sea point at a time.

After. This PR combines three pieces of work from goal abbb0456 and goal 5afd9379. Each piece was checked at the old pin 50a10730, and the combined branch was checked again at the new pin fac74069.

  1. Strict W3OUTG, POISON in every run, strict mode only when a run names its parts (the follow-up of Production single-node Kokkos runtime: no-copy resident VA, strict mode, preset, packaging, GPU-run plan #191's review findings 1 to 3). The fallback of W3OUTG to the Fortran goes through KOKKOS_FALLBACK: a strict run stops there, and a run without strict mode counts the fallback. This is the fork commit fac74069 on ww3-gpu/strict-outg, PATCH.md section 31, and the WW3/ pin moves to it. scripts/ww3-gpu.sh and the regtest matrix set WW_KOKKOS_VA_POISON=1 in every Kokkos run and unset it in the Fortran runs. Strict mode applies only to a Kokkos run that names its parts, so all is a coverage run again. .gitmodules names ww3-gpu/kokkos-pnt as the fork branch that holds the pin. Record: docs/log/2026-10-18-strict-outg-poison.md.
  2. The resident restart pair, first run (readiness gap 10). scripts/restart_pair.sh now reads log.ww3. This fixes only the plumbing; what the script compares is unchanged. Case full under SHRD, OpenMP with 4 threads, strict mode and POISON on. A run from the 06:00 restart gives the same 12:00 output and restart as the continuous run, for the Fortran and for the resident Kokkos, and every Kokkos file equals the Fortran's (cmp). This holds at 50a10730 and again at fac74069. Not covered ⚠: MPI, CUDA, ww3_multi, a moving grid. Record: docs/log/2026-10-18-resident-restart-pair.md.
  3. W3SDS4 with one sea point per vector lane (plan 005 unit L3, after kokkos: W3SIN4's stress scan with one sea point per vector lane… #199's L2). w3sds4_point is split into parts that hold its lines unchanged. w3sds4_lanes<8> runs the mean-square-slope sums, the saturation integral and the turbulence and cumulative loops on 8-wide GCC vectors. The libmvec sites stay one grouped call per point. The V8/M8 types move to st4_vec8.hpp, shared with W3SIN4. Record: docs/log/2026-10-18-srce-lanes-sds4.md, plan 005.

Evidence

All runs used the OpenMP backend on the CPU. Nothing ran on a device; --gpu appears only as --build-only.

Check At fac74069 (combined HEAD)
WW_KOKKOS_SRCE_LANES={1,8} WW_KOKKOS_VA_POISON=1 bash scripts/ww3-gpu.sh --ww3-gpu {smoke,full} --cpu --kokkos resident (v) every model file byte-identical to the Fortran of the same executable; every file but the NetCDF byte-identical to --ww3's; nccmp-tol PASS
The same with WW_KOKKOS_STRICT=1 full: (v) the same verdict. smoke: exit 1, stopped by design in W3OUTG because smoke asks for a skewness field the translation leaves to the Fortran
WW_KOKKOS_VA_POISON=1 OMP_NUM_THREADS=4 bash scripts/restart_pair.sh (v) exit 0, every comparison identical, no call left on the Fortran for SRCE, CPRT, OUTG
just regtest-selected (v) PASS (ww3_tp2.15 x4 and ww3_ts2 ST4_PR3_UQ; the L3 log has the per-entry test.comp counts, 0 files different)
WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=2 just ww3-gpu-matrix 4 serial (at fac74069, before L3) (v) 5 PASS; the two ww3_tp2.6 entries FAIL by design (unstructured grid, kept FAIL by the owner)
just kokkos-test serial-debug, openmp-release; CUDA --build-only of the preset (v) pass; build and link clean
gates (v) bash -n, shellcheck, both --self-tests, ww3-gpu-matrix-selftest, port_score.py --check, unittest, wfip check, figures.py check, vale 0 errors; security-reviewer: no findings

Time of the one-thread srce.batch.kernel, full, taskset -c 2, Intel Core i9-14900, at 50a10730, median of three alternated runs: 104.99 s with 1 point per team, 80.55 s with 8 (L2 and L3 together, 23.3 %). Regtest pending ⚠: only the selected entries, not the whole matrix. The first run with 1 point per team started at a load of 7.85 ⚠. Command and loads: the L3 log.

Needs a person's review

AGENTS.md: a change to a comparison script, a gate or how they are invoked is reviewed by a person.

  • POISON in every Kokkos run (scripts/ww3-gpu.sh, regtests/cmake/ww3gpu_common.cmake). A correct run writes the same bits with or without it, and a caller's =0 turns it off.
  • Strict mode only when a run names its parts (ww3gpu_common.cmake, run_entry.cmake, selftest.cmake). This relaxes when strict mode applies: under all, a fallback to the Fortran is counted in the exit line rather than stopping the run. Confirm that this hides nothing you want stopped.
  • The new stop: strict mode now stops a run whose output asks for WBT or SKW, as smoke does. A strict run of smoke with the resident preset now exits 1 until those fields are translated.
  • scripts/restart_pair.sh: the plumbing fix described above. Is one SHRD case enough to call gap 10 covered on the CPU?
  • W3SDS4 lanes: the loop nesting (IK and ITH outside, the lanes inside, one accumulator per direction and lane); GCC vector types rather than std::simd. COS/SIN(USDIR) is computed once per lane, not once per ITH as the Fortran text has it. It is the same pure call on the same argument, so the bits are the same. Say whether "translate, don't improve" wants the per-ITH call kept.
  • Not covered by full ⚠: SSDSC(3) > 0, SDSDTH >= 180, cases 2 and 3 of W3SDS4, and the fallback when the transposed tables are missing. The first three were forced by a temporary override and gave the same bits with 1 and 8 points per team. That compares the Kokkos code with itself, not with the Fortran.

Decisions left to the owner

  • The lane loops on the host call the same glibc libm and libmvec per lane, not the transcribed device libm (docs/GPU_LIBM_BITS_202610.md, decision 2).
  • The zero weights of the saturation integral are multiplied, not skipped: skipping is not bit-neutral (0 × Inf = NaN).
  • Done 2026-10-18: the fork branch ww3-gpu/kokkos-pnt, which the new .gitmodules names, was fast-forwarded 50a10730..fac74069, so it holds the pin.

Links: the abbb0456 handover, the c3e7eecf handover, #191, #199, gap 10 of docs/PRODUCTION_READINESS_202610.md. Goals abbb0456-8241 and 5afd9379 (session ed126573-8f64-46a0-9c43-4a8d5afd9379).

🤖 Generated with Claude Code

h0ffmann and others added 10 commits October 18, 2026 17:32
Goal abbb0456-8241, the follow-up of #191 (code review finding 2). Since
#191 the "still newer on the device at finalize" check only prints, and
its replacement, WW_KOKKOS_VA_POISON=1, was set by no script: a host
reader that missed its column sync passed unless the stale bytes changed
an output file. Every Kokkos run of scripts/ww3-gpu.sh (run_model, with
--kokkos or WW3GPU_ROUTINES) and of the regtest matrix (ww3gpu_kokkos_env,
the K case) now has WW_KOKKOS_VA_POISON=1, so the stale host VA is NaN
after each kernel write and a missed sync fails the byte comparison. The
Fortran runs (F, G, and ww3-gpu.sh's k=0) unset it. A caller's
WW_KOKKOS_VA_POISON=0 still turns it off: ww3-gpu.sh reads it with
${WW_KOKKOS_VA_POISON:-1}, run_entry.cmake reads it once before the runs
rewrite the environment. The poison fills only a copy a correct reader
never reads, so a correct run's bits do not change. This changes a gate's
invocation (AGENTS.md, "Never cheat a regtest tool"): a person reviews it.

Tested: bash -n and shellcheck scripts/ww3-gpu.sh (clean); bash
scripts/ww3-gpu.sh --self-test (ok, with the new va_poison and run_model
checks); just ww3-gpu-matrix-selftest (ok, with the new POISON
expectations); ww3-gpu.sh --ww3-gpu smoke and full --cpu --kokkos
resident on the pinned fork commit: byte-identical to the Fortran and to
--ww3 with POISON on (2161 and 97 fills)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts parts

Goal abbb0456-8241, the follow-up of #191 (code review finding 1). With
WW3GPU_KOKKOS=all the Kokkos run had WW_KOKKOS_ALL=1 and WW_KOKKOS_STRICT=1
together, so KOKKOS_PART's fallback for a part the entry's switch does not
cover (MAP3 and MAPT with PR3, SRCE without LN1, BT1 or DB1) stopped every
such entry in W3INIT, while ww3gpu_parts_line_check(ALL) accepts partial
coverage and ww3-gpu.sh --kokkos all sets no strict mode. ww3gpu_kokkos_env
now sets WW_KOKKOS_STRICT=1 for the K run only when its flags do not hold
ALL (WW3GPU_KOKKOS=resident or a part list); under ALL the fallback is
counted at exit. The selftest expectation "env, Kokkos run of all" follows.
This changes a gate's invocation (AGENTS.md, "Never cheat a regtest
tool"): a person reviews it.

Tested: just ww3-gpu-matrix-selftest (ok); WW3GPU_KOKKOS=all just
ww3-gpu-matrix-list serial (363 entries; 38 with WW3GPU_SWITCH_WORDS=PR0;SHRD);
WW3GPU_KOKKOS=all WW3GPU_SWITCH_WORDS="PR0;SHRD" WW3GPU_OMP_THREADS=8 just
ww3-gpu-matrix 1 serial -R ww3_tnc1: entry 38 runs and passes, test.comp
0/30, 10 of 16 parts on Kokkos, "calls left on the Fortran: MAP3=1 MAPT=1
SRCE=1"
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Goal abbb0456-8241. .gitmodules still had branch = ww3-gpu/kokkos-wire-base
for WW3, so git submodule update --remote would have moved the pin back to
4a94a8f6; it now names ww3-gpu/kokkos-pnt, which the owner fast-forwarded
to the pin 50a10730. The comments and messages that named older fork
branches (ww3-gpu/kokkos-wire-base, kokkos-st4, kokkos-resident) as "the
fork" now name kokkos-pnt: ww3-gpu.sh's header and its "not the fork"
message, whose hint is now git submodule update --init WW3 (the pin, not a
branch head), the justfile, health.yml, regtests/CMakeLists.txt,
kokkos/src/fortran_iface/CMakeLists.txt, resident_c.hpp and srce_shim.cpp.
Text only; no logic changes. PATCH.md's dated history keeps the names it
recorded.

Tested: bash -n and shellcheck scripts/ww3-gpu.sh (clean); bash
scripts/ww3-gpu.sh --self-test (ok); just ww3-gpu-matrix-selftest (ok);
git grep -nE 'ww3-gpu/kokkos-(wire-base|st4|resident)' -- .gitmodules
scripts justfile regtests .github kokkos/src finds only PATCH.md's history
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…191

Stopped by the owner before the last step; another session takes it over.
Done and recorded in docs/log/2026-10-18-strict-outg-poison.md: the
fork commit fac74069 (W3KOKKOS_OUTG's fallback through KOKKOS_FALLBACK),
POISON by default in Kokkos runs, strict only where a run names its
parts, .gitmodules on ww3-gpu/kokkos-pnt. Not done: the resident serial
matrix result (MATRIXRESULT placeholder in the log), port_score --write
after the pin bump, the security review, the PR.

Tested: per the log entry: gfortran -E -P of 143 files without KOKKOS; smoke and full resident byte-identical to the Fortran and to --ww3 with POISON on; full with SKW stops under WW_KOKKOS_STRICT=1 and counts OUTG=3 without; all-mode ww3_tnc1 test.comp 0/30. Not run here: the resident matrix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fills the matrix row of the 2026-10-18 strict-outg-poison entry with the
real verdicts at the pin fac74069 merged with origin/main: five of seven
entries pass, the two ww3_tp2.6 entries fail by design (unstructured
mesh, kept FAIL by the owner). The branch's earlier wip: commit holds
the docs, PATCH.md section 31 and the pin; this commit completes it.

Tested: WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=2 just ww3-gpu-matrix 4 serial (5 PASS, 2 FAIL by design, 729.7 s); bash -n and bash scripts/ww3-gpu.sh --self-test ok; just ww3-gpu-matrix-selftest ok; just ww3-gpu-matrix-list 7 entries; port_score.py --check current; wfip.py check 0 problems; python3 -m unittest discover tests ok; just kokkos-test serial-debug and openmp-release 100% of 11; shellcheck and vale not installed here (not run)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… record gap 10

Goal 5afd9379-8152. scripts/restart_pair.sh read the "Kokkos parts" line from ww3_shel.out,
where WW3 does not print it (log.ww3 has it), and grep under pipefail ended the script silently
with exit 1. It now reads log.ww3. What the script compares is unchanged.

The pair then ran on main b828041 with WW3 at 50a10730 (SHRD KOKKOS build, OpenMP, 4 threads,
WW_KOKKOS_VA_POISON=1, WW_KOKKOS_STRICT=1): b against a identical for the Fortran and for the
resident Kokkos (out_grd.ww3 and the 12:00 restart), and every file of Kokkos against Fortran
identical. Gap 10 of the readiness list and a dated log say what that covers and what it does not.

Tested: scripts/restart_pair.sh exit 0 (all comparisons identical); restart_pair.sh --self-test
passed; bash -n and shellcheck clean; just vale 0 errors.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… lane (plan 005 L3)

Unit L3 of plan 005, goal 5afd9379-4417. w3sds4_point is split into sds4_prep, sds4_mss,
sds4_sat, sds4_case1, sds4_case2, sds4_case3, sds4_turb and sds4_whitecap, the routine's lines
moved without a change (only the case bodies are dedented), and w3sds4_point calls them in the
routine's order. w3sds4_lanes<8> runs them for up to eight points: the mean-square-slope sums
(w3src4md.F90:2228-2254), the saturation integral (:2262-2290) and the wave-turbulence and
cumulative loops (:2533-2575) with one point per lane of GCC vector types, each lane keeping its
own sums in the routine's order and every zero weight multiplied; branches on BTH0 are selects;
powf, atan2f and sincosf are called per lane and the libmvec sites stay one grouped call per
point. COS/SIN(USDIR) is computed once per lane instead of per ITH (the same pure call). The
vector types move to st4_vec8.hpp for W3SIN4 and W3SDS4. w3srce_lanes calls W3SDS4 once per
step for the points still in the loop; the scalar path is unchanged.

Tested: just kokkos-test serial-debug and openmp-release, 11/11 each, no warning; bash scripts/ww3-gpu.sh --ww3-gpu smoke and full --cpu --kokkos resident with WW_KOKKOS_STRICT=1 WW_KOKKOS_VA_POISON=1 and WW_KOKKOS_SRCE_LANES=1 and 8: every model file byte-identical to the Fortran of the same executable, nccmp-tol PASS; bash scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident --build-only compiles, no warning
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rt report and plan updated

The log records the question, the parts and the three lane loops, the bit verdicts on smoke and
full with 1 and 8 points per team, the branches the full case does not reach (checked lane path
against scalar path with a temporary parameter override, removed), WW3's regtests of the
selected entries, the one-thread kernel timing (104.99 s to 80.55 s, L2 and L3 together) and the
four open questions for the owner. The port report gains the L3 section and the plan marks L3.

Tested: just regtest-selected (5 entries PASS, 8/8 ctest, smoke byte-identical); python3 scripts/port_score.py --write then --check, current; vale 3.17.1 on the four changed Markdown files, 0 errors, 0 warnings
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ot run" lines fixed

The three pieces of the branch were verified at WW3 50a10730. At fac74069 strict mode stops a
run when W3OUTG falls back to the Fortran. The restart pair (full) and full pass under strict;
smoke stops in W3OUTG because it requests a skewness field, so its bit verdict at fac74069 is
the non-strict run. Both logs get a dated section; gap 10 and the two stale lines of the NOAA
progress record now say the pair ran.

Tested: just kokkos-test serial-debug and openmp-release, exit 0; WW_KOKKOS_SRCE_LANES=1 and 8 with WW_KOKKOS_VA_POISON=1, smoke and full --cpu --kokkos resident: non-strict byte-identical to the Fortran and nccmp-tol PASS, strict full the same, strict smoke stops in W3OUTG (exit 1); scripts/restart_pair.sh at fac74069 exit 0, every comparison identical; just regtest-selected passed; smoke --gpu --build-only exit 0; bash -n, shellcheck, both --self-test, just ww3-gpu-matrix-selftest, port_score.py --check, unittest discover tests, just wfip check, figures.py check, just vale (3.17.1) all exit 0

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@h0ffmann h0ffmann changed the title ww3-gpu.sh, regtest matrix: WW_KOKKOS_VA_POISON=1 in every Kokkos run Strict W3OUTG and POISON in every run, the resident restart pair, W3SDS4 on vector lanes Oct 18, 2026
@h0ffmann
h0ffmann merged commit 053f538 into main Oct 19, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant