Repository navigation
Strict W3OUTG and POISON in every run, the resident restart pair, W3SDS4 on vector lanes - #201
Merged
Merged
Conversation
Goal abbb0456-8241, the follow-up of #191 (code review finding 2). Since #191 the "still newer on the device at finalize" check only prints, and its replacement, WW_KOKKOS_VA_POISON=1, was set by no script: a host reader that missed its column sync passed unless the stale bytes changed an output file. Every Kokkos run of scripts/ww3-gpu.sh (run_model, with --kokkos or WW3GPU_ROUTINES) and of the regtest matrix (ww3gpu_kokkos_env, the K case) now has WW_KOKKOS_VA_POISON=1, so the stale host VA is NaN after each kernel write and a missed sync fails the byte comparison. The Fortran runs (F, G, and ww3-gpu.sh's k=0) unset it. A caller's WW_KOKKOS_VA_POISON=0 still turns it off: ww3-gpu.sh reads it with ${WW_KOKKOS_VA_POISON:-1}, run_entry.cmake reads it once before the runs rewrite the environment. The poison fills only a copy a correct reader never reads, so a correct run's bits do not change. This changes a gate's invocation (AGENTS.md, "Never cheat a regtest tool"): a person reviews it. Tested: bash -n and shellcheck scripts/ww3-gpu.sh (clean); bash scripts/ww3-gpu.sh --self-test (ok, with the new va_poison and run_model checks); just ww3-gpu-matrix-selftest (ok, with the new POISON expectations); ww3-gpu.sh --ww3-gpu smoke and full --cpu --kokkos resident on the pinned fork commit: byte-identical to the Fortran and to --ww3 with POISON on (2161 and 97 fills) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts parts Goal abbb0456-8241, the follow-up of #191 (code review finding 1). With WW3GPU_KOKKOS=all the Kokkos run had WW_KOKKOS_ALL=1 and WW_KOKKOS_STRICT=1 together, so KOKKOS_PART's fallback for a part the entry's switch does not cover (MAP3 and MAPT with PR3, SRCE without LN1, BT1 or DB1) stopped every such entry in W3INIT, while ww3gpu_parts_line_check(ALL) accepts partial coverage and ww3-gpu.sh --kokkos all sets no strict mode. ww3gpu_kokkos_env now sets WW_KOKKOS_STRICT=1 for the K run only when its flags do not hold ALL (WW3GPU_KOKKOS=resident or a part list); under ALL the fallback is counted at exit. The selftest expectation "env, Kokkos run of all" follows. This changes a gate's invocation (AGENTS.md, "Never cheat a regtest tool"): a person reviews it. Tested: just ww3-gpu-matrix-selftest (ok); WW3GPU_KOKKOS=all just ww3-gpu-matrix-list serial (363 entries; 38 with WW3GPU_SWITCH_WORDS=PR0;SHRD); WW3GPU_KOKKOS=all WW3GPU_SWITCH_WORDS="PR0;SHRD" WW3GPU_OMP_THREADS=8 just ww3-gpu-matrix 1 serial -R ww3_tnc1: entry 38 runs and passes, test.comp 0/30, 10 of 16 parts on Kokkos, "calls left on the Fortran: MAP3=1 MAPT=1 SRCE=1" Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Goal abbb0456-8241. .gitmodules still had branch = ww3-gpu/kokkos-wire-base for WW3, so git submodule update --remote would have moved the pin back to 4a94a8f6; it now names ww3-gpu/kokkos-pnt, which the owner fast-forwarded to the pin 50a10730. The comments and messages that named older fork branches (ww3-gpu/kokkos-wire-base, kokkos-st4, kokkos-resident) as "the fork" now name kokkos-pnt: ww3-gpu.sh's header and its "not the fork" message, whose hint is now git submodule update --init WW3 (the pin, not a branch head), the justfile, health.yml, regtests/CMakeLists.txt, kokkos/src/fortran_iface/CMakeLists.txt, resident_c.hpp and srce_shim.cpp. Text only; no logic changes. PATCH.md's dated history keeps the names it recorded. Tested: bash -n and shellcheck scripts/ww3-gpu.sh (clean); bash scripts/ww3-gpu.sh --self-test (ok); just ww3-gpu-matrix-selftest (ok); git grep -nE 'ww3-gpu/kokkos-(wire-base|st4|resident)' -- .gitmodules scripts justfile regtests .github kokkos/src finds only PATCH.md's history Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…191 Stopped by the owner before the last step; another session takes it over. Done and recorded in docs/log/2026-10-18-strict-outg-poison.md: the fork commit fac74069 (W3KOKKOS_OUTG's fallback through KOKKOS_FALLBACK), POISON by default in Kokkos runs, strict only where a run names its parts, .gitmodules on ww3-gpu/kokkos-pnt. Not done: the resident serial matrix result (MATRIXRESULT placeholder in the log), port_score --write after the pin bump, the security review, the PR. Tested: per the log entry: gfortran -E -P of 143 files without KOKKOS; smoke and full resident byte-identical to the Fortran and to --ww3 with POISON on; full with SKW stops under WW_KOKKOS_STRICT=1 and counts OUTG=3 without; all-mode ww3_tnc1 test.comp 0/30. Not run here: the resident matrix Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fills the matrix row of the 2026-10-18 strict-outg-poison entry with the real verdicts at the pin fac74069 merged with origin/main: five of seven entries pass, the two ww3_tp2.6 entries fail by design (unstructured mesh, kept FAIL by the owner). The branch's earlier wip: commit holds the docs, PATCH.md section 31 and the pin; this commit completes it. Tested: WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=2 just ww3-gpu-matrix 4 serial (5 PASS, 2 FAIL by design, 729.7 s); bash -n and bash scripts/ww3-gpu.sh --self-test ok; just ww3-gpu-matrix-selftest ok; just ww3-gpu-matrix-list 7 entries; port_score.py --check current; wfip.py check 0 problems; python3 -m unittest discover tests ok; just kokkos-test serial-debug and openmp-release 100% of 11; shellcheck and vale not installed here (not run) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… record gap 10 Goal 5afd9379-8152. scripts/restart_pair.sh read the "Kokkos parts" line from ww3_shel.out, where WW3 does not print it (log.ww3 has it), and grep under pipefail ended the script silently with exit 1. It now reads log.ww3. What the script compares is unchanged. The pair then ran on main b828041 with WW3 at 50a10730 (SHRD KOKKOS build, OpenMP, 4 threads, WW_KOKKOS_VA_POISON=1, WW_KOKKOS_STRICT=1): b against a identical for the Fortran and for the resident Kokkos (out_grd.ww3 and the 12:00 restart), and every file of Kokkos against Fortran identical. Gap 10 of the readiness list and a dated log say what that covers and what it does not. Tested: scripts/restart_pair.sh exit 0 (all comparisons identical); restart_pair.sh --self-test passed; bash -n and shellcheck clean; just vale 0 errors. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… lane (plan 005 L3) Unit L3 of plan 005, goal 5afd9379-4417. w3sds4_point is split into sds4_prep, sds4_mss, sds4_sat, sds4_case1, sds4_case2, sds4_case3, sds4_turb and sds4_whitecap, the routine's lines moved without a change (only the case bodies are dedented), and w3sds4_point calls them in the routine's order. w3sds4_lanes<8> runs them for up to eight points: the mean-square-slope sums (w3src4md.F90:2228-2254), the saturation integral (:2262-2290) and the wave-turbulence and cumulative loops (:2533-2575) with one point per lane of GCC vector types, each lane keeping its own sums in the routine's order and every zero weight multiplied; branches on BTH0 are selects; powf, atan2f and sincosf are called per lane and the libmvec sites stay one grouped call per point. COS/SIN(USDIR) is computed once per lane instead of per ITH (the same pure call). The vector types move to st4_vec8.hpp for W3SIN4 and W3SDS4. w3srce_lanes calls W3SDS4 once per step for the points still in the loop; the scalar path is unchanged. Tested: just kokkos-test serial-debug and openmp-release, 11/11 each, no warning; bash scripts/ww3-gpu.sh --ww3-gpu smoke and full --cpu --kokkos resident with WW_KOKKOS_STRICT=1 WW_KOKKOS_VA_POISON=1 and WW_KOKKOS_SRCE_LANES=1 and 8: every model file byte-identical to the Fortran of the same executable, nccmp-tol PASS; bash scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident --build-only compiles, no warning Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rt report and plan updated The log records the question, the parts and the three lane loops, the bit verdicts on smoke and full with 1 and 8 points per team, the branches the full case does not reach (checked lane path against scalar path with a temporary parameter override, removed), WW3's regtests of the selected entries, the one-thread kernel timing (104.99 s to 80.55 s, L2 and L3 together) and the four open questions for the owner. The port report gains the L3 section and the plan marks L3. Tested: just regtest-selected (5 entries PASS, 8/8 ctest, smoke byte-identical); python3 scripts/port_score.py --write then --check, current; vale 3.17.1 on the four changed Markdown files, 0 errors, 0 warnings Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ot run" lines fixed The three pieces of the branch were verified at WW3 50a10730. At fac74069 strict mode stops a run when W3OUTG falls back to the Fortran. The restart pair (full) and full pass under strict; smoke stops in W3OUTG because it requests a skewness field, so its bit verdict at fac74069 is the non-strict run. Both logs get a dated section; gap 10 and the two stale lines of the NOAA progress record now say the pair ran. Tested: just kokkos-test serial-debug and openmp-release, exit 0; WW_KOKKOS_SRCE_LANES=1 and 8 with WW_KOKKOS_VA_POISON=1, smoke and full --cpu --kokkos resident: non-strict byte-identical to the Fortran and nccmp-tol PASS, strict full the same, strict smoke stops in W3OUTG (exit 1); scripts/restart_pair.sh at fac74069 exit 0, every comparison identical; just regtest-selected passed; smoke --gpu --build-only exit 0; bash -n, shellcheck, both --self-test, just ww3-gpu-matrix-selftest, port_score.py --check, unittest discover tests, just wfip check, figures.py check, just vale (3.17.1) all exit 0 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before. After #191, a passing Kokkos run did not prove that
W3OUTGran on Kokkos. For the fields the translation leaves out (WBT,SKW),W3KOKKOS_OUTGran the FortranW3OUTGwithout telling strict mode. No script set the missed-sync detectorWW_KOKKOS_VA_POISON. UnderWW3GPU_KOKKOS=all, the matrix set strict mode, so every entry whose switch file a part does not cover stopped inW3INIT. The resident restart had never been regtested, andscripts/restart_pair.shhad never run: it read the Kokkos-parts line from the wrong file and exited 1 without a message.W3SDS4, 24.8 % of the one-thread batchedW3SRCEin the S0 profile, ran one sea point at a time.After. This PR combines three pieces of work from goal
abbb0456and goal5afd9379. Each piece was checked at the old pin50a10730, and the combined branch was checked again at the new pinfac74069.W3OUTG,POISONin every run, strict mode only when a run names its parts (the follow-up of Production single-node Kokkos runtime: no-copy resident VA, strict mode, preset, packaging, GPU-run plan #191's review findings 1 to 3). The fallback ofW3OUTGto the Fortran goes throughKOKKOS_FALLBACK: a strict run stops there, and a run without strict mode counts the fallback. This is the fork commitfac74069onww3-gpu/strict-outg, PATCH.md section 31, and theWW3/pin moves to it.scripts/ww3-gpu.shand the regtest matrix setWW_KOKKOS_VA_POISON=1in every Kokkos run and unset it in the Fortran runs. Strict mode applies only to a Kokkos run that names its parts, soallis a coverage run again..gitmodulesnamesww3-gpu/kokkos-pntas the fork branch that holds the pin. Record:docs/log/2026-10-18-strict-outg-poison.md.scripts/restart_pair.shnow readslog.ww3. This fixes only the plumbing; what the script compares is unchanged. CasefullunderSHRD, OpenMP with 4 threads, strict mode andPOISONon. A run from the 06:00 restart gives the same 12:00 output and restart as the continuous run, for the Fortran and for the resident Kokkos, and every Kokkos file equals the Fortran's (cmp). This holds at50a10730and again atfac74069. Not covered ⚠:MPI, CUDA,ww3_multi, a moving grid. Record:docs/log/2026-10-18-resident-restart-pair.md.W3SDS4with one sea point per vector lane (plan 005 unit L3, after kokkos: W3SIN4's stress scan with one sea point per vector lane… #199's L2).w3sds4_pointis split into parts that hold its lines unchanged.w3sds4_lanes<8>runs the mean-square-slope sums, the saturation integral and the turbulence and cumulative loops on 8-wide GCC vectors. The libmvec sites stay one grouped call per point. TheV8/M8types move tost4_vec8.hpp, shared withW3SIN4. Record:docs/log/2026-10-18-srce-lanes-sds4.md, plan 005.Evidence
All runs used the OpenMP backend on the CPU. Nothing ran on a device;
--gpuappears only as--build-only.fac74069(combined HEAD)WW_KOKKOS_SRCE_LANES={1,8} WW_KOKKOS_VA_POISON=1 bash scripts/ww3-gpu.sh --ww3-gpu {smoke,full} --cpu --kokkos resident--ww3's;nccmp-tolPASSWW_KOKKOS_STRICT=1full: (v) the same verdict.smoke: exit 1, stopped by design inW3OUTGbecausesmokeasks for a skewness field the translation leaves to the FortranWW_KOKKOS_VA_POISON=1 OMP_NUM_THREADS=4 bash scripts/restart_pair.shSRCE,CPRT,OUTGjust regtest-selectedww3_tp2.15x4 andww3_ts2ST4_PR3_UQ; the L3 log has the per-entrytest.compcounts, 0 files different)WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=2 just ww3-gpu-matrix 4 serial(atfac74069, before L3)ww3_tp2.6entries FAIL by design (unstructured grid, kept FAIL by the owner)just kokkos-test serial-debug,openmp-release; CUDA--build-onlyof the presetbash -n, shellcheck, both--self-tests,ww3-gpu-matrix-selftest,port_score.py --check,unittest,wfip check,figures.py check, vale 0 errors; security-reviewer: no findingsTime of the one-thread
srce.batch.kernel,full,taskset -c 2, Intel Core i9-14900, at50a10730, median of three alternated runs: 104.99 s with 1 point per team, 80.55 s with 8 (L2 and L3 together, 23.3 %). Regtest pending ⚠: only the selected entries, not the whole matrix. The first run with 1 point per team started at a load of 7.85 ⚠. Command and loads: the L3 log.Needs a person's review
AGENTS.md: a change to a comparison script, a gate or how they are invoked is reviewed by a person.
POISONin every Kokkos run (scripts/ww3-gpu.sh,regtests/cmake/ww3gpu_common.cmake). A correct run writes the same bits with or without it, and a caller's=0turns it off.ww3gpu_common.cmake,run_entry.cmake,selftest.cmake). This relaxes when strict mode applies: underall, a fallback to the Fortran is counted in the exit line rather than stopping the run. Confirm that this hides nothing you want stopped.WBTorSKW, assmokedoes. A strict run ofsmokewith the resident preset now exits 1 until those fields are translated.scripts/restart_pair.sh: the plumbing fix described above. Is oneSHRDcase enough to call gap 10 covered on the CPU?W3SDS4lanes: the loop nesting (IKandITHoutside, the lanes inside, one accumulator per direction and lane); GCC vector types rather thanstd::simd.COS/SIN(USDIR)is computed once per lane, not once perITHas the Fortran text has it. It is the same pure call on the same argument, so the bits are the same. Say whether "translate, don't improve" wants the per-ITHcall kept.full⚠:SSDSC(3) > 0,SDSDTH >= 180, cases 2 and 3 ofW3SDS4, and the fallback when the transposed tables are missing. The first three were forced by a temporary override and gave the same bits with 1 and 8 points per team. That compares the Kokkos code with itself, not with the Fortran.Decisions left to the owner
libmandlibmvecper lane, not the transcribed device libm (docs/GPU_LIBM_BITS_202610.md, decision 2).ww3-gpu/kokkos-pnt, which the new.gitmodulesnames, was fast-forwarded50a10730..fac74069, so it holds the pin.Links: the
abbb0456handover, thec3e7eecfhandover, #191, #199, gap 10 ofdocs/PRODUCTION_READINESS_202610.md. Goalsabbb0456-8241and5afd9379(sessioned126573-8f64-46a0-9c43-4a8d5afd9379).🤖 Generated with Claude Code