From 43a4c1db8723c3eae9ebdf1a85255b8314806990 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 18:51:39 -0300 Subject: [PATCH 1/4] kokkos: W3SIN4's stress scan with one sea point per vector lane (plan 005 L2) Unit L2 of plan 005, goal c3e7eecf-6142, Claude Code session ec4548a3-46a8-46e2-be0a-2d74c3e7eecf. w3sin4_point is split into sin4_prep, sin4_scan and sin4_tail with its lines moved verbatim. sin4_scan_lanes<8> runs the scan (w3src4md.F90:696-797) for up to eight points, the ITH loop on GCC vector types of eight REAL: each vector operation is the scalar statement's, lane by lane, operands in the same order; a branch is a select of the value its taken side computes; expf and powf are called per lane where the branch calls them; each lane keeps its own stress sums in ITH order. w3srce_lanes calls it at both W3SIN4 sites of the integration step. Explicit vectors because g++ leaves the plain lane loop scalar under its default -ftrapping-math, and that flag is not changed for the whole library. Tested: just kokkos-test serial-debug and openmp-release, 11/11 each, no warning; bash scripts/ww3-gpu.sh --ww3-gpu smoke and full --cpu --kokkos resident with WW_KOKKOS_SRCE_LANES=8 WW_KOKKOS_STRICT=1 WW_KOKKOS_VA_POISON=1 on the final code: every model file byte-identical to the Fortran of the same executable, every file but the NetCDF to --ww3; bash scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident --build-only, compiles, no warning, nothing run on a device; srce.batch.kernel one thread pinned (taskset -c 2), 3 alternated runs each: 105.17 s with 1 point per team, 99.77 s with 8 (the log entry's script); the three moved ranges found verbatim; python3 scripts/port_score.py --write then --check; scripts/vale.sh and lychee --offline on the changed Markdown, 0 errors Co-Authored-By: Claude Opus 5.5 --- ...026-10-18-005-srce-points-in-lanes-plan.md | 2 +- docs/log/2026-10-18-srce-lanes-sin4.md | 100 ++++++ docs/log/README.md | 1 + kokkos/PORT_STATUS.md | 2 +- kokkos/port_report/W3SRCE_KOKKOS.md | 15 + kokkos/src/fortran_iface/srce_shim.cpp | 2 +- kokkos/src/ww_kokkos/srce_kernel.hpp | 80 ++++- kokkos/src/ww_kokkos/st4_sin4.hpp | 318 +++++++++++++++++- site/data/port_score.tsv | 2 +- 9 files changed, 488 insertions(+), 34 deletions(-) create mode 100644 docs/log/2026-10-18-srce-lanes-sin4.md diff --git a/docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md b/docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md index 3eec5895..8f26e7f0 100644 --- a/docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md +++ b/docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md @@ -87,7 +87,7 @@ change a bit. |---|---|---| | S0 | One-thread profile of today's batch, libm separated from the rest | the shares below are measured; the go/no-go of S0 is applied | | L1 | The lane layer: teams of `W` points (`WW_KOKKOS_SRCE_LANES`, 1 to 8, 1 reproduces today), each section run for the team's points in turn with today's scalar routines | **Done** 2026-10-18: `full` and `smoke` byte-identical with 1 and 8 (`kokkos/port_report/W3SRCE_KOKKOS.md`, "Points per team"). The point-minor scratch moves to L2, the first unit that reads it | -| L2 | `W3SIN4` on lanes (its stress loop, `st4_sin4.hpp:198-249`) | byte-identical; kernel time of `W3SRCE` measured with and without, 1 thread | +| L2 | `W3SIN4` on lanes (its stress loop, `st4_sin4.hpp:198-249`) | **Done** 2026-10-18: byte-identical on `smoke` and `full`; one-thread `W3SRCE` kernel 105.17 s to 99.77 s, 5.1 % ([log](../../log/2026-10-18-srce-lanes-sin4.md)) | | L3 | `W3SDS4` on lanes, the libmvec sites left as lane loops | the same | | L4 | The integration and the initialisation (`srce_kernel.hpp:177-400`) | the same | | L5 | `W3SPR4`, `W3SLN1`, `W3SDB1`, `W3SBT1` | the same | diff --git a/docs/log/2026-10-18-srce-lanes-sin4.md b/docs/log/2026-10-18-srce-lanes-sin4.md new file mode 100644 index 00000000..5e1a9682 --- /dev/null +++ b/docs/log/2026-10-18-srce-lanes-sin4.md @@ -0,0 +1,100 @@ +# `W3SIN4`'s stress scan with one sea point per vector lane: bit-identical, 5.1 % off the one-thread `W3SRCE` kernel + +2026-10-18, for M. Hoffmann (goal `c3e7eecf-6142`), unit L2 of +[the points-in-lanes plan](../compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md). + +## Question + +Does the stress scan of `W3SIN4` (`w3src4md.F90:696-797` at `WW3@50a10730`, the loop over `IK` +and `ITH` that builds `DSTAB`, `LLWS` and the wave-supported stress) give the Fortran's bits and +take less time when eight sea points run it at once, one per vector lane? + +## What was done + +- `w3sin4_point` (`kokkos/src/ww_kokkos/st4_sin4.hpp`) split into `sin4_prep`, `sin4_scan` and + `sin4_tail`, whose bodies are the routine's lines moved without a change (v, each of the three + ranges found verbatim, in order, by a line comparison); `sin4_detail::Mid` carries the locals + from one part to the next by copy. +- `sin4_scan_lanes<8>`: the scan for up to eight points, the `IK` loop outside, then per `IK` a + loop over the lanes for the per-`IK` values (`powf`, `atan2f`, `sincosf`, `logf` called per + lane), then the `ITH` loop on GCC vector types of eight `REAL` (`V8`, `M8`). Each vector + operation is the scalar statement's, lane by lane, operands in the same order; a branch is a + select of the value its taken side computes; `expf` and `powf` are called per lane, for the + lanes whose branch calls them; each lane keeps its own stress sums, added in `ITH` order. +- `w3sin4_lanes<8>` runs `sin4_prep` and `sin4_tail` per point around the lane scan, and + `w3srce_lanes` (`srce_kernel.hpp`) calls it at both `W3SIN4` sites of the integration step + (`:1239` and `:2007`) for the points still in the loop. The call in the initialisation + (`:1100`, only with `SINTAIL4 > 0.5`) stays per point. +- Why explicit vectors: written as plain loops over the lanes, g++ left the `ITH` body scalar + ("unsupported control flow in loop", `-fopt-info-vec-all`). It sinks the values only one side of + a select uses into a branch and, under its default `-ftrapping-math`, does not speculate them + back (v: the same loop in a standalone file vectorises with `-fno-trapping-math` and not + without). Changing that flag would change it for every kernel of the library; the vector types + change only this loop. + +## Result + +- (v) Bits, 8 points per team (`WW_KOKKOS_SRCE_LANES=8`), `WW_KOKKOS_STRICT=1 + WW_KOKKOS_VA_POISON=1`: `bash scripts/ww3-gpu.sh --ww3-gpu smoke --cpu --kokkos resident` and the + same on `full` give every model file byte-identical to the Fortran of the same executable + (`out_grd.ww3`, `out_pnt.ww3`, `restart.ww3`, `ww3.196806.nc`; `out_grd.ww3`, `ww3.field.nc`), and + every file but the NetCDF byte-identical to `--ww3`'s, `nccmp-tol` PASS. Both the scalar + transcription and the vector form were run. +- (v) Time: `srce.batch.kernel` (`WW_KOKKOS_TIMERS=1`), `full`, one thread pinned to a performance + core (`taskset -c 2`), runs alternated, Intel Core i9-14900, 2026-10-18 21:33 to 21:57 UTC, load + 9.8 at the first run's start and 1.9 to 3.5 after: + +| `WW_KOKKOS_SRCE_LANES` | Runs (s) | Median | +|---|---|---| +| 1 | 105.21, 105.17, 105.11 | 105.17 s | +| 8 | 98.67, 100.42, 99.77 | 99.77 s | + +- (v) Every run's `out_grd.ww3` is byte-identical to the Fortran-matched output of the timing + build (`cmp`). +- ⚠ 5.4 s of the kernel's 105.2 s, 5.1 %. `W3SIN4`'s own code was 12.9 % of the batch in the S0 + profile ([2026-10-18-srce-lanes-s0-profile](2026-10-18-srce-lanes-s0-profile.md)), so the lane + scan removes about 40 % of `W3SIN4`'s time: derived from S0's share, not measured on its own. +- (v) `bash scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident --build-only`: the CUDA + build compiles and links with no warning. The lane code is host-only; nothing ran on a device. +- (v) `just kokkos-test serial-debug` and `openmp-release`: 11/11 each, no warning. The first + `serial-debug` build warned `-Wpsabi` (a function returning a 32-byte vector without AVX, as + that preset has no `-march`); `vload` now writes through a reference, and the bits and CUDA + checks above were run again on that final code. +- ⚠ The timing below ran on the code before that change to `vload`, which only changes how an + inline helper hands back its vector; it was not timed again. +- ⚠ An unpinned one-thread run is no measure on this CPU: the same code took 113 s to 155 s in four + earlier runs, as the thread moved between performance and efficiency cores. No 32-thread + figure is given: the machine was shared during those runs. + +## What it changes + +The lane layer pays where a per-point loop has ordered sums and branches g++ will not vectorise. +`W3SDS4` (24.8 % of the batch in S0) is the next unit. + +## How to repeat + +`lanes_1t.sh`, after `bash scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos resident` (which +builds and stages the case): + +```bash +#!/usr/bin/env bash +# One-thread W3SRCE kernel time, WW_KOKKOS_SRCE_LANES 1 against 8, alternated, full case. +set -euo pipefail +repo=/home/hoffmann/code/ww3-gpu-2 +exe="$repo/build/ww3-gpu-openmp/ww3/bin/ww3_shel" +case_dir="$repo/build/runs/ww3-gpu-full-openmp/kokkos" +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lanes-1t" +mkdir -p "$here" +for rep in 1 2 3; do + for w in 1 8; do + d="$here/w$w-r$rep"; rm -rf "$d"; mkdir -p "$d" + for f in ST4TABUHF2.bin depth.inp mask.inp mod_def.ww3 namelists.nml ww3_shel.nml; do cp "$case_dir/$f" "$d/"; done + l0=$(cut -d' ' -f1 /proc/loadavg) + (cd "$d" && taskset -c 2 env OMP_NUM_THREADS=1 OMP_PROC_BIND=false WW_KOKKOS_ALL=0 WW_KOKKOS_PRESET=resident \ + WW_KOKKOS_STRICT=1 WW_KOKKOS_TIMERS=1 WW_KOKKOS_SRCE_LANES=$w "$exe" > ww3_shel.out 2> ww3_shel.err) + k=$(grep -E "srce.batch.kernel" "$d/ww3_shel.err" | head -1 | tr -s " ") + same=$(cmp -s "$d/out_grd.ww3" "$case_dir/out_grd.ww3" && echo identical || echo DIFFERENT) + echo "rep $rep lanes $w: [$k], out_grd.ww3 $same, load $l0 -> $(cut -d' ' -f1 /proc/loadavg)" + done +done +``` diff --git a/docs/log/README.md b/docs/log/README.md index 421c9282..b91fe64f 100644 --- a/docs/log/README.md +++ b/docs/log/README.md @@ -73,3 +73,4 @@ Date, author, issue/PR/WFIP. | 2026-10-18 | [`W3CPRT` on the resident `VA`: partitioning without copying the spectrum back](2026-10-18-cprt-resident.md) | plan 2026-10-18-002 U9, WFIP-0005 | | 2026-10-18 | [U3 on the integrated pin: the resident serial, OMPG, `ww3_multi` and `ww3_tnc1` entries; the restart pair designed, not run](2026-10-18-u3-regtests-v2.md) | plan 2026-10-18-002 U3, #191 | | 2026-10-18 | [One thread of the resident `W3SRCE` batch: 91 % in the port's own code, 9 % in libm](2026-10-18-srce-lanes-s0-profile.md) | plan 2026-10-18-005 S0, #197 | +| 2026-10-18 | [`W3SIN4`'s stress scan with one sea point per vector lane: bit-identical, 5.1 % off the one-thread `W3SRCE` kernel](2026-10-18-srce-lanes-sin4.md) | plan 2026-10-18-005 L2 | diff --git a/kokkos/PORT_STATUS.md b/kokkos/PORT_STATUS.md index 2bfebcfd..f2cb0976 100644 --- a/kokkos/PORT_STATUS.md +++ b/kokkos/PORT_STATUS.md @@ -49,7 +49,7 @@ Profile: full (bench_small, 120 x 80, 9 520 sea points, 12 h), Release -pg, swit | ported | **99.3 %** | 17270 of 263375 (6.56 %) | 63 of 1023 (6.16 %) | 25 of 147 | WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag | | regtested | **48.6 %** | 2689 of 263375 (1.02 %) | 12 of 1023 (1.17 %) | 6 of 147 | WW3's regtests ran with it on Kokkos and agreed with the Fortran | -The port itself: 59696 lines in 350 files, the C++ under kokkos/src (every .cpp and .hpp line). +The port itself: 60034 lines in 350 files, the C++ under kokkos/src (every .cpp and .hpp line). On a GPU: none yet. The CUDA build links; the RTX 4090 run, `just smoke-ww3-gpu gpu`, is the owner's, supervised. diff --git a/kokkos/port_report/W3SRCE_KOKKOS.md b/kokkos/port_report/W3SRCE_KOKKOS.md index 81aa6c42..7f3bbcdc 100644 --- a/kokkos/port_report/W3SRCE_KOKKOS.md +++ b/kokkos/port_report/W3SRCE_KOKKOS.md @@ -219,6 +219,21 @@ One run each on a machine shared with another session's regtests (load 9 to 25): run took 19.20 s with 1 point per team and 19.23 s and 19.60 s with 8 ⚠, so the layer costs nothing visible, and gains nothing yet, as designed. +### `W3SIN4` with one point per vector lane (plan 005, unit L2) + +With more than one point per team, `w3srce_lanes` calls `st4::w3sin4_lanes<8>` at both `W3SIN4` +sites of the integration step (`:1239`, `:2007`): section 1 and the rest of `W3SIN4` per point, +the stress scan (`w3src4md.F90:696-797`) on vectors of eight `REAL`, one point per lane, each +lane's operations those of the scalar statement in its order, each lane's stress sums its own +(`kokkos/src/ww_kokkos/st4_sin4.hpp`, `sin4_scan_lanes`). The scalar `w3sin4_point` is now +`sin4_prep`, `sin4_scan` and `sin4_tail` with its lines moved verbatim (v, a line comparison). + +| Check, 2026-10-18 | Result | +|---|---| +| `smoke`, `full`, `WW_KOKKOS_SRCE_LANES=8`, strict, poisoned | every model file byte-identical to the Fortran of the same executable; every file but the NetCDF to `--ww3`'s (v) | +| `srce.batch.kernel`, `full`, one thread on a performance core, 3 alternated runs each | 105.17 s with 1 point per team, 99.77 s with 8: 5.1 % less (v, [log](../../docs/log/2026-10-18-srce-lanes-sin4.md)) | +| `bash scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident --build-only` | compiles and links, no warning; the lane path is host-only and a device keeps one point per team; nothing run on a device (v) | + ## Benchmarks One CPU measurement of the batched path, after the replay above passed on the same tree diff --git a/kokkos/src/fortran_iface/srce_shim.cpp b/kokkos/src/fortran_iface/srce_shim.cpp index 6f8d9a75..b3a0880a 100644 --- a/kokkos/src/fortran_iface/srce_shim.cpp +++ b/kokkos/src/fortran_iface/srce_shim.cpp @@ -576,7 +576,7 @@ void run_batch(GridCtx& g, const st4::Params& q, const st4::Fields& f, const snl team.team_barrier(); const snl1::TeamScratch w = snl1::snl1_team_scratch(team, level, nc, nt); // SrcePoint holds references, so the team's points are built in place. - alignas(SrcePoint) unsigned char mem[kSrceMaxLanes * sizeof(SrcePoint)]; + alignas(SrcePoint) unsigned char mem[kSrceMaxLanes * sizeof(SrcePoint)] = {}; SrcePoint* const x = reinterpret_cast(mem); for (int l = 0; l < nl; ++l) { const int i = i0 + l; diff --git a/kokkos/src/ww_kokkos/srce_kernel.hpp b/kokkos/src/ww_kokkos/srce_kernel.hpp index 23cd46eb..0681fdea 100644 --- a/kokkos/src/ww_kokkos/srce_kernel.hpp +++ b/kokkos/src/ww_kokkos/srce_kernel.hpp @@ -284,16 +284,16 @@ struct SrcePoint { st.done = 0; } - /// 2. One step of the dynamic integration loop (:1210-2042), up to the DIA. - KOKKOS_INLINE_FUNCTION void step_a() const { - using srce_detail::mn; - using srce_detail::mni; - using srce_detail::mx; - using srce_detail::mxi; + /// 2. One step of the dynamic integration loop (:1210-2042), up to the DIA: step_a_sln1, + /// then W3SIN4 (:1239). + KOKKOS_INLINE_FUNCTION void step_a_sln1() const { st.nsteps = st.nsteps + 1; // :1212 w3sln1_point(p, g.sig, g.ecos, g.esin, wn1, st.fhigh, o.ustar, in.u10dir, vsln, dirf, wnf); // :1223 - sin4(); // :1239 + } + KOKKOS_INLINE_FUNCTION void step_a() const { + step_a_sln1(); + sin4(); // :1239 } /// The step's DIA (:1259-1263), run by the whole team. IQTPE <= 0 (GQM) is left to the @@ -304,8 +304,14 @@ struct SrcePoint { const snl1::TeamScratch& nw) const { snl1::snl1_team(team, nc, nt, nsig, spec, cg1, o.wnmean * st.depth, vsnl, vdnl, nw); } - /// The rest of the step, after the DIA; sets st.done after the last one. + /// The rest of the step, after the DIA: step_c_main, W3SIN4 (:2007), step_c_end, which + /// sets st.done after the last step. KOKKOS_INLINE_FUNCTION void step_c() const { + step_c_main(); + sin4(); // :2007 + step_c_end(); + } + KOKKOS_INLINE_FUNCTION void step_c_main() const { using srce_detail::mn; using srce_detail::mni; using srce_detail::mx; @@ -419,7 +425,8 @@ struct SrcePoint { for (int ik = st.nkh; ik < nk; ++ik) for (int ith = 0; ith < nth; ++ith) spec[ith + ik * nth] = spec[ith + (ik - 1) * nth] * p.fachfa + static_cast(0.); - sin4(); // :2007 + } + KOKKOS_INLINE_FUNCTION void step_c_end() const { if (sintail4 < static_cast(0.5)) spr4(); // :2012 st.done = st.dttot >= static_cast(0.9999) * in.dtg ? 1 : 0; // :2031 } @@ -543,21 +550,60 @@ KOKKOS_INLINE_FUNCTION void w3srce_lanes(const TeamMember& team, const snl1::Con for (int l = 0; l < nl; ++l) x[l].init(); }); team.team_barrier(); + // W3SIN4 for the points listed in `on`, one point per vector lane (st4::w3sin4_lanes), with + // each point's arguments of SrcePoint::sin4. + const auto sin4_lanes = [&](const int* on, const int n) { + if (n == 0) return; + const Real *a[kSrceMaxLanes] = {}, *cg[kSrceMaxLanes] = {}, *k[kSrceMaxLanes] = {}; + Real u[kSrceMaxLanes] = {}, ustar[kSrceMaxLanes] = {}, drat[kSrceMaxLanes] = {}, + as[kSrceMaxLanes] = {}, usdir[kSrceMaxLanes] = {}, z0[kSrceMaxLanes] = {}; + Real *tauwx[kSrceMaxLanes] = {}, *tauwy[kSrceMaxLanes] = {}, *tauwnx[kSrceMaxLanes] = {}, + *tauwny[kSrceMaxLanes] = {}, *s[kSrceMaxLanes] = {}, *d[kSrceMaxLanes] = {}, + *work[kSrceMaxLanes] = {}; + int* llws[kSrceMaxLanes] = {}; + for (int j = 0; j < n; ++j) { + const SrcePoint& y = x[on[j]]; + a[j] = y.spec; + cg[j] = y.cg1; + k[j] = y.wn2; + u[j] = y.in.u10abs; + ustar[j] = y.o.ustar; + drat[j] = y.st.drat; + as[j] = y.in.as; + usdir[j] = y.in.u10dir; + z0[j] = y.st.z0; + tauwx[j] = &y.o.tauwx; + tauwy[j] = &y.o.tauwy; + tauwnx[j] = &y.st.tauwax; + tauwny[j] = &y.st.tauway; + s[j] = y.vsin; + d[j] = y.vdin; + llws[j] = y.llws; + work[j] = y.st4w; + } + st4::w3sin4_lanes(x[on[0]].q, x[on[0]].f, n, a, cg, k, u, ustar, drat, as, + usdir, z0, tauwx, tauwy, tauwnx, tauwny, s, d, llws, work); + }; while (true) { - // st.done changes only in step_c, so whether a point is in the loop holds for the whole step. + // st.done changes only in step_c_end, so the points in the loop are the same for the whole + // step. + int on[kSrceMaxLanes]; + int non = 0; + for (int l = 0; l < nl; ++l) + if (!x[l].st.done) on[non++] = l; Kokkos::single(Kokkos::PerTeam(team), [&]() { - for (int l = 0; l < nl; ++l) - if (!x[l].st.done) x[l].step_a(); + for (int j = 0; j < non; ++j) x[on[j]].step_a_sln1(); + sin4_lanes(on, non); // :1239 }); team.team_barrier(); - for (int l = 0; l < nl; ++l) { - if (x[l].st.done) continue; - x[l].dia(team, nc, nt, nsig, nw); + for (int j = 0; j < non; ++j) { + x[on[j]].dia(team, nc, nt, nsig, nw); team.team_barrier(); } Kokkos::single(Kokkos::PerTeam(team), [&]() { - for (int l = 0; l < nl; ++l) - if (!x[l].st.done) x[l].step_c(); + for (int j = 0; j < non; ++j) x[on[j]].step_c_main(); + sin4_lanes(on, non); // :2007 + for (int j = 0; j < non; ++j) x[on[j]].step_c_end(); }); team.team_barrier(); bool all = true; diff --git a/kokkos/src/ww_kokkos/st4_sin4.hpp b/kokkos/src/ww_kokkos/st4_sin4.hpp index 6a0bd568..87a80b9b 100644 --- a/kokkos/src/ww_kokkos/st4_sin4.hpp +++ b/kokkos/src/ww_kokkos/st4_sin4.hpp @@ -41,6 +41,25 @@ inline constexpr int kJtot = 50; // JTOT, :574 inline constexpr Real kKm = static_cast(363.); // KM, :575 inline constexpr Real kCmm = static_cast(0.2325); // CMM, :575 +/// W3SIN4's locals that cross from section 1 to the stress scan of section 2, and from the scan +/// to the rest; the parts copy them in and out, which changes no bit. +struct Mid { + Real const1, const0, uorb, pturb, pvisc, fw, fu, fud, ustarsigma; + Real ust, taux, tauy, cosu, sinu, ustp, usdirp; + Real stressstab[3][2], stressstabn[3][2]; +}; + + +// 8 lanes of REAL and their masks, GCC's vector extensions (host code only: the lane path runs +// on host backends). Arithmetic on them is lane-wise IEEE; with -ffp-contract=off nothing fuses. +using V8 = Real __attribute__((vector_size(8 * sizeof(Real)))); +using M8 = int __attribute__((vector_size(8 * sizeof(int)))); +static_assert(sizeof(Real) == sizeof(int), "a mask lane per REAL lane"); +// By reference, not by value: a function returning a 32-byte vector warns (-Wpsabi) in a build +// without AVX (serial-debug has no -march). +inline void vload(V8& v, const Real* x) { __builtin_memcpy(&v, x, sizeof v); } +inline void vstore(Real* x, const V8& v) { __builtin_memcpy(x, &v, sizeof v); } + } // namespace sin4_detail /// Reals and ints of per-point work storage w3sin4_point needs (no allocation in a kernel). @@ -48,22 +67,13 @@ inline constexpr Real kCmm = static_cast(0.2325); // CMM, :575 inline int sin4_work_reals(const Params& p) { return 3 * p.nspec; } inline int sin4_work_ints(const Params& p) { (void)p; return 0; } -/// W3SIN4 for one point. a = A(1:NSPEC), cg = CG(1:NK), k = K(1:NSPEC) (WN2), -/// s, d = S, D(1:NSPEC), llws = LLWS(1:NSPEC) as 0/1, all 0-based. IX, IY and -/// BRLAMBDA are not read by the routine and are not passed. The OUT arguments -/// arrive holding the caller's values. -KOKKOS_INLINE_FUNCTION void w3sin4_point(const Params& p, const Fields& f, const Real* a, - const Real* cg, const Real* k, Real u, Real ustar, - Real drat, Real as, Real usdir, Real z0, Real cd, - Real& tauwx, Real& tauwy, Real& tauwnx, Real& tauwny, - Real* s, Real* d, int* llws, Real* work, int* iwork) { +/// Section 1 of W3SIN4 (:600-675) and the stability correction of section 2 (:683-692). +KOKKOS_INLINE_FUNCTION void sin4_prep(const Params& p, const Fields& f, const Real* a, + const Real* cg, Real u, Real ustar, Real drat, Real as, + Real z0, Real* work, sin4_detail::Mid& m) { using namespace sin4_detail; - (void)cd; // CD is an argument of W3SIN4 but never read. - (void)iwork; const int nk = p.nk, nth = p.nth, nspec = p.nspec; Real* dstab = work; - auto DSTAB = [&](int istab, int is) -> Real& { return dstab[(istab - 1) * nspec + (is - 1)]; }; - // 1. Preparations (:600-602) for (int i = 0; i < 3 * nspec; ++i) dstab[i] = static_cast(0.); Real stressstab[3][2] = {{0, 0}, {0, 0}, {0, 0}}; @@ -144,6 +154,35 @@ KOKKOS_INLINE_FUNCTION void w3sin4_point(const Params& p, const Fields& f, const } ustarsigma = (static_cast(1.0) + u / (static_cast(10.) + u)) * usigma; } + m.const1 = const1; + m.const0 = const0; + m.uorb = uorb; + m.pturb = pturb; + m.pvisc = pvisc; + m.fw = fw; + m.fu = fu; + m.fud = fud; + m.ustarsigma = ustarsigma; + for (int i = 0; i < 3; ++i) + for (int j = 0; j < 2; ++j) { + m.stressstab[i][j] = stressstab[i][j]; + m.stressstabn[i][j] = stressstabn[i][j]; + } +} + +/// The stress scan of section 2 (:696-797): DSTAB, LLWS and the stresses, IK after IK. +KOKKOS_INLINE_FUNCTION void sin4_scan(const Params& p, const Fields& f, const Real* a, + const Real* cg, const Real* k, Real ustar, Real drat, + Real usdir, Real z0, int* llws, Real* work, + sin4_detail::Mid& m) { + using namespace sin4_detail; + const int nk = p.nk, nth = p.nth, nspec = p.nspec; + Real* dstab = work; + auto DSTAB = [&](int istab, int is) -> Real& { return dstab[(istab - 1) * nspec + (is - 1)]; }; + Real(&stressstab)[3][2] = m.stressstab; + Real(&stressstabn)[3][2] = m.stressstabn; + const Real const0 = m.const0, uorb = m.uorb, pturb = m.pturb, pvisc = m.pvisc, fw = m.fw, + fu = m.fu, fud = m.fud, ustarsigma = m.ustarsigma; Real ust = ustar; Real taux = static_cast(0), tauy = static_cast(0); Real cosu = static_cast(0), sinu = static_cast(0); @@ -230,6 +269,222 @@ KOKKOS_INLINE_FUNCTION void w3sin4_point(const Params& p, const Fields& f, const } } } + m.ust = ust; + m.taux = taux; + m.tauy = tauy; + m.cosu = cosu; + m.sinu = sinu; + m.ustp = ustp; + m.usdirp = usdirp; +} + +/// sin4_scan for the `n` points of a team at once (1 <= n <= W), one point per vector lane +/// (plan 2026-10-18-005, unit L2). Lane l computes the statements sin4_scan computes for point +/// l, in its order: the loops over IK and ITH stay outside, the loop over the lanes is innermost, +/// each lane keeps its own stress sums, so no sum mixes two points. A branch becomes a select of +/// the value its taken side computes; expf and powf are called per lane, for the lanes whose +/// branch calls them. Lanes n..W-1 repeat point 0's inputs and their results are not stored. +template +inline void sin4_scan_lanes(const Params& p, const Fields& f, const int n, + const Real* const* a_, const Real* const* cg_, + const Real* const* k_, const Real* ustar_, + const Real* drat_, const Real* usdir_, + const Real* z0_, int* const* llws, Real* const* work, + sin4_detail::Mid* m) { + using namespace sin4_detail; + static_assert(W == 8, "the ITH loop is written for V8"); + const int nk = p.nk, nth = p.nth, nspec = p.nspec; + const Real* a[W]; + const Real* cg[W]; + const Real* k[W]; + Real ustar[W], drat[W], usdir[W], z0[W]; + Real const0[W], uorb[W], pturb[W], pvisc[W], fw[W], fu[W], fud[W], ustarsigma[W]; + Real stressstab[3][2][W], stressstabn[3][2][W]; + for (int l = 0; l < W; ++l) { + const int q = l < n ? l : 0; + a[l] = a_[q]; + cg[l] = cg_[q]; + k[l] = k_[q]; + ustar[l] = ustar_[q]; + drat[l] = drat_[q]; + usdir[l] = usdir_[q]; + z0[l] = z0_[q]; + const0[l] = m[q].const0; + uorb[l] = m[q].uorb; + pturb[l] = m[q].pturb; + pvisc[l] = m[q].pvisc; + fw[l] = m[q].fw; + fu[l] = m[q].fu; + fud[l] = m[q].fud; + ustarsigma[l] = m[q].ustarsigma; + for (int i = 0; i < 3; ++i) + for (int j = 0; j < 2; ++j) { + stressstab[i][j][l] = m[q].stressstab[i][j]; + stressstabn[i][j][l] = m[q].stressstabn[i][j]; + } + } + Real ust[W], taux[W], tauy[W], cosu[W], sinu[W], ustp[W], usdirp[W]; + for (int l = 0; l < W; ++l) { + ust[l] = ustar[l]; + taux[l] = static_cast(0); + tauy[l] = static_cast(0); + cosu[l] = static_cast(0); + sinu[l] = static_cast(0); + ustp[l] = static_cast(0); + usdirp[l] = static_cast(0); + } + // Without W3_STAB3 a single pass with ISTAB=3; with it, ISTAB=1,2 (:696-701). + const int istab_first = p.stab3 ? 1 : 3; + const int istab_last = p.stab3 ? 2 : 3; + for (int istab = istab_first; istab <= istab_last; ++istab) { + Real(&ss)[2][W] = stressstab[istab - 1]; + Real(&sn)[2][W] = stressstabn[istab - 1]; + for (int l = 0; l < W; ++l) { + if (istab == 1) ust[l] = ustar[l] * (static_cast(1.) - ustarsigma[l]); + if (istab == 2) ust[l] = ustar[l] * (static_cast(1.) + ustarsigma[l]); + Real cu, su; + dm::sincosf(usdir[l], su, cu); + taux[l] = (ust[l] * ust[l]) * cu; + tauy[l] = (ust[l] * ust[l]) * su; + ss[0][l] = static_cast(0.); + ss[1][l] = static_cast(0.); + sn[0][l] = static_cast(0.); + sn[1][l] = static_cast(0.); + } + // Sequential scan over IK: each IK reads the stress of the earlier ones (:713-797). + for (int ik = 1; ik <= nk; ++ik) { + const int is0 = 1 + (ik - 1) * nth; + const Real sig2is = f.sig2(is0 - 1); + Real cm[W], ucn[W], const2[W], cnst[W], zcn[W], swellcoefv[W], swellcoeft[W]; + for (int l = 0; l < W; ++l) { + const Real taupx = taux[l] - Kokkos::fabs(p.ttauwshelter) * ss[0][l]; + const Real taupy = tauy[l] - Kokkos::fabs(p.ttauwshelter) * ss[1][l]; + // (..)**0.25 is powf(x, 0.25) in the dump (:717). + ustp[l] = min_r(dm::powf((taupx * taupx + taupy * taupy), static_cast(0.25)), + max_r(ust[l], static_cast(0.3))); + usdirp[l] = dm::atan2f(taupy, taupx); + dm::sincosf(usdirp[l], sinu[l], cosu[l]); + const Real kis = k[l][is0 - 1]; + cm[l] = kis / sig2is; + ucn[l] = ustp[l] * cm[l] + p.zzalp; + const2[l] = f.dden2(is0 - 1) / cg[l][ik - 1] * kGrav / (f.sig(ik) / kis * drat[l]); + cnst[l] = sig2is * const0[l]; + zcn[l] = dm::logf(kis * z0[l]); + // -(...) of the whole product (:737-738); sqrt(2*NU_AIR*SIG2) with 2*NU_AIR folded. + swellcoefv[l] = -(p.sswellf[4] * drat[l] * static_cast(2) * kis * + Kokkos::sqrt(kTwoNuAir * sig2is)); + swellcoeft[l] = + -(drat[l] * p.sswellf[0] * static_cast(16) * (sig2is * sig2is) / kGrav); + } + // The ITH loop on 8-wide vectors, lane l being point l. Each vector operation is the IEEE + // operation of the scalar statement it replaces, lane by lane, with its operands in the + // same order; a comparison gives a lane mask and `m ? x : y` selects per lane. ss and sn + // stay in registers for the whole ITH loop, one sum per lane, added in ITH order. + V8 cosu_v, sinu_v, ucn_v, zcn_v, cnst_v, cm_v, ustar_v, swc_v, swt_v, fw_v, uorb_v, fu_v, + fud_v, ustp_v, pturb_v, pvisc_v, const2_v; + vload(cosu_v, cosu); + vload(sinu_v, sinu); + vload(ucn_v, ucn); + vload(zcn_v, zcn); + vload(cnst_v, cnst); + vload(cm_v, cm); + vload(ustar_v, ustar); + vload(swc_v, swellcoefv); + vload(swt_v, swellcoeft); + vload(fw_v, fw); + vload(uorb_v, uorb); + vload(fu_v, fu); + vload(fud_v, fud); + vload(ustp_v, ustp); + vload(pturb_v, pturb); + vload(pvisc_v, pvisc); + vload(const2_v, const2); + const M8 s0nz = p.sswellf[0] != static_cast(0) ? ~M8{} : M8{}; + const M8 s2pos = p.sswellf[2] > static_cast(0) ? ~M8{} : M8{}; + V8 sn0, sn1, ss0, ss1; + vload(sn0, sn[0]); + vload(sn1, sn[1]); + vload(ss0, ss[0]); + vload(ss1, ss[1]); + for (int ith = 1; ith <= nth; ++ith) { + const int is = ith + (ik - 1) * nth; + const Real ecos = f.ecos(is - 1), esin = f.esin(is - 1); + const V8 coswind = (ecos * cosu_v + esin * sinu_v); + const M8 wind = coswind > static_cast(0.01); + const V8 x = coswind * ucn_v; + const V8 zarg = kKappa / x; + const V8 zlog = zcn_v + zarg; + const M8 grow = wind & (zlog < static_cast(0.)); + V8 ex = {}, pw = {}; + for (int l = 0; l < W; ++l) + if (grow[l]) { + ex[l] = dm::expf(zlog[l]); + // COSWIND**SSINTHP: REAL exponent, powf in the dump (:756). + pw[l] = dm::powf(coswind[l], p.ssinthp); + } + const V8 zlog2 = zlog * zlog; + const V8 grown = cnst_v * ex * (zlog2 * zlog2) * ucn_v * ucn_v * pw; + V8 dst = grow ? grown : V8{}; + // Uses USTAR, not USTP (:769), as ECWAM implsch.F does. + const M8 gust = + wind & (static_cast(28.) * cm_v * ustar_v * coswind >= static_cast(1)); + const M8 lw = grow | gust; + // COSWIND here is the value of this IS even when it is <= 0.01 (:777-783). + const M8 swell = (s0nz & (dst < static_cast(1E-7) * sig2is)) | s2pos; + const V8 dturb = swt_v * (fw_v * uorb_v + (fu_v + fud_v * coswind) * ustp_v); + const V8 withswell = dst + pturb_v * dturb + pvisc_v * swc_v; + dst = swell ? withswell : dst; + // Wave-supported stress, "direction to", hence the plus sign (:785-795). + V8 av; + for (int l = 0; l < W; ++l) av[l] = a[l][is - 1]; + const V8 temp2 = const2_v * dst * av; + const M8 neg = dst < static_cast(0); + sn0 = neg ? sn0 + temp2 * ecos : sn0; + sn1 = neg ? sn1 + temp2 * esin : sn1; + ss0 = neg ? ss0 : ss0 + temp2 * ecos; + ss1 = neg ? ss1 : ss1 + temp2 * esin; + for (int l = 0; l < n; ++l) { + work[l][(istab - 1) * nspec + (is - 1)] = dst[l]; + llws[l][is - 1] = lw[l] ? 1 : 0; + } + } + vstore(sn[0], sn0); + vstore(sn[1], sn1); + vstore(ss[0], ss0); + vstore(ss[1], ss1); + } + } + for (int l = 0; l < n; ++l) { + m[l].ust = ust[l]; + m[l].taux = taux[l]; + m[l].tauy = tauy[l]; + m[l].cosu = cosu[l]; + m[l].sinu = sinu[l]; + m[l].ustp = ustp[l]; + m[l].usdirp = usdirp[l]; + for (int i = 0; i < 3; ++i) + for (int j = 0; j < 2; ++j) { + m[l].stressstab[i][j] = stressstab[i][j][l]; + m[l].stressstabn[i][j] = stressstabn[i][j][l]; + } + } +} + +/// The rest of W3SIN4 (:799-976): D and S, the tail stress and the stress limit. +KOKKOS_INLINE_FUNCTION void sin4_tail(const Params& p, const Fields& f, const Real* a, + const Real* cg, Real ustar, Real z0, Real& tauwx, + Real& tauwy, Real& tauwnx, Real& tauwny, Real* s, Real* d, + Real* work, sin4_detail::Mid& m) { + using namespace sin4_detail; + const int nk = p.nk, nth = p.nth, nspec = p.nspec; + Real* dstab = work; + auto DSTAB = [&](int istab, int is) -> Real& { return dstab[(istab - 1) * nspec + (is - 1)]; }; + Real(&stressstab)[3][2] = m.stressstab; + Real(&stressstabn)[3][2] = m.stressstabn; + const Real const1 = m.const1; + Real const0 = m.const0; + Real ust = m.ust, taux = m.taux, tauy = m.tauy, cosu = m.cosu, sinu = m.sinu, ustp = m.ustp, + usdirp = m.usdirp; Real xstress, ystress; if (!p.stab3) { // :799-803 for (int is = 1; is <= nspec; ++is) d[is - 1] = DSTAB(3, is); @@ -373,4 +628,41 @@ KOKKOS_INLINE_FUNCTION void w3sin4_point(const Params& p, const Fields& f, const } } +/// W3SIN4 for one point. a = A(1:NSPEC), cg = CG(1:NK), k = K(1:NSPEC) (WN2), +/// s, d = S, D(1:NSPEC), llws = LLWS(1:NSPEC) as 0/1, all 0-based. IX, IY and +/// BRLAMBDA are not read by the routine and are not passed. The OUT arguments +/// arrive holding the caller's values. +KOKKOS_INLINE_FUNCTION void w3sin4_point(const Params& p, const Fields& f, const Real* a, + const Real* cg, const Real* k, Real u, Real ustar, + Real drat, Real as, Real usdir, Real z0, Real cd, + Real& tauwx, Real& tauwy, Real& tauwnx, Real& tauwny, + Real* s, Real* d, int* llws, Real* work, int* iwork) { + (void)cd; // CD is an argument of W3SIN4 but never read. + (void)iwork; + sin4_detail::Mid m; + sin4_prep(p, f, a, cg, u, ustar, drat, as, z0, work, m); + sin4_scan(p, f, a, cg, k, ustar, drat, usdir, z0, llws, work, m); + sin4_tail(p, f, a, cg, ustar, z0, tauwx, tauwy, tauwnx, tauwny, s, d, work, m); +} + +/// W3SIN4 for the `n` points of a team (1 <= n <= W), each argument an array of the n points' +/// arguments of w3sin4_point: section 1 and the rest per point, the stress scan with one point +/// per vector lane (sin4_scan_lanes). Every point gets the bits w3sin4_point gives it. +template +inline void w3sin4_lanes(const Params& p, const Fields& f, const int n, + const Real* const* a, const Real* const* cg, + const Real* const* k, const Real* u, const Real* ustar, + const Real* drat, const Real* as, const Real* usdir, + const Real* z0, Real* const* tauwx, Real* const* tauwy, + Real* const* tauwnx, Real* const* tauwny, Real* const* s, + Real* const* d, int* const* llws, Real* const* work) { + sin4_detail::Mid m[W]; + for (int l = 0; l < n; ++l) + sin4_prep(p, f, a[l], cg[l], u[l], ustar[l], drat[l], as[l], z0[l], work[l], m[l]); + sin4_scan_lanes(p, f, n, a, cg, k, ustar, drat, usdir, z0, llws, work, m); + for (int l = 0; l < n; ++l) + sin4_tail(p, f, a[l], cg[l], ustar[l], z0[l], *tauwx[l], *tauwy[l], *tauwnx[l], *tauwny[l], + s[l], d[l], work[l], m[l]); +} + } // namespace ww::st4 diff --git a/site/data/port_score.tsv b/site/data/port_score.tsv index 0673fd79..b82f16bd 100644 --- a/site/data/port_score.tsv +++ b/site/data/port_score.tsv @@ -7,4 +7,4 @@ of the run time, regtested 48.6 % 12 of 1 023 routines WW3's regtests ran with i of the lines of code, translated 25.80 % 67 947 of 263 375 Fortran lines of the translated routines, of every line in WW3's model/src, in 65 of 147 files of the lines of code, ported 6.56 % 17 270 of 263 375 Fortran lines of the ported routines, of every line in WW3's model/src, in 25 of 147 files of the lines of code, regtested 1.02 % 2 689 of 263 375 Fortran lines of the regtested routines, of every line in WW3's model/src, in 6 of 147 files -lines of Kokkos C++ 59 696 in 350 files under kokkos/src: the port itself +lines of Kokkos C++ 60 034 in 350 files under kokkos/src: the port itself From f99b998b44afc7f2344cc3fb2c41c739f4534523 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 18:56:57 -0300 Subject: [PATCH 2/4] docs/log: handover of goal c3e7eecf, what was merged and what is open What this session finished (#197: the NOAA record and its ten-run median, plan 005, S0, L1; the merge that unblocked #198), what is open (#199's regtest run and review, plan 005 L3 to L7, the uncommitted opportunity and GFS-Wave analysis), what waits on the owner, and the traps met: unpinned one-thread timings on a hybrid CPU, the load average counting the job itself, g++'s -ftrapping-math and lane loops, -Wpsabi without -march, pkill -f killing its own shell. Tested: scripts/vale.sh on the entry and docs/log/README.md, exit 0, 0 errors; lychee --offline on both, 0 errors Co-Authored-By: Claude Opus 5.5 --- docs/log/2026-10-18-goal-c3e7eecf-handover.md | 66 +++++++++++++++++++ docs/log/README.md | 1 + 2 files changed, 67 insertions(+) create mode 100644 docs/log/2026-10-18-goal-c3e7eecf-handover.md diff --git a/docs/log/2026-10-18-goal-c3e7eecf-handover.md b/docs/log/2026-10-18-goal-c3e7eecf-handover.md new file mode 100644 index 00000000..c224458c --- /dev/null +++ b/docs/log/2026-10-18-goal-c3e7eecf-handover.md @@ -0,0 +1,66 @@ +# Handover of goal `c3e7eecf`: the NOAA record, the ten-run median, points in lanes up to `W3SIN4` + +2026-10-18, an agent for M. Hoffmann (Claude Code session `ec4548a3-46a8-46e2-be0a-2d74c3e7eecf`). +What this session finished, what is open, and what it learned, for whoever continues. It picks up +where [the `abbb0456` handover](2026-10-18-goal-abbb0456-handover.md) stops; every claim names its +record, and nothing here was re-run for this entry. + +## What worked (merged) + +| Result | Record | +|---|---| +| The NOAA progress record after #191: pin `50a10730`, seven preset parts with `CPRT`, strict mode, the NaN fill, one copy of `VA` to the device on `full`, 22 regtest entries with 0 files different and the four FAIL rows kept, a status table up front; the READMEs link it; gaps 6, 7 and 13 of the readiness list marked at the pin | PR #197; `docs/NOAA_PROGRESS_RECORD_202610.md`, `docs/PRODUCTION_READINESS_202610.md` | +| The record's speed figure: median of ten alternated runs on `main`'s code (`5cd0140`), `full`, 32 threads, NOAA WW3 26.18 s, Kokkos resident 15.84 s, 1.65 times less wall time, every output byte-identical to run 1's; regtest pending ⚠ (22 entries, not the whole matrix) | PR #197; the record, section 3 | +| Plan 005, sea points in the vector lanes of the batched `W3SRCE`, with its per-lane parity argument and units S0 to L7 | `docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md` | +| S0: one thread of the resident batch is 51.1 % scalar per-point code lanes can reach, 39.0 % DIA, 8.9 % libm and libmvec | `docs/log/2026-10-18-srce-lanes-s0-profile.md` | +| L1: `SrcePoint` sections with the kernel's bodies moved verbatim, `w3srce_lanes`, `WW_KOKKOS_SRCE_LANES` (1 to 8, default 1, host only); `smoke` and `full` byte-identical with 1 and 8 | `kokkos/port_report/W3SRCE_KOKKOS.md`, "Points per team" | +| PR #198's conflict with `main` resolved by a merge (no history rewritten) | merge commit `41f4197` | + +## Open work + +| Item | State | Next step | +|---|---|---| +| PR #199, plan 005 L2: `W3SIN4`'s stress scan on 8-wide vectors | `smoke`, `full` byte-identical with 8 points per team; one-thread kernel 105.17 s to 99.77 s (5.1 %); CUDA `--build-only` clean; merged with `main` | the regtest matrix with `WW_KOKKOS_SRCE_LANES=8` (the matrix passes the variable through to its Kokkos runs, `regtests/cmake/ww3gpu_common.cmake`, `ww3gpu_kokkos_env`); the owner's review of the loop nesting and of the explicit vector types (the PR body) | +| Plan 005 L3: `W3SDS4` on lanes | not started; 24.8 % of the batch in S0, the largest unit left | split `w3sds4_point` as L2 split `W3SIN4`; the five libmvec sites (`st4_libmvec.hpp`) stay a loop over the lanes, each lane calling today's grouped code; time it pinned (`taskset -c 2`) as L2 was | +| Plan 005 L4 to L7 | not started | the integration and `w3srce_team` (9.9 %), `W3SPR4` and the small terms, the DIA layout, then a median of three at 32 threads on an idle machine | +| The opportunity and GFS-Wave analysis of 2026-10-18 | written by a sub-agent into the session's scratch directory, not committed; three of its claims checked (the `Kokkos::single` sites, the batched `W3SRCE` switch gate refusing `FLD2` and `SEED` and needing `LN1`, `7a23cd77` being plan 004's GFS.v17 commit) ⚠ the rest not checked | if wanted, redo it into `docs/compound/plans/`; its order for GFS-Wave was: read GFS.v17's switch, grid and compiler from the UFS weather model; plan 004; `W3FLD2` and the `SEED` and no-`LN1` paths in the batched `W3SRCE`; a `SHRD` build of `ww3_ufs1.1`; profile, restart pair, the Intel compiler; an ADR before any MPI work (ADR-0017) | + +## Waiting on the owner + +- The decisions plan 005 names: whether the host may call the transcribed vector libm + (`libm_device.hpp`) inside lane loops, and whether a sampled check is enough for `powf` and + `atan2f` (`docs/GPU_LIBM_BITS_202610.md`); whether to skip `W3SDS4`'s zero weights (not + bit-neutral: `0 x Inf` is NaN). +- The restart pair (`scripts/restart_pair.sh`), still never run. +- The `bench/results/` record of the ten-run median, once WW3's regtests have passed on `5cd0140` + beyond the 22 entries. +- Whether the regtests of #191 move the `regtested` cells of `kokkos/ports.tsv` for the resident + parts to "yes"; the ledger still says "no". +- From the `abbb0456` handover, still open: the strict-OUTG/POISON follow-up branch + (`goal/abbb0456-8241-strict-outg-poison`, no PR), the Fortran A/B harness branch, #196. + +## Left on disk, not in the repository + +- `~/code/ww3-gpu-2/noaa-timing.sh` (untracked): the timing of the record, `bash noaa-timing.sh + [N]`, N alternated runs each, the load printed at the start and the end. It moves `WW3/` to the + checked-out commit's pin first. +- `~/code/ww3-gpu-2` is on `goal/c3e7eecf-6142-srce-lanes-sin4`, `WW3/` at `50a10730`. + +## Traps met (for whoever continues) + +- **A one-thread timing on the i9-14900 is no measure unless pinned.** The same code took 113 s to + 155 s in four unpinned runs as the thread moved between performance and efficiency cores; pinned + to CPU 2 (`taskset -c 2`), three runs agreed within 0.1 s. +- **The load average counts the timed job's own threads.** A 32-thread run shows a load of 25 to 27 + with nothing else on the machine; read the load before the job starts. +- **g++ leaves a lane loop with selects scalar under its default `-ftrapping-math`**: it sinks the + values only one side of a select uses into a branch and will not speculate them back + ("unsupported control flow in loop"). Plan 005 uses GCC vector types instead of changing that + flag for every kernel; `-fopt-info-vec-all` on the shim's translation unit, inside the pinned + shell, tells which loops vectorised. +- **`serial-debug` has no `-march`**: a function that returns a 32-byte vector by value warns + `-Wpsabi` there; pass vectors by reference. +- **`pkill -f ` from a tool shell kills that shell too** when the pattern is in its own + command line; kill by PID. +- **Merged means finished**: work after a merged PR goes on a new `goal/` branch from `main`, as L2 + did after #197 merged. diff --git a/docs/log/README.md b/docs/log/README.md index 0fb31226..5c7c7b79 100644 --- a/docs/log/README.md +++ b/docs/log/README.md @@ -75,3 +75,4 @@ Date, author, issue/PR/WFIP. | 2026-10-18 | [One thread of the resident `W3SRCE` batch: 91 % in the port's own code, 9 % in libm](2026-10-18-srce-lanes-s0-profile.md) | plan 2026-10-18-005 S0, #197 | | 2026-10-18 | [Handover of goal `abbb0456`: what #191 delivered, the open branches (the review follow-up, plan 006, the Fortran A/B harness), what waits on the owner, the traps met](2026-10-18-goal-abbb0456-handover.md) | PR #191, PR #198 | | 2026-10-18 | [`W3SIN4`'s stress scan with one sea point per vector lane: bit-identical, 5.1 % off the one-thread `W3SRCE` kernel](2026-10-18-srce-lanes-sin4.md) | plan 2026-10-18-005 L2 | +| 2026-10-18 | [Handover of goal `c3e7eecf`: the NOAA record and its ten-run median, plan 005 up to L2 (#199), what waits on the owner, the traps met](2026-10-18-goal-c3e7eecf-handover.md) | PR #197, PR #199 | From 919c5cf9fe104345d3127c6e35ff621d542a55c5 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 19:05:13 -0300 Subject: [PATCH 3/4] kokkos: eight points per team by default on host backends (plan 005) The owner asked for 8 as the default of WW_KOKKOS_SRCE_LANES, so the lane W3SIN4 runs without a variable. A launch with fewer points than 8 x threads takes one point per team, so a small grid keeps every thread busy (ww3_ts1, one sea point, does); like the schedule, this decides only which points share a team. A device keeps one point per team. At 32 threads on full the default takes 2.6 % less ww3_shel wall time than one point per team. Tested: just kokkos-test serial-debug and openmp-release, 11/11 each, no warning; bash scripts/ww3-gpu.sh --ww3-gpu smoke and full --cpu --kokkos resident with no WW_KOKKOS_SRCE_LANES, WW_KOKKOS_STRICT=1 WW_KOKKOS_VA_POISON=1: every model file byte-identical to the Fortran of the same executable, every file but the NetCDF to --ww3; full, 32 threads, 3 alternated runs each: 15.88 s with 1 point per team, 15.47 s with the default, every out_grd.ww3 identical; python3 scripts/port_score.py --write then --check; scripts/vale.sh and lychee --offline on the changed Markdown, 0 errors Co-Authored-By: Claude Opus 5.5 --- docs/GLOSSARY.md | 2 +- ...026-10-18-005-srce-points-in-lanes-plan.md | 2 +- docs/log/2026-10-18-goal-c3e7eecf-handover.md | 2 +- docs/log/2026-10-18-srce-lanes-sin4.md | 43 +++++++++++++++++++ kokkos/PORT_STATUS.md | 2 +- kokkos/port_report/W3SRCE_KOKKOS.md | 7 ++- kokkos/src/fortran_iface/srce_shim.cpp | 21 +++++---- site/data/port_score.tsv | 2 +- 8 files changed, 65 insertions(+), 16 deletions(-) diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index 32605d35..b6e0bc1d 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -704,7 +704,7 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | resident VA / `ww_va_host_read` / `ww_va_host_write` / `ww_va_device` / `ww_va_report` / `ww_srce_batch_va` | the device copy of the spectrum `VA(NSPEC,0:NSEALM)` that lives for the run (WFIP-0005), with a state per grid saying which copy holds the newer bits (`HOST`, `DEVICE`, `BOTH`); the C calls W3WAVE makes before a host read or write of `VA`, the one a kernel shim makes before reading the device copy, the traffic report, and the batched `W3SRCE` that works in place on the device copy `(v)` [kokkos/src/fortran_iface/resident_c.hpp](../kokkos/src/fortran_iface/resident_c.hpp) | The protocol of `Kokkos::DualView` (modify, sync) over WW3's own array; with no kernel on the device copy no call copies anything. Wrapped by `W3KOKKOS_VA_HOST_READ` / `_WRITE` in `w3kokkos_residentmd.F90` on the fork. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | | `WW_KOKKOS_VA_RESIDENT` / `WW_KOKKOS_VA_COUNT` | diagnostics of the resident VA, ignoring `WW_KOKKOS_ALL`: `=1` makes every host sync call push a host-newer `VA` to the device and copy it back, so a host write without its sync call shows in the output; the count prints the traffic line and the copy counters at exit `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset. On the CPU both copies are `memcpy`, so the bits cannot change. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | | `WW_KOKKOS_VA_CHECK` / `WW_KOKKOS_VA_POISON` | diagnostics of the resident VA, read once, ignoring `WW_KOKKOS_ALL`, which compare or overwrite only the stale copy: `VA_CHECK=1` makes every sync call check what the state claims before it copies (unless `HOST`, the host `VA` still holds the bits of the last sync, so a host write without its `W3KOKKOS_VA_HOST_WRITE` stops the model; with `BOTH`, `d_va` holds them too, so a kernel that wrote `d_va` without marking it does); `VA_POISON=1` fills the host `VA` with a quiet NaN after every kernel write of `d_va`, so a host read without its sync, or a column the partial read misses, puts NaN in the output `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset; a correct run writes the same bits with either. Added by the mutation audit of 2026-10-17. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | -| `WW_KOKKOS_SRCE_LANES` | sea points per team of the batched `W3SRCE` launch on a host backend, 1 to 8, read once; each point runs the statements it runs alone, the points of a team take turns `(v)` [kokkos/src/fortran_iface/srce_shim.cpp](../kokkos/src/fortran_iface/srce_shim.cpp) | Default 1, today's one point per team; a device always uses 1. The lane layer of plan 005: later units vectorise across the points of a team. | [docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md](compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md) | +| `WW_KOKKOS_SRCE_LANES` | the most sea points per team of the batched `W3SRCE` launch on a host backend, 1 to 8, read once; each point runs the statements it runs alone, the points of a team take turns `(v)` [kokkos/src/fortran_iface/srce_shim.cpp](../kokkos/src/fortran_iface/srce_shim.cpp) | Default 8 (the owner, 2026-10-18), 1 for one point per team; a launch with fewer points than lanes times threads uses 1, and a device always uses 1. The lane layer of plan 005, which `W3SIN4`'s stress scan vectorises across (L2). | [docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md](compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md) | | `WW_KOKKOS_OUTG_STAGED` | a diagnostic of the outg unit, read once: `=1` makes a host backend take the device's path of `ww_outg_va`, the inputs copied into the shim's own buffers, the fields computed into buffers, and only the fields a host reader needs copied back, so a field the copy-back selection misses keeps its old bits in WW3's array and shows in `out_grd.ww3` `(v)` [kokkos/src/fortran_iface/outg_shim.cpp](../kokkos/src/fortran_iface/outg_shim.cpp) | Default unset: on Serial and OpenMP the kernel writes WW3's arrays in place. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | | `WW_KOKKOS_STRICT` | environment variable read once by `ww_kokkos_strict()`: `=1` makes every place where a part asked on Kokkos would run the Fortran stop the model with `EXTCDE` and a `NDSE` line naming the part and the reason: `KOKKOS_PART` under `WW_KOKKOS_ALL=1`, the two `W3SRCE` set-up fallbacks and the per-step `W3SRCE` gate (`W3KOKKOS_SRCE_STEP`). Unset, those places run the Fortran as before and the shim prints one line at exit, `ww_kokkos: calls left on the Fortran (...): SRCE= ...` `(v)` [kokkos/src/fortran_iface/part_flags.cpp](../kokkos/src/fortran_iface/part_flags.cpp) | Default unset. The regtest matrix's Kokkos run sets it, its Fortran runs unset it, so a pair cannot compare the Fortran with itself unseen. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 24 | | `WW_KOKKOS_PRESET` / `presets.txt` | environment variable naming a line `PRESET ` of `kokkos/src/fortran_iface/presets.txt`: `ww_kokkos_flag` turns its parts on as if each `WW_KOKKOS_` were `1`; a set `WW_KOKKOS_` still wins, and `W3INIT` stops on a name the file lacks. `resident` is the only preset (`SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT`) `(v)` [kokkos/src/fortran_iface/presets.txt](../kokkos/src/fortran_iface/presets.txt) | The file is the one list: CMake compiles it into the shim, `scripts/ww3-gpu.sh --kokkos resident` and the matrix's `WW3GPU_KOKKOS=resident` read it. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 25 | diff --git a/docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md b/docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md index 8f26e7f0..38681659 100644 --- a/docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md +++ b/docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md @@ -86,7 +86,7 @@ change a bit. | Unit | What | Done when | |---|---|---| | S0 | One-thread profile of today's batch, libm separated from the rest | the shares below are measured; the go/no-go of S0 is applied | -| L1 | The lane layer: teams of `W` points (`WW_KOKKOS_SRCE_LANES`, 1 to 8, 1 reproduces today), each section run for the team's points in turn with today's scalar routines | **Done** 2026-10-18: `full` and `smoke` byte-identical with 1 and 8 (`kokkos/port_report/W3SRCE_KOKKOS.md`, "Points per team"). The point-minor scratch moves to L2, the first unit that reads it | +| L1 | The lane layer: teams of `W` points (`WW_KOKKOS_SRCE_LANES`, 1 to 8, 1 reproduces today, default 8 since L2), each section run for the team's points in turn with today's scalar routines | **Done** 2026-10-18: `full` and `smoke` byte-identical with 1 and 8 (`kokkos/port_report/W3SRCE_KOKKOS.md`, "Points per team"). The point-minor scratch moves to L2, the first unit that reads it | | L2 | `W3SIN4` on lanes (its stress loop, `st4_sin4.hpp:198-249`) | **Done** 2026-10-18: byte-identical on `smoke` and `full`; one-thread `W3SRCE` kernel 105.17 s to 99.77 s, 5.1 % ([log](../../log/2026-10-18-srce-lanes-sin4.md)) | | L3 | `W3SDS4` on lanes, the libmvec sites left as lane loops | the same | | L4 | The integration and the initialisation (`srce_kernel.hpp:177-400`) | the same | diff --git a/docs/log/2026-10-18-goal-c3e7eecf-handover.md b/docs/log/2026-10-18-goal-c3e7eecf-handover.md index c224458c..eb6e1726 100644 --- a/docs/log/2026-10-18-goal-c3e7eecf-handover.md +++ b/docs/log/2026-10-18-goal-c3e7eecf-handover.md @@ -13,7 +13,7 @@ record, and nothing here was re-run for this entry. | The record's speed figure: median of ten alternated runs on `main`'s code (`5cd0140`), `full`, 32 threads, NOAA WW3 26.18 s, Kokkos resident 15.84 s, 1.65 times less wall time, every output byte-identical to run 1's; regtest pending ⚠ (22 entries, not the whole matrix) | PR #197; the record, section 3 | | Plan 005, sea points in the vector lanes of the batched `W3SRCE`, with its per-lane parity argument and units S0 to L7 | `docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md` | | S0: one thread of the resident batch is 51.1 % scalar per-point code lanes can reach, 39.0 % DIA, 8.9 % libm and libmvec | `docs/log/2026-10-18-srce-lanes-s0-profile.md` | -| L1: `SrcePoint` sections with the kernel's bodies moved verbatim, `w3srce_lanes`, `WW_KOKKOS_SRCE_LANES` (1 to 8, default 1, host only); `smoke` and `full` byte-identical with 1 and 8 | `kokkos/port_report/W3SRCE_KOKKOS.md`, "Points per team" | +| L1: `SrcePoint` sections with the kernel's bodies moved verbatim, `w3srce_lanes`, `WW_KOKKOS_SRCE_LANES` (1 to 8, host only; default 1 in #197, 8 since #199); `smoke` and `full` byte-identical with 1 and 8 | `kokkos/port_report/W3SRCE_KOKKOS.md`, "Points per team" | | PR #198's conflict with `main` resolved by a merge (no history rewritten) | merge commit `41f4197` | ## Open work diff --git a/docs/log/2026-10-18-srce-lanes-sin4.md b/docs/log/2026-10-18-srce-lanes-sin4.md index 5e1a9682..0cb622a5 100644 --- a/docs/log/2026-10-18-srce-lanes-sin4.md +++ b/docs/log/2026-10-18-srce-lanes-sin4.md @@ -66,6 +66,17 @@ take less time when eight sea points run it at once, one per vector lane? earlier runs, as the thread moved between performance and efficiency cores. No 32-thread figure is given: the machine was shared during those runs. +- (v) 32 threads, the owner's default of 8 points per team (2026-10-18), `full`, `ww3_shel` wall + time, Kokkos run only, alternated, 22:13 to 22:20 UTC: 15.88, 15.92, 15.84 s with + `WW_KOKKOS_SRCE_LANES=1`, 15.44, 15.54, 15.47 s with the default; medians 15.88 and 15.47 s, + 2.6 % less, every `out_grd.ww3` byte-identical. The 1-minute load read before each run rose from + 22 to 38: these back-to-back 32-thread runs themselves ⚠ (inferred; each set agrees within + 0.1 s). The script is `default8.sh` below. +- (v) With the default and no variable set: `smoke` and `full` byte-identical to the Fortran of the + same executable (strict, poisoned), `just kokkos-test serial-debug` and `openmp-release` 11/11, + no warning. `smoke` (`ww3_ts1`, one sea point) takes one point per team under the default: a + launch with fewer points than 8 x threads does. + ## What it changes The lane layer pays where a per-point loop has ordered sums and branches g++ will not vectorise. @@ -98,3 +109,35 @@ for rep in 1 2 3; do done done ``` + +`default8.sh`, the 32-thread comparison of the default (its first part runs the gates and the +parity checks above): + +```bash +#!/usr/bin/env bash +set -uo pipefail +cd /home/hoffmann/code/ww3-gpu-2 +S=/tmp/claude-1000/-home-hoffmann-code-ww3-gpu-2/ec4548a3-46a8-46e2-be0a-2d74c3e7eecf/scratchpad +for b in serial-debug openmp-release; do + just kokkos-test $b > $S/kt-$b.log 2>&1; echo "$b exit=$? $(grep -E 'tests passed' $S/kt-$b.log | tail -1) warnings=$(grep -c 'warning:' $S/kt-$b.log)" +done +for c in smoke full; do + env -u WW_KOKKOS_SRCE_LANES WW_KOKKOS_STRICT=1 WW_KOKKOS_VA_POISON=1 bash scripts/ww3-gpu.sh --ww3-gpu $c --cpu --kokkos resident > $S/$c-default.log 2>&1 + echo "$c exit=$?"; grep -E "points per team|^>> (every|the two)" $S/$c-default.log +done +# 32 threads, full, Kokkos only, alternated: lanes 1 against the default +exe=build/ww3-gpu-openmp/ww3/bin/ww3_shel; case_dir=build/runs/ww3-gpu-full-openmp/kokkos +for rep in 1 2 3; do + for w in 1 default; do + d=$S/t32/$w-$rep; rm -rf $d; mkdir -p $d + for f in ST4TABUHF2.bin depth.inp mask.inp mod_def.ww3 namelists.nml ww3_shel.nml; do cp $case_dir/$f $d/; done + l0=$(cut -d' ' -f1 /proc/loadavg) + if [ $w = 1 ]; then lv=(WW_KOKKOS_SRCE_LANES=1); else lv=(); fi + t0=$(date +%s.%N) + (cd $d && env -u WW_KOKKOS_SRCE_LANES OMP_PROC_BIND=false WW_KOKKOS_ALL=0 WW_KOKKOS_PRESET=resident "${lv[@]}" /home/hoffmann/code/ww3-gpu-2/$exe > out 2> err) + t1=$(date +%s.%N) + same=$(cmp -s $d/out_grd.ww3 $case_dir/out_grd.ww3 && echo identical || echo DIFFERENT) + echo "rep $rep lanes $w: ww3_shel $(echo "$t1 - $t0" | bc) s, out_grd.ww3 $same, load $l0" + done +done +``` diff --git a/kokkos/PORT_STATUS.md b/kokkos/PORT_STATUS.md index f2cb0976..acf904e5 100644 --- a/kokkos/PORT_STATUS.md +++ b/kokkos/PORT_STATUS.md @@ -49,7 +49,7 @@ Profile: full (bench_small, 120 x 80, 9 520 sea points, 12 h), Release -pg, swit | ported | **99.3 %** | 17270 of 263375 (6.56 %) | 63 of 1023 (6.16 %) | 25 of 147 | WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag | | regtested | **48.6 %** | 2689 of 263375 (1.02 %) | 12 of 1023 (1.17 %) | 6 of 147 | WW3's regtests ran with it on Kokkos and agreed with the Fortran | -The port itself: 60034 lines in 350 files, the C++ under kokkos/src (every .cpp and .hpp line). +The port itself: 60037 lines in 350 files, the C++ under kokkos/src (every .cpp and .hpp line). On a GPU: none yet. The CUDA build links; the RTX 4090 run, `just smoke-ww3-gpu gpu`, is the owner's, supervised. diff --git a/kokkos/port_report/W3SRCE_KOKKOS.md b/kokkos/port_report/W3SRCE_KOKKOS.md index 7f3bbcdc..ad155043 100644 --- a/kokkos/port_report/W3SRCE_KOKKOS.md +++ b/kokkos/port_report/W3SRCE_KOKKOS.md @@ -204,8 +204,10 @@ four bodies found verbatim, in order, in the new file, 2026-10-18). `w3srce_team for one point as before. `w3srce_lanes` runs them for up to eight points per team: each section for every point still in the integration loop, one after the other, and each point's DIA on the whole team. A point reads and writes only its own arguments and work arrays, so the order in which the -points of a team take their turns cannot change a bit. `WW_KOKKOS_SRCE_LANES` (1 to 8, default 1, -host backends only) sets the number; the exit counters print it. This unit vectorises nothing yet: +points of a team take their turns cannot change a bit. `WW_KOKKOS_SRCE_LANES` (1 to 8, host +backends only) sets the most points per team; the exit counters print it. It defaulted to 1 in +L1; since L2 the default is 8 (the owner, 2026-10-18), and a launch with fewer points than +8 x threads takes one point per team, so every thread still gets a team. This unit vectorises nothing yet: it is the layer the later units of [plan 005](../../docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md) vectorise across. @@ -232,6 +234,7 @@ lane's operations those of the scalar statement in its order, each lane's stress |---|---| | `smoke`, `full`, `WW_KOKKOS_SRCE_LANES=8`, strict, poisoned | every model file byte-identical to the Fortran of the same executable; every file but the NetCDF to `--ww3`'s (v) | | `srce.batch.kernel`, `full`, one thread on a performance core, 3 alternated runs each | 105.17 s with 1 point per team, 99.77 s with 8: 5.1 % less (v, [log](../../docs/log/2026-10-18-srce-lanes-sin4.md)) | +| `ww3_shel`, `full`, 32 threads, 3 alternated runs each | 15.88 s with 1, 15.47 s with the default 8: 2.6 % less (v, the same log) | | `bash scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident --build-only` | compiles and links, no warning; the lane path is host-only and a device keeps one point per team; nothing run on a device (v) | ## Benchmarks diff --git a/kokkos/src/fortran_iface/srce_shim.cpp b/kokkos/src/fortran_iface/srce_shim.cpp index b3a0880a..b6bcb744 100644 --- a/kokkos/src/fortran_iface/srce_shim.cpp +++ b/kokkos/src/fortran_iface/srce_shim.cpp @@ -133,21 +133,21 @@ const Flags& flags() { env_is_one("WW_KOKKOS_SRCE_STAGED")}; return f; } -/// Points per team of the batched launch on a host backend: WW_KOKKOS_SRCE_LANES, 1 to -/// kSrceMaxLanes, read once; 1, the default, is one point per team. A team of several points -/// runs each point's statements as one point alone runs them (w3srce_lanes), so the value -/// changes which points share a team, never a bit (plan 2026-10-18-005). A device keeps one -/// point per team. +/// The most points per team of the batched launch on a host backend: WW_KOKKOS_SRCE_LANES, 1 to +/// kSrceMaxLanes, read once; kSrceMaxLanes by default (the owner, 2026-10-18), 1 for one point +/// per team. A team of several points runs each point's statements as one point alone runs them +/// (w3srce_lanes), so the value changes which points share a team, never a bit (plan +/// 2026-10-18-005). A device keeps one point per team. int srce_lanes() { static const int lanes = [] { - int w = 1; + int w = kSrceMaxLanes; if (const char* v = std::getenv("WW_KOKKOS_SRCE_LANES")) { char* end = nullptr; const long x = std::strtol(v, &end, 10); if (end != v && *end == '\0' && x >= 1 && x <= kSrceMaxLanes) w = static_cast(x); else - std::fprintf(stderr, "ww_kokkos: WW_KOKKOS_SRCE_LANES='%s' ignored, using 1\n", v); + std::fprintf(stderr, "ww_kokkos: WW_KOKKOS_SRCE_LANES='%s' ignored, using %d\n", v, w); } return w; }(); @@ -160,7 +160,7 @@ void print_count() { std::fprintf(stderr, "ww_kokkos: W3SRCE batched calls: %lld\n", g_batch_calls.load()); std::fprintf(stderr, "ww_kokkos: W3SRCE batched launches: %lld\n", g_batch_launches.load()); std::fprintf(stderr, "ww_kokkos: W3SRCE batched points: %lld\n", g_batch_points.load()); - std::fprintf(stderr, "ww_kokkos: W3SRCE batch points per team (WW_KOKKOS_SRCE_LANES): %d\n", + std::fprintf(stderr, "ww_kokkos: W3SRCE batch points per team at most (WW_KOKKOS_SRCE_LANES): %d\n", Kokkos::SpaceAccessibility::accessible ? srce_lanes() : 1); @@ -562,7 +562,10 @@ void run_batch(GridCtx& g, const st4::Params& q, const st4::Fields& f, const snl // schedule decides only which thread runs a point, never the point's arithmetic. bool launched = false; if constexpr (Kokkos::SpaceAccessibility::accessible) { - const int lanes = srce_lanes(); + // Fewer points than lanes x threads: one point per team, so every thread still gets work + // (ww3_ts1 has one sea point, full 9 520). Like the schedule below, this decides only which + // points share a team. + const int lanes = n >= srce_lanes() * ex.concurrency() ? srce_lanes() : 1; if (lanes > 1) { const int league = (n + lanes - 1) / lanes; const auto body_lanes = KOKKOS_LAMBDA(const Member& team) { diff --git a/site/data/port_score.tsv b/site/data/port_score.tsv index b82f16bd..4911a9e1 100644 --- a/site/data/port_score.tsv +++ b/site/data/port_score.tsv @@ -7,4 +7,4 @@ of the run time, regtested 48.6 % 12 of 1 023 routines WW3's regtests ran with i of the lines of code, translated 25.80 % 67 947 of 263 375 Fortran lines of the translated routines, of every line in WW3's model/src, in 65 of 147 files of the lines of code, ported 6.56 % 17 270 of 263 375 Fortran lines of the ported routines, of every line in WW3's model/src, in 25 of 147 files of the lines of code, regtested 1.02 % 2 689 of 263 375 Fortran lines of the regtested routines, of every line in WW3's model/src, in 6 of 147 files -lines of Kokkos C++ 60 034 in 350 files under kokkos/src: the port itself +lines of Kokkos C++ 60 037 in 350 files under kokkos/src: the port itself From d02b92efd4397bd1c450cf15de32506919c43331 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 19:16:07 -0300 Subject: [PATCH 4/4] docs/log: WW3's regtests of PR #199 with eight points per team by default just regtest-selected at 919c5cf: ww3_tp2.15 x4 and ww3_ts2 ST4_PR3_UQ give test.comp 0 files different and nccmp-tol PASS, smoke byte-identical, the verdicts of main's run. The four ww3_tp2.15 entries ran the lane path (276 and 5 480 points per launch against the 48 of 8 lanes x 6 threads); ww3_ts2, 24 points per launch, ran one point per team. Tested: just regtest-selected (5 entries PASS, 8/8 ctest, smoke byte-identical); scripts/vale.sh on the entry, exit 0 Co-Authored-By: Claude Opus 5.5 --- docs/log/2026-10-18-srce-lanes-sin4.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/log/2026-10-18-srce-lanes-sin4.md b/docs/log/2026-10-18-srce-lanes-sin4.md index 0cb622a5..0cdb1bb9 100644 --- a/docs/log/2026-10-18-srce-lanes-sin4.md +++ b/docs/log/2026-10-18-srce-lanes-sin4.md @@ -77,6 +77,18 @@ take less time when eight sea points run it at once, one per vector lane? no warning. `smoke` (`ww3_ts1`, one sea point) takes one point per team under the default: a launch with fewer points than 8 x threads does. +- (v) WW3's regtests with the default, `just regtest-selected` at `919c5cf` (`WW3@50a10730`, + 2026-10-18 22:05 to 22:15 UTC, the resident preset, lab switch, strict, 6 threads per run): + `ww3_tp2.15` default, `-g 5km`, `input_const`, `input_rho` and `ww3_ts2` `ST4_PR3_UQ` give + `test.comp` 0/23, 0/23, 0/23, 0/25, 0/22, `nccmp-tol` PASS, no second Fortran run needed, and + `smoke` byte-identical; the same verdicts as `main`'s run of the same selection (the owner's, + `5cd0140`). The Kokkos logs give 276 points per batched launch for the four `ww3_tp2.15` entries + (5 480 for `-g 5km`), at least the 48 of 8 lanes x 6 threads, so they ran the lane path; + `ww3_ts2` has 24 points per launch and ran one point per team. +- ⚠ Entries 3, 5, 6 and 7 took 100 to 112 s against 40 to 53 s in `main`'s run. `ww3_ts2` did not + take the lane path and slowed by the same factor, and the load was 17.7 when the run started, + so the machine and not the lanes is taken as the cause; not measured. + ## What it changes The lane layer pays where a per-point loop has ordered sums and branches g++ will not vectorise.