Strict W3OUTG and POISON in every run, the resident restart pair, W3SDS4 on vector lanes

.github/workflows/health.yml

@@ -1,6 +1,6 @@
 # health — the combinations of scripts/ww3-gpu.sh that CI keeps alive. For now one, by the owner's
 # word (2026-10-14): "h1 - --ww3-gpu + --cpu ... this combination is the only one that needs to be
-# present in CI for now". h1 builds WW3 from the WW3/ submodule (h0ffmann/WW3, ww3-gpu/kokkos-wire-base)
+# present in CI for now". h1 builds WW3 from the WW3/ submodule (h0ffmann/WW3, ww3-gpu/kokkos-pnt)
 # with the KOKKOS switch and the ww_kokkos package on OpenMP, runs regtest ww3_ts1 with the
 # Fortran and the Kokkos DIA, then builds --ww3 and compares pair to pair (nccmp-tol). The other
 # combinations and why they are not here: README.md, "Health".

.gitmodules

@@ -6,7 +6,7 @@
 [submodule "WW3"]
 	path = WW3
 	url = git@github.com:h0ffmann/WW3.git
-	branch = ww3-gpu/kokkos-wire-base
+	branch = ww3-gpu/kokkos-pnt
 [submodule "WW4"]
 	path = WW4
 	url = git@github.com:h0ffmann/WW4.git

AGENTS.md

@@ -308,7 +308,10 @@ those transfers.
   are not resident.
 - **How it is checked.** A change to the resident sync reports `WW_KOKKOS_VA_COUNT=1` (bytes host
   to device and device to host) on `full` before and after; the whole-spectrum copies per step
-  must not grow, or the change is refused.
+  must not grow, or the change is refused. Every Kokkos run of `scripts/ww3-gpu.sh` and of
+  `just ww3-gpu-matrix` has `WW_KOKKOS_VA_POISON=1`: the stale host `VA` is filled with NaN after
+  each kernel write, so a host reader that misses its sync changes an output file and the byte
+  comparison fails. A caller's `WW_KOKKOS_VA_POISON=0` turns it off; the Fortran runs never have it.
 
 ## Bit for bit with the Fortran (hard rule)
 

CONTRIBUTING.md

@@ -146,7 +146,9 @@ A mistake in a figure is as welcome as one in the text: the gallery links an iss
   the host only before an output or a restart that needs it. A whole-spectrum copy per step, per
   grid or per `W3WAVE` call is refused, even when it is the simpler fix: on a GPU it is a bus
   transfer every step. A change to the sync reports `WW_KOKKOS_VA_COUNT=1` on `full` before and
-  after.
+  after. `scripts/ww3-gpu.sh` and `just ww3-gpu-matrix` set `WW_KOKKOS_VA_POISON=1` in every Kokkos
+  run, so a host reader that misses its sync reads NaN and fails the byte comparison;
+  `WW_KOKKOS_VA_POISON=0` in the caller's environment turns it off, and the Fortran runs never have it.
 - `codemeta.json` is generated from `CITATION.cff` and `.zenodo.json` (`just codemeta`); never
   edit it by hand.
 - Run `just kokkos-test serial-debug` after touching `kokkos/`, and `bash -n` plus

docs/GLOSSARY.md

@@ -703,10 +703,10 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md).
 | `WW_KOKKOS_SRCE_STAGED` / input shadow | a diagnostic of `ww_srce_batch_va`, read once: `=1` makes a host backend stage the per-step inputs (`BJSEA`, `BWN1`, `BCG1`, `BPIN`, `BPSC`, `BWC`, `BTB`, `BTI`) into the batch buffers as a device does, skipping an input whose bits equal its *shadow* (the host copy of what the buffer holds, compared with `memcmp`), instead of reading them in place; a wrong skip would show as a stale input in the output. With `WW_KOKKOS_SRCE_COUNT=1` the shim prints the input copies and bytes H2D, the unchanged inputs, and the copies back D2H, projected for a device on a host backend `(v)` [kokkos/src/fortran_iface/srce_shim.cpp](../kokkos/src/fortran_iface/srce_shim.cpp) | Default unset: on Serial and OpenMP the kernel reads and writes the caller's packed arrays in place, with no copy (WFIP-0005 unit 4). | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) |
 | resident VA / `ww_va_host_read` / `ww_va_host_write` / `ww_va_device` / `ww_va_report` / `ww_srce_batch_va` | the device copy of the spectrum `VA(NSPEC,0:NSEALM)` that lives for the run (WFIP-0005), with a state per grid saying which copy holds the newer bits (`HOST`, `DEVICE`, `BOTH`); the C calls W3WAVE makes before a host read or write of `VA`, the one a kernel shim makes before reading the device copy, the traffic report, and the batched `W3SRCE` that works in place on the device copy `(v)` [kokkos/src/fortran_iface/resident_c.hpp](../kokkos/src/fortran_iface/resident_c.hpp) | The protocol of `Kokkos::DualView` (modify, sync) over WW3's own array; with no kernel on the device copy no call copies anything. Wrapped by `W3KOKKOS_VA_HOST_READ` / `_WRITE` in `w3kokkos_residentmd.F90` on the fork. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) |
 | `WW_KOKKOS_VA_RESIDENT` / `WW_KOKKOS_VA_COUNT` | diagnostics of the resident VA, ignoring `WW_KOKKOS_ALL`: `=1` makes every host sync call push a host-newer `VA` to the device and copy it back, so a host write without its sync call shows in the output; the count prints the traffic line and the copy counters at exit `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset. On the CPU both copies are `memcpy`, so the bits cannot change. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) |
-| `WW_KOKKOS_VA_CHECK` / `WW_KOKKOS_VA_POISON` | diagnostics of the resident VA, read once, ignoring `WW_KOKKOS_ALL`, which compare or overwrite only the stale copy: `VA_CHECK=1` makes every sync call check what the state claims before it copies (unless `HOST`, the host `VA` still holds the bits of the last sync, so a host write without its `W3KOKKOS_VA_HOST_WRITE` stops the model; with `BOTH`, `d_va` holds them too, so a kernel that wrote `d_va` without marking it does); `VA_POISON=1` fills the host `VA` with a quiet NaN after every kernel write of `d_va`, so a host read without its sync, or a column the partial read misses, puts NaN in the output `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset; a correct run writes the same bits with either. Added by the mutation audit of 2026-10-17. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) |
+| `WW_KOKKOS_VA_CHECK` / `WW_KOKKOS_VA_POISON` | diagnostics of the resident VA, read once, ignoring `WW_KOKKOS_ALL`, which compare or overwrite only the stale copy: `VA_CHECK=1` makes every sync call check what the state claims before it copies (unless `HOST`, the host `VA` still holds the bits of the last sync, so a host write without its `W3KOKKOS_VA_HOST_WRITE` stops the model; with `BOTH`, `d_va` holds them too, so a kernel that wrote `d_va` without marking it does); `VA_POISON=1` fills the host `VA` with a quiet NaN after every kernel write of `d_va`, so a host read without its sync, or a column the partial read misses, puts NaN in the output `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset in the shim; a correct run writes the same bits with either. Since 2026-10-18 `scripts/ww3-gpu.sh` and the regtest matrix set `WW_KOKKOS_VA_POISON=1` in every Kokkos run (a caller's `=0` turns it off) and unset it in the Fortran runs. Added by the mutation audit of 2026-10-17. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) |
 | `WW_KOKKOS_SRCE_LANES` | the most sea points per team of the batched `W3SRCE` launch on a host backend, 1 to 8, read once; each point runs the statements it runs alone, the points of a team take turns `(v)` [kokkos/src/fortran_iface/srce_shim.cpp](../kokkos/src/fortran_iface/srce_shim.cpp) | Default 8 (the owner, 2026-10-18), 1 for one point per team; a launch with fewer points than lanes times threads uses 1, and a device always uses 1. The lane layer of plan 005, which `W3SIN4`'s stress scan vectorises across (L2). | [docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md](compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md) |
 | `WW_KOKKOS_OUTG_STAGED` | a diagnostic of the outg unit, read once: `=1` makes a host backend take the device's path of `ww_outg_va`, the inputs copied into the shim's own buffers, the fields computed into buffers, and only the fields a host reader needs copied back, so a field the copy-back selection misses keeps its old bits in WW3's array and shows in `out_grd.ww3` `(v)` [kokkos/src/fortran_iface/outg_shim.cpp](../kokkos/src/fortran_iface/outg_shim.cpp) | Default unset: on Serial and OpenMP the kernel writes WW3's arrays in place. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) |
-| `WW_KOKKOS_STRICT` | environment variable read once by `ww_kokkos_strict()`: `=1` makes every place where a part asked on Kokkos would run the Fortran stop the model with `EXTCDE` and a `NDSE` line naming the part and the reason: `KOKKOS_PART` under `WW_KOKKOS_ALL=1`, the two `W3SRCE` set-up fallbacks and the per-step `W3SRCE` gate (`W3KOKKOS_SRCE_STEP`). Unset, those places run the Fortran as before and the shim prints one line at exit, `ww_kokkos: calls left on the Fortran (...): SRCE=<n> ...` `(v)` [kokkos/src/fortran_iface/part_flags.cpp](../kokkos/src/fortran_iface/part_flags.cpp) | Default unset. The regtest matrix's Kokkos run sets it, its Fortran runs unset it, so a pair cannot compare the Fortran with itself unseen. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 24 |
+| `WW_KOKKOS_STRICT` | environment variable read once by `ww_kokkos_strict()`: `=1` makes every place where a part asked on Kokkos would run the Fortran stop the model with `EXTCDE` and a `NDSE` line naming the part and the reason: `KOKKOS_PART` under `WW_KOKKOS_ALL=1`, the two `W3SRCE` set-up fallbacks, the per-step `W3SRCE` gate (`W3KOKKOS_SRCE_STEP`), `W3KOKKOS_CPRT`'s two fallbacks and `W3KOKKOS_OUTG`'s call of the Fortran `W3OUTG` for a field its translation leaves to the Fortran (U3 Stokes, skewness, WBT, an unallocated 3-D field). Unset, those places run the Fortran as before and the shim prints one line at exit, `ww_kokkos: calls left on the Fortran (...): SRCE=<n> ...` `(v)` [kokkos/src/fortran_iface/part_flags.cpp](../kokkos/src/fortran_iface/part_flags.cpp) | Default unset. The regtest matrix's Kokkos run sets it when it names its parts (a preset or a part list), not under `WW3GPU_KOKKOS=all`, where a fallback is counted; its Fortran runs unset it, so a pair cannot compare the Fortran with itself unseen. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 24 |
 | `WW_KOKKOS_PRESET` / `presets.txt` | environment variable naming a line `PRESET <name> <PART,...>` of `kokkos/src/fortran_iface/presets.txt`: `ww_kokkos_flag` turns its parts on as if each `WW_KOKKOS_<PART>` were `1`; a set `WW_KOKKOS_<PART>` still wins, and `W3INIT` stops on a name the file lacks. `resident` is the only preset (`SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT`) `(v)` [kokkos/src/fortran_iface/presets.txt](../kokkos/src/fortran_iface/presets.txt) | The file is the one list: CMake compiles it into the shim, `scripts/ww3-gpu.sh --kokkos resident` and the matrix's `WW3GPU_KOKKOS=resident` read it. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 25 |
 | mutation audit / mutant (equivalent, surviving) | a deliberate, minimal breakage of a piece of the port, applied in a scratch tree and never committed, to ask whether the checks that call the piece correct would fail without it: a mutant is *detected* when a check fails, *surviving* when every check passes, and *equivalent* when it cannot change any bit the case computes (it is not a defect the case can show) `(v)` [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | First run 2026-10-17, on the CPU. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) |
 | `WW_KOKKOS_ALL` / `ww_kokkos_flag` | the master run-time flag of every Kokkos part, and the one C function that resolves each `WW_KOKKOS_<PART>`: a set part flag wins (`1` on, anything else off), an unset one follows `WW_KOKKOS_ALL=1` (on) or stays off `(v)` [kokkos/src/fortran_iface/part_flags.cpp](../kokkos/src/fortran_iface/part_flags.cpp) | "100 %" means every wired part, not all of WW3; under the master a part whose translation does not cover the switch file stays Fortran. `W3INIT` logs one "Kokkos parts" line; `scripts/ww3-gpu.sh --kokkos all\|none\|<PART,...>` sets it. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) |

docs/NOAA_PROGRESS_RECORD_202610.md

@@ -181,7 +181,7 @@ tables in `docs/data/ww3gpu_matrix_*_pin-50a10730_2026-10-18.tsv`.
   `ww3_grid` namelist read against the test switch). None is marked expected-to-fail.
 - **Checked (v).** The DIA (`W3SNL1`) alone passes the `NL1` entries of the regtest matrix.
 - **Not shown.** Parity with the Intel compiler; parity on a GPU (judged by tolerance); restart
-  reproduction (`scripts/restart_pair.sh` exists and has not run); `ww3_multi` under MPI.
+  reproduction beyond one `SHRD` case (`scripts/restart_pair.sh` ran on `full` at `WW3/` `50a10730` and `fac74069`, OpenMP backend, 12 hours, every comparison identical; [log](log/2026-10-18-resident-restart-pair.md)); `ww3_multi` under MPI.
 
 ## 5. Where the CPU time goes
 
@@ -380,7 +380,7 @@ Sections 3 and 4 of `W3SNL1` read the spectrum at the quadruplet bins through th
 | Median of three, alternated with NOAA WW3 | add `--repeat 3` |
 | Stop on any fallback; catch a missed sync | `WW_KOKKOS_STRICT=1 WW_KOKKOS_VA_POISON=1` in the environment |
 | The regtest entries of section 4 | the commands of `docs/log/2026-10-18-u3-regtests-v2.md`, "What was done" |
-| Restart reproduction, `SHRD` | `scripts/restart_pair.sh` (not run yet) |
+| Restart reproduction, `SHRD` | `scripts/restart_pair.sh` (ran at `WW3/` `50a10730` and `fac74069`, [log](log/2026-10-18-resident-restart-pair.md)) |
 | The same on the point case | `bash scripts/ww3-gpu.sh --ww3-gpu smoke --cpu --kokkos resident` |
 | Per-part timers | add `WW_KOKKOS_TIMERS=1` to the Kokkos run, or `--timers` |
 | DIA kernel alone | `OMP_PROC_BIND=false OMP_NUM_THREADS=<t> kokkos/build/openmp-release/tests/ww_bench_snl1` |

docs/PRODUCTION_READINESS_202610.md

@@ -47,11 +47,11 @@ regtest matrix yet"). 10 of the 310 rows of `kokkos/ports.tsv` are regtested and
 | 3 | MPI: four of the six resident parts are `SHRD` only, the preset stops on `DIST MPI`, the communicator is ignored, the device id is not per rank | Blocker | Major | A resident preset without `XYP3`, `OUTG`, `IOPE`, `UPDV`; one run on `switch_lab_mpi` |
 | 4 | Bit parity measured with gfortran and g++ only; the operational compiler and the GPU are not measured | Blocker | Major | Static argument for the Intel build's math calls; one smoke with it |
 | 5 | Build and packaging: a separate static library, C++20 and CMake 3.25; the install-only configure and the exact Kokkos check are done (U6) | Major | Major | A shared-library build and a recipe for the module systems of an operational centre |
-| 6 | `WW_KOKKOS_ALL=1` and `W3KOKKOS_SRCE_ON` fall back to the Fortran without a word | Major | Minor | Done, at the pin `50a10730` since #191: `WW_KOKKOS_STRICT=1` stops, otherwise one fallback-count line at exit; the matrix's Kokkos runs set it |
-| 7 | Configuration is 61 environment variables, no namelist; the preset lives in a lab script | Major | Major | Smallest form done, at the pin `50a10730` since #191: `WW_KOKKOS_PRESET=resident` from one table, `presets.txt`; next, a namelist group read by `W3INIT` |
+| 6 | `WW_KOKKOS_ALL=1` and `W3KOKKOS_SRCE_ON` fall back to the Fortran without a word | Major | Minor | Done: `WW_KOKKOS_STRICT=1` stops, otherwise one fallback-count line at exit, `W3OUTG`'s fallback included since `ww3-gpu/strict-outg`; the matrix's Kokkos runs set it when they name their parts (section 6) |
+| 7 | Configuration is 61 environment variables, no namelist; the preset lives in a lab script | Major | Major | Smallest form done on `ww3-gpu/u2-strict-preset`: `WW_KOKKOS_PRESET=resident` from one table, `presets.txt`; next, a namelist group read by `W3INIT` |
 | 8 | Observability: log lines on every rank, counters at exit to `stderr`, no provenance in the output files | Major | Minor | Gate the lines by `NAPLOG`; one NetCDF global attribute |
 | 9 | OpenMP inside MPI: the per-point parts are serialised by `!$OMP CRITICAL`; the thread budget per rank is not set by the shim | Major | Minor | Measure the hybrid entries of `ww3_ufs1.1` before claiming a speed for them |
-| 10 | Restart with the resident spectrum: covered by the sync calls, never regtested; every `matrix.base` restart pair is `MPI` | Major | Minor | Run `scripts/restart_pair.sh` (a `SHRD` restart pair on `full`, written 2026-10-18, not run) |
+| 10 | Restart with the resident spectrum: covered by the sync calls, never regtested; every `matrix.base` restart pair is `MPI` | Major | Minor | Run `scripts/restart_pair.sh` (a `SHRD` restart pair on `full`); it passed at `WW3/` `50a10730` and again at `fac74069` (strict mode there stops a run that asks for `WBT` or `SKW`; this case asks for neither), [log](log/2026-10-18-resident-restart-pair.md) |
 | 11 | `PDLIB` and unstructured grids: translated, not wired, gated out | Minor ⚠ | Major | None for GFS-Wave v16; a wiring unit for `PDLIB` when the owner asks |
 | 12 | The fork patch: 4 082 lines of new Fortran, 819 changed lines in three WW3 files, 58 218 lines of C++ in another repository, most of it LGPL-3.0-or-later, some MIT, 76 files with no licence line | Minor | Blocker | ADR-0018 (Proposed) names the three places `ww_kokkos` could live; the owner chooses, after a legal reading ⚠ |
 | 13 | A stale contract remark: "one spectral grid per process"; edited on the fork branch `ww3-gpu/u1-multi-va` (section 13) | Minor | Minor | Closed: at the pin `50a10730` since #191 |
@@ -307,6 +307,17 @@ asked for Kokkos and got the Fortran learns it only from one log line or a `*_CO
   `W3KOKKOS_SRCE_ON` is unchanged. The regtest matrix's Kokkos run sets `WW_KOKKOS_STRICT=1`
   (`regtests/cmake/ww3gpu_common.cmake`, `ww3gpu_kokkos_env`). `PATCH.md` section 24 has the
   runs that show it.
+- **Update, 2026-10-18 (the follow-up of #191, its code review's findings 1 to 3).** Three changes,
+  [log](log/2026-10-18-strict-outg-poison.md):
+  - `W3KOKKOS_OUTG`'s call of the Fortran `W3OUTG` for a field its translation leaves to the
+    Fortran (U3 Stokes, skewness, `WBT`, an unallocated 3-D field) now goes through
+    `KOKKOS_FALLBACK`: it stops under `WW_KOKKOS_STRICT=1` and is counted otherwise (fork branch
+    `ww3-gpu/strict-outg`, `PATCH.md` section 31). Before, strict mode did not see it.
+  - The matrix sets `WW_KOKKOS_STRICT=1` only for a Kokkos run that names its parts
+    (`WW3GPU_KOKKOS=resident` or a part list); under `WW3GPU_KOKKOS=all` a part the entry's
+    switch does not cover is counted, not stopped, so an entry without `PR3` runs.
+  - `WW_KOKKOS_VA_POISON=1`, the missed-sync detector, is on in every Kokkos run of
+    `scripts/ww3-gpu.sh` and of the matrix, off in the Fortran runs; a caller's `=0` turns it off.
 
 ## 7. Configuration surface
 
@@ -393,7 +404,16 @@ the UFS case (`work_c`, then `work_d` from its restart, compared by `test.comp`;
 restarts a `SHRD` case. [`scripts/restart_pair.sh`](../scripts/restart_pair.sh) is the smallest
 equivalent on `full` with `switch_lab_shrd`: a 00:00 to 12:00 run and a 06:00 to 12:00 run from
 its own 06:00 restart, for the Fortran and the resident Kokkos of one executable, compared with
-`cmp`. It was not run ⚠ ([log](log/2026-10-18-u3-regtests-v2.md)); the gap stays Major for (a).
+`cmp`. It was not run when written ⚠ ([log](log/2026-10-18-u3-regtests-v2.md)).
+
+**Update, 2026-10-18 (restart pair run).** The pair ran once at `b828041` with `WW3/` at `50a10730`,
+4 threads, `WW_KOKKOS_VA_POISON=1`, `WW_KOKKOS_STRICT=1`: the 06:00-to-12:00 run from the 06:00
+restart gives the bytes of the run cut from the cold start (`out_grd.ww3` and the 12:00 restart),
+for the Fortran and for the resident Kokkos, and every file of the Kokkos runs equals the
+Fortran's (v, `cmp`, 9 comparisons, exit 0;
+[log](log/2026-10-18-resident-restart-pair.md)). It covers one `SHRD` case on the OpenMP backend,
+one restart and 12 hours; `MPI` restart, the CUDA backend and `ww3_multi` are not covered ⚠. The
+gap stays open for those, and for (a) until the owner accepts one case as enough.
 
 ## 11. `PDLIB` and unstructured grids