Production single-node Kokkos runtime: no-copy resident VA, strict mode, preset, packaging, GPU-run plan

.claude/skills/ww3-port/SKILL.md

@@ -46,6 +46,10 @@ compile the module with the flags in `<build>/model/src/CMakeFiles/ww3_lib.dir/f
   (`w3gdatmd.F90:2278`). A shim mirrors that with one context per grid: initialised with W3INIT's
   `IMOD`, selected at the call by `IGRID` (`snl1_shim.cpp`, `contexts()`). One file-static
   context silently breaks `ww3_multi`.
+- **Resident `VA`: sync what the host wrote, never the whole spectrum.** A host write between
+  device kernels (`ww3_multi`'s `WMIOHG`, `WMIOEG`) needs a sync at that write, of what it wrote;
+  marking the host newer at every `W3WAVE` return copies the whole spectrum per call and is
+  refused (`AGENTS.md`, "Resident spectrum"). Count with `WW_KOKKOS_VA_COUNT=1`.
 - **Initialisation follows the Fortran's condition.** `INSNL1` runs only when `IQTPE > 0`
   (`w3iogrmd.F90:1801`); with `IQTYPE <= 0` an `NL1` switch computes GQM and never reaches the port.
 

AGENTS.md

@@ -30,6 +30,10 @@ looser.
   routine it replaces, inside the same WW3 executable; `test.comp` finds no file different on any
   regtest entry that exercises it, and a tolerance is a diagnostic, never the verdict
   ([Bit for bit](#bit-for-bit-with-the-fortran-hard-rule)).
+- **No full-spectrum copy to keep the copies in sync**: once `VA` is resident, no code path
+  copies the whole spectrum between host and device per step, per grid or per `W3WAVE` call to
+  keep the two copies consistent; a host write is followed by a sync of what it wrote, where it
+  writes it ([Resident spectrum](#resident-spectrum-hard-rule)).
 - **Speed is the goal, parity is its condition**: the repository exists to make WW3 run faster
   with Kokkos and CUDA without changing the Fortran's arithmetic; a speed figure carries the
   machine, the command and the median of three runs, and says "regtest pending ⚠" until WW3's
@@ -285,6 +289,27 @@ broken most often:
   keeps the spectrum resident. Never use Unified Memory to avoid thinking about transfers, and never
   read the CUDA kernel column as "the model will be this much faster".
 
+## Resident spectrum (hard rule)
+
+The owner, 18 October 2026: a defect of the resident spectrum is never fixed by copying all of
+it. A whole-spectrum copy is not the optimal solution, and the resident design exists to remove
+those transfers.
+
+- **What it covers.** `VA` and any other array the resident preset binds on the device
+  (`kokkos/src/fortran_iface/resident_shim.cpp`), inside `W3WAVE`'s time loop and between
+  `W3WAVE` calls of `ww3_shel` and `ww3_multi`.
+- **What to do instead.** Sync at the host write that made the copies differ, and only the part
+  written when the shim can (it already moves columns, `resident_shim.cpp:270`, `:290`). Read the
+  device copy to the host only where the host reads it (an output, a restart). A whole-array copy
+  is never chosen because it is simpler.
+- **What is allowed.** The copy when `VA` is bound and the one at the end of the run; the reads
+  before an output or a restart write that needs the whole field; `WW_KOKKOS_VA_CHECK`, a
+  diagnostic, off by default; the per-point parts of phase 1, which copy per call by design and
+  are not resident.
+- **How it is checked.** A change to the resident sync reports `WW_KOKKOS_VA_COUNT=1` (bytes host
+  to device and device to host) on `full` before and after; the whole-spectrum copies per step
+  must not grow, or the change is refused.
+
 ## Bit for bit with the Fortran (hard rule)
 
 The owner, 15 October 2026: every ported code of WW3 is bit-to-bit equal to the behaviour of the

CONTRIBUTING.md

@@ -140,6 +140,13 @@ A mistake in a figure is as welcome as one in the text: the gallery links an iss
   WW3's own regtest run with the Fortran routine and with the Kokkos routine; CTest is the runner,
   not a test of the port's own. A change to `regtests/` runs `just ww3-gpu-matrix-selftest` and
   `just ww3-gpu-matrix-list`. Anything tried that did not become a number in a table, failures included, goes in a dated entry in `docs/log/`.
+- The resident spectrum is never kept consistent by copying all of it (`AGENTS.md`, "Resident
+  spectrum"). When the host writes `VA` between two device kernels (`ww3_multi`'s boundary and
+  nesting updates, for one), sync what was written, where it is written; read the device copy to
+  the host only before an output or a restart that needs it. A whole-spectrum copy per step, per
+  grid or per `W3WAVE` call is refused, even when it is the simpler fix: on a GPU it is a bus
+  transfer every step. A change to the sync reports `WW_KOKKOS_VA_COUNT=1` on `full` before and
+  after.
 - `codemeta.json` is generated from `CITATION.cff` and `.zenodo.json` (`just codemeta`); never
   edit it by hand.
 - Run `just kokkos-test serial-debug` after touching `kokkos/`, and `bash -n` plus

README.md

@@ -149,7 +149,7 @@ with `--kokkos`:
 | `none` | none | the Fortran only, in the Kokkos build |
 | `all` | all the parts of the build (`WW_KOKKOS_ALL=1`), but not the opt-in parts below | the widest test of the translation |
 | `gpu` | `SRCE` | the default of `--gpu`: all of `W3SRCE` in each call ([ADR-0013](docs/ADRs/ADR-0013-operational-parts-default-to-kokkos.md)) |
-| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV` | the full time step on Kokkos, with the spectrum kept on the Kokkos side ([next section](#what---kokkos-resident-does)) |
+| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT` | the full time step on Kokkos, with the spectrum kept on the Kokkos side ([next section](#what---kokkos-resident-does)) |
 | a list, for example `SNL1,SDS4` | the parts in the list | a test of one part at a time |
 
 ### what `--kokkos resident` does

README.pt-BR.md

@@ -150,7 +150,7 @@ duas rodadas. Escolha as partes com `--kokkos`:
 | `none` | nenhuma | só o Fortran, no build com Kokkos |
 | `all` | todas as partes do build (`WW_KOKKOS_ALL=1`), menos as opcionais abaixo | o teste mais amplo da tradução |
 | `gpu` | `SRCE` | o padrão do `--gpu`: o `W3SRCE` inteiro em cada chamada ([ADR-0013](docs/ADRs/ADR-0013-operational-parts-default-to-kokkos.md)) |
-| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV` | o passo de tempo inteiro em Kokkos, com o espectro guardado do lado do Kokkos ([próxima seção](#o-que-o---kokkos-resident-faz)) |
+| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT` | o passo de tempo inteiro em Kokkos, com o espectro guardado do lado do Kokkos ([próxima seção](#o-que-o---kokkos-resident-faz)) |
 | uma lista, por exemplo `SNL1,SDS4` | as partes da lista | o teste de uma parte por vez |
 
 ### o que o `--kokkos resident` faz

docs/ADRs/ADR-0018-where-ww-kokkos-lives-for-upstream.md

@@ -0,0 +1,131 @@
+# ADR-0018: Where `ww_kokkos` lives if WW3 upstream takes the Kokkos runtime
+
+| | |
+|---|---|
+| **Status** | Proposed |
+| **Date** | 2026-10-18 |
+| **Deciders** | Hoffmann |
+| **Written by** | an agent, from gap 12 of [`PRODUCTION_READINESS_202610.md`](../PRODUCTION_READINESS_202610.md) and unit U6 of the [production single-node plan](../compound/plans/2026-10-18-002-production-single-node-plan.md); no option is chosen |
+| **Scope** | Where the C++ library `ww_kokkos` and its C ABI would live for a WW3 upstream that takes the `KOKKOS` switch: an external package WW3 finds, a submodule of WW3, or sources inside `model/src`. Not whether upstream wants it, not the Fortran half (it is in WW3 in every option), not the single-node scope ([ADR-0017](ADR-0017-kokkos-port-is-single-node-mpi-stays-fortran.md)) |
+| **Related** | [ADR-0010](ADR-0010-kokkos-validated-by-ww3-regtests-only.md); [ADR-0016](ADR-0016-speed-first-regtests-postponed-one-person-loop.md); [ADR-0017](ADR-0017-kokkos-port-is-single-node-mpi-stays-fortran.md); [`kokkos/README.md`](../../kokkos/README.md), "Install-only build"; [`kokkos/src/fortran_iface/PATCH.md`](../../kokkos/src/fortran_iface/PATCH.md) |
+
+## Context
+
+The port has two halves. The Fortran half is on the fork (`h0ffmann/WW3`, pinned at
+`WW3@80644ba9`); the C++ half is `kokkos/` in this repository. `diff -rq WW3-noaa/model WW3/model`
+(v, 2026-10-18) lists what the fork adds to NOAA's WW3 at the commit it branched from:
+
+| Fork change | Size (v, 2026-10-18) |
+|---|---|
+| Six new modules, `model/src/w3kokkos*md.F90` | 4 082 lines |
+| `w3initmd.F90`, `w3srcemd.F90`, `w3wavemd.F90` | 114, 456 and 249 changed lines (`diff \| grep -c '^[<>]'`) |
+| `model/src/CMakeLists.txt` | 64 changed lines, the `WW_KOKKOS` block |
+| `model/src/cmake/switches.json` | the `kokkos` category |
+
+The C++ that the `KOKKOS` switch links is larger (v, `cat kokkos/src/*/*.{cpp,hpp} | wc -l`,
+2026-10-18): 48 571 lines in `kokkos/src/ww_kokkos/` (the kernels) and 9 647 in
+`kokkos/src/fortran_iface/` (the C ABI and shims), 347 files.
+
+How WW3 reaches it today: `find_package(ww_kokkos 0.9 REQUIRED)` and `target_link_libraries(ww3_lib
+PUBLIC ww_kokkos::ww_kokkos)` (v, `model/src/CMakeLists.txt:43-44` at `WW3@80644ba9`). Since this
+change the package can be configured with Kokkos as its one dependency
+(`-DWW_KOKKOS_BUILD_TESTS=OFF`), and its config asks for the exact Kokkos version, backends and
+architecture it was built with (v, `kokkos/README.md`, "Install-only build", configure traced
+2026-10-18). Its own version contract is `SameMajorVersion` at 0.9.0 with entry points only added
+(v, `kokkos/CMakeLists.txt`); nothing checks the "only added" ⚠.
+
+Licences, as the files state them (v, `grep SPDX-License-Identifier`, 2026-10-18):
+
+- WW3 is under the GNU Lesser General Public License v3 (v, `WW3/LICENSE.md`, first lines). The
+  fork's new modules carry `LGPL-3.0-or-later` (v, `model/src/w3kokkosmd.F90:33`).
+- Of the 347 C++ files under `kokkos/src/`, 249 carry `LGPL-3.0-or-later`, all in `ww_kokkos/`:
+  the kernels translated from WW3's Fortran. 22 carry `MIT`: the shims, the timers, the fixture
+  reader and `real.hpp`. 76 carry no line (71 in `ww_kokkos/`, 5 in `fortran_iface/`); the
+  repository is MIT (v, `LICENSE`), but whether a file translated from WW3 without the line is
+  MIT or a derived work under the LGPL is not settled here ⚠.
+- Gap 12 of `PRODUCTION_READINESS_202610.md` read "the C++ it calls is MIT"; the count above
+  shows most of the C++ is LGPL-3.0-or-later, as `README.md`, "licensing", already says of the
+  translated kernels.
+
+⚠ No legal reading was made for any statement on what a licence permits in this ADR. Each option
+names the licence question it raises; a person qualified to answer it has to.
+
+## Decision
+
+None yet. This ADR records the three options with the evidence above, for the owner and, if it
+comes to that, for WW3's maintainers to choose from. What the evidence supports today:
+
+- Option (a) is what exists and was built end to end on 2026-10-18: install-only configure,
+  install, WW3 built against the prefix, `smoke` byte-identical pair to pair
+  (`PRODUCTION_READINESS_202610.md`, gap 5).
+- Options (b) and (c) were not built. Their costs below are read from the files, not measured ⚠.
+
+## Alternatives considered
+
+### (a) An external package WW3 finds (today's layout)
+
+WW3 keeps `find_package(ww_kokkos <major.minor> REQUIRED)` behind `-DWW_KOKKOS=ON`; `ww_kokkos` is
+released and installed on its own, like NetCDF or ESMF.
+
+- The upstream diff is the Fortran half only: 4 082 new and 819 changed lines of Fortran, 64 lines
+  of CMake, one `switches.json` category.
+- The C++ keeps its own repository, licence files, CMake minimum (3.25; WW3 asks for 3.19, v,
+  `WW3/CMakeLists.txt:4`) and C++20 requirement, and its own release cadence.
+- Two version contracts must move together: WW3's `find_package` minimum and the ABI version of
+  `ww_kokkos`. Today a WW3 commit that needs a new entry point raises the minimum by hand.
+- An operational centre installs one more library, built against the same Kokkos as its WW3; the
+  `EXACT` check makes a mismatch a configure error instead of a link-time surprise.
+- WW3's exported targets name `ww_kokkos::ww_kokkos` (`ww3_lib` links it `PUBLIC`), so a consumer
+  of an installed WW3 must find the package too ⚠ (no consumer built).
+- Licence: the LGPL-derived kernels are distributed from a repository whose top licence is MIT,
+  each file with its own line. Whether that is enough is the question marked ⚠ above.
+
+### (b) A submodule of WW3
+
+WW3 adds `ww_kokkos` as a git submodule at a pinned commit and builds it with `add_subdirectory`
+or `ExternalProject_Add` when `KOKKOS` is set.
+
+- WW3 has no submodule at the pin (v, no `.gitmodules` in `WW3/`); this would be its first, with
+  `git clone --recurse-submodules` in every build recipe that uses the switch.
+- The gitlink pins the C++ exactly, so the two-version problem of (a) becomes one commit; the
+  `find_package` minimum is no longer needed.
+- `add_subdirectory` brings this tree's CMake 3.25 minimum and C++20 into WW3's configure, and the
+  install-only option is what keeps GoogleTest and NetCDF out of it. `ExternalProject_Add` keeps
+  the trees apart at the cost of a second configure step ⚠ (neither built).
+- The C++ stays in its own repository with its own licence files, as in (a).
+- Who maintains the submodule's repository, and who may move the gitlink, is a governance
+  question WW3's maintainers answer, not this repository.
+
+### (c) Sources inside `model/src`
+
+The C++ is copied into WW3, for instance `model/src/kokkos/`, and built by WW3's CMake.
+
+- The upstream diff grows by about 58 000 lines of C++ in 347 files, about 14 times the 4 082 new lines of the Fortran half.
+- One repository, one version, one licence file: the LGPL-derived kernels would sit in an LGPL
+  project. The 22 MIT files would sit in it too ⚠ (whether WW3 accepts MIT files, or relicences
+  them, is for its maintainers and a legal reading).
+- WW3's CMake would need C++ in its languages, the contraction flags (`-ffp-contract=off`,
+  `--fmad=false`) on the C++ sources only, and the CUDA `libm` table generator; today those live
+  in `kokkos/src/ww_kokkos/CMakeLists.txt`.
+- The lab's tools and intro programs stay here; the port's validation is WW3's regtests in every
+  option (ADR-0010), so no test suite moves.
+- Every change to the C++ becomes a WW3 pull request; this repository would then track WW3 for
+  the C++ as it does for the Fortran.
+
+## Consequences
+
+Whatever is chosen:
+
+- The Fortran half is upstream's in every option; its 819 changed lines are what a reviewer reads,
+  and with `KOKKOS` off they compile to the original (`PRODUCTION_READINESS_202610.md`, gap 12).
+- The licence of the files without an `SPDX` line has to be settled before any option ships ⚠.
+- The `EXACT` Kokkos check stays: in (b) and (c) the Kokkos is still an external package, and a
+  build against another Kokkos still links code compiled for one Kokkos with another's runtime.
+- Gap 12 stays a Blocker for target (b) of `PRODUCTION_READINESS_202610.md` until an option is
+  accepted. Gap 5's configure dependencies are closed by the install-only option; its other
+  items (a static archive only, C++20, CMake 3.25) remain.
+
+## Revisit when
+
+The owner opens a conversation with WW3's maintainers about taking the runtime, or a legal reading
+of the licence question is available, or WW3 adds submodules or C++ for another reason.

docs/ADRs/README.md

@@ -35,3 +35,4 @@ fills the `/adrs/` page of the book's website on every build.
 | [ADR-0015](ADR-0015-triton-kernels-allowed-under-three-conditions.md) | Triton GPU kernels are allowed as an exception to "no Python in lab code" when better fitted than the Kokkos kernel, integrated with the Fortran/C++ code behind a run-time flag, and regtested by WW3's regtests | Accepted | 2026-10-17 |
 | [ADR-0016](ADR-0016-speed-first-regtests-postponed-one-person-loop.md) | The goal is WW3's run time on Kokkos and CUDA; WW3's regtests move to a validation phase the owner calls, speed figures before it are marked "regtest pending", and engineering work follows Compound Engineering's loop sized for one person (supersedes ADR-0011) | Accepted | 2026-10-17 |
 | [ADR-0017](ADR-0017-kokkos-port-is-single-node-mpi-stays-fortran.md) | The Kokkos port is single node (`SHRD`, with OpenMP, on the CPU and one GPU); an MPI build of WW3 keeps the Fortran path, and no MPI path is translated, wired or regtested while it stands | Accepted | 2026-10-18 |
+| [ADR-0018](ADR-0018-where-ww-kokkos-lives-for-upstream.md) | Where `ww_kokkos` lives if WW3 upstream takes the Kokkos runtime: an external package WW3 finds (today's layout, built end to end), a submodule of WW3, or sources inside `model/src`; no option chosen, the licence of the translated C++ marked ⚠ | Proposed | 2026-10-18 |