diff --git a/.claude/skills/ww3-port/SKILL.md b/.claude/skills/ww3-port/SKILL.md index c5ae44a0..d023db00 100644 --- a/.claude/skills/ww3-port/SKILL.md +++ b/.claude/skills/ww3-port/SKILL.md @@ -46,6 +46,10 @@ compile the module with the flags in `/model/src/CMakeFiles/ww3_lib.dir/f (`w3gdatmd.F90:2278`). A shim mirrors that with one context per grid: initialised with W3INIT's `IMOD`, selected at the call by `IGRID` (`snl1_shim.cpp`, `contexts()`). One file-static context silently breaks `ww3_multi`. +- **Resident `VA`: sync what the host wrote, never the whole spectrum.** A host write between + device kernels (`ww3_multi`'s `WMIOHG`, `WMIOEG`) needs a sync at that write, of what it wrote; + marking the host newer at every `W3WAVE` return copies the whole spectrum per call and is + refused (`AGENTS.md`, "Resident spectrum"). Count with `WW_KOKKOS_VA_COUNT=1`. - **Initialisation follows the Fortran's condition.** `INSNL1` runs only when `IQTPE > 0` (`w3iogrmd.F90:1801`); with `IQTYPE <= 0` an `NL1` switch computes GQM and never reaches the port. diff --git a/AGENTS.md b/AGENTS.md index 512aa06d..3d67b069 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,6 +30,10 @@ looser. routine it replaces, inside the same WW3 executable; `test.comp` finds no file different on any regtest entry that exercises it, and a tolerance is a diagnostic, never the verdict ([Bit for bit](#bit-for-bit-with-the-fortran-hard-rule)). +- **No full-spectrum copy to keep the copies in sync**: once `VA` is resident, no code path + copies the whole spectrum between host and device per step, per grid or per `W3WAVE` call to + keep the two copies consistent; a host write is followed by a sync of what it wrote, where it + writes it ([Resident spectrum](#resident-spectrum-hard-rule)). - **Speed is the goal, parity is its condition**: the repository exists to make WW3 run faster with Kokkos and CUDA without changing the Fortran's arithmetic; a speed figure carries the machine, the command and the median of three runs, and says "regtest pending ⚠" until WW3's @@ -285,6 +289,27 @@ broken most often: keeps the spectrum resident. Never use Unified Memory to avoid thinking about transfers, and never read the CUDA kernel column as "the model will be this much faster". +## Resident spectrum (hard rule) + +The owner, 18 October 2026: a defect of the resident spectrum is never fixed by copying all of +it. A whole-spectrum copy is not the optimal solution, and the resident design exists to remove +those transfers. + +- **What it covers.** `VA` and any other array the resident preset binds on the device + (`kokkos/src/fortran_iface/resident_shim.cpp`), inside `W3WAVE`'s time loop and between + `W3WAVE` calls of `ww3_shel` and `ww3_multi`. +- **What to do instead.** Sync at the host write that made the copies differ, and only the part + written when the shim can (it already moves columns, `resident_shim.cpp:270`, `:290`). Read the + device copy to the host only where the host reads it (an output, a restart). A whole-array copy + is never chosen because it is simpler. +- **What is allowed.** The copy when `VA` is bound and the one at the end of the run; the reads + before an output or a restart write that needs the whole field; `WW_KOKKOS_VA_CHECK`, a + diagnostic, off by default; the per-point parts of phase 1, which copy per call by design and + are not resident. +- **How it is checked.** A change to the resident sync reports `WW_KOKKOS_VA_COUNT=1` (bytes host + to device and device to host) on `full` before and after; the whole-spectrum copies per step + must not grow, or the change is refused. + ## Bit for bit with the Fortran (hard rule) The owner, 15 October 2026: every ported code of WW3 is bit-to-bit equal to the behaviour of the diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 619a2c9b..b08ca89d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -140,6 +140,13 @@ A mistake in a figure is as welcome as one in the text: the gallery links an iss WW3's own regtest run with the Fortran routine and with the Kokkos routine; CTest is the runner, not a test of the port's own. A change to `regtests/` runs `just ww3-gpu-matrix-selftest` and `just ww3-gpu-matrix-list`. Anything tried that did not become a number in a table, failures included, goes in a dated entry in `docs/log/`. +- The resident spectrum is never kept consistent by copying all of it (`AGENTS.md`, "Resident + spectrum"). When the host writes `VA` between two device kernels (`ww3_multi`'s boundary and + nesting updates, for one), sync what was written, where it is written; read the device copy to + the host only before an output or a restart that needs it. A whole-spectrum copy per step, per + grid or per `W3WAVE` call is refused, even when it is the simpler fix: on a GPU it is a bus + transfer every step. A change to the sync reports `WW_KOKKOS_VA_COUNT=1` on `full` before and + after. - `codemeta.json` is generated from `CITATION.cff` and `.zenodo.json` (`just codemeta`); never edit it by hand. - Run `just kokkos-test serial-debug` after touching `kokkos/`, and `bash -n` plus diff --git a/README.md b/README.md index a87c85e9..688f0065 100644 --- a/README.md +++ b/README.md @@ -149,7 +149,7 @@ with `--kokkos`: | `none` | none | the Fortran only, in the Kokkos build | | `all` | all the parts of the build (`WW_KOKKOS_ALL=1`), but not the opt-in parts below | the widest test of the translation | | `gpu` | `SRCE` | the default of `--gpu`: all of `W3SRCE` in each call ([ADR-0013](docs/ADRs/ADR-0013-operational-parts-default-to-kokkos.md)) | -| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV` | the full time step on Kokkos, with the spectrum kept on the Kokkos side ([next section](#what---kokkos-resident-does)) | +| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT` | the full time step on Kokkos, with the spectrum kept on the Kokkos side ([next section](#what---kokkos-resident-does)) | | a list, for example `SNL1,SDS4` | the parts in the list | a test of one part at a time | ### what `--kokkos resident` does diff --git a/README.pt-BR.md b/README.pt-BR.md index d3cde672..89a8d871 100644 --- a/README.pt-BR.md +++ b/README.pt-BR.md @@ -150,7 +150,7 @@ duas rodadas. Escolha as partes com `--kokkos`: | `none` | nenhuma | só o Fortran, no build com Kokkos | | `all` | todas as partes do build (`WW_KOKKOS_ALL=1`), menos as opcionais abaixo | o teste mais amplo da tradução | | `gpu` | `SRCE` | o padrão do `--gpu`: o `W3SRCE` inteiro em cada chamada ([ADR-0013](docs/ADRs/ADR-0013-operational-parts-default-to-kokkos.md)) | -| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV` | o passo de tempo inteiro em Kokkos, com o espectro guardado do lado do Kokkos ([próxima seção](#o-que-o---kokkos-resident-faz)) | +| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT` | o passo de tempo inteiro em Kokkos, com o espectro guardado do lado do Kokkos ([próxima seção](#o-que-o---kokkos-resident-faz)) | | uma lista, por exemplo `SNL1,SDS4` | as partes da lista | o teste de uma parte por vez | ### o que o `--kokkos resident` faz diff --git a/WW3 b/WW3 index 80644ba9..50a10730 160000 --- a/WW3 +++ b/WW3 @@ -1 +1 @@ -Subproject commit 80644ba9fcb0498f8ef2cc15eff70cb2bd12b60c +Subproject commit 50a107307aad05207f751f399f5f6481becfc2eb diff --git a/docs/ADRs/ADR-0018-where-ww-kokkos-lives-for-upstream.md b/docs/ADRs/ADR-0018-where-ww-kokkos-lives-for-upstream.md new file mode 100644 index 00000000..1116e880 --- /dev/null +++ b/docs/ADRs/ADR-0018-where-ww-kokkos-lives-for-upstream.md @@ -0,0 +1,131 @@ +# ADR-0018: Where `ww_kokkos` lives if WW3 upstream takes the Kokkos runtime + +| | | +|---|---| +| **Status** | Proposed | +| **Date** | 2026-10-18 | +| **Deciders** | Hoffmann | +| **Written by** | an agent, from gap 12 of [`PRODUCTION_READINESS_202610.md`](../PRODUCTION_READINESS_202610.md) and unit U6 of the [production single-node plan](../compound/plans/2026-10-18-002-production-single-node-plan.md); no option is chosen | +| **Scope** | Where the C++ library `ww_kokkos` and its C ABI would live for a WW3 upstream that takes the `KOKKOS` switch: an external package WW3 finds, a submodule of WW3, or sources inside `model/src`. Not whether upstream wants it, not the Fortran half (it is in WW3 in every option), not the single-node scope ([ADR-0017](ADR-0017-kokkos-port-is-single-node-mpi-stays-fortran.md)) | +| **Related** | [ADR-0010](ADR-0010-kokkos-validated-by-ww3-regtests-only.md); [ADR-0016](ADR-0016-speed-first-regtests-postponed-one-person-loop.md); [ADR-0017](ADR-0017-kokkos-port-is-single-node-mpi-stays-fortran.md); [`kokkos/README.md`](../../kokkos/README.md), "Install-only build"; [`kokkos/src/fortran_iface/PATCH.md`](../../kokkos/src/fortran_iface/PATCH.md) | + +## Context + +The port has two halves. The Fortran half is on the fork (`h0ffmann/WW3`, pinned at +`WW3@80644ba9`); the C++ half is `kokkos/` in this repository. `diff -rq WW3-noaa/model WW3/model` +(v, 2026-10-18) lists what the fork adds to NOAA's WW3 at the commit it branched from: + +| Fork change | Size (v, 2026-10-18) | +|---|---| +| Six new modules, `model/src/w3kokkos*md.F90` | 4 082 lines | +| `w3initmd.F90`, `w3srcemd.F90`, `w3wavemd.F90` | 114, 456 and 249 changed lines (`diff \| grep -c '^[<>]'`) | +| `model/src/CMakeLists.txt` | 64 changed lines, the `WW_KOKKOS` block | +| `model/src/cmake/switches.json` | the `kokkos` category | + +The C++ that the `KOKKOS` switch links is larger (v, `cat kokkos/src/*/*.{cpp,hpp} | wc -l`, +2026-10-18): 48 571 lines in `kokkos/src/ww_kokkos/` (the kernels) and 9 647 in +`kokkos/src/fortran_iface/` (the C ABI and shims), 347 files. + +How WW3 reaches it today: `find_package(ww_kokkos 0.9 REQUIRED)` and `target_link_libraries(ww3_lib +PUBLIC ww_kokkos::ww_kokkos)` (v, `model/src/CMakeLists.txt:43-44` at `WW3@80644ba9`). Since this +change the package can be configured with Kokkos as its one dependency +(`-DWW_KOKKOS_BUILD_TESTS=OFF`), and its config asks for the exact Kokkos version, backends and +architecture it was built with (v, `kokkos/README.md`, "Install-only build", configure traced +2026-10-18). Its own version contract is `SameMajorVersion` at 0.9.0 with entry points only added +(v, `kokkos/CMakeLists.txt`); nothing checks the "only added" ⚠. + +Licences, as the files state them (v, `grep SPDX-License-Identifier`, 2026-10-18): + +- WW3 is under the GNU Lesser General Public License v3 (v, `WW3/LICENSE.md`, first lines). The + fork's new modules carry `LGPL-3.0-or-later` (v, `model/src/w3kokkosmd.F90:33`). +- Of the 347 C++ files under `kokkos/src/`, 249 carry `LGPL-3.0-or-later`, all in `ww_kokkos/`: + the kernels translated from WW3's Fortran. 22 carry `MIT`: the shims, the timers, the fixture + reader and `real.hpp`. 76 carry no line (71 in `ww_kokkos/`, 5 in `fortran_iface/`); the + repository is MIT (v, `LICENSE`), but whether a file translated from WW3 without the line is + MIT or a derived work under the LGPL is not settled here ⚠. +- Gap 12 of `PRODUCTION_READINESS_202610.md` read "the C++ it calls is MIT"; the count above + shows most of the C++ is LGPL-3.0-or-later, as `README.md`, "licensing", already says of the + translated kernels. + +⚠ No legal reading was made for any statement on what a licence permits in this ADR. Each option +names the licence question it raises; a person qualified to answer it has to. + +## Decision + +None yet. This ADR records the three options with the evidence above, for the owner and, if it +comes to that, for WW3's maintainers to choose from. What the evidence supports today: + +- Option (a) is what exists and was built end to end on 2026-10-18: install-only configure, + install, WW3 built against the prefix, `smoke` byte-identical pair to pair + (`PRODUCTION_READINESS_202610.md`, gap 5). +- Options (b) and (c) were not built. Their costs below are read from the files, not measured ⚠. + +## Alternatives considered + +### (a) An external package WW3 finds (today's layout) + +WW3 keeps `find_package(ww_kokkos REQUIRED)` behind `-DWW_KOKKOS=ON`; `ww_kokkos` is +released and installed on its own, like NetCDF or ESMF. + +- The upstream diff is the Fortran half only: 4 082 new and 819 changed lines of Fortran, 64 lines + of CMake, one `switches.json` category. +- The C++ keeps its own repository, licence files, CMake minimum (3.25; WW3 asks for 3.19, v, + `WW3/CMakeLists.txt:4`) and C++20 requirement, and its own release cadence. +- Two version contracts must move together: WW3's `find_package` minimum and the ABI version of + `ww_kokkos`. Today a WW3 commit that needs a new entry point raises the minimum by hand. +- An operational centre installs one more library, built against the same Kokkos as its WW3; the + `EXACT` check makes a mismatch a configure error instead of a link-time surprise. +- WW3's exported targets name `ww_kokkos::ww_kokkos` (`ww3_lib` links it `PUBLIC`), so a consumer + of an installed WW3 must find the package too ⚠ (no consumer built). +- Licence: the LGPL-derived kernels are distributed from a repository whose top licence is MIT, + each file with its own line. Whether that is enough is the question marked ⚠ above. + +### (b) A submodule of WW3 + +WW3 adds `ww_kokkos` as a git submodule at a pinned commit and builds it with `add_subdirectory` +or `ExternalProject_Add` when `KOKKOS` is set. + +- WW3 has no submodule at the pin (v, no `.gitmodules` in `WW3/`); this would be its first, with + `git clone --recurse-submodules` in every build recipe that uses the switch. +- The gitlink pins the C++ exactly, so the two-version problem of (a) becomes one commit; the + `find_package` minimum is no longer needed. +- `add_subdirectory` brings this tree's CMake 3.25 minimum and C++20 into WW3's configure, and the + install-only option is what keeps GoogleTest and NetCDF out of it. `ExternalProject_Add` keeps + the trees apart at the cost of a second configure step ⚠ (neither built). +- The C++ stays in its own repository with its own licence files, as in (a). +- Who maintains the submodule's repository, and who may move the gitlink, is a governance + question WW3's maintainers answer, not this repository. + +### (c) Sources inside `model/src` + +The C++ is copied into WW3, for instance `model/src/kokkos/`, and built by WW3's CMake. + +- The upstream diff grows by about 58 000 lines of C++ in 347 files, about 14 times the 4 082 new lines of the Fortran half. +- One repository, one version, one licence file: the LGPL-derived kernels would sit in an LGPL + project. The 22 MIT files would sit in it too ⚠ (whether WW3 accepts MIT files, or relicences + them, is for its maintainers and a legal reading). +- WW3's CMake would need C++ in its languages, the contraction flags (`-ffp-contract=off`, + `--fmad=false`) on the C++ sources only, and the CUDA `libm` table generator; today those live + in `kokkos/src/ww_kokkos/CMakeLists.txt`. +- The lab's tools and intro programs stay here; the port's validation is WW3's regtests in every + option (ADR-0010), so no test suite moves. +- Every change to the C++ becomes a WW3 pull request; this repository would then track WW3 for + the C++ as it does for the Fortran. + +## Consequences + +Whatever is chosen: + +- The Fortran half is upstream's in every option; its 819 changed lines are what a reviewer reads, + and with `KOKKOS` off they compile to the original (`PRODUCTION_READINESS_202610.md`, gap 12). +- The licence of the files without an `SPDX` line has to be settled before any option ships ⚠. +- The `EXACT` Kokkos check stays: in (b) and (c) the Kokkos is still an external package, and a + build against another Kokkos still links code compiled for one Kokkos with another's runtime. +- Gap 12 stays a Blocker for target (b) of `PRODUCTION_READINESS_202610.md` until an option is + accepted. Gap 5's configure dependencies are closed by the install-only option; its other + items (a static archive only, C++20, CMake 3.25) remain. + +## Revisit when + +The owner opens a conversation with WW3's maintainers about taking the runtime, or a legal reading +of the licence question is available, or WW3 adds submodules or C++ for another reason. diff --git a/docs/ADRs/README.md b/docs/ADRs/README.md index 4ff3f9f0..d2e360fa 100644 --- a/docs/ADRs/README.md +++ b/docs/ADRs/README.md @@ -35,3 +35,4 @@ fills the `/adrs/` page of the book's website on every build. | [ADR-0015](ADR-0015-triton-kernels-allowed-under-three-conditions.md) | Triton GPU kernels are allowed as an exception to "no Python in lab code" when better fitted than the Kokkos kernel, integrated with the Fortran/C++ code behind a run-time flag, and regtested by WW3's regtests | Accepted | 2026-10-17 | | [ADR-0016](ADR-0016-speed-first-regtests-postponed-one-person-loop.md) | The goal is WW3's run time on Kokkos and CUDA; WW3's regtests move to a validation phase the owner calls, speed figures before it are marked "regtest pending", and engineering work follows Compound Engineering's loop sized for one person (supersedes ADR-0011) | Accepted | 2026-10-17 | | [ADR-0017](ADR-0017-kokkos-port-is-single-node-mpi-stays-fortran.md) | The Kokkos port is single node (`SHRD`, with OpenMP, on the CPU and one GPU); an MPI build of WW3 keeps the Fortran path, and no MPI path is translated, wired or regtested while it stands | Accepted | 2026-10-18 | +| [ADR-0018](ADR-0018-where-ww-kokkos-lives-for-upstream.md) | Where `ww_kokkos` lives if WW3 upstream takes the Kokkos runtime: an external package WW3 finds (today's layout, built end to end), a submodule of WW3, or sources inside `model/src`; no option chosen, the licence of the translated C++ marked ⚠ | Proposed | 2026-10-18 | diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index feee1a45..096ca35c 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -432,6 +432,7 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | W3UTAU / W3URHO / W3UIC1 | "Interpolate atmosphere momentum fields to the given time", "Interpolate air density field to the given time", "Update ice thickness in the wave model" `(v)` `w3updtmd.F90:835`, `:2626`, `:1555` | Forcing updates in `W3WAVE` not translated yet; hooks of the Kokkos `W3WAVE`. | [kokkos/port_report/W3WAVE_KOKKOS.md](../kokkos/port_report/W3WAVE_KOKKOS.md) | | W3UBPT | "Update spectra at the active boundary points" `(v)` `w3updtmd.F90:1329` | Hooks of the Kokkos `W3WAVE`. | [kokkos/port_report/W3WAVE_KOKKOS.md](../kokkos/port_report/W3WAVE_KOKKOS.md) | | W3IOPE / W3IOSF / W3CPRT | "Extract point output data and store in output COMMONs", "Write partitioned spectral data to file", "Partitioning of spectra into fields for all grid points that are locally stored" `(v)` `w3iopomd.F90:873`, `w3iosfmd.F90:360`, `:109` | The output and boundary I/O `W3WAVE` calls; hooks of the Kokkos `W3WAVE`. | [kokkos/port_report/W3WAVE_KOKKOS.md](../kokkos/port_report/W3WAVE_KOKKOS.md) | +| ICPRT / DTPRT / FINSIZ | `W3ODATMD`'s partition table: `ICPRT(JSEA,1)` the number of partitions plus one of point `JSEA` (0 when skipped), `ICPRT(JSEA,2)` its first column of `DTPRT(DIMP,FINSIZ)`, which holds `XP(1:15,0:NP)` of every point one after the other; built by `W3CPRT`, read by `W3OUTG` section 6 and `W3IOSF` `(v)` `w3iosfmd.F90:221-326` | A few values per point and partition, against `NTH*NK` of the spectrum. | [kokkos/port_report/W3CPRT_KOKKOS.md](../kokkos/port_report/W3CPRT_KOKKOS.md) | | MAPFS | the sea-point index of a grid point, `MAPFS(IY,IX)`; "Some array start at 0 because MAPFS(IY,IX)=0 for missing points" `(v)` `w3gdatmd.F90:1723`, `:1735` | `0` on land, so `W3IOPE` reads element 0 of the `(0:NSEA)` arrays for a land corner, with weight 0. | [kokkos/port_report/W3IOPE_KOKKOS.md](../kokkos/port_report/W3IOPE_KOKKOS.md) | | LPDLIB | "Logical for using the PDLIB library" `(v)` `constants.F90:102` | A run-time flag, `.FALSE.` unless WW3 was built with `PDLIB` (`w3initmd.F90:642-645`); `W3IOPE` interpolates `ICEFO` with a different sum under it. | [kokkos/port_report/W3IOPE_KOKKOS.md](../kokkos/port_report/W3IOPE_KOKKOS.md) | | SPPNT | "Point output buffer" `(v)` `w3adatmd.F90:306`, allocated `SPPNT(NTH,NK,4)` `:1451` | The four corner spectra `W3IOPE` copies from `A` (as `SP`) before interpolating them. | [kokkos/port_report/W3IOPE_KOKKOS.md](../kokkos/port_report/W3IOPE_KOKKOS.md) | @@ -484,7 +485,8 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | AINIT / AINIT2 / INFLAGS1 | "Flag for array initialization" (`AINIT(2)`) and "Array consolidating the above six" input flags `(v)` `w3adatmd.F90:318`, `w3idatmd.F90:99` | The conditions under which `W3SETA` re-points its alias groups. | [kokkos/port_report/W3SETA_KOKKOS.md](../kokkos/port_report/W3SETA_KOKKOS.md) | | NAPROC / UNDEF | "Number of processors for computation" / "Value for undefined parameters in gridded output fields" `(v)` `w3odatmd.F90:102`, `:82-83` | `W3XDMA` sizes the full-grid X arrays as `NSEALM*NAPROC` and fills most with `UNDEF`. | [kokkos/port_report/W3XDMA_KOKKOS.md](../kokkos/port_report/W3XDMA_KOKKOS.md) | | OUTFLAGS / NXXX / X* arrays | `W3XDMA`'s `LOGICAL OUTFLAGS(NOGRP,NGRPP)` argument, its local full-grid size `NSEALM*NAPROC`, and the 93 full-grid output copies of `WADAT` (`XHS` ... `XUSERO`) `(v)` `w3adatmd.F90:1555`, `:1560`, `:1596`, `:414-497` | Allocated by `W3XDMA`, gathered into by `DO_OUTPUT_EXCHANGES`, named by `W3XETA`'s aliases; `XUSERO`'s shape follows `OUTFLAGS(10,2)` alone, a WW3 defect kept as written. | [kokkos/port_report/W3XDMA_KOKKOS.md](../kokkos/port_report/W3XDMA_KOKKOS.md) | -| WMWAVE / WMIOHG / WMIOEG | "Run multi-grid version of WAVEWATCH III" / "Gather internal high-to-low data for a given model" / "Gather internal same-rank data for a given model" `(v)` `model/src/wmwavemd.F90:85`, `wminiomd.F90:1714`, `:2814` | `ww3_multi`'s driver and two of its exchanges between grids; the exchanges write a grid's `VA` on the host between its `W3WAVE` calls, which the resident spectrum's sync calls do not see. | [docs/PRODUCTION_READINESS_202610.md](PRODUCTION_READINESS_202610.md) | +| WMWAVE / WMIOHG / WMIOEG | "Run multi-grid version of WAVEWATCH III" / "Gather internal high-to-low data for a given model" / "Gather internal same-rank data for a given model" `(v)` `model/src/wmwavemd.F90:85`, `wminiomd.F90:1714`, `:2814` | `ww3_multi`'s driver and two of its exchanges between grids; the exchanges write a grid's `VA` on the host between its `W3WAVE` calls; with the resident `VA` they store the columns they wrote into the device copy (`kokkos/src/fortran_iface/PATCH.md`, section 26). | [docs/PRODUCTION_READINESS_202610.md](PRODUCTION_READINESS_202610.md) | +| WMIOBS / WMIOHS / WMIOES / W3MPIP | "Stage internal boundary data in the data structure BPSTGE" / "Stage internal high-to-low data in the data structure HGSTGE" / "Stage internal same-rank data in the data structure EQSTGE" / "Prepare MPI persistent communication needed for WAVEWATCH I/O routines" `(v)` `WW3-noaa/model/src/wminiomd.F90:121`, `:1402`, `:2516`, `w3initmd.F90:5412` at `03adda91` | The sending side of `ww3_multi`'s exchanges, which read a grid's `VA` on the host between its `W3WAVE` calls, and the set-up of the MPI requests that send point-output spectra from `VA`; with the resident `VA` each reads its columns back first (`kokkos/src/fortran_iface/PATCH.md`, section 26). | [docs/log/2026-10-18-ww3-multi-resident-va.md](log/2026-10-18-ww3-multi-resident-va.md) | | W3FLDG / W3FLDO | "Update input fields in the WAVEWATCH III generic shell from a WAVEWATCH III shell data file or write from preprocessor" / "Open and prepare WAVEWATCH III field files as used by the generic shell and the field preprocessor" `(v)` `w3fldsmd.F90:964-965`, `:116-117` | `W3FLDG` reads or writes one time record and the field records of `wind.ww3`, `level.ww3`, `current.ww3` and the others, gfortran unformatted big-endian; `W3FLDO` opens the unit and writes or checks its 40-byte header record (file id, field id, grid dimensions, filler, tide flag). | [kokkos/port_report/W3FLDG_KOKKOS.md](../kokkos/port_report/W3FLDG_KOKKOS.md), [kokkos/port_report/W3FLDO_KOKKOS.md](../kokkos/port_report/W3FLDO_KOKKOS.md) | ## HPC and software @@ -520,6 +522,8 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | NVMe | ⚠ Non-Volatile Memory Express (not expanded in the repo) | Fast local storage named among today's lab-machine features. | [pubs/proposal/pt/05-justification.md](../pubs/proposal/pt/05-justification.md) | | FP32 / FP64 / float32 | single / double precision; WW3's default `REAL` is 4 bytes, `ww::Real` is `float` `(v)` [kokkos/src/ww_kokkos/real.hpp](../kokkos/src/ww_kokkos/real.hpp) | The port keeps `float` state and accumulates in `double`. | [course/11-kokkos-and-modern-cpp.md](../course/11-kokkos-and-modern-cpp.md) | | FMA / `-ffp-contract` / `--fmad` | fused multiply-add and the GCC / nvcc flags that control contraction `(v)` [kokkos/README.md](../kokkos/README.md) | One rounding where the Fortran does two; `ww_kokkos` builds with `-ffp-contract=off` and `--fmad=false` for bit parity. | [course/12-porting-a-kernel-w3snl1.md](../course/12-porting-a-kernel-w3snl1.md), [kokkos/PORT_STATUS.md](../kokkos/PORT_STATUS.md) | +| `--expt-relaxed-constexpr` / nvcc `#20013-D` | nvcc's flag that lets device code call a `constexpr` host function, and the warning it prints without it `(v)` the CUDA build log, [docs/log/2026-10-18-cuda-build-only.md](log/2026-10-18-cuda-build-only.md) | Not set in the lab's CUDA Kokkos; the 48 warnings of 2026-10-18 are the host-only lambdas of the translated, unwired `DO_OUTPUT_EXCHANGES`. | [docs/log/2026-10-18-cuda-build-only.md](log/2026-10-18-cuda-build-only.md) | +| `#pragma GCC novector` | a GCC 14+ pragma that stops the vectoriser on the next loop `(v)` `g++ -fopt-info-vec-all` ("dont-vectorize loop") | Keeps the Fortran's operand order where g++'s vector body swaps an add; nvcc warns `#1675-D` and still passes it to the host g++. | [kokkos/src/ww_kokkos/uost_uost_sourceterm_compute.hpp](../kokkos/src/ww_kokkos/uost_uost_sourceterm_compute.hpp) | | ULP | ⚠ unit in the last place (not expanded in the repo) | The granularity of float rounding; the budget of a bit-parity test. | [kokkos/src/ww_kokkos/snl1_config.hpp](../kokkos/src/ww_kokkos/snl1_config.hpp) | | `powi()` / `pow11()` | integer-power helpers that reproduce gfortran's multiplication chain for `x**n` `(v)` [kokkos/README.md](../kokkos/README.md) | Why the port is bit-for-bit rather than "close": `x**n` is not `std::pow`. | [course/12-porting-a-kernel-w3snl1.md](../course/12-porting-a-kernel-w3snl1.md) | | Kokkos | ⚠ the Sandia-led C++ performance-portability library (not expanded; 5.2.0 pinned `(v)` [kokkos/README.md](../kokkos/README.md)) | One kernel source runs on Serial, OpenMP or CUDA backends chosen at build time; the abstraction WW4 proposes too. | [course/11-kokkos-and-modern-cpp.md](../course/11-kokkos-and-modern-cpp.md) | @@ -700,6 +704,8 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | `WW_KOKKOS_VA_RESIDENT` / `WW_KOKKOS_VA_COUNT` | diagnostics of the resident VA, ignoring `WW_KOKKOS_ALL`: `=1` makes every host sync call push a host-newer `VA` to the device and copy it back, so a host write without its sync call shows in the output; the count prints the traffic line and the copy counters at exit `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset. On the CPU both copies are `memcpy`, so the bits cannot change. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | | `WW_KOKKOS_VA_CHECK` / `WW_KOKKOS_VA_POISON` | diagnostics of the resident VA, read once, ignoring `WW_KOKKOS_ALL`, which compare or overwrite only the stale copy: `VA_CHECK=1` makes every sync call check what the state claims before it copies (unless `HOST`, the host `VA` still holds the bits of the last sync, so a host write without its `W3KOKKOS_VA_HOST_WRITE` stops the model; with `BOTH`, `d_va` holds them too, so a kernel that wrote `d_va` without marking it does); `VA_POISON=1` fills the host `VA` with a quiet NaN after every kernel write of `d_va`, so a host read without its sync, or a column the partial read misses, puts NaN in the output `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset; a correct run writes the same bits with either. Added by the mutation audit of 2026-10-17. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | | `WW_KOKKOS_OUTG_STAGED` | a diagnostic of the outg unit, read once: `=1` makes a host backend take the device's path of `ww_outg_va`, the inputs copied into the shim's own buffers, the fields computed into buffers, and only the fields a host reader needs copied back, so a field the copy-back selection misses keeps its old bits in WW3's array and shows in `out_grd.ww3` `(v)` [kokkos/src/fortran_iface/outg_shim.cpp](../kokkos/src/fortran_iface/outg_shim.cpp) | Default unset: on Serial and OpenMP the kernel writes WW3's arrays in place. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | +| `WW_KOKKOS_STRICT` | environment variable read once by `ww_kokkos_strict()`: `=1` makes every place where a part asked on Kokkos would run the Fortran stop the model with `EXTCDE` and a `NDSE` line naming the part and the reason: `KOKKOS_PART` under `WW_KOKKOS_ALL=1`, the two `W3SRCE` set-up fallbacks and the per-step `W3SRCE` gate (`W3KOKKOS_SRCE_STEP`). Unset, those places run the Fortran as before and the shim prints one line at exit, `ww_kokkos: calls left on the Fortran (...): SRCE= ...` `(v)` [kokkos/src/fortran_iface/part_flags.cpp](../kokkos/src/fortran_iface/part_flags.cpp) | Default unset. The regtest matrix's Kokkos run sets it, its Fortran runs unset it, so a pair cannot compare the Fortran with itself unseen. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 24 | +| `WW_KOKKOS_PRESET` / `presets.txt` | environment variable naming a line `PRESET ` of `kokkos/src/fortran_iface/presets.txt`: `ww_kokkos_flag` turns its parts on as if each `WW_KOKKOS_` were `1`; a set `WW_KOKKOS_` still wins, and `W3INIT` stops on a name the file lacks. `resident` is the only preset (`SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT`) `(v)` [kokkos/src/fortran_iface/presets.txt](../kokkos/src/fortran_iface/presets.txt) | The file is the one list: CMake compiles it into the shim, `scripts/ww3-gpu.sh --kokkos resident` and the matrix's `WW3GPU_KOKKOS=resident` read it. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 25 | | mutation audit / mutant (equivalent, surviving) | a deliberate, minimal breakage of a piece of the port, applied in a scratch tree and never committed, to ask whether the checks that call the piece correct would fail without it: a mutant is *detected* when a check fails, *surviving* when every check passes, and *equivalent* when it cannot change any bit the case computes (it is not a defect the case can show) `(v)` [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | First run 2026-10-17, on the CPU. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | | `WW_KOKKOS_ALL` / `ww_kokkos_flag` | the master run-time flag of every Kokkos part, and the one C function that resolves each `WW_KOKKOS_`: a set part flag wins (`1` on, anything else off), an unset one follows `WW_KOKKOS_ALL=1` (on) or stays off `(v)` [kokkos/src/fortran_iface/part_flags.cpp](../kokkos/src/fortran_iface/part_flags.cpp) | "100 %" means every wired part, not all of WW3; under the master a part whose translation does not cover the switch file stays Fortran. `W3INIT` logs one "Kokkos parts" line; `scripts/ww3-gpu.sh --kokkos all\|none\|` sets it. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | | W3KOKKOS_UNITSMD / W3KOKKOS_REGISTER / `parts.d` | the Fortran module the fork's CMake generates from every `model/src/w3kokkos_md.F90` (it `USE`s each unit module and calls each unit's set-up from `W3INIT`), the routine a unit's set-up calls to put a part on the "Kokkos parts" line, and the per-unit files `kokkos/src/fortran_iface/parts.d/.txt` that `scripts/ww3-gpu.sh` reads its parts from `(v)` h0ffmann/WW3 `ww3-gpu/kokkos-wire-base` | A wiring unit adds files and its own call-site hunks and edits no shared list. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | @@ -709,11 +715,12 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | `WW_KOKKOS_OUTG` / `WW_KOKKOS_OUTG_COUNT` / `ww_outg_va` / W3KOKKOS_OUTG | the run-time flag, counter and C entry point of the outg wiring unit (`W3OUTG`'s integrals on the resident VA, WFIP-0005 unit 5), and its Fortran wrapper on the fork; opt-in only: `WW_KOKKOS_ALL=1` does not turn it on; the wrapper calls the Fortran `W3OUTG` when partition, `CALC_U3STOKES`, `SKEWNESS` or `CALC_WBT` fields are requested; the count prints calls, sea points and the fields and bytes copied back at exit `(v)` [kokkos/src/fortran_iface/outg_shim.cpp](../kokkos/src/fortran_iface/outg_shim.cpp) | Wired on h0ffmann/WW3 `ww3-gpu/kokkos-outg`, 2026-10-17; run byte-identical on the CPU, not regtested. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | | `WW_KOKKOS_IOPE` / `ww_va_host_read_cols` / W3KOKKOS_IOPE_READ | the run-time flag, C entry point and Fortran wrapper of the pnt wiring unit (WFIP-0005): before `W3IOPE` (point output) the host copy of the resident `VA` is refreshed only in the columns `W3IOPE` reads, the sea points `MAPFS` gives at the four interpolation corners `IPTINT` of each output point, gathered on the device and copied back; the resident state is not changed, so the other columns stay stale until the next full host read; opt-in only: `WW_KOKKOS_ALL=1` does not turn it on; counted under `WW_KOKKOS_VA_COUNT` (`VA partial host reads`, `... device-to-host copies`, `... columns copied`) `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Wired on h0ffmann/WW3 `ww3-gpu/kokkos-pnt`, 2026-10-17; run byte-identical on the CPU, not regtested. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | | `WW_KOKKOS_UPDV` / `ww_va_device_zero_cols` / W3KOKKOS_UICE | the run-time flag, C entry point and Fortran wrapper of the updv wiring unit (WFIP-0005): at an ice update the Fortran `W3UICE` runs on the host `VA`, the columns it set to 0 are found from the ice bit of `MAPST2` before and after, and the same columns are set to 0 in the resident `VA` on the device, so no `VA` crosses; with `ww_va_host_update` before it; under `WW_KOKKOS_VA_RESIDENT=1` the two copies are compared after each update and a difference stops the model; opt-in only: `WW_KOKKOS_ALL=1` does not turn it on; `W3ULEV` is not covered `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Wired on h0ffmann/WW3 `ww3-gpu/kokkos-pnt`, 2026-10-17; run byte-identical on the CPU, not regtested. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | +| `WW_KOKKOS_CPRT` / `ww_cprt_va` / `ww_cprt_fetch` / `ww_outg_part` / W3KOKKOS_CPRT | the run-time flag, C entry points and Fortran wrapper of the cprt wiring unit: at an output time with partitioning, `W3CPRT` (the watershed partitioning of every local sea point, `W3PART` per point) runs on the resident `VA` as a Kokkos launch over the points, each point the Fortran's serial code, and only the table `ICPRT` and the used columns of `DTPRT` come back to the host; `ww_outg_part` fills `W3OUTG`'s partition fields (section 6) from that table without `VA`, so a gridded output with partition fields no longer falls back to the Fortran `W3OUTG`; a NaN depth or a spectrum with no peak falls back to the Fortran `W3CPRT` after a host read; opt-in only: `WW_KOKKOS_ALL=1` does not turn it on; `WW_KOKKOS_CPRT_COUNT=1` prints its counters, `WW_KOKKOS_CPRT_CHUNK` the points per launch `(v)` [kokkos/src/fortran_iface/cprt_shim.cpp](../kokkos/src/fortran_iface/cprt_shim.cpp) | Wired on h0ffmann/WW3 `ww3-gpu/u9-cprt`, 2026-10-18; run byte-identical on the CPU. | [kokkos/port_report/W3CPRT_KOKKOS.md](../kokkos/port_report/W3CPRT_KOKKOS.md) | | `scripts/libm_bits.sh` / `kokkos/tools/libm_bits/` / `ww::libm` | the checker and the host-and-device copies of the libm functions the `W3SRCE` unit calls per sea point (glibc 2.44's `expf`, `logf`, `powf`, `sincosf`, libmvec's `_ZGVbN4v_expf` and `_ZGVbN4vv_powf`, CORE-MATH's `tanhf`, `log10f`, `atan2f`, `coshf`, `sinhf`, and double `exp`, `tanh`), in namespace `ww::libm` `(v)` [scripts/libm_bits.sh](../scripts/libm_bits.sh) header, #177, #178 | `probe` and `fingerprint` name this CPU's glibc build, `all` runs every host check, `--nvcc` compiles the device check, `--device` runs it under the owner only (exit 7 without a typed "yes"). On the host every copy equals glibc on every input checked; on a device it is ⚠ until `--device` runs. | [docs/GPU_LIBM_BITS_202610.md](GPU_LIBM_BITS_202610.md), [docs/WFIPs/WFIP-0006-device-libm-bits.md](WFIPs/WFIP-0006-device-libm-bits.md) | | `libm_device.hpp` / `ww::dm` / `WW_KOKKOS_LIBM` | the header through which every libm call of the kernels `WW_KOKKOS_ALL` runs goes, its namespace, and the CMake option (CUDA builds only, `ON` by default) that points its device branch at `ww::libm`; the host branch is the Kokkos call the kernel made before, so the CPU build calls glibc as it did `(v)` [kokkos/src/ww_kokkos/libm_device.hpp](../kokkos/src/ww_kokkos/libm_device.hpp), [kokkos/src/ww_kokkos/CMakeLists.txt](../kokkos/src/ww_kokkos/CMakeLists.txt) | WFIP-0006 units 1 to 4 (#178). Overloads for the other argument type are deleted, so a float/double promotion does not compile. `-DWW_KOKKOS_LIBM=OFF` keeps libdevice and ADR-0014's verdict by tolerance. | [docs/WFIPs/WFIP-0006-device-libm-bits.md](WFIPs/WFIP-0006-device-libm-bits.md), [AGENTS.md](../AGENTS.md) | | `gen_tables` / `ww_libm_tables.hpp` / `WW_LIBM_HOST_FMA` | a host program the CUDA build compiles with the host g++ (nvcc's front end fails on it) and runs at build time; it writes a header with libmvec's `powf` tables read from the loaded `libmvec.so.1`, this host's `rcpps` values, and `WW_LIBM_HOST_FMA`, glibc's fma or sse2 build decided from glibc's own `exp` `(v)` [kokkos/src/ww_kokkos/CMakeLists.txt](../kokkos/src/ww_kokkos/CMakeLists.txt), [kokkos/tools/libm_bits/gen_tables.cpp](../kokkos/tools/libm_bits/gen_tables.cpp) | It follows `GLIBC_TUNABLES`; the run host must be the build host, or have the same glibc build and `rcpps`, which nothing checks at run time ⚠. | [docs/WFIPs/WFIP-0006-device-libm-bits.md](WFIPs/WFIP-0006-device-libm-bits.md) | | `WW_KOKKOS_TIMERS` / `--timers` / `ww::timers::Region` | a diagnostic environment variable, read once and ignoring `WW_KOKKOS_ALL`: `=1` makes every shim time its regions, named `.` (`.call` the entry point, `.h2d` the copies to the device, `.kernel` the launch, `.d2h` the copies back; `st4.tables.call` the ST4 table rebuild), fencing the default execution space at each start and stop, and print one line per region at exit `(v)` [kokkos/src/fortran_iface/timers.hpp](../kokkos/src/fortran_iface/timers.hpp) | `scripts/ww3-gpu.sh --ww3-gpu --timers` sets it and prints each region's share of `ww3_shel` and the time outside Kokkos; refused with `--repeat`, because a fenced run is a breakdown, not a timing. Off, a region is one branch (#175). | [scripts/ww3-gpu.sh](../scripts/ww3-gpu.sh), [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | -| `--kokkos gpu` / `--kokkos resident` / `just perf-ww3-gpu` | the two part-list presets of `scripts/ww3-gpu.sh`: `gpu` is `SRCE` alone (ADR-0013), `resident` is `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV`, the parts that keep `VA` on the device (WFIP-0005 units 1 to 5); and the recipe that times a case with `--gpu --repeat` `(v)` [scripts/ww3-gpu.sh](../scripts/ww3-gpu.sh) `kokkos_preset`, [justfile](../justfile) | `resident` is not a default: none of its parts is regtested yet (`kokkos/ports.tsv`). With it the full case copies `VA` to the device once and back once, on the CPU (#179); nothing has run on a GPU. `just perf-ww3-gpu` asks the supervising human for "yes". | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), [bench/README.md](../bench/README.md) | +| `--kokkos gpu` / `--kokkos resident` / `just perf-ww3-gpu` | the two part-list presets of `scripts/ww3-gpu.sh`: `gpu` is `SRCE` alone (ADR-0013), `resident` is `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT`, the parts that keep `VA` on the device (WFIP-0005 units 1 to 5, and `W3CPRT`, PATCH.md section 27); and the recipe that times a case with `--gpu --repeat` `(v)` [scripts/ww3-gpu.sh](../scripts/ww3-gpu.sh) `kokkos_preset`, [justfile](../justfile) | `resident` is not a default: none of its parts is regtested yet (`kokkos/ports.tsv`). With it the full case copies `VA` to the device once and back once, on the CPU (#179); nothing has run on a GPU. `just perf-ww3-gpu` asks the supervising human for "yes". | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), [bench/README.md](../bench/README.md) | | `ww_srce_init` / `ww_srce` / `ww_srce_batch` / `ww_srce_enabled` / `ww_srce_last_error` | the C entry points of the W3SRCE port; `ww_srce_batch` takes every active sea point of a time step in one call (`W3KOKKOS_SRCE_BATCH` on the fork) `(v)` [kokkos/src/fortran_iface/ww_kokkos_c.hpp](../kokkos/src/fortran_iface/ww_kokkos_c.hpp) | `ww_srce_init` needs the DIA and ST4 contexts of the same grid | | GIMPLE | GCC's three-address intermediate representation; `-fdump-tree-optimized` writes it after the optimisations `(v)` [ADR-0011](ADRs/ADR-0011-port-in-units-regtest-at-checkpoints.md) | The static-parity argument compares the Fortran's and the C++'s, statement by statement | | `maxss` / `minss` / `maxps` | the x86 SSE scalar and packed float max/min instructions; `maxss a,b` keeps `b` only when `b > a`, so a NaN or a `+0`/`-0` tie returns `a` `(v)` [kokkos/port_report/W3SRCE_KOKKOS.md](../kokkos/port_report/W3SRCE_KOKKOS.md) | The operand order a Fortran `MAX`/`MIN` gets in the object, which the port copies; GCC may swap it after GIMPLE | @@ -721,6 +728,7 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | `WW_KOKKOS_DEVICE_ID` | environment variable selecting the GPU (default 0), read instead of `MPI_Comm_rank()` because phase 1 links no MPI `(v)` [kokkos/README.md](../kokkos/README.md) | Set per rank by the launcher in an MPI job. | [kokkos/src/fortran_iface/ww_kokkos_c.hpp](../kokkos/src/fortran_iface/ww_kokkos_c.hpp) | | `WW_DETERMINISTIC` | CMake option: "force serial reductions for bit-reproducible validation builds" `(v)` [kokkos/CMakeLists.txt](../kokkos/CMakeLists.txt) | On in `serial-debug`; changes nothing for the DIA, which has no reduction. | [course/11-kokkos-and-modern-cpp.md](../course/11-kokkos-and-modern-cpp.md) | | `WW_ENABLE_FORTRAN` / `WW_WW3_BUILD_DIR` | build the Fortran module, reference and drivers (default ON) / a configured WW3 build (`libww3.a` + `mod/`) that enables the `ww3_lib` cross-check `(v)` [kokkos/README.md](../kokkos/README.md) | The second is what `just l1-crosscheck` sets. | [kokkos/README.md](../kokkos/README.md) | +| `WW_KOKKOS_BUILD_TESTS` / `WW3GPU_KOKKOS_PREFIX` | CMake option of `kokkos/` (default ON): OFF is the install-only configure, `ww_kokkos` and its package config with Kokkos the one dependency searched, no GoogleTest, NetCDF, tools or Fortran / the variable that makes `scripts/ww3-gpu.sh --ww3-gpu` build WW3 against a `ww_kokkos` package installed elsewhere `(v)` [kokkos/CMakeLists.txt](../kokkos/CMakeLists.txt), [scripts/ww3-gpu.sh](../scripts/ww3-gpu.sh) | The installed `ww_kokkosConfig.cmake` asks for the Kokkos version it was built with `EXACT` and refuses other backends or architecture. | [kokkos/README.md](../kokkos/README.md) | | `WW_FIXTURE_DIR` / `WW_TEST_ENVIRONMENT` | compile definition pointing tests at `tests/fixtures/` / the CTest environment `OMP_PROC_BIND=false;OMP_NUM_THREADS=2` `(v)` [kokkos/CMakeLists.txt](../kokkos/CMakeLists.txt) | Test plumbing. | [kokkos/tests/CMakeLists.txt](../kokkos/tests/CMakeLists.txt) | | `serial-debug` / `openmp-release` / `cuda-release` | the three presets: Debug + sanitizers + `WW_DETERMINISTIC` / Release `-O3 -march=x86-64-v3` / Release with `nvcc_wrapper`, arch 89 `(v)` [kokkos/CMakePresets.json](../kokkos/CMakePresets.json) | The name says build type, not backend. | [course/11-kokkos-and-modern-cpp.md](../course/11-kokkos-and-modern-cpp.md) | | `cuda-h100-release` / `WW_KOKKOS_ARCH` | the H100 preset: `cuda-release` with `CMAKE_CUDA_ARCHITECTURES=90` and `WW_KOKKOS_ARCH=HOPPER90`; `WW_KOKKOS_ARCH`, a CMake cache variable of `kokkos/`, names a Kokkos architecture the found Kokkos must have, checked with Kokkos' `kokkos_check` at configure time `(v)` [kokkos/CMakePresets.json](../kokkos/CMakePresets.json), [kokkos/CMakeLists.txt](../kokkos/CMakeLists.txt) | Not the default: the `#cuda` shell's Kokkos is ADA89, so the preset stops at configure there (v, 2026-10-17); `scripts/ww3-gpu.sh --gpu` builds `cuda-release`. | [kokkos/README.md](../kokkos/README.md) | @@ -1053,6 +1061,7 @@ Every term above, alphabetically, with the section it lives in. - EXPAND: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - EXPND2: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - EXTCDE: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) +- `--expt-relaxed-constexpr` / nvcc `#20013-D`: [HPC and software](#hpc-and-software) - `extern "C"`: [HPC and software](#hpc-and-software) - EXTIOF / EXTOPN: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - F, E: [Wave physics and the spectrum](#wave-physics-and-the-spectrum) @@ -1126,6 +1135,7 @@ Every term above, alphabetically, with the section it lives in. - F_ZHAO_V1: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - GAULEG / INSNLGQM: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - GAULEG: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) +- `#pragma GCC novector`: [HPC and software](#hpc-and-software) - gdb stack sampling: [HPC and software](#hpc-and-software) - GEBCO: [HPC and software](#hpc-and-software) - GEFS-Wave: [Models, projects and institutions](#models-projects-and-institutions) @@ -1174,6 +1184,7 @@ Every term above, alphabetically, with the section it lives in. - IC3TABLE_CHENG: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - ICL DAP: [HPC and software](#hpc-and-software) - ICLO / ICLO_NONE / ICLO_GRDI / ICLO_GRDJ / ICLO_TRDL / ICLO_TRPL: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) +- ICPRT / DTPRT / FINSIZ: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - IFILE4: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - IJKCel3: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - IJKUFc: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) @@ -2140,6 +2151,7 @@ Every term above, alphabetically, with the section it lives in. - wm*md.F90 (WMESMFMD: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - wm*md.F90 (WMESMFMD, WMSCRPMD, …): [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - WMSCRPMD: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) +- WMIOBS / WMIOHS / WMIOES / W3MPIP: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - WMWAVE / WMIOHG / WMIOEG: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - WW3GPU_LAB_SWITCH: [This repository's own names](#this-repositorys-own-names) - WW_KOKKOS_STRICT / WW_BUILD_TESTS (proposed): [This repository's own names](#this-repositorys-own-names) @@ -2226,6 +2238,8 @@ Every term above, alphabetically, with the section it lives in. - `ww_kokkos`: [This repository's own names](#this-repositorys-own-names) - `ww_kokkos_c.hpp`: [This repository's own names](#this-repositorys-own-names) - `ww_kokkos_c.hpp` / `snl1_shim.cpp` / `w3kokkosmd.F90`: [This repository's own names](#this-repositorys-own-names) +- `WW_KOKKOS_BUILD_TESTS` / `WW3GPU_KOKKOS_PREFIX`: [This repository's own names](#this-repositorys-own-names) +- `WW_KOKKOS_CPRT` / `ww_cprt_va` / `ww_cprt_fetch` / `ww_outg_part` / W3KOKKOS_CPRT: [This repository's own names](#this-repositorys-own-names) - `WW_KOKKOS_DEVICE_ID`: [This repository's own names](#this-repositorys-own-names) - `WW_KOKKOS_TIMERS` / `--timers` / `ww::timers::Region`: [This repository's own names](#this-repositorys-own-names) - `WW_KOKKOS_ERR_RUNTIME`: [This repository's own names](#this-repositorys-own-names) diff --git a/docs/GPU_LIBM_BITS_202610.md b/docs/GPU_LIBM_BITS_202610.md index 3602e2a6..f4a64183 100644 --- a/docs/GPU_LIBM_BITS_202610.md +++ b/docs/GPU_LIBM_BITS_202610.md @@ -280,7 +280,9 @@ reference, not of the port. It is not needed now. ## What is not shown -- No device run: that the device's bits equal the host's is ⚠ until `--device` runs. +- No device run: that the device's bits equal the host's is ⚠ until `--device` runs. The + owner's supervised run does it first, and records it here: + [`GPU_SUPERVISED_RUN_CHECKLIST_202610.md`](GPU_SUPERVISED_RUN_CHECKLIST_202610.md), step 1. - No kernel of `kokkos/src` calls these headers yet; [WFIP-0006](WFIPs/WFIP-0006-device-libm-bits.md) is the design, and until it is done ADR-0014's verdict by tolerance stands. - `mvec_powf` was checked against one CPU's libmvec; a second vendor's `rcpps` was not seen ⚠. diff --git a/docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md b/docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md new file mode 100644 index 00000000..432f04f4 --- /dev/null +++ b/docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md @@ -0,0 +1,176 @@ +# Checklist for the owner's supervised GPU run, October 2026 + +The steps, in order, for the first runs of the Kokkos port on Kokkos' CUDA backend, and what each +one must leave on record. Only the owner runs them, at a terminal, watching the run (AGENTS.md, +"Running on the GPU"); every command below asks for a typed "yes" and exits 7 without one. An +agent never runs them, never sets `WW3GPU_IN_SHELL` or `WW3GPU_GPU_CONFIRMED`, and never pipes +the answer. + +A GPU run is judged by `nccmp-tol` with `kokkos/tools/nccmp-tol/tolerances.txt`, against the +Fortran of the same executable and against `--ww3`, and a tolerance is never widened to make it +pass ([ADR-0014](ADRs/ADR-0014-gpu-runs-judged-by-tolerance.md)). Why the bits can differ, site +by site, is [`GPU_LIBM_BITS_202610.md`](GPU_LIBM_BITS_202610.md). This page is unit U5 of +[`compound/plans/2026-10-18-002-production-single-node-plan.md`](compound/plans/2026-10-18-002-production-single-node-plan.md) +and the GPU half of gap 4 of [`PRODUCTION_READINESS_202610.md`](PRODUCTION_READINESS_202610.md). + +Nothing on this page has run on a GPU. The build of step 0 was compiled on 2026-10-18 without a +run ([`log/2026-10-18-cuda-build-only.md`](log/2026-10-18-cuda-build-only.md)); every statement +about a device result is ⚠ until the step that answers it has run. + +## 0. Before the first command + +**The build.** Both commands compile only; neither touches the device. + +```bash +git submodule update --init WW3 WW3-noaa && just submodule-init +nix develop ./picasso/labs/pratico#cuda --command cmake -S kokkos --preset cuda-release +nix develop ./picasso/labs/pratico#cuda --command cmake --build kokkos/build/cuda-release -j8 +scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident --build-only +``` + +Do not use `just kokkos-cuda-test` for this step: its last line runs `ctest` with CUDA as the +default execution space, which is a device run. On 2026-10-18 both builds passed with 51 nvcc +warnings, none in a part the fork calls, and 8 gfortran warnings in WW3's own files (v, the log +entry above). A new warning in the build of the run's commit is read before step 1. + +**Write down, once, in the log entry of the run:** + +| What | Command | +|---|---| +| ww3-gpu commit, and whether the tree is clean | `git rev-parse --short HEAD`, `git status --porcelain` | +| `WW3/` pin (fork), `WW3-noaa/` pin, `picasso/` pin | `git submodule status WW3 WW3-noaa picasso` | +| CPU model and hardware threads | `grep -m1 'model name' /proc/cpuinfo`, `nproc --all` | +| GPU model and driver version | `nvidia-smi --query-gpu=name,driver_version --format=csv,noheader` | +| the CUDA version the driver supports | the header of `nvidia-smi` | +| nvcc, Kokkos, the host compilers | `nvcc --version`, the `kokkos-cuda-` store path in `kokkos/build/cuda-release/CMakeCache.txt`, `gfortran --version`, and the "CXX compiler identification" line of the configure (nvcc_wrapper's host g++, 14.4.0 on 2026-10-18, while the `#ww3` shell's is 16.2.0) | +| whether the GPU drives a display | `nvidia-smi --query-gpu=display_active --format=csv,noheader` | +| the machine's load before and after | `cut -d' ' -f1 /proc/loadavg` | + +A GPU that drives a display has a watchdog that ends a long kernel with +`cudaErrorLaunchTimeout`; `WW_KOKKOS_SRCE_BATCH_MIB` (default 128 MiB on a device) keeps a +`W3SRCE` launch short for that reason, and its limit was never measured +(`kokkos/src/fortran_iface/PATCH.md`, the `WW_KOKKOS_SRCE_BATCH_MIB` paragraph). Record the value +used when it is set. A launch timeout is a stop, not a reason to change the case. + +**Stop rules for every step.** A step that fails stops the checklist. The failure is recorded as +it is, with its output, in the log entry. A tolerance, a tolerance file, a comparison script and +the way it is called stay as they are; a change to any of them is its own pull request reviewed +by a person (AGENTS.md, "Never cheat a regtest tool"). + +## 1. The device math: `scripts/libm_bits.sh --device` + +```bash +scripts/libm_bits.sh --device +``` + +It runs `device_check` on GPU 0: every transcribed function (`expf`, `logf`, `sincosf`, `powf`, +the libmvec copies, `tanhf`, `log10f`, `atan2f`, `coshf`, `sinhf`, `exp`, `tanh`), every input +of each against the host's glibc, with CUDA's own functions printed beside them. Exit +0: no difference; 1: a difference; 2: a CUDA error; 7: not confirmed. + +Record: + +- the full output, per function: inputs checked, inputs different, the first differing input; +- the exit code; +- in [`GPU_LIBM_BITS_202610.md`](GPU_LIBM_BITS_202610.md), a dated "Results on the device" + section beside "Results on the host", and the "What is not shown" line about the device updated + only by what the run showed. + +A difference here does not stop steps 2 to 4 (ADR-0014 already judges them by tolerance), but it +says which `⚠` of the libm page stays. + +## 2. The smoke case on the GPU + +```bash +just smoke-ww3-gpu gpu +scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident +``` + +The first is `--kokkos gpu` (`SRCE` alone, ADR-0013), the default of `--gpu`; the second is the +resident preset (`SRCE,XYP3,KTP3,OUTG,IOPE,UPDV`). Both run `ww3_ts1` twice in one executable +(the Fortran, then Kokkos), then `--ww3`, pure NOAA WW3 from `WW3-noaa/` on the CPU. + +Record, for each of the two commands: + +- the `nccmp-tol` table against the Fortran of the same executable, and the one against `--ww3`, + every field, listed or not: `max|d|`, rms, largest relative difference, and the verdict of the + judged ones (`hs` on the smoke: "1 judged, 25 unlisted", ADR-0014, "Consequences"); +- which model files `cmp` found different, against each reference (on CUDA the script reports + them and goes on); +- the parts line the Kokkos run printed, which shows the parts the preset asked for ran on + Kokkos; +- the exit code. + +The tables go into `docs/data/ww3gpu_cuda_smoke_.md`, the command and the verdict +into the log entry. A failed `nccmp-tol` stops the checklist. + +## 3. The full case on the GPU + +```bash +just full-ww3-gpu gpu +scripts/ww3-gpu.sh --ww3-gpu full --gpu --kokkos resident +``` + +`bench_small`: 120 x 80, 9 520 sea points, NK = NTH = 24, 12 hours. Record the same items as in +step 2, in `docs/data/ww3gpu_cuda_full_.md`; the full case judges `hs` of five +listed fields (ADR-0014). The wall times these runs print are one run each: a check that the +pieces work, never a number for a table. + +**The timing, only after step 3 passed on the same commit:** + +```bash +just perf-ww3-gpu full 3 resident +``` + +Three runs of each, alternated, every output compared with the first run's, and the medians; the +script writes `perf.tsv` only after its own `nccmp-tol` comparisons passed. On a quiet machine +(load average recorded before and after). Then `python3 scripts/results.py record ` +writes the `bench/results/` record, whose `regtest` says "GPU: judged by tolerance, not bits" +(ADR-0014, point 4), and `just results check`. A figure quoted anywhere names the machine, the +command and the median of three, and says "regtest pending ⚠" until the matrix of step 4 has +passed on its commit (ADR-0016). + +## 4. WW3's regtest matrix on the GPU: `just ww3-gpu-matrix-cuda` + +On the entries the CPU regtests of this pull request passed: + +> **Entries: filled by U3 of this PR.** The integrator writes here the CTest names +> (`NNN::`) of the entries that passed on the CPU with the resident preset and the +> lab switch, and the record they came from in `docs/data/`. + +```bash +just ww3-gpu-matrix-cuda 2 serial -R '' +``` + +Two things about the recipe as it is on 2026-10-18 (v, `justfile`, the `ww3-gpu-matrix-cuda` +recipe, and `regtests/CMakeLists.txt`): + +- It forwards `WW3GPU_ROUTINES` only. `WW3GPU_KOKKOS=resident` and `WW3GPU_LAB_SWITCH=ON`, which + the CPU entries of U3 used, are not passed to its CMake configure, so the recipe runs each + entry's own `matrix.base` switch with the routines of `WW3GPU_ROUTINES` (`SNL1` by default), + not the resident preset. Running the U3 entries as U3 ran them needs those two variables + forwarded, a change to how a regtest driver is called and so reviewed by a person first. + Until then, record which configuration ran. +- `WW3GPU_STRICT` defaults to ON, so an entry fails when `test.comp` finds a file different. On + CUDA that verdict is expected and is not the ADR-0014 verdict: read the `nccmp_tol` and + `judged` columns of `summary.tsv` for that, and record both. + +Record: + +- `build/ww3-gpu-matrix-cuda-[...]/summary.tsv`, copied to + `docs/data/ww3gpu_matrix_cuda__.tsv` with its header lines (date, + commits, machine, command, switch, Kokkos flags); +- per entry: `test_comp`, `nccmp_tol`, `judged`, `kokkos_spectra` (an entry with 0 did not + exercise the port and is reported as such), `ww3_self`; +- the log entry: which entries passed `nccmp-tol`, which differ and in which field. + +## After the run + +- One dated entry in `docs/log/` with every record above, indexed in + [`log/README.md`](log/README.md), and linked from the pull request that commits it. +- [`PRODUCTION_READINESS_202610.md`](PRODUCTION_READINESS_202610.md), gap 4: the "GPU" paragraph + updated by what was measured. +- ADR-0014's "Revisit when" names the first supervised run: the owner reads its `nccmp-tol` + tables, every field, and decides whether the ADR stands. +- `kokkos/ports.tsv` changes only where the record supports it, then + `python3 scripts/port_score.py --write` and `--check`. diff --git a/docs/PRODUCTION_READINESS_202610.md b/docs/PRODUCTION_READINESS_202610.md index dea60643..52b3d126 100644 --- a/docs/PRODUCTION_READINESS_202610.md +++ b/docs/PRODUCTION_READINESS_202610.md @@ -38,18 +38,18 @@ regtest matrix yet"). 10 of the 310 rows of `kokkos/ports.tsv` are regtested and | # | Gap | (a) | (b) | Smallest next step | |---|---|---|---|---| | 1 | No regtest switch has the source-term set the Kokkos `W3SRCE` translates; NOAA's UFS regtests miss it too | Blocker | Blocker | Read GFS-Wave's operational switch file at its tag; run the `WW3GPU_LAB_SWITCH` matrix on the MPI and `ww3_multi` entries | -| 2 | `ww3_multi` writes `VA` on the host between `W3WAVE` calls with no sync | Blocker | Blocker | Mark the host newer when `W3WAVE` returns; run `mww3_test_0x` with `WW_KOKKOS_VA_CHECK=1` | +| 2 | `ww3_multi` writes `VA` on the host between `W3WAVE` calls with no sync; fixed under `SHRD` on 2026-10-18 by column syncs at the exchanges, passed again at the integrated pin `50a10730` (section 2) | Minor ⚠ | Minor ⚠ | `mww3_test_08` under MPI when the owner runs MPI | | 3 | MPI: four of the six resident parts are `SHRD` only, the preset stops on `DIST MPI`, the communicator is ignored, the device id is not per rank | Blocker | Major | A resident preset without `XYP3`, `OUTG`, `IOPE`, `UPDV`; one run on `switch_lab_mpi` | | 4 | Bit parity measured with gfortran and g++ only; the operational compiler and the GPU are not measured | Blocker | Major | Static argument for the Intel build's math calls; one smoke with it | -| 5 | Build and packaging: a separate static library whose install needs GoogleTest and NetCDF, an unversioned Kokkos, C++20 and CMake 3.25 | Major | Blocker | An install-only configure with Kokkos as the one dependency, its version recorded in `ww_kokkosConfig.cmake` | -| 6 | `WW_KOKKOS_ALL=1` and `W3KOKKOS_SRCE_ON` fall back to the Fortran without a word | Major | Minor | A strict flag that turns every fallback into a stop, and a per-run fallback count | -| 7 | Configuration is 61 environment variables, no namelist; the preset lives in a lab script | Major | Major | A namelist group read by `W3INIT`, the environment as override, the effective set printed once | +| 5 | Build and packaging: a separate static library, C++20 and CMake 3.25; the install-only configure and the exact Kokkos check are done (U6) | Major | Major | A shared-library build and a recipe for the module systems of an operational centre | +| 6 | `WW_KOKKOS_ALL=1` and `W3KOKKOS_SRCE_ON` fall back to the Fortran without a word | Major | Minor | Done on fork branch `ww3-gpu/u2-strict-preset`, not yet at the pin: `WW_KOKKOS_STRICT=1` stops, otherwise one fallback-count line at exit; the matrix's Kokkos runs set it | +| 7 | Configuration is 61 environment variables, no namelist; the preset lives in a lab script | Major | Major | Smallest form done on `ww3-gpu/u2-strict-preset`: `WW_KOKKOS_PRESET=resident` from one table, `presets.txt`; next, a namelist group read by `W3INIT` | | 8 | Observability: log lines on every rank, counters at exit to `stderr`, no provenance in the output files | Major | Minor | Gate the lines by `NAPLOG`; one NetCDF global attribute | | 9 | OpenMP inside MPI: the per-point parts are serialised by `!$OMP CRITICAL`; the thread budget per rank is not set by the shim | Major | Minor | Measure the hybrid entries of `ww3_ufs1.1` before claiming a speed for them | -| 10 | Restart with the resident spectrum: covered by the sync calls, never regtested | Major | Minor | The restart-reproduction pair of `ww3_ufs1.1` (`work_c` against `work_d`) under the lab switch | +| 10 | Restart with the resident spectrum: covered by the sync calls, never regtested; every `matrix.base` restart pair is `MPI` | Major | Minor | Run `scripts/restart_pair.sh` (a `SHRD` restart pair on `full`, written 2026-10-18, not run) | | 11 | `PDLIB` and unstructured grids: translated, not wired, gated out | Minor ⚠ | Major | None for GFS-Wave v16; a wiring unit for `PDLIB` when the owner asks | -| 12 | The fork patch: 4 082 lines of new Fortran, 819 changed lines in three WW3 files, the C++ in another repository under another licence | Minor | Blocker | An ADR on where `ww_kokkos` lives for upstream | -| 13 | A stale contract remark: "one spectral grid per process" | Minor | Minor | Edit the remark in the fork | +| 12 | The fork patch: 4 082 lines of new Fortran, 819 changed lines in three WW3 files, 58 218 lines of C++ in another repository, most of it LGPL-3.0-or-later, some MIT, 76 files with no licence line | Minor | Blocker | ADR-0018 (Proposed) names the three places `ww_kokkos` could live; the owner chooses, after a legal reading ⚠ | +| 13 | A stale contract remark: "one spectral grid per process"; edited on the fork branch `ww3-gpu/u1-multi-va` (section 13) | Minor | Minor | Move the pin | The sections below give the evidence of each gap. The ranked top five closes the page. @@ -131,6 +131,33 @@ per grid too (v, `snl1_shim.cpp:83-89`), so several grids in one process are sup costs one host-to-device copy of `VA` at the next call; run the `mww3_test_01` to `_03` entries with `WW_KOKKOS_VA_CHECK=1` under the lab switch. +**Update, 2026-10-18 (plan 002, U1).** Reproduced and fixed under `SHRD` +([log](log/2026-10-18-ww3-multi-resident-va.md), `PATCH.md` section 26). With the lab switch +and the resident preset, `WW_KOKKOS_VA_CHECK=1` stopped `mww3_test_02`, `_03` and `_04` +(`grdset_a`), and without it they differed from the Fortran in 5, 10 and 4 files, with +`ww3_self` 0 (v). The step above was not taken: on a GPU it adds a whole copy of `VA` per grid +per call, which the owner refused. Instead each `ww3_multi` routine that reads or writes `VA` +between `W3WAVE` calls (`WMIOBS`, `WMIOHS`, `WMIOES`, `WMIOHG`, `WMIOEG`, and `WMWAVE`'s +unified point output) syncs only the columns of its own loop, and `W3WAVE` no longer reads +`VA` back at its return, since `ww3_shel` reads none after it. This adds `wminiomd.F90` and +`wmwavemd.F90` to the fork's diff. After, the six `mww3_test_01` to `_05` entries gave +`test.comp` 0 files different, also with the host copy poisoned, and no entry's whole-array +copies went up: `_04` went from 864 to 0 device-to-host copies, and `ww3_shel` full from 1 to 0 +(v). Left open: `mww3_test_08` (MPI), whose six differing files the same defect explains by +reading ⚠, not run (ADR-0017); the fork branch waits for the pin bump. + +- **(a), (b) now Minor** under `SHRD` ⚠: shown on the four `PR3 UQ` `grdset_a`-type entries and + one moving grid, with the lab switch's source terms, not on GFS-Wave's grids. + +**Update, 2026-10-18 (plan 002, U3).** On the integrated pin (`WW3/` `50a10730`, `CPRT` in the +preset), agent A's commands again gave `test.comp` 0 files different with the host copy poisoned +on `mww3_test_01` to `_04` (resident) and on both `mww3_test_05` moving-grid entries +(`SRCE,OUTG,IOPE,UPDV`), and the five `mww3_test_05` `ST4` OMPG entries passed with `CPRT` too (v, +[log](log/2026-10-18-u3-regtests-v2.md)). Whole-`VA` copies device to host: `_02` 8 → 0, `_03` +216 → 3, the others unchanged (v). `mww3_test_01` to `_04` have `FLSOU` false, so they test the +exchanges and the propagation, not `W3SRCE` (v). `mww3_test_07` fails in the Fortran run as well +(its `SDS4` namelist under the lab switch). `mww3_test_08` (MPI) is still not run. + ## 3. MPI *Scope, 2026-10-18:* out of scope by decision. The port is single node, and an MPI build keeps @@ -195,6 +222,10 @@ WFIP-0006) with the host, and every flag's parity note says "GPU ⚠ never run" (ADR-0014). WCOSS2 has no GPUs (the `WCOSS2` row of [`GLOSSARY.md`](GLOSSARY.md)), so for (a) today the GPU is not the path; GFS-Wave's spectral grid, 50 x 36, also needs the DIA's level-1 scratch on the RTX 4090 (v, `kokkos/PORT_STATUS.md:63`), a speed matter, not a parity one. +The CUDA build of the fork for the resident preset compiles, build only, with no warning in a +wired part (v, [`log/2026-10-18-cuda-build-only.md`](log/2026-10-18-cuda-build-only.md)); the +owner's supervised run follows +[`GPU_SUPERVISED_RUN_CHECKLIST_202610.md`](GPU_SUPERVISED_RUN_CHECKLIST_202610.md). **Real kind.** The interface is `REAL(C_FLOAT)`; "a WW3 built with an 8-byte default REAL needs a different ABI" (v, `w3kokkosmd.F90:30-31`). @@ -214,9 +245,9 @@ What a WW3 build needs today: |---|---|---| | A separately built, installed `ww_kokkos` | `find_package(ww_kokkos 0.9 REQUIRED)` (v, `WW3 model/src/CMakeLists.txt:43`) | Upstream has no place to fetch it from | | A static archive only | `add_library(ww_kokkos STATIC)` (v, `kokkos/CMakeLists.txt:52`) | No shared build for a coupled system that links WW3 as a library | -| GoogleTest, NetCDF C and Fortran, a Fortran compiler to configure the library | `find_package(GTest REQUIRED)`, `pkg_check_modules(NETCDF ...)`, `NETCDFF` (v, `kokkos/CMakeLists.txt:12`, `:33-37`) | The install needs the test and tool dependencies | +| GoogleTest, NetCDF C and Fortran, a Fortran compiler to configure the library | `find_package(GTest REQUIRED)`, `pkg_check_modules(NETCDF ...)`, `NETCDFF` under `WW_KOKKOS_BUILD_TESTS` (default ON), and C and Fortran enabled only then (v, `kokkos/CMakeLists.txt`) | Closed (U6): `-DWW_KOKKOS_BUILD_TESTS=OFF` configures with C++ only and searches Kokkos only (see below) | | C++20, CMake 3.25 | v, `kokkos/CMakeLists.txt:11`, `:14` | WW3 asks for CMake 3.19 (v, `WW3/CMakeLists.txt:4`) | -| Kokkos, unversioned | `find_package(Kokkos REQUIRED)` and `find_dependency(Kokkos)` (v, `kokkos/CMakeLists.txt:26`, `:83-85`) | WW3's configure may find a Kokkos other than the one the archive was built with; only `WW_KOKKOS_ARCH`, optional, checks the architecture (v, `:27-32`) | +| Kokkos, pinned by the package | `kokkos/cmake/ww_kokkosConfig.cmake.in`: `find_dependency(Kokkos EXACT)`, then `Kokkos_DEVICES` and `Kokkos_ARCH` compared with the values recorded at build time (v) | Closed (U6): another Kokkos is a configure error of WW3's `find_package(ww_kokkos 0.9 REQUIRED)`, unchanged in the fork | | A version contract | `SameMajorVersion` at 0.9.0, entry points only added (v, `kokkos/CMakeLists.txt:86-90`) | Sound for a 0.x series, but no symbol check enforces "only added" ⚠ | | Contraction off in the kernels | `-ffp-contract=off`, `--fmad=false`, `PRIVATE` to `ww_kokkos` (v, `kokkos/src/ww_kokkos/CMakeLists.txt:297-303`) | Correct where it is; WW3's side does not check it was built this way | | `ww3_lib` links `ww_kokkos::ww_kokkos` `PUBLIC` | v, `WW3 model/src/CMakeLists.txt:44` | WW3's exported targets (v, `:285-299`) then name a package a consumer must also find ⚠ (no consumer built) | @@ -226,9 +257,19 @@ What a WW3 build needs today: - **(a) Major.** NCEP can build it by hand; it cannot pin it as one more library in its stack without the items above. - **(b) Blocker.** An upstream optional dependency must configure with its own dependencies only. -- **Next step.** An option (`WW_BUILD_TESTS`, default on here) that skips GoogleTest, NetCDF, - the tools and the Fortran language for an install-only configure; write `Kokkos_VERSION` and - the enabled backends into `ww_kokkosConfig.cmake` and check them in WW3's `find_package`. +- **State (U6, 2026-10-18).** `WW_KOKKOS_BUILD_TESTS=OFF` is the install-only configure + (`kokkos/README.md`, "Install-only build"). Traced with `cmake --trace-expand` in the `#ww3` + shell, it ran `find_package(Kokkos REQUIRED)` and, inside Kokkos' config, + `find_package(OpenMP REQUIRED COMPONENTS CXX)`, and no other `find_package` or + `pkg_check_modules`; `Kokkos_DIR` is the only package directory in its cache (v). The + installed config records Kokkos 5.2.2, `OPENMP;SERIAL`, no architecture (v). WW3 built + against that prefix with `WW3GPU_KOKKOS_PREFIX= scripts/ww3-gpu.sh --ww3-gpu smoke + --cpu` (v, 2026-10-18): WW3's cache names that prefix as `ww_kokkos_DIR`; the Kokkos and the + Fortran runs of the same executable wrote byte-identical files (`out_grd.ww3`, `out_pnt.ww3`, + `restart.ww3`, the NetCDF), and against `--ww3` every binary model file was byte-identical and + `nccmp-tol` found 0 difference in every NetCDF field. No fork change was needed. +- **Next step.** A shared build of `ww_kokkos` for a coupled system, and the CMake and C++ + minimums against what an operational centre's module system provides ⚠. ## 6. Failure behaviour @@ -252,6 +293,15 @@ asked for Kokkos and got the Fortran learns it only from one log line or a `*_CO - **(b) Minor.** The default is the Fortran anyway. - **Next step.** `WW_KOKKOS_STRICT=1`: `KOKKOS_PART` and the `SRCE` gate stop instead of falling back, and `W3KOKKOS_SRCE_ON` counts its Fortran calls for one line at the end of the run. +- **Done (2026-10-18), not yet at the pin.** Fork branch `ww3-gpu/u2-strict-preset` (`f0cbe70d`): + `KOKKOS_PART` under `WW_KOKKOS_ALL=1`, the two `W3SRCE` set-up fallbacks and a new per-step gate + `W3KOKKOS_SRCE_STEP` call `KOKKOS_FALLBACK`, which stops with `EXTCDE` and a `NDSE` line + naming the part and the reason under `WW_KOKKOS_STRICT=1`, and otherwise counts; the shim + prints `ww_kokkos: calls left on the Fortran (WW_KOKKOS_STRICT=1 stops instead): SRCE= ...` + at exit on every process, as the `*_COUNT` lines do. The flag is read once; the per-point + `W3KOKKOS_SRCE_ON` is unchanged. The regtest matrix's Kokkos run sets `WW_KOKKOS_STRICT=1` + (`regtests/cmake/ww3gpu_common.cmake`, `ww3gpu_kokkos_env`). `PATCH.md` section 24 has the + runs that show it. ## 7. Configuration surface @@ -269,6 +319,14 @@ script (v, `scripts/ww3-gpu.sh:190`), so a production job card would repeat it b ⚠ (convention, not a rule read today). - **Next step.** A `&KOKKOS_NML` group read in `W3INIT` that sets the same flags, the environment kept as override for tests, and the effective set written once to the log. +- **Done in its smallest form (2026-10-18), not yet at the pin.** `WW_KOKKOS_PRESET=resident`, + read by `W3INIT` through the shim (fork `237ca0de` on `ww3-gpu/u2-strict-preset`): the part + list is one line of `kokkos/src/fortran_iface/presets.txt`, which CMake compiles into + `part_flags.cpp` and `scripts/ww3-gpu.sh --kokkos resident` and the matrix's + `WW3GPU_KOKKOS=resident` read, so they cannot drift. A set `WW_KOKKOS_` still wins, an + unknown name stops the model, and the parts line stays the effective set, with the preset's + name on the line before it. A job card then carries one variable instead of six. The + namelist group remains the next step. `PATCH.md` section 25. ## 8. Observability @@ -324,6 +382,14 @@ the UFS case (`work_c`, then `work_d` from its restart, compared by `test.comp`; - **(b) Minor.** - **Next step.** That pair under `WW3GPU_LAB_SWITCH=ON` with the per-point parts and `SRCE`. +**Update, 2026-10-18 (plan 002, U3).** `ww3_ufs1.1` is an `MPI` case, as is every other +`rstrt_b4b` pair of `matrix.base` (`ww3_tp2.17` with `PDLIB`; v, `matrix.base:2048-2067`, +`:2171-2205` at `50a10730`), and the resident preset stops under `MPI` (ADR-0017). No entry +restarts a `SHRD` case. [`scripts/restart_pair.sh`](../scripts/restart_pair.sh) is the smallest +equivalent on `full` with `switch_lab_shrd`: a 00:00 to 12:00 run and a 06:00 to 12:00 run from +its own 06:00 restart, for the Fortran and the resident Kokkos of one executable, compared with +`cmp`. It was not run ⚠ ([log](log/2026-10-18-u3-regtests-v2.md)); the gap stays Major for (a). + ## 11. `PDLIB` and unstructured grids 29 rows of `kokkos/ports.tsv` translate `PDLIB` routines (`pdlib_w3profsmd`, `w3parall`); none @@ -342,16 +408,22 @@ batched `W3SRCE` is compiled only `!defined(W3_PDLIB)` (v, `w3wavemd.F90:452`, ` `diff -r` of the fork against `WW3-noaa/` (v, today): six new modules, 4 082 lines (`w3kokkos*md.F90`), and 114, 456 and 249 changed lines in `w3initmd.F90`, `w3srcemd.F90`, `w3wavemd.F90`, plus a `kokkos` category in `switches.json` and the CMake block of section 5. The -Fortran is LGPL-3.0-or-later (v, `w3kokkosmd.F90:33`); the C++ it calls is MIT (v, the 12 -`SPDX` lines of `kokkos/src/fortran_iface/*.cpp`) and lives here, not in WW3. The `#ifdef +Fortran is LGPL-3.0-or-later (v, `w3kokkosmd.F90:33`). The C++ it calls lives here, not in WW3: +58 218 lines in 347 files under `kokkos/src/`, of which 249 files carry `LGPL-3.0-or-later` (the +kernels translated from WW3, all in `ww_kokkos/`), 22 carry `MIT` (the shims, timers, fixture +reader) and 76 carry no line (v, `grep SPDX-License-Identifier`, 2026-10-18). An earlier version +of this section called the C++ MIT; the count corrects it. The `#ifdef W3_KOKKOS` hunks inside `W3WAVE`'s time loop are what a reviewer reads; with the `KOKKOS` switch off they compile to the original. - **(a) Minor.** Operations can run the fork. - **(b) Blocker.** Upstream needs to know where the C++ comes from, who maintains it, and how its version is pinned. -- **Next step.** An ADR: `ww_kokkos` as an external package WW3 finds, as a submodule of WW3, or - inside `model/src`; the licences permit each ⚠ (no legal reading). +- **State (U6, 2026-10-18).** [ADR-0018](ADRs/ADR-0018-where-ww-kokkos-lives-for-upstream.md), + Proposed, sets out the three options (an external package WW3 finds, a submodule of WW3, + sources inside `model/src`) with the size and licence of each; only the first was built. +- **Next step.** The owner's choice among them, after a legal reading of the licence of the + translated C++ and of the files with no licence line ⚠. ## 13. A stale remark in the contract @@ -360,6 +432,10 @@ grid's tables". The shim keeps one context per grid index in a map (v, `snl1_shi `W3INIT` calls `WW_SNL1_INIT` with `IMOD` (v, `w3initmd.F90:764`). The remark describes phase 1 and misleads a reviewer of (b). Remark b of the same block, on concurrent calls, is still true. +**Update, 2026-10-18 (plan 002, U7).** Remark a now says one context per grid, keyed by the grid +index, with `W3INIT` passing `IMOD`; remark b is kept (v, fork commit `f330dc53` on +`ww3-gpu/u1-multi-va`, `w3kokkosmd.F90:99-104`). Closed once the pin moves. + ## Ranked top five Ranked by what blocks both targets first, then by how much of GFS-Wave's run it decides. @@ -371,7 +447,9 @@ Ranked by what blocks both targets first, then by how much of GFS-Wave's run it 2. **`ww3_multi` writes `VA` behind the resident copy.** `wminiomd.F90:2264`, `:3261-3330` write it between `W3WAVE` calls; `W3WAVE` returns with "both equal" (`w3wavemd.F90:3153`, `resident_shim.cpp:221-240`). GFS-Wave is a `ww3_multi` mosaic. Next: a host write at - `W3WAVE`'s return, then `mww3_test_0x` with `WW_KOKKOS_VA_CHECK=1`. + `W3WAVE`'s return, then `mww3_test_0x` with `WW_KOKKOS_VA_CHECK=1`. Update, 2026-10-18: + fixed under `SHRD` with column syncs at the exchanges and no read at the return, the + `mww3_test_01` to `_05` entries byte-identical (section 2); `mww3_test_08` (MPI) not run. 3. **The resident preset cannot run under `DIST MPI`.** `XYP3`, `OUTG`, `IOPE`, `UPDV` are `SHRD` only and the preset sets them explicitly, which stops the model (`w3kokkos_prop3md.F90:127-129`, `w3kokkosmd.F90:1197-1201`); the batched `W3SRCE` never ran @@ -381,11 +459,10 @@ Ranked by what blocks both targets first, then by how much of GFS-Wave's run it (`model/src/CMakeLists.txt:113-115`) calls another math library; no GPU run compared the device libm. Next: the static argument for the Intel build's transcendental calls, then one smoke. -5. **The library cannot be installed as a plain dependency.** Its configure needs GoogleTest, - NetCDF and Fortran, C++20 and CMake 3.25, and records no Kokkos version - (`kokkos/CMakeLists.txt:11-37`, `:83-85`); the C++ lives outside WW3 under another licence. - Next: an install-only option, the Kokkos version in the package config, and an ADR on where - `ww_kokkos` lives. +5. **Where the library lives for upstream is open.** Since U6 it installs with Kokkos as its one + dependency and pins that Kokkos exactly (`kokkos/README.md`, "Install-only build"); it still + needs C++20 and CMake 3.25, and its C++ lives outside WW3, mostly under WW3's licence. + Next: the owner's choice among the options of ADR-0018, after a legal reading. ## How the checks were made diff --git a/docs/compound/plans/2026-10-18-002-production-single-node-plan.md b/docs/compound/plans/2026-10-18-002-production-single-node-plan.md new file mode 100644 index 00000000..243450c1 --- /dev/null +++ b/docs/compound/plans/2026-10-18-002-production-single-node-plan.md @@ -0,0 +1,182 @@ +--- +title: Production-ready single-node Kokkos runtime, CPU, GPU-prepared - Plan +type: feat +date: 2026-10-18 +artifact_contract: ce-unified-plan/v1 +product_contract_source: owner brief, session of 2026-10-18 +execution: code +--- + +# Production-ready single-node Kokkos runtime - Plan + +## Goal Capsule + +- **Objective:** the shared-memory (single-node) Kokkos runtime of the `h0ffmann/WW3` fork gives + the Fortran's bits on every WW3 driver it accepts, says when it falls back, is configured + without the lab script, builds for CUDA and installs as one optional dependency. +- **Means:** seven units (U1 to U7 below, the owner's numbering), fork changes as one commit + series on `ww3-gpu/kokkos-pnt` and one pin bump, WW3's own regtests under + `WW3GPU_LAB_SWITCH=ON` as the only tests (ADR-0010). +- **Authority:** AGENTS.md over this plan. ADR-0016 (this is a validation phase the owner + called, 2026-10-18), ADR-0017 (MPI stays Fortran: no MPI port), ADR-0014 (a GPU run is the + owner's, supervised; agents only `--build-only`). +- **Stop conditions:** a `test.comp` difference that a Fortran/Fortran run does not reproduce + stops the unit and is reported; no tolerance is widened, no entry is dropped or marked + expected-to-fail, no `--gpu` run. +- **Execution profile:** large loop; branch `goal/abbb0456-6043-production-single-node` from + `origin/main` (`5b48e08`); independent units as sub-agents in worktrees, their commits + cherry-picked onto the branch; one PR through `just pr`. + +--- + +## Brainstorm outcome + +Decisions taken before planning, each with the reason; the owner can reverse any of them in +review. + +- **B1. The `ww3_multi` fix marks the host newer only where `ww3_multi` writes `VA`** (the + owner, 2026-10-18: the fix must not cost speed). `W3WAVE` keeps its exit read + (`w3wavemd.F90:3151-3153` at `80644ba9`); the host writes of `WMIOHG` (`wminiomd.F90:2264`) + and `WMIOEG` (`:3261`, `:3278`, `:3326`, `:3330`) are each followed by + `W3KOKKOS_VA_HOST_WRITE` behind `W3_KOKKOS`. `ww3_shel` pays no copy, `ww3_multi` pays one + only on a step that writes. Rejected: marking the host newer at `W3WAVE`'s exit, one + host-to-device copy of `VA` per grid per step, which on a GPU is a bus transfer the resident + design exists to avoid. The cost is one more WW3 file in the fork diff (gap 12). A partial + update of the columns written is a further option, taken only if small and clearly correct. +- **B2. Strict mode is one environment flag, `WW_KOKKOS_STRICT=1`**, read where the parts are + resolved: `KOKKOS_PART` under `WW_KOKKOS_ALL=1` and the per-call `W3SRCE` gate stop with + `EXTCDE` and a `NDSE` line naming the part and the reason, instead of falling back. Without + it, behaviour is today's, plus one line at the end of the run with the per-part count of + Fortran fallback calls. The matrix's Kokkos run sets `WW_KOKKOS_STRICT=1`; its Fortran run + does not. +- **B3. Configuration, smallest useful form: `WW_KOKKOS_PRESET=`**, read by `W3INIT` + through the shim, with `resident` the only preset (its part list moved from + `scripts/ww3-gpu.sh:190` into one table in `kokkos/src/fortran_iface/`, which the script then + reads, so the two cannot drift). A per-part variable still overrides the preset. The effective + set is the existing parts line, printed once on rank 0. No namelist: the brief says one only if + needed, and a `&KOKKOS_NML` group adds a WW3 input-file format for one user; recorded as the + next step of gap 7. +- **B4. Packaging: `WW_KOKKOS_BUILD_TESTS` (default ON here)**; OFF drops GoogleTest, NetCDF, + the tools and the Fortran language from the configure. `ww_kokkosConfig.cmake` records + `Kokkos_VERSION` and the enabled backends, and `find_dependency(Kokkos EXACT)`. + WW3's `find_package(ww_kokkos 0.9 REQUIRED)` is unchanged; the check lives in the package. +- **B5. The ADR on where `ww_kokkos` lives** is drafted Proposed with three options (external + package WW3 finds; submodule of WW3; inside `model/src`), no recommendation beyond what the + evidence of gap 12 supports, and the licence question marked ⚠. +- **B6. GPU readiness is a build and a checklist.** `--build-only` of the fork on the CUDA + backend with the resident preset's parts, warnings read and listed; the checklist is a file the + owner follows during a supervised run. + +## Requirements + +- R1. A SHRD `ww3_multi` entry under the lab switch with the resident preset and + `WW_KOKKOS_VA_CHECK=1` passes `test.comp` against the Fortran, before and after the fix is + shown (the reproduction fails first, or is shown not to fail with the reason). +- R2. With `WW_KOKKOS_STRICT=1` no run can complete on the Fortran for a part it asked on + Kokkos; the matrix's Kokkos runs use it. +- R3. The 7 resident serial entries, the OMPG entries with the part subset, and one restart + pair pass under the lab switch, each recorded in `docs/data/` and `docs/log/`. +- R4. `WW_KOKKOS_PRESET=resident` alone gives the same parts line as `--kokkos resident`. +- R5. The fork and `ww_kokkos` build on Kokkos' CUDA backend, `--build-only`; a checklist for + the owner's supervised run exists. +- R6. `ww_kokkos` configures and installs with Kokkos as its one dependency; the ADR is Proposed. +- R7. The stale remark of gap 13 is corrected. + +## Scope boundaries + +- Out: MPI (ADR-0017), `PDLIB`, any GPU run, any timing in a table (the owner's, median of + three on a quiet machine), the profile's speed proposals (each its own PR with L2 evidence). +- A change to `regtests/` or to how a comparison is invoked (U2) is reviewed by a human before + merge; the PR body says so. + +## Implementation units + +Fork work: each unit that changes the fork works on its own fork branch from `80644ba9` +(`ww3-gpu/u1-multi-va`, `ww3-gpu/u2-strict-preset`, `ww3-gpu/u6-package`), and the integrator +merges them, in that order, into `ww3-gpu/kokkos-pnt`, then bumps the pin once. Every fork change +gets a section in `kokkos/src/fortran_iface/PATCH.md`. + +### U1 + U7. `ww3_multi` and the resident `VA`; the stale remark (agent A) + +- **Files:** fork `model/src/w3wavemd.F90`, `w3kokkosmd.F90:99-104`; `PATCH.md`; possibly + `kokkos/src/fortran_iface/resident_shim.cpp`. +- **Approach:** reproduce first: the `mww3_test_0x` entries that are SHRD with + `WW3GPU_LAB_SWITCH=ON`, the resident preset, `WW_KOKKOS_VA_CHECK=1`; and a part subset + where `XYP3` refuses (`SRCE,KTP3,OUTG,IOPE,UPDV`). Then B1, then the same entries + byte-identical. Say whether this explains `mww3_test_08` (MPI): read its path, do not port MPI. +- **Done:** a `docs/log/` entry with the commands, before and after; the fork commits pushed. + +### U2 + U4. Strict mode and the preset (agent B) + +- **Files:** fork `w3kokkosmd.F90` (`KOKKOS_PART`, `W3KOKKOS_SRCE_ON`), `w3initmd.F90`; + `kokkos/src/fortran_iface/part_flags.cpp`, a preset table; `scripts/ww3-gpu.sh`; + `regtests/cmake/run_entry.cmake` (Kokkos run sets `WW_KOKKOS_STRICT=1`) and + `regtests/cmake/selftest.cmake`; `PATCH.md`; `docs/GLOSSARY.md`. +- **Done:** `just ww3-gpu-matrix-selftest`, `bash scripts/ww3-gpu.sh --self-test`, shellcheck, + `just kokkos-test serial-debug` and `openmp-release`; `just smoke-ww3-gpu` with the preset + from `WW_KOKKOS_PRESET` only, parts line identical to `--kokkos resident`. + +### U3. Wider CPU regtest coverage (agent E, after integration) + +- **Approach:** on the integrated pin, `WW3GPU_LAB_SWITCH=ON`: the resident serial selection; + the OMPG selection with the part subset; the restart-reproduction pair of a SHRD case (gap 10). + Failures go through U4a of `2026-10-18-001-cpu-resident-validation-plan.md`. +- **Done:** summaries in `docs/data/`, one `docs/log/` entry, `kokkos/ports.tsv` `regtested` + only where the record supports it (`port_score.py --write`, `--check`). + +### U5. GPU readiness, build-only (agent D; the final build again after integration) + +- **Files:** `docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md`; the build log summary in `docs/log/`. +- **Done:** `scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --build-only` with the resident preset and + `just kokkos-test cuda-release` built (no run on the device), warnings listed; the checklist + names `scripts/libm_bits.sh --device`, `just smoke-ww3-gpu gpu`, `just full-ww3-gpu gpu`, + `just ww3-gpu-matrix-cuda` on the entries U3 passed, and what to record for each (ADR-0014). + +### U6. Packaging and the ADR (agent C) + +- **Files:** `kokkos/CMakeLists.txt`, `kokkos/cmake/ww_kokkosConfig.cmake.in`, `kokkos/README.md`, + `docs/ADRs/ADR-0018-*.md` (Proposed), the ADR index. +- **Done:** an install-only configure and install in a scratch prefix with only Kokkos found; + the installed config records the Kokkos version and backends; WW3 builds against that prefix + (`just smoke-ww3-gpu` on it); `just kokkos-test serial-debug` still passes. + +### U8. No whole-spectrum copy in the output readers (agent F) + +The owner, 2026-10-18: the resident spectrum is the design, with no copies; where code still +needs one, port what removes it. `W3WAVE` stays Fortran and calls the Kokkos parts, so every +host statement that reads `VA` needs a current host copy. U8 replaces the whole-array reads +before `W3IOTR`, `W3IOBC 'WRITE'` and the OASIS send (`w3wavemd.F90:2934-2975` at `80644ba9`) +with column reads of the points each one reads, as `W3KOKKOS_IOPE_READ` does. The restart +reads stay (the rule allows them). Every run uses `WW_KOKKOS_VA_POISON=1`: `WW_KOKKOS_VA_CHECK` +catches an unsynced host write, not a stale host read. + +### U9. Partitioning on the device (agent G) + +`W3CPRT` reads every point's spectrum at each output time with partitioning on +(`w3wavemd.F90:2691`). U9 translates it per point onto the resident `VA` and copies only the +partition fields back, or stops before code with the obstacle for the owner. + +### U1, extended. The `ww3_multi` exchange on the device (agent A) + +The gather and scatter of the exchanged points of `WMIOBG`, `WMIOHG` and `WMIOEG` run as a +kernel on the resident `VA`, so only those points' spectra cross. The exit read of `W3WAVE` +(`w3wavemd.F90:3151-3153`) is removed. + +### Next, not in this PR: zero copies on a host backend + +On Kokkos' OpenMP backend the device copy of `VA` is a second host array. An unmanaged View over +the Fortran `VA` would remove the copies there entirely ⚠ (the measured copy cost on `full` is +0.002 s, `docs/log/2026-10-18-resident-profile.md`, so this is about memory, not time). + +## Integration and gates (the integrator) + +Merge the fork branches, push `ww3-gpu/kokkos-pnt`, bump `WW3/`, cherry-pick the units' commits, +update `docs/PRODUCTION_READINESS_202610.md` (each gap's state), then the gates of AGENTS.md for +what changed, `security-reviewer` (scripts/ changes), a report-only code review, `just pr`. + +## Owner steps (listed in the PR body) + +- U5 timing: median of three on a quiet machine, then the `bench/results/` record. +- U6: whether the lab-switch evidence lets `kokkos/ports.tsv` rows move to `regtested` "yes". +- Any GPU run of U5. +- Whether to act on the profile's speed proposals. diff --git a/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md new file mode 100644 index 00000000..39474216 --- /dev/null +++ b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md @@ -0,0 +1,328 @@ +--- +title: The first supervised GPU run of the resident preset, and the five plans after it - Plan +type: feat +date: 2026-10-18 +artifact_contract: ce-unified-plan/v1 +product_contract_source: owner, session of 2026-10-18 (goal abbb0456-6043, PR #191) +execution: code +--- + +# The first supervised GPU run, and the five plans after it - Plan + +## Goal Capsule + +- **Objective:** answer, with one measured number, whether WW3 with its time step resident on + the GPU beats the same WW3 resident on the CPU, and how far the GPU's output is from the + Fortran's; then pick the next plan from that answer. +- **Means:** plan 1 below: agent preparation with no device run (phase A), one session the owner + supervises at the terminal (phase B), records and a decision (phase C). +- **Authority:** AGENTS.md over this plan; "Running on the GPU (hard rule)" (no `--gpu` run + without the owner), ADR-0014 (a GPU run is judged by `nccmp-tol`'s tolerances, never widened), + ADR-0017 (single node), "Resident spectrum (hard rule)". +- **Stop conditions:** any `nccmp-tol` failure, any NaN in the GPU output where the Fortran has a + value, a watchdog kill that a smaller batch does not cure, or a run that is not reproducible run + to run: record it and stop that step; never change a tolerance or a gate during the session. +- **Execution profile:** large. Phase A in the next session, on a branch from `main` after PR + #191 merges; phase B when the owner says so; phase C in the same PR as phase A's records or a + follow-up. + +--- + +## Context: NOAA's most common use of WW3 + +GFS-Wave, and GEFS-Wave built the same way, is a `ww3_multi` mosaic of three grids (global 10 +arcmin, Southern Ocean 15 arcmin, Arctic 9 km), 50 x 36 spectral components, ST4 with the DIA, +to 384 h, run with MPI on WCOSS2, whose nodes are AMD EPYC 7742 with no GPU, moving to Google +Cloud by December 2027 (v, `docs/GLOSSARY.md` rows GFS-Wave and WCOSS2, sources read +2026-10-16). + +Against that use, the Kokkos runtime has three gaps after PR #191 is written: + +- `ww3_multi`: the resident path gave wrong numbers between `W3WAVE` calls (gap 2 of + `docs/PRODUCTION_READINESS_202610.md`); U1 of PR #191 fixes it, and its regtests decide. +- MPI: out of scope (ADR-0017). +- GFS-Wave's own switch file was never read. If it has `SEED` or `FLD2`, the Kokkos `W3SRCE` + does not run there (gap 1) ⚠. + +## Could WW3 entirely on the GPU beat ww3-gpu on the CPU? + +On `full` (`bench_small`, 9 520 sea points, 24 x 24 spectral components, 12 h), very likely, by +an order of magnitude ⚠. No WW3 run on a GPU exists yet (`kokkos/PORT_STATUS.md`); this is a +projection from these measurements: + +| Fact | Number | Source | +|---|---|---| +| CPU resident, 32 threads, `full` | 14.36 s wall; `W3SRCE` kernel 12.45 s (87 %); outside Kokkos 0.22 s (1.5 %) | `docs/log/2026-10-18-resident-profile.md`, one run, load 3.5 ⚠ | +| One thread, port against Fortran | 174.78 s against 174.27 s | same log, one run each ⚠ | +| `W3SRCE` kernel, 1 to 32 threads | 12.6x faster (the i9-14900 has 24 cores) | same log | +| DIA kernel alone, 1000 points, OpenMP 32 threads against RTX 4090 | 5.02 ms against 0.047 ms, 107x | `bench/results/snl1-i9-14900-rtx4090-2026-09-15.json`, median of three (v) | +| The same call with the per-call copies of phase 1 | 5.95 ms against 0.75 ms, 7.9x | same record (v) | + +The second row says the CPU gain is all in spreading work over threads: the per-point code costs +what the Fortran costs. The last two rows bracket the GPU: with the spectrum resident (the design +since WFIP-0005, and with no copies after PR #191), the end-to-end ratio moves from 7.9x toward +the kernel's 107x. If the whole step ran 20 to 50 times faster on the GPU, `full` would take 0.5 +to 1 s against 14.4 s ⚠. Three assumptions carry that figure, and plan 1 tests each: + +1. **The whole `W3SRCE` kernel scales like the DIA alone.** It does not, as written: the body on + `main` is `w3srce_team`, `TeamPolicy(n, AUTO)`, 168 registers, most of the work in + `Kokkos::single`, so by ptxas' estimate about three threads per SM do the non-DIA work + (`origin/goal/42-gpu-step-srceteam`, `438ab8b`, `12685ff`, not merged). It also runs with + `--fmad=false` and the device copies of glibc's functions (`WW_KOKKOS_LIBM`), both slower than + CUDA's defaults. +2. **Launches are cheap.** Each step launches several kernels; on 9 520 points their latency may + dominate. Smoke (one point, 2 161 launches) is slower on a GPU by construction. +3. **The host remainder stays small.** 1.5 % on the CPU becomes the floor once kernels shrink: + 0.22 s against a 0.5 s target. + +**"Without any CPU" is not possible.** Set-up, namelists, I/O and the time-loop driver stay on the +host; a GPU is driven from a CPU. The target is the whole time step on the device, the host only +orchestrating and writing outputs: plan 4. + +**More CPU threads do not change the picture.** At 12.6x on 32 threads of a 24-core part, the +i9 is near its ceiling. Beyond one node is MPI (ADR-0017). A 128-core EPYC node may scale further +⚠ (not measured). Inside a node, `W3XYP3`'s threads are idle in 57 % of its samples, which is +1.1 s at most. + +**Correctness costs.** The GPU is not yet bit for bit: a GPU run is judged by tolerance +(ADR-0014) until `scripts/libm_bits.sh --device` shows the device copies of glibc's functions +give glibc's bits, and the host C++ of the CUDA build is compiled by g++ 14.4.0, not the 16.2.0 of +every parity argument so far (`docs/log/2026-10-18-cuda-build-only.md`) ⚠. + +## The five plans, ranked + +| # | Plan | Attacks | Why this rank | Owner decision needed | +|---|---|---|---|---| +| 1 | The first supervised GPU run (this file, below) | the unknown: GPU against CPU, and the GPU's distance from the Fortran | cheapest, most information; decides whether 2 and 4 aim at the CPU or the GPU | the run itself | +| 2 | Several sea points per vector lane in `W3SRCE` | DIA, `W3SDS4`, `W3SIN4`, the integration: 12.0 of 14.4 s on the CPU | the only CPU plan that attacks the per-point cost, now all of it; on a GPU the same layout gives coalesced loads | yes: it changes loop nesting (AGENTS.md, "Porting"); its own PR with the per-lane parity argument | +| 3 | GFS-Wave's own configuration | correctness on what NOAA runs | without it every speed-up is shown on a configuration NOAA does not run | which tag to read | +| 4 | The whole time step resident on the GPU | the host remainder, launch count, readers left after PR #191 | what makes plan 1's number hold on cases with more output than `full` | lane Q's merge | +| 5 | Thread efficiency in one node, and a many-core baseline | `W3XYP3` imbalance, pinning, scaling on a quiet machine and a WCOSS2-like or cloud EPYC node | the honest CPU baseline a GPU result must beat for NOAA's move to the cloud | access to such a node | + +Not ranked: skipping `W3SDS4`'s zero-weight terms (12 of 23 products, profile log) changes the +result for an infinite or NaN input, so it is the owner's decision and its own PR. + +**Work in progress elsewhere (2026-10-18).** PR #193 (branch +`goal/eb10363f-4821-noaa-progress-record`, another session, merged 2026-10-18) builds the `W3SDS4` saturation-integral loop +interchange (`kokkos/src/ww_kokkos/st4_sds4.hpp`), a DIA direction-halo layout that turns the +index-table gathers into contiguous loads (`snl1_team.hpp`, `snl1_tables.{hpp,cpp}`), and +`-fno-math-errno` for the host build (`kokkos/src/ww_kokkos/CMakeLists.txt`). The halo layout +attacks the same gathers as plan 2 by another route. That session reports the DIA kernel alone +(`ww_bench_snl1`, medians of seven) 1.93x faster at one thread and 1.19x at 32 with the halo, and +the `W3SDS4` interchange about 7 % faster unless `SATWEIGHTS` and `SATINDICES` are transposed; +end to end not measured ⚠ (PR #193's own record is the source). Plan 2 starts from that +result, not before it. Its timing and the GPU run of plan 1 need a resident regtest pass on the +tree that holds both PRs. + +### Plan 2 in short + +A team of eight points, one per AVX2 lane, each lane its own point's arithmetic in the +Fortran's order, spectra point-minor in team scratch (profile log, proposal 1). No sum crosses a +lane, so the parity argument is the per-lane GIMPLE. Covers ranks 1 to 4 of the profile if built +as a layer for the whole `W3SRCE` team body. Trap: the five libmvec sites of ST4 +(`st4_libmvec.hpp`) keep the Fortran's groups of four directions per call, so those loops stay. +Gain ⚠: the profile does not separate gathers from arithmetic. + +### Plan 3 in short + +Read the switch of the GFS-Wave release tag and record it in gap 1; if it has `SEED` or `FLD2`, +translate their `W3SRCE` branches (ADR-0002 order permitting); build a three-grid `ww3_multi` +regtest case with that switch from WW3's inputs, and add it to the matrix under +`WW3GPU_LAB_SWITCH=OFF`. + +### Plan 4 in short + +After PR #191 (readers U8, U9, the `ww3_multi` exchange of U1): review and land lane Q +(`w3srce_split`, 96 threads per point; three surviving mutants in `12685ff`'s message to resolve +first); move the forcing updates left on the host; cut launches per step (merge the small +kernels, or CUDA graphs if Kokkos' graph API covers them ⚠); count with `WW_KOKKOS_VA_COUNT=1` +and the timers. + +### Plan 5 in short + +`W3XYP3` component scheduling (dynamic over components, or a team size from the component +count), `OMP_PROC_BIND`/`OMP_PLACES` on a quiet machine, scaling at 8, 16, 24, 32 threads, median +of three; one run on a many-core EPYC node if the owner has one. + +--- + +## Plan 1 in detail: the first supervised GPU run + +### What the run must answer + +- Q1. Wall time of `ww3_shel`, `full`, `--kokkos resident`, on the RTX 4090 against the same + preset on the CPU (32 threads) and against `--ww3`, median of three, same commit. +- Q2. Per-field distance of the GPU output from the Fortran of the same executable and from + `--ww3` (`max|d|`, rms, largest relative difference), and whether it passes + `kokkos/tools/nccmp-tol/tolerances.txt` unchanged. +- Q3. Where the GPU wall time goes (timers): `W3SRCE`, propagation, outputs, host remainder, + copies. +- Q4. Whether the device copies of glibc's functions give glibc's bits (`libm_bits --device`). +- Q5. Whether the run is reproducible run to run on the GPU. + +### Facts the plan is built on (each checked in a file, 2026-10-18) + +- `scripts/ww3-gpu.sh --gpu` asks for a typed "yes" at a terminal (`:359-370` at `fe3cbc8`), exit + 7 without one, exit 4 without a driver; it then re-executes inside + `nix develop picasso/labs/pratico#cuda`. `--build-only` skips the prompt and is the agents'. +- `--gpu` alone means the `gpu` preset, `SRCE` only (`:296-298`); `resident` is + `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV` (`:191-197`). `just smoke-ww3-gpu gpu`, `just full-ww3-gpu gpu` + and `just perf-ww3-gpu full 3` run `gpu`, not `resident`; `perf-ww3-gpu` takes the parts as + its third argument (`justfile:215`). +- `--timers` fences the device (a breakdown, not a timing) and cannot combine with `--repeat`. + `--repeat N` alternates `--ww3` and Kokkos runs and requires every Kokkos NetCDF to equal run + 1's (`:709-721`). +- Comparison: `nccmp-tol` on the first `ww3.*.nc` only (`:673-677`), against the Fortran of the + same executable then against `--ww3` (`:695`); `cmp` of every model file is reported, not fatal, + on CUDA (`:682`, `:698`). `full` writes only `HS` (`bench_small/ww3_ounf.nml:10`); `coast` + writes every `OUTG` field. +- `nccmp-tol` counts a NaN or fill in the test where the reference has a value as "dropped", says + so on `stderr` and still passes the variable (`kokkos/tools/nccmp-tol/nccmp_tol.cpp:52-56`). + `compare.cpp` has no 0/360 degree wrap for `dir` and `dp`. +- `counts_resident.txt` expects `W3SRCE batch inputs unchanged, not copied: 422` on `coast`. At + the device's 128 MiB batch cap `coast` takes two launches per step, so no shadow is kept and + the line differs: the script exits 1 before `nccmp-tol`. With `WW_KOKKOS_SRCE_BATCH_MIB` set the + check is skipped (`:651-662`). +- The 4090 drives the display; the driver's watchdog ends a long kernel + (`cudaErrorLaunchTimeout`); the batch cap's estimate assumes the GPU runs at least a sixth of the + CPU's speed (`kokkos/src/fortran_iface/PATCH.md` section 19). +- `just ww3-gpu-matrix-cuda` forwards `WW3GPU_ROUTINES` only, not `WW3GPU_KOKKOS` or + `WW3GPU_LAB_SWITCH`, and `WW3GPU_STRICT` is on, so every entry fails on any byte difference + (`justfile:143-159`). +- `results.py record` needs `--repeat` of at least 3, writes the ADR-0014 wording for CUDA, and + refuses a second study `ww3-vs-ww3gpu--cuda-` on the same day (`:121-171`); it keeps + no per-field statistics. +- `libm_bits.sh --device` asks its own "yes", builds `device_check.cu` (`--fmad=false`, sm_89), + checks 15 functions on 2^32 inputs each with a single-threaded host reference + (`device_check.cu:146-168`): hours for the full set ⚠ (not timed); `--device expf` checks one. +- `WW_KOKKOS_LIBM`'s tables are generated from the build host's libmvec, so build and run happen + on the same workstation (`docs/TOOLCHAIN.md`). +- The CUDA builds in the main checkout are from 2026-10-16, before `WW_KOKKOS_LIBM`, and on the + `SHRD` switch: they rebuild from scratch. The toolchain stamp includes `CMAKE_PREFIX_PATH`, so + `build/ww3` also rebuilds inside `#cuda` and again on the next CPU run (`:480-487`). +- `run_dir`s are deleted at every start (`build/runs/ww3-gpu--cuda/`); `WW3GPU_RUNS` + moves them. +- Docker's CUDA image is another toolchain (CUDA 13.4, no OpenMP, counts check skipped) and, since + PR #192, builds with `WW_KOKKOS_LIBM=OFF` (`docker/Dockerfile.cuda`), so its kernels call CUDA's + libm, not the device copies of glibc's functions: not used for this run. Which value each kind of + build should use is the owner's decision in + [`KOKKOS_LIBM_ON_VS_OFF.md`](../../KOKKOS_LIBM_ON_VS_OFF.md) (PR #194); A3 records the value the + run's build has (`WW_KOKKOS_LIBM` in `CMakeCache.txt`). + +### Phase A: preparation by agents (next session, no device run) + +Each unit ends with its gates and a commit; nothing in phase A types "yes" or launches a kernel. + +**A1. Freeze the commit.** A branch `goal/-gpu-first-run` from `main` after PR #191 merged. +The commit that phase B runs is its tip at the end of phase A, written into the run card (A7). +No other session works in that checkout during phase B: a separate worktree, `WW3GPU_RUNS` under +it. + +**A2. Make the tools able to judge this run.** Each change makes a comparison stricter or lets +it run the configuration asked; each is a change to a comparison or its invocation, so a person +reviews it before merge (AGENTS.md, "Never cheat a regtest tool"), and it merges before phase B. + +- A2a. `nccmp-tol`: a NaN or fill in the test where the reference has a finite value fails the + variable (today: passes with a note). Stricter, never looser. +- A2b. `nccmp-tol`: judge every `ww3.*.nc` the case writes, not the first only. +- A2c. `counts_resident.txt` on a device: the expected `not copied` count follows the number of + launches per step (or the line is checked on the host backend only, and the device run prints + its count for the record). Decide with the owner which; the CPU check stays as it is. +- A2d. `ww3-gpu-matrix-cuda` forwards `WW3GPU_KOKKOS` and `WW3GPU_LAB_SWITCH`, and a GPU entry's + verdict comes from `nccmp_tol`/`judged` with `test.comp` recorded (ADR-0014), selftest fixtures + for both. +- A2e. Ask, do not do: a 0/360 wrap for `dir`/`dp` in `compare.cpp`. A wrap changes what passes + near north, so it is the owner's decision; until then, `coast`'s direction fields are read by + hand when they fail near 0 or 360. +- A2f. `smoke-ww3-gpu`, `full-ww3-gpu`: accept a parts argument as `coast-ww3-gpu` does, so the + run card can use `just` and the preset is visible in the command. If not, the card uses + `scripts/ww3-gpu.sh` directly. + +**A3. Build everything the session will run, on the frozen commit, on the workstation.** +`JOBS=8 bash scripts/ww3-gpu.sh --ww3-gpu full --gpu --kokkos resident --build-only`, the same for +`smoke` and `coast` if their builds differ, and `scripts/libm_bits.sh --nvcc` (build only, if the +script separates it; else record that its build happens in the session). Then build `--ww3` in +the `#cuda` shell so the toolchain stamp matches (`--ww3 full --build-only` if supported ⚠). +Record nvcc, Kokkos, g++ host, gfortran, driver, and the `CMakeCache.txt` lines for +`WW_KOKKOS_LIBM`, `CMAKE_CUDA_ARCHITECTURES`, contraction flags. Read the warnings against +`docs/log/2026-10-18-cuda-build-only.md`'s list; a new warning in a wired part stops phase B. + +**A3b. The host compiler of the CUDA build.** gfortran is 16.2.0 in both builds, but +`nvcc_wrapper` compiles the host half of every `.cpp` with g++ 14.4.0, against 16.2.0 for the CPU +build and every static parity argument (`docs/log/2026-10-18-cuda-build-only.md:25-26`, +`:74-77`). Before phase B, one of two things: build the library on the OpenMP backend with g++ +14.4.0 and run `smoke` and `full` resident pair to pair with the Fortran (byte-identical, or a +defect to fix before the GPU run); or compare the `-O3` GIMPLE of the host-side shim paths under +both compilers. Record why the `#cuda` shell pins g++ 14.4 (presumably the newest host GCC nvcc +12.9 accepts ⚠, from NVIDIA's support matrix). Without it, a GPU difference cannot be split into +the device's share and the host compile's. + +**A4. The same commands on the CPU first.** On the frozen commit: +`bash scripts/ww3-gpu.sh --ww3-gpu {smoke,full,coast} --cpu --kokkos resident`, each byte-identical +to the Fortran and to `--ww3` (NetCDF attribute aside); `WW_KOKKOS_VA_COUNT=1` and +`WW_KOKKOS_VA_POISON=1` on `full` and `coast`. These are phase B's reference numbers and prove +the commit is sound before the GPU is involved. + +**A5. Static readiness of the device kernels.** ptxas `--resource-usage` (registers, spills, +shared memory) of every kernel the preset launches, from A3's build; occupancy per kernel from +the 4090's limits; the expected launches per step on `full`. From them, the watchdog risk of +`W3SRCE` at 128 MiB and the batch size for the first run (32 MiB, as the handover proposed, +unless A5 says otherwise). Written into the run card; nothing tuned. + +**A6. Lane Q stays out of the first run.** The first run measures `main`'s `w3srce_team`, so +its number is the baseline lane Q must beat (plan 4). A6 records lane Q's state and its three +surviving mutants, nothing more. + +**A7. The run card.** Update `docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md` into one ordered card +with the exact commands of phase B (no `just` recipe whose default preset is not `resident`), the +frozen commit, the expected CPU numbers from A4, the per-field template for +`docs/data/ww3gpu_cuda_{smoke,full,coast}_.md`, the stop rule of each step, and the +`script -a` capture of the whole session. + +**A8. Phase A's PR.** Through `just pr`, the A2 changes flagged for human review in the body; +phase B starts after it merges. + +### Phase B: the owner's session (in order; the owner types every "yes") + +| Step | Command (from the run card) | Records | Stop rule | +|---|---|---|---| +| B0 | quiet machine: load below 1, no other session, `nvidia-smi -q` snapshot, `nvidia-smi --query-gpu=display_active --format=csv`, `script -a ` | machine block of the card | load above 1: wait | +| B1 | `scripts/libm_bits.sh --device expf` | TSV line, exit code | exit 2: CUDA error, stop and diagnose | +| B2 | `WW_KOKKOS_SRCE_BATCH_MIB=32 bash scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident` | parts line all `=K`, `nccmp-tol` both references, every field's statistics; speed not read | any failure: stop, record | +| B3 | `WW_KOKKOS_SRCE_BATCH_MIB=32 bash scripts/ww3-gpu.sh --ww3-gpu full --gpu --kokkos resident --timers` | timers breakdown (Q3), `nccmp-tol` | `cudaErrorLaunchTimeout`: halve the batch once; again: stop | +| B4 | B3 without `--timers`, twice, with `WW3GPU_RUNS` apart; `cmp` the two Kokkos outputs | Q5 | not identical: record, skip B5's `--repeat` | +| B5 | `just perf-ww3-gpu full 3 resident` (with B3's batch variable if B3 needed it) | median of three (Q1); `results.py record` | `nccmp-tol` failure: no record | +| B6 | `WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1` on B4's command | NaN check, bytes moved (Resident spectrum rule) | a NaN in the output: a missed sync, stop | +| B7 | `coast` as B2, then without the batch variable once A2c is merged | every `OUTG` field, directions by hand near 0/360 | as B2 | +| B8 | `just ww3-gpu-matrix-cuda` on the entries U3 of PR #191 passed, if A2d merged | per-entry `nccmp_tol`, `test.comp`, `judged` | as the matrix reports | +| B9 | `scripts/libm_bits.sh --device` for all 15 functions, left running at the end | the full TSV (Q4) | none: a record | + +### Phase C: records and the decision + +- `docs/data/ww3gpu_cuda_{smoke,full,coast}_.md` (per-field statistics), + `bench/results/` from `results.py record` (`just results check`), a `docs/log/` entry with the + session's log and every stop rule that fired, `docs/GPU_LIBM_BITS_202610.md` with B1 and B9. +- `kokkos/PORT_STATUS.md` "On a GPU" and `port_score.py --write`; WFIP-0005 unit 6's done + criterion corrected to the resident preset (`just perf-ww3-gpu full 3 resident`); ADR-0014's + revisit after the first run, as it asks. +- The decision on plans 2 and 4, from B5's median against A4's CPU median, both on the same + commit: + +| GPU / CPU resident, `full` | What it says | Next | +|---|---|---| +| at least 5x | the device step is the faster design here | plan 4 (lane Q, launches, host remainder), then plan 2 for the GPU layout | +| 2x to 5x | the GPU wins but something bounds it | read B3's timers: `W3SRCE` dominant means lane Q first; launches or host means plan 4's launch work | +| below 2x | the GPU does not pay on this case yet | plan 2 on the CPU, and a larger case for the GPU before any further GPU work ⚠ | + +Whatever the ratio, the GPU number enters a table only with B5's `nccmp-tol` pass (ADR-0014) and +says "regtest pending" until WW3's regtests passed on its commit. + +## Risks the plan cannot remove + +- The 4090 is a desktop part with a display; an H100 has no watchdog and other limits, and + `cuda-h100-release` needs a HOPPER90 Kokkos picasso does not build. The ratio does not transfer + to an H100 ⚠. +- `full` is a 9 520-point regional case; GFS-Wave's global grid is larger, which favours the GPU, + and writes more output per step, which does not ⚠. +- g++ 14.4 for the CUDA build's host code: a host branch of the CUDA build may round differently + from the CPU build; A4 on the CPU build does not cover it ⚠. diff --git a/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md b/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md new file mode 100644 index 00000000..0d4ed186 --- /dev/null +++ b/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md @@ -0,0 +1,87 @@ +--- +title: Pin WW3 to NOAA's production/GFS.v17 branch - Plan +type: feat +date: 2026-10-18 +artifact_contract: ce-unified-plan/v1 +product_contract_source: "owner, session of 2026-10-18 (goal abbb0456-6043), asked for a stable WW3 pin; GFS.v17 in its own PR after #191" +execution: code +--- + +# Pin WW3 to NOAA's `production/GFS.v17` - Plan + +## Goal Capsule + +- **Objective:** `WW3-noaa/` and the fork's KOKKOS patch sit on the WW3 that NOAA's next + operational GFS runs, so every parity result, timing and regtest of ww3-gpu is about the code + GFS-Wave will use. +- **Authority:** AGENTS.md ("WW3 source is read-only", "Bit for bit", "Translate, don't + improve"); the owner chose `production/GFS.v17` on 2026-10-18, in its own PR after #191. +- **Stop conditions:** a translated routine whose Fortran changed between the two bases and whose + translation no longer matches it; a regtest difference that a Fortran/Fortran run does not + reproduce. Either stops the pin move; it is never solved by keeping old Fortran. +- **Execution profile:** large; its own branch from `main` after PR #191 merges. + +## Why GFS.v17 and not a release tag + +Facts read 2026-10-18 (v, `git ls-remote --tags` and `gh release list -R NOAA-EMC/WW3`, +`git rev-list` in `WW3-noaa/`): + +| Candidate | Commit, date | Against the pin `03adda91` (`develop`, 2026-10-05) | +|---|---|---| +| `develop` | `03adda91`, 2026-10-05 | the pin itself | +| release `6.07.1`, the only tag | 2019-04-19 | 404 commits older; before the CMake build the patch uses | +| `production/GFS.v16.3`, operational GFS-Wave today | `41e7c9ca`, 2025-02-21 | split 2020-03-04; 372 of the pin's commits missing, 18 of its own | +| `production/GFS.v17` | `7a23cd77`, 2026-05-14 | split 2025-11-26 (`9d0cb947`); 143 of its own, 31 of the pin's missing | + +NOAA ships WW3 through `production/*` branches, not release tags. That `GFS.v17` is the next +operational GFS, and its implementation date, are not read in a NOAA notice ⚠ (the `noaa-notices` +skill is the place to check). + +## What moves + +`git diff --stat 03adda91 origin/production/GFS.v17 -- model/src`: 64 files, 11 661 insertions, +2 878 deletions (v). Every file of the KOKKOS patch changed upstream: `CMakeLists.txt`, +`cmake/switches.json`, `w3initmd.F90`, `w3srcemd.F90`, `w3wavemd.F90` (v). So did the Fortran of +two ported routines' modules, `w3pro3md.F90` (`W3XYP3`, `W3KTP3`) and `w3iobcmd.F90` (the +boundary output of U8) (v). `w3src4md.F90` (ST4) and `w3snl1md.F90` (the DIA), 66 % of the +resident wall time, did not change (v, absent from the name list). `wminiomd.F90` and +`w3iotrmd.F90` did not change either. + +## Units + +1. **Read GFS.v17's switch files and record them** (plan 3 of + `2026-10-18-003-gpu-first-supervised-run-plan.md`): `model/bin/switch_NCEP_st4` and the switch + the GFS.v17 workflow builds with ⚠ (where that file lives is not read). Run the `W3SRCE` gate + of gap 1 on it: if `SEED` or `FLD2` is there, the Kokkos `W3SRCE` does not run for GFS-Wave. +2. **Diff the Fortran of every translated routine** between `03adda91` and `7a23cd77`, from + `kokkos/ports.tsv`'s file:line columns. Each routine whose lines changed is listed with the + hunk; its translation is updated in its own commit (translate, don't improve), with the static + parity argument of ADR-0016 redone. +3. **Rebase the fork.** `WW3-noaa/` to `7a23cd77`; a new fork branch from it carrying the KOKKOS + patch (`ww3-gpu/kokkos-pnt` after #191's pin bump), each hunk re-placed, `PATCH.md`'s line + numbers updated. `scripts/ww3-gpu.sh`'s fork-versus-NOAA check must pass unchanged. +4. **Rerun the evidence on the new base:** `smoke`, `full`, `coast` resident pair to pair; the + resident matrix with `WW3GPU_LAB_SWITCH=ON` (the entries of #191's U3); a `ww3_multi` entry; + `WW_KOKKOS_VA_POISON=1` and `WW_KOKKOS_VA_COUNT=1`. Then `port_score.py --write`, and every + page that cites `model/src/.F90:` at `80644ba9` or `03adda91` updated to the new + pin (AGENTS.md says the pin's commit at the time of writing). +5. **The coupling caps.** `production/GFS.v17` carries the UFS NUOPC cap, which the fork at + `50a10730` does not: `wav_comp_nuopc.F90` calls `W3WAVE`, and `wav_import_export.F90` and + `wav_restart_mod.F90` use `VA` 16 and 13 times (v, `git show` at `7a23cd77`). Since PR #191, + `W3WAVE` returns with the resident `VA` possibly newer on the device, so every host use of `VA` + in those files is listed with its file:line and synced where it is (columns where the cap reads + columns, a whole-field read only before an export or a restart that needs every point), as + PATCH.md section 29 did for the ESMF cap. Then one coupled case with + `WW_KOKKOS_VA_POISON=1`, which fills a stale host `VA` with NaN, so a missed sync shows in the + exports; which UFS case runs on this machine is not known ⚠. +6. **Docs:** AGENTS.md, CONTRIBUTING.md and `docs/TOOLCHAIN.md` name the new pin and why it is + a production branch; `README.md` and `README.pt-BR.md` together. + +## Risks + +- `GFS.v17` lacks 31 commits of `develop`; any of them that the KOKKOS patch or a translation + relies on must be found in unit 2 or 3 ⚠ (not read). +- A production branch moves with NOAA's fixes; the pin is a commit, and moving it again is a + deliberate change, as for every submodule here. +- The other session's NOAA study and its `W3SDS4` patch are written against `03adda91`; the + patch is checked again on the new base with `scripts/ww3_fortran_ab.sh`. diff --git a/docs/data/ww3gpu_matrix_multi_pin-50a10730_2026-10-18.tsv b/docs/data/ww3gpu_matrix_multi_pin-50a10730_2026-10-18.tsv new file mode 100644 index 00000000..e4a367fa --- /dev/null +++ b/docs/data/ww3gpu_matrix_multi_pin-50a10730_2026-10-18.tsv @@ -0,0 +1,44 @@ +# Unit U3, item 3: agent A's ww3_multi commands of docs/log/2026-10-18-ww3-multi-resident-va.md on the integrated pin; record: docs/log/2026-10-18-u3-regtests-v2.md. +# Block 1: the SHRD serial entries, resident preset. Block 2: entry 61 (mww3_test_07) alone, to keep its ww3_grid error. Block 3: the moving-grid OMPG entries of mww3_test_05. +# Entry 61 fails in the Fortran run too: ww3_grid stops with "ERROR IN READING SDS4 FROM FILE, IOSTAT = 5010" (work_F_61/ww3_grid_refug.out), the entry's namelist against the lab switch's ST4. +# The VA diagnostics were set in the environment of the U3 agent's runs and the summary does not record them: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1. +# date: 2026-10-18T18:31:52Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 serial -R ^0(33|34|38|43|61): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +33 mww3_test_01 PR3_UQ yes yes yes 0/22 PASS hs 107 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ +34 mww3_test_02 PR3_UQ yes yes yes 0/23 PASS dp,fp,hs 127 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +38 mww3_test_03 PR3_UQ yes yes yes 0/34 PASS dp,fp,hs 126 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +43 mww3_test_04 PR3_UQ yes yes yes 0/25 PASS dir,dp,fp,hs 201 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +61 mww3_test_07 default yes no no n/a n/a - 90 0 - -w work_PR3_UQ -m grdset +# +# date: 2026-10-18T18:38:02Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 1 serial -R ^061: +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +61 mww3_test_07 default yes no no n/a n/a - 60 0 - -w work_PR3_UQ -m grdset +# +# date: 2026-10-18T18:32:18Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,MGP,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 omp -R ^00(2|8): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words PR3 UQ SHRD MGP !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV = SRCE,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +2 mww3_test_05 ST1_PR3_UQ_OMP yes yes yes 0/27 PASS hs 352 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST1_PR3_UQ_OMP -w work_ST1_PR3_UQ_OMP -m grdset_a -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes yes yes 0/27 PASS hs 351 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np diff --git a/docs/data/ww3gpu_matrix_multi_va_2026-10-18.tsv b/docs/data/ww3gpu_matrix_multi_va_2026-10-18.tsv new file mode 100644 index 00000000..4a3395de --- /dev/null +++ b/docs/data/ww3gpu_matrix_multi_va_2026-10-18.tsv @@ -0,0 +1,124 @@ +# The regtest matrix summaries of docs/log/2026-10-18-ww3-multi-resident-va.md, as written by +# record:summary, one block per run. The VA diagnostics were set in the environment and the +# summary does not record them: each block's first line names them. Before is WW3 80644ba9; +# after is WW3 8f6ba104, whose tree equals ww3-gpu/u1-multi-va f330dc53 (git diff --stat empty). +# +# === before_res_check: WW_KOKKOS_VA_CHECK=1 WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T14:53:39Z +# ww3-gpu commit: 4f87fec (dirty) +# WW3 commit: 80644ba9 (origin/ww3-gpu/kokkos-pnt) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 serial -R ^0(33|34|38|43): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +33 mww3_test_01 PR3_UQ yes yes yes 0/22 PASS hs 28 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ +34 mww3_test_02 PR3_UQ yes fortran-only yes n/a n/a - 41 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +38 mww3_test_03 PR3_UQ yes fortran-only yes n/a n/a - 46 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +43 mww3_test_04 PR3_UQ yes fortran-only yes n/a n/a - 74 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +# +# === before_res_nocheck: WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:01:19Z +# ww3-gpu commit: 4f87fec (dirty) +# WW3 commit: 80644ba9 (origin/ww3-gpu/kokkos-pnt) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 serial -R ^0(33|34|38|43): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +33 mww3_test_01 PR3_UQ yes yes yes 0/22 PASS hs 31 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ +34 mww3_test_02 PR3_UQ yes yes yes 5/23 FAIL dp,fp,hs 61 SRCE:0,SDS4:0,SIN4:0,SPR4:0 0/23 -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +38 mww3_test_03 PR3_UQ yes yes yes 10/34 FAIL dp,fp,hs 72 SRCE:0,SDS4:0,SIN4:0,SPR4:0 0/34 -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +43 mww3_test_04 PR3_UQ yes yes yes 4/25 FAIL dir,dp,fp,hs 150 SRCE:0,SDS4:0,SIN4:0,SPR4:0 0/25 -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +# +# === before_sub_check: WW_KOKKOS_VA_CHECK=1 WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:07:44Z +# ww3-gpu commit: 4f87fec (dirty) +# WW3 commit: 80644ba9 (origin/ww3-gpu/kokkos-pnt) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,MGP,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 omp -R ^00(2|8): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words PR3 UQ SHRD MGP !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV = SRCE,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +2 mww3_test_05 ST1_PR3_UQ_OMP yes fortran-only yes n/a n/a - 184 SRCE:10147,SDS4:0,SIN4:0,SPR4:0 - -s ST1_PR3_UQ_OMP -w work_ST1_PR3_UQ_OMP -m grdset_a -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes fortran-only yes n/a n/a - 181 SRCE:10147,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np +# +# === before_sub_nocheck: WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:11:49Z +# ww3-gpu commit: 4f87fec (dirty) +# WW3 commit: 80644ba9 (origin/ww3-gpu/kokkos-pnt) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,MGP,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 omp -R ^00(2|8): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words PR3 UQ SHRD MGP !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV = SRCE,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +2 mww3_test_05 ST1_PR3_UQ_OMP yes yes yes 0/27 PASS hs 210 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST1_PR3_UQ_OMP -w work_ST1_PR3_UQ_OMP -m grdset_a -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes yes yes 0/27 PASS hs 217 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np +# +# === after_res_poison: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_CHECK=1 WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:47:56Z +# ww3-gpu commit: 1ebd62f +# WW3 commit: 8f6ba104 () +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 serial -R ^0(33|34|38|43): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +33 mww3_test_01 PR3_UQ yes yes yes 0/22 PASS hs 31 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ +34 mww3_test_02 PR3_UQ yes yes yes 0/23 PASS dp,fp,hs 48 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +38 mww3_test_03 PR3_UQ yes yes yes 0/34 PASS dp,fp,hs 57 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +43 mww3_test_04 PR3_UQ yes yes yes 0/25 PASS dir,dp,fp,hs 121 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +# +# === after_res_nocheck: WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:54:32Z +# ww3-gpu commit: 1ebd62f +# WW3 commit: 8f6ba104 () +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 serial -R ^0(33|34|38|43): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +33 mww3_test_01 PR3_UQ yes yes yes 0/22 PASS hs 34 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ +34 mww3_test_02 PR3_UQ yes yes yes 0/23 PASS dp,fp,hs 43 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +38 mww3_test_03 PR3_UQ yes yes yes 0/34 PASS dp,fp,hs 56 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +43 mww3_test_04 PR3_UQ yes yes yes 0/25 PASS dir,dp,fp,hs 120 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +# +# === after_sub_poison: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_CHECK=1 WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:51:48Z +# ww3-gpu commit: 1ebd62f +# WW3 commit: 8f6ba104 () +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,MGP,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 omp -R ^00(2|8): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words PR3 UQ SHRD MGP !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV = SRCE,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +2 mww3_test_05 ST1_PR3_UQ_OMP yes yes yes 0/27 PASS hs 200 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST1_PR3_UQ_OMP -w work_ST1_PR3_UQ_OMP -m grdset_a -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes yes yes 0/27 PASS hs 199 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np +# +# === after_sub_nocheck: WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T16:00:52Z +# ww3-gpu commit: 1ebd62f +# WW3 commit: 8f6ba104 () +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,MGP,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 omp -R ^00(2|8): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words PR3 UQ SHRD MGP !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV = SRCE,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +2 mww3_test_05 ST1_PR3_UQ_OMP yes yes yes 0/27 PASS hs 345 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST1_PR3_UQ_OMP -w work_ST1_PR3_UQ_OMP -m grdset_a -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes yes yes 0/27 PASS hs 347 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np diff --git a/docs/data/ww3gpu_matrix_omp_pin-50a10730_2026-10-18.tsv b/docs/data/ww3gpu_matrix_omp_pin-50a10730_2026-10-18.tsv new file mode 100644 index 00000000..60f2992e --- /dev/null +++ b/docs/data/ww3gpu_matrix_omp_pin-50a10730_2026-10-18.tsv @@ -0,0 +1,24 @@ +# Unit U3, item 2; record: docs/log/2026-10-18-u3-regtests-v2.md. The U3 agent's run, as written by record:summary. +# The OMPG entries of matrix.base with ST4 and SHRD, without PDLIB; XYP3 and KTP3 left out (they refuse a moving grid, w3kokkos_prop3md.F90:123-125 at 50a10730). +# selection:count failed: CMakeLists.txt has no expected count for this word list, so the check has nothing to compare; every entry ran and passed. +# The VA diagnostics were set in the environment of the U3 agent's runs and the summary does not record them: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1. +# date: 2026-10-18T18:24:51Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV,CPRT WW3GPU_SWITCH_WORDS=ST4,SHRD,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 4 omp +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words ST4 SHRD !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV,CPRT = SRCE,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +1 ww3_ts3 ST4_PR1_OMP yes yes yes 0/20 PASS hs 217 SRCE:42025,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR1_OMP -w work_ST4_PR1_OMP -f -O -n $np +2 mww3_test_05 ST4_PR1_OMP yes yes yes 0/27 PASS hs 395 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR1_OMP -w work_ST4_PR1_OMP -m grdset_a -f -O -n $np +3 ww3_ts3 ST4_PR2_UQ_OMP yes yes yes 0/20 PASS hs 213 SRCE:42025,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR2_UQ_OMP -w work_ST4_PR2_UQ_OMP -f -O -n $np +4 mww3_test_05 ST4_PR2_UQ_OMP yes yes yes 0/27 PASS hs 413 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR2_UQ_OMP -w work_ST4_PR2_UQ_OMP -m grdset_a -f -O -n $np +5 ww3_ts3 ST4_PR2_UNO_OMP yes yes yes 0/20 PASS hs 215 SRCE:42025,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR2_UNO_OMP -w work_ST4_PR2_UNO_OMP -f -O -n $np +6 mww3_test_05 ST4_PR2_UNO_OMP yes yes yes 0/27 PASS hs 373 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR2_UNO_OMP -w work_ST4_PR2_UNO_OMP -m grdset_a -f -O -n $np +7 ww3_ts3 ST4_PR3_UQ_OMP yes yes yes 0/20 PASS hs 183 SRCE:42025,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes yes yes 0/27 PASS hs 295 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np +9 ww3_ts3 ST4_PR3_UNO_OMP yes yes yes 0/20 PASS hs 185 SRCE:42025,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UNO_OMP -w work_ST4_PR3_UNO_OMP -f -O -n $np +10 mww3_test_05 ST4_PR3_UNO_OMP yes yes yes 0/27 PASS hs 215 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UNO_OMP -w work_ST4_PR3_UNO_OMP -m grdset_a -f -O -n $np diff --git a/docs/data/ww3gpu_matrix_serial_pin-50a10730_2026-10-18.tsv b/docs/data/ww3gpu_matrix_serial_pin-50a10730_2026-10-18.tsv new file mode 100644 index 00000000..c9fd897d --- /dev/null +++ b/docs/data/ww3gpu_matrix_serial_pin-50a10730_2026-10-18.tsv @@ -0,0 +1,42 @@ +# Unit U3 of docs/compound/plans/2026-10-18-002-production-single-node-plan.md; record: docs/log/2026-10-18-u3-regtests-v2.md. +# Block 1: the owner's run, in the worktree ~/code/ww3-gpu-pr191 at daaef07d, as written by record:summary. Block 2: the same command by the U3 agent in its worktree, with OMP_NUM_THREADS=4. +# Rows 1 and 2 (ww3_tp2.6 ST4, ST4_ASCII) are FAIL in both blocks: the Kokkos run stops in W3INIT with "WW_KOKKOS_XYP3=1, BUT ITS KOKKOS TRANSLATION DOES NOT COVER THIS SWITCH FILE" +# (logs/1.K.log:160 of the owner's build): LIMON is an unstructured grid, which W3KOKKOS_PROP3_SETUP refuses (WW3/model/src/w3kokkos_prop3md.F90:122,134 at 50a10730), as docs/log/2026-10-18-resident-regtest-lab-switch.md:51 records. +# The owner decided on 2026-10-18 to keep both as FAIL rows, with no change to the selection, the verdict or the parts; issue #196 is the port of the unstructured path. +# +# date: 2026-10-18T18:29:19Z +# ww3-gpu commit: daaef07d +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=8 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON just ww3-gpu-matrix 4 serial +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words ST4 NL1 PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +1 ww3_tp2.6 ST4 yes fortran-only no n/a n/a - 202 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s ST4 -w work_ST4 +2 ww3_tp2.6 ST4_ASCII yes fortran-only no n/a n/a - 197 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s ST4_ASCII -w work_ST4_ASCII +3 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 159 SRCE:6900,SDS4:0,SIN4:0,SPR4:0 - -w work_PR3_UQ +4 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 979 SRCE:137000,SDS4:0,SIN4:0,SPR4:0 - -w work_5km -g 5km +5 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 141 SRCE:6624,SDS4:0,SIN4:0,SPR4:0 - -i input_const -w work_PR3_UQ_CONST +6 ww3_tp2.15 default yes yes yes 0/25 PASS dir,dp,fp,hs 153 SRCE:6900,SDS4:0,SIN4:0,SPR4:0 - -i input_rho -w work_PR3_UQ_RHO +7 ww3_ts2 ST4_PR3_UQ yes yes yes 0/22 n/a - 117 SRCE:4632,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ -w work_ST4_PR3_UQ +# +# The VA diagnostics were set in the environment of the U3 agent's runs and the summary does not record them: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1. +# date: 2026-10-18T18:09:13Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 4 serial +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words ST4 NL1 PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +1 ww3_tp2.6 ST4 yes fortran-only no n/a n/a - 127 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s ST4 -w work_ST4 +2 ww3_tp2.6 ST4_ASCII yes fortran-only no n/a n/a - 132 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s ST4_ASCII -w work_ST4_ASCII +3 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 95 SRCE:6900,SDS4:0,SIN4:0,SPR4:0 - -w work_PR3_UQ +4 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 545 SRCE:137000,SDS4:0,SIN4:0,SPR4:0 - -w work_5km -g 5km +5 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 87 SRCE:6624,SDS4:0,SIN4:0,SPR4:0 - -i input_const -w work_PR3_UQ_CONST +6 ww3_tp2.15 default yes yes yes 0/25 PASS dir,dp,fp,hs 82 SRCE:6900,SDS4:0,SIN4:0,SPR4:0 - -i input_rho -w work_PR3_UQ_RHO +7 ww3_ts2 ST4_PR3_UQ yes yes yes 0/22 n/a - 72 SRCE:4632,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ -w work_ST4_PR3_UQ diff --git a/docs/data/ww3gpu_matrix_tnc1_pin-50a10730_2026-10-18.tsv b/docs/data/ww3gpu_matrix_tnc1_pin-50a10730_2026-10-18.tsv new file mode 100644 index 00000000..3abd0eab --- /dev/null +++ b/docs/data/ww3gpu_matrix_tnc1_pin-50a10730_2026-10-18.tsv @@ -0,0 +1,27 @@ +# Unit U3, item 4: ww3_tnc1 (PR0, partition fields); record: docs/log/2026-10-18-u3-regtests-v2.md. +# Block 1: SRCE,OUTG,IOPE,UPDV,CPRT. Block 2: resident, which stops in W3INIT by design ("WW_KOKKOS_XYP3=1, BUT ITS KOKKOS TRANSLATION DOES NOT COVER THIS SWITCH FILE", +# KOKKOS_PART at WW3/model/src/w3kokkosmd.F90:1244-1245, asked from the no-PR3 branch at w3kokkos_prop3md.F90:143 at 50a10730). +# The VA diagnostics were set in the environment of the U3 agent's runs and the summary does not record them: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1. +# date: 2026-10-18T18:34:09Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV,CPRT WW3GPU_SWITCH_WORDS=PR0;SHRD WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 1 serial -R ^038: +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR0 SHRD, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV,CPRT = SRCE,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +38 ww3_tnc1 default yes yes yes 0/30 n/a - 57 SRCE:405,SDS4:0,SIN4:0,SPR4:0 - -N +# +# date: 2026-10-18T18:36:08Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR0;SHRD WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 1 serial -R ^038: +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR0 SHRD, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +38 ww3_tnc1 default yes fortran-only no n/a n/a - 70 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -N diff --git a/docs/log/2026-10-18-cprt-resident.md b/docs/log/2026-10-18-cprt-resident.md new file mode 100644 index 00000000..404a4a5f --- /dev/null +++ b/docs/log/2026-10-18-cprt-resident.md @@ -0,0 +1,93 @@ +# W3CPRT on the resident VA: partitioning without copying the spectrum back + +2026-10-18, agent G of the production single-node goal (unit U9 of +[`docs/compound/plans/`](../compound/plans/), branch `goal/abbb0456-6043-u9-cprt`; fork branch +`ww3-gpu/u9-cprt` of h0ffmann/WW3, on `80644ba9`). + +## Question + +At every output time with partitioning on (gridded partition fields, or the partition file), +`W3WAVE` read the whole `VA` back from the device for `W3CPRT` (`w3wavemd.F90:2688-2693` at +`80644ba9`), and `W3KOKKOS_OUTG` fell back to the Fortran `W3OUTG` when a partition field was +requested. Can `W3CPRT` run on the resident `VA`, with only its table coming back, bit for bit? + +## What was done + +- `W3PART` and everything it calls were translated on 2026-10-16 and never wired + ([`W3PART_KOKKOS.md`](../../kokkos/port_report/W3PART_KOKKOS.md)). `W3CPRT` itself is 236 lines: + per sea point a conversion of `N` to `E = N 2 pi sigma / c_g`, one `W3PART` call, and the + storage of `XP(1:15,0:NP)` one point after the other. Points are independent, so a launch over + points with each point's serial code is the translation; the table is placed by an integer + prefix sum. Kernel [`iosf_w3cprt.hpp`](../../kokkos/src/ww_kokkos/iosf_w3cprt.hpp), shim + [`cprt_shim.cpp`](../../kokkos/src/fortran_iface/cprt_shim.cpp), fork unit + `w3kokkos_cprtmd.F90`, details in + [`PATCH.md` section 27](../../kokkos/src/fortran_iface/PATCH.md). +- The first run with `CPRT` on still showed 13 device-to-host copies of `VA` in 12 hours: the + host read before `W3IOSF` (`w3wavemd.F90:2963-2966`). `W3IOSF` reads no `VA` (its `USE` lists, + `w3iosfmd.F90:429-453`, v), so that read is skipped under `KOKKOS_CPRT`. +- `W3OUTG`'s section 6 (the partition fields from the table) is now its own entry point, + `ww_outg_part`, so `W3KOKKOS_OUTG` no longer falls back for `FLPART`. +- Static parity: the `-O3` GIMPLE of `w3iosfmd.F90` and of the shim + ([`W3CPRT_KOKKOS.md`](../../kokkos/port_report/W3CPRT_KOKKOS.md)). +- CPU only, never `--gpu`. i9-14900, shared with other agents' jobs, `OMP_NUM_THREADS=4`, + `JOBS=4`, gfortran/g++ 16.2.0, Kokkos 5.2.2 OpenMP backend. + +Runs. `bench_small` (`kokkos/tests/fixtures/bench_small`: 120 x 80, 9 520 sea points, +`NK = NTH = 24`, 12 hours) writes no output, so two throwaway variants of its `ww3_shel.nml` (not +committed, restored after each run) turned partitioning on, both with hourly field output and the +hourly partition file (`DATE%PARTITION`, `TYPE%PARTITION` over the whole grid, unformatted): + +- `p6`: fields `HS T02 DIR` (which `W3OUTG` on Kokkos computes) and the partition file; +- `pf`: the same plus all 17 partition fields (`PHS PTP PLP PDIR PSPR PWS PDP PQP PPE PGW PSW + PTM10 PT01 PT02 PEP TWS PNR`). + +Each ran as `WW3GPU_SWITCH=switch_lab_ WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1 JOBS=4 +OMP_NUM_THREADS=4 bash scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos `, "before" with +`SRCE,XYP3,KTP3,OUTG,IOPE,UPDV` and "after" with `CPRT` added, and `partition.ww3` was compared +with `cmp` by hand (the script does not list it). WW3's regtest `ww3_tpt1.1` (one multi-modal +spectrum on 18 sea points, one grid per partitioning method `PTM = 1..5`, not in `matrix.base`) +ran through the matrix driver's `regtests/cmake/run_entry.cmake` with the configuration of +`WW3GPU_KOKKOS=CPRT WW3GPU_SWITCH_WORDS=SHRD just ww3-gpu-matrix-list serial`, options `-N -g +PTM`, `WW_KOKKOS_VA_POISON=1`. + +## Result + +| Run | `VA` device-to-host copies, 12 h, 13 output times | Files | +|---|---|---| +| `p6`, `switch_lab_shrd`, before | 13 (275.71 MB) | every model file and `partition.ww3` byte-identical to the Fortran of the same executable and to `--ww3` (v) | +| `p6`, `switch_lab_shrd`, after | 1 (21.21 MB, `W3WAVE`'s read at its return, which U1 removes on its own branch) | the same (v) | +| `pf`, `switch_lab_shrd`, before (with `ww_outg_part`) | 13 | the same (v) | +| `pf`, `switch_lab_shrd`, after | 1 | the same (v) | +| `pf`, `switch_lab_ompg`, after | 1 | the same (v) | + +- (v) In every run `VA host copies poisoned` was 97: the host `VA` was filled with NaN after each + kernel write, and no output changed, so no host reader read a stale `VA`. +- (v) After, per output time, what a device would move instead of the 21.21 MB of `VA` + (`WW_KOKKOS_CPRT_COUNT=1` and `WW_KOKKOS_OUTG_COUNT=1`, projected on the host backend, totals + divided by 13 calls): the table back, 1 135 640 B (`ICPRT` 76 168 B and the used `DTPRT` + columns); the inputs in, 2 212 324 B (`CG` and `WN` are 1.98 MB of it); with the 17 partition + fields requested, those fields back, 3 387 072 B. 0 fallbacks to the Fortran. +- (v) `ww3_tpt1.1`, `PTM = 1` to `5`: `test.comp` found 0 of 22 files different and `nccmp-tol` + passed on each; each Kokkos run made 2 `W3CPRT` Kokkos calls on 18 sea points, 0 fallbacks. + The staged switch had `PR3 UQ` in place of the test's `PR0` ⚠: the fork's `KOKKOS` build does + not compile with `PR0` (`w3kokkosmd.F90:3183` at `80644ba9` uses `W3ADATMD`'s PR3 arrays), so + neither `ww3_tpt1.1` as written nor `ww3_tnc1` (the one serial `matrix.base` entry with partition + fields, `PR0`) can be built with `KOKKOS`; both runs of each pair used the one executable. Two of + the five entries first failed to link (`DSO missing from command line`, `libgfortran`) in a + build directory left by an interrupted run; removing `build_903` and `build_905` fixed it ⚠. +- (v) A WW3 defect found on the way: the formatted partition file (`TYPE%PARTITION%FORMAT = T`) + stops `ww3_shel` with a gfortran format error in the Fortran itself. Format 942 is + `(I3,3F8.2,2F9.2,F7.2)` (`w3iosfmd.F90:739`) and `:706` writes `IP` and 15 values of + `DTPRT(:,IP+IOFF)`; the format reverts to `I3` for a real (backtrace under `gdb`, Fortran run of + the same executable). The runs above use `FORMAT = F`. Not changed: WW3 is read-only. +- ⚠ Nothing ran on a GPU, and the CUDA build of the new code was not compiled. The wall times the + script prints are one run each on a shared machine and are not quoted. + +## What it changes + +`W3CPRT` and `W3OUTG`'s partition fields no longer copy the spectrum back: with `CPRT` on, an +output time with partitioning moves only the table and the fields written. Open: `CPRT` is not +in the `resident` preset (section 25 of PATCH.md), the fallbacks are not reported to +`WW_KOKKOS_STRICT` (section 24; this fork branch predates it), the regtest matrix has no +`KOKKOS`-buildable entry with partitioning, and the formatted partition file is broken in WW3 +at the pin. diff --git a/docs/log/2026-10-18-cuda-build-only.md b/docs/log/2026-10-18-cuda-build-only.md new file mode 100644 index 00000000..57527d09 --- /dev/null +++ b/docs/log/2026-10-18-cuda-build-only.md @@ -0,0 +1,91 @@ +# The CUDA build of `ww_kokkos` and of the fork for the resident preset, build only: 51 nvcc warnings, none in a wired part + +2026-10-18, an agent for M. Hoffmann; unit U5 of +[`2026-10-18-002-production-single-node-plan.md`](../compound/plans/2026-10-18-002-production-single-node-plan.md), +[ADR-0014](../ADRs/ADR-0014-gpu-runs-judged-by-tolerance.md). No device run. + +## Question + +Do `ww_kokkos` and the `h0ffmann/WW3` fork build on Kokkos' CUDA backend for the resident preset +(`--kokkos resident`, `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV`), and does any warning of nvcc, the Kokkos +wrapper or gfortran touch the bits: contraction, a conversion, a math call, a host function +called from device code? The answer is the precondition of the owner's supervised run, +[`GPU_SUPERVISED_RUN_CHECKLIST_202610.md`](../GPU_SUPERVISED_RUN_CHECKLIST_202610.md). + +## What was done + +On branch `goal/abbb0456-6043-u5-gpu-ready` at `4f87fec` (from +`goal/abbb0456-6043-production-single-node`), `WW3/` at `80644ba9` (`ww3-gpu/kokkos-pnt`), +`WW3-noaa/` at `03adda91`, `picasso/` at `be76132`. Machine: Intel Core i9-14900, 32 threads, +shared with three other agents (at most 8 build jobs); NVIDIA GeForce RTX 4090, driver 615.71.09 +(`nvidia-smi --query-gpu=name,driver_version`, a driver query, no kernel launched). + +Toolchain, all from the `#cuda` shell of `picasso/labs/pratico` (v, the configure log): nvcc +12.9.86 (`cuda_12.9.r12.9/compiler.36037853_0`), Kokkos 5.2.2 (`kokkos-cuda-5.2.2`, devices +`OPENMP;SERIAL;CUDA`, `-arch=sm_89`), nvcc_wrapper's host compiler g++ 14.4.0, gfortran and gcc +16.2.0, CMake 4.4.3, glibc 2.44, `WW_KOKKOS_LIBM=ON`. + +```bash +nix develop ./picasso/labs/pratico#cuda --command cmake -S kokkos --preset cuda-release +nix develop ./picasso/labs/pratico#cuda --command cmake --build kokkos/build/cuda-release -j8 -- -v +JOBS=8 scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident --build-only +``` + +`just kokkos-cuda-test` was not used: its last line runs `ctest` with CUDA as the default +execution space, a device run. `just kokkos-test cuda-release` was not used either: it builds in +the `#ww3` shell and also runs `ctest`. The `--build-only` command rebuilt `cuda-release` (the +script's toolchain stamp differs from a hand configure) and built WW3 with the switch +`switch_lab_ompg` plus `KOKKOS`, `-DCMAKE_CXX_COMPILER=nvcc_wrapper`, Fortran `-O3`. The logs +are in the worktree's `build/u5/` and `build/ww3-gpu-cuda/{configure,build}.log` (git-ignored). + +## Result + +Both builds passed (v): the 73 steps of `ww_kokkos` with its tools and intro programs, and the 21 +WW3 programs (`ww3_shel`, `ww3_multi`, ...) in `build/ww3-gpu-cuda/ww3/bin`. The second build of +`cuda-release` printed the same 53 lines matching `warning` as the first (v, `grep -c`). + +**Contraction and math flags** (v, the `-v` build log and `nvcc_wrapper -dryrun` on a test file): +every one of the 25 translation units of `libww_kokkos.a` is compiled with `-ffp-contract=off +--fmad=false`; nvcc_wrapper passes `-ffp-contract=off` to the host g++, ptxas gets `--fmad +false`, and cicc `-ftz=0 -prec_div=1 -prec_sqrt=1 -fmad=0`. No `--use_fast_math` anywhere. + +**nvcc warnings, 51 in the Kokkos code, grouped:** + +| File:line | Warning | Count | Can it touch the bits | +|---|---|---|---| +| `kokkos/src/ww_kokkos/pdlib_do_output_exchanges.hpp`:151, 153, 156, 159, 217-220, 247, 248, 265-267, 271, 272, 291 | `#20013-D` calling a `constexpr __host__` function (`operator()`) from a `__host__ __device__` function (`walk_columns`, or the packing lambda of `do_output_exchanges`); "the experimental flag `--expt-relaxed-constexpr` can be used to allow this" | 48 (16 sites, 3 each) | The lambdas `one`, `e3`, `swell`, `cols` of `walk_columns` are host-only to nvcc, and the packing kernel calls them on the device. It compiled; whether the device code calls them correctly is ⚠ (never run). No floating-point arithmetic in this routine (`kokkos/ports.tsv`, row `pdlib_field_vec:do_output_exchanges`), so a defect would misplace values, not round them. The routine is translated only: header-only, no shim, no `W3WAVE` hunk (v, `ports.tsv`), instantiated only by `wave_support.cpp:305` for the compile check. Not in the resident preset and not called by the fork. | +| `kokkos/src/ww_kokkos/uost_uost_sourceterm_compute.hpp`:54, 56 | `#1675-D` unrecognized GCC pragma (`#pragma GCC novector`) | 2 | nvcc's front end does not know the pragma, and passes it on: on a test file the host g++ 14.4.0 under nvcc_wrapper reports "splitting region at dont-vectorize loop" (v, `-Xcompiler -fopt-info-vec-all`). The pragma keeps the Fortran's operand order of `S_LD + S_SE` when both are NaN (the header's comment, lines 14-16); it is off in device code (`!defined(__CUDA_ARCH__)`). `UOST_SOURCETERM` is translated only, no shim (v, `ports.tsv`). | +| `kokkos/src/ww_kokkos/wavset_fd_compute_lh_stress.hpp`:45 | `#550-D` variable `syy_x` was set but never used | 1 | No: a dead store in a translation of `W3WAVSET`, translated only (v, `ports.tsv`). Not changed: translate, don't improve. | + +No warning about a double-to-float conversion, an unsupported math function, a host function +called from a kernel of a wired part, or contraction, in either build (v, `grep -i warning`). + +**CMake, one warning:** Kokkos' package "does not support CXX extensions. Forcing +`-DCMAKE_CXX_EXTENSIONS=Off`" at WW3's `find_package(ww_kokkos)`. No effect on the arithmetic ⚠ +(it selects `-std=c++20` over `-std=gnu++20`). + +**gfortran, 8 warnings, all in WW3's own files:** "Legacy Extension: Missing comma in FORMAT +string" at `w3dispmd.F90:913`, `w3iorsmd.F90:490`, `w3iogomd.F90:316`, `w3iogomd.F90:2734`, +`w3iopomd.F90:2415`, `ww3_gspl.F90:3029`, `ww3_ounp.F90:1344`, `ww3_prnc.F90:2430`. A FORMAT +string formats text output; it does not touch the model's arithmetic. None in the fork's +`w3kokkos*md.F90` modules (v, `grep`). + +**One thing that is not a warning and may touch the bits:** the CUDA build's host C++ is +compiled by g++ 14.4.0 (nvcc_wrapper's host compiler), while the CPU build and every static +parity argument of `kokkos/ports.tsv` use g++ 16.2.0. A host branch of a kernel in the CUDA +build (a fallback that runs on the host, or the `Kokkos::Serial` path a shim takes) was never +compared with the Fortran under g++ 14 ⚠. On the device the question is the libm one of +[`GPU_LIBM_BITS_202610.md`](../GPU_LIBM_BITS_202610.md). + +Nothing ran on the GPU: no `ctest`, no `ww3_shel`, no `libm_bits --device`. + +## What it changes + +- The checklist [`GPU_SUPERVISED_RUN_CHECKLIST_202610.md`](../GPU_SUPERVISED_RUN_CHECKLIST_202610.md) + starts from this build. +- `just ww3-gpu-matrix-cuda` forwards `WW3GPU_ROUTINES` only, not `WW3GPU_KOKKOS` or + `WW3GPU_LAB_SWITCH`, so it cannot run the resident-preset entries of the CPU matrix as they ran + (v, `justfile`). Left for the integrator and a person's review: it changes how a regtest + driver is called. +- The `#20013-D` sites matter once `DO_OUTPUT_EXCHANGES` is wired on a device; the wiring unit + reads them first. Not fixed here. diff --git a/docs/log/2026-10-18-resident-output-readers.md b/docs/log/2026-10-18-resident-output-readers.md new file mode 100644 index 00000000..ee09b09f --- /dev/null +++ b/docs/log/2026-10-18-resident-output-readers.md @@ -0,0 +1,83 @@ +# 2026-10-18: the boundary and track outputs read only their columns of the resident `VA` + +Goal `abbb0456-6043`, unit U8 of the production single-node plan +([`docs/compound/plans/2026-10-18-002-production-single-node-plan.md`](../compound/plans/2026-10-18-002-production-single-node-plan.md)). +Fork branch `ww3-gpu/u8-out-readers` of h0ffmann/WW3: `31a09522`, `791e8c34` on `80644ba9`. +The change is [`PATCH.md` section 23](../../kokkos/src/fortran_iface/PATCH.md#23-the-boundary-and-track-outputs-read-only-their-columns-ww3-gpuu8-out-readers). + +## What was tried + +The output block of `W3WAVE` copied the whole device `VA` to the host before each of `W3IOTR`, +`W3IOBC ('WRITE')`, `W3IOSF` and the OASIS coupling (`w3wavemd.F90:2934-2975` at `80644ba9`). +Read in the Fortran at the pin: + +- `W3IOBC ('WRITE')` reads the columns `ISBPO(1:NBO2(NFBPO))` (`w3iobcmd.F90:660-668`): now a + column read. +- `W3IOTR` reads `MAPFS(IY,IX)` for the points `MASK1` marks with `MAPSTA > 0` + (`w3iotrmd.F90:728-819`), known only after it reads the track file: now a column read inside + `W3IOTR`, before its section 4. +- `W3IOSF` partitions every sea point with `MAPSTA >= 0` (`w3iosfmd.F90:243-265`): the full read + stays. Partitioning on the device belongs to the `W3CPRT` unit. +- The OASIS coupling reads no `VA` (v, `grep` of `w3agcmmd.F90`, `w3ogcmmd.F90`, + `w3igcmmd.F90`, `w3oacpmd.F90`): its sync is removed; not compiled here ⚠. + +The new column reads are under `WW_KOKKOS_IOPE=1`, the opt-in flag of the point output's column +read. + +## Commands + +The superproject at `96ef800` plus this change, `WW3/` at the fork commit named, on an Intel Core +i9-14900 shared with other sessions, `OMP_NUM_THREADS=4`, never `--gpu`: + +```bash +# the matrix entry with track output, before (WW3 at 80644ba9) and after (791e8c34) +nix develop ./picasso/labs/pratico#ww3 --command cmake -S regtests -B build/m-after \ + -DWW3GPU_SELECTION=serial -DWW3GPU_OMP_THREADS=4 -DWW3GPU_DATA= \ + -DWW3GPU_KOKKOS=resident -DWW3GPU_SWITCH_WORDS='PR3;UQ;SHRD;!PDLIB;!MGP;!MGG' \ + -DWW3GPU_LAB_SWITCH=ON -DWW3GPU_STRICT=ON +WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1 \ + nix develop ./picasso/labs/pratico#ww3 --command ctest --test-dir build/m-after -R tp2.2 +# the entry with boundary output, through run_entry.cmake (see "Not resolved"), with a copy of +# build/m-after/ww3gpu_config.cmake whose WW3GPU_KOKKOS is SRCE,OUTG,IOPE,UPDV +WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1 nix develop ./picasso/labs/pratico#ww3 --command \ + cmake -DCONFIG=build/m-after/ww3gpu_config_tr1.cmake -DN=901 -DTEST=ww3_tr1 \ + '-DOPTS=-i input_std -w work_std' -P regtests/cmake/run_entry.cmake +# pair to pair with --ww3 +WW_KOKKOS_VA_COUNT=1 bash scripts/ww3-gpu.sh --ww3-gpu smoke --cpu --kokkos resident +WW_KOKKOS_VA_COUNT=1 bash scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos resident +``` + +## What it showed + +| Entry | test.comp after | `VA` before | `VA` after | +|---|---|---|---| +| `ww3_tp2.2 -s PR3_UQ`, resident, track every 4 h | 0/24 different, `track_o.ww3` and `ww3.202206_trck.nc` identical (v) | D2H 4 (60.07 MB), host reads 8, partial reads 3 | D2H 4 (60.07 MB), host reads 4, partial reads 7 (3 track copies, 5 columns, 4.4 kB) | +| `ww3_tr1 -i input_std`, `SRCE,OUTG,IOPE,UPDV`, boundary every 1 h | 0/23 different, `nest1.ww3` to `nest3.ww3` identical (v) | D2H 7 (0.57 MB), host reads 8, writes 6, partial reads 7 | D2H 7 (0.57 MB), host reads 1, writes 6, partial reads 14 (7 boundary copies of 8 columns, 70 kB) | +| `smoke`, `full`, resident | byte-identical to the Fortran and to `--ww3`, `nccmp-tol` PASS (v) | D2H 37, 1 (section 17) | D2H 37, 1 | + +With `WW_KOKKOS_VA_POISON=1` the stale host `VA` is NaN after every device write, so a column a +reader needs and the list missed would show in its output file; none did. The whole-spectrum +copies per run did not drop in either entry: the next full sync of the same step copies `VA` +anyway (in `ww3_tp2.2` the exit read of `W3WAVE` at the wind times, which are the track times; in +`ww3_tr1` the host write before the `PR1` Fortran propagation at every step). What changed is +what each output reader copies: the columns it reads, not the whole spectrum. + +## Found on the way + +- No KOKKOS switch without `PR3` compiled: `W3KOKKOS_MAP3` and `W3KOKKOS_MAPT` named + `W3ADATMD` variables declared only under `W3_PR3` (`w3adatmd.F90:520-525`, `:648-653`). Fixed in + `31a09522`; `ww3_tr1` (`SHRD RTD PR1`) builds with it (v). +- `W3KOKKOS_IOTR_READ` writes `VA` while `W3IOTR` holds it as `INTENT(IN)`; the `-O3` dump loads + nothing of it before the call (v, `PATCH.md` section 23), which is the compiler's code, not a + guarantee of the standard ⚠. + +## Not resolved + +- The matrix driver drops the empty options of a first-kept entry: `ww3gpu_select_entries` + (`regtests/cmake/ww3gpu_common.cmake:337-343`) appends `""` to an empty list, which adds nothing, + so `E_OPTS` is one shorter than `E_TESTS` and the configure stops ("list index: 3 out of range") + for the `SHRD;RTD;!PDLIB` selection, whose first entry is `ww3_tr1` with no options. Not fixed: + `regtests/` changes need a person's review ⚠. +- No runnable entry writes boundary output with `PR3`: `ww3_tp2.17` is `MPI`, and `ww3_tp2.6` and + `ww3_tp2.7` write it once, at the start, before the device holds newer bits ⚠. +- `W3IOSF` keeps its full read until partitioning runs on the device. diff --git a/docs/log/2026-10-18-u3-regtests-v2.md b/docs/log/2026-10-18-u3-regtests-v2.md new file mode 100644 index 00000000..27dce2c5 --- /dev/null +++ b/docs/log/2026-10-18-u3-regtests-v2.md @@ -0,0 +1,163 @@ +# U3 on the integrated pin: the resident serial, OMPG, `ww3_multi` and `ww3_tnc1` entries; the restart pair designed, not run + +2026-10-18, the U3 agent for M. Hoffmann (goal `abbb0456-6043`), unit U3 of +[the production single-node plan](../compound/plans/2026-10-18-002-production-single-node-plan.md), +PR #191; gaps 2 and 10 of [`PRODUCTION_READINESS_202610.md`](../PRODUCTION_READINESS_202610.md). +The owner ran item 1 himself. + +## Question + +On the branch rebased onto `main` (`daaef07`, `WW3/` at `50a10730`, with #193's DIA-halo and +`W3SDS4` kernel changes and `CPRT` in the resident preset), does every Kokkos run of WW3's +regtests that the preset can select give the Fortran's bytes, with the host copy of `VA` poisoned? +And is there a restart-reproduction pair for a `SHRD` case? + +## What was done + +The owner's Intel Core i9-14900 (32 hardware threads, shared with the integrator's jobs: load +average from 2.4 to 17.0 over the runs, read with `uptime` at each start and end), the pinned `ww3` +shell (gfortran and g++ 16.2.0), Kokkos' OpenMP backend, never `--gpu`. Every run had +`WW3GPU_LAB_SWITCH=ON` and `WW3GPU_STRICT=ON` (the default); the matrix's Kokkos run sets +`WW_KOKKOS_STRICT=1` itself. The U3 agent's runs also had `WW_KOKKOS_VA_POISON=1` and +`WW_KOKKOS_VA_COUNT=1` in the environment and at most 4 jobs of 4 threads. + +1. **Resident, serial**, run by the owner in `~/code/ww3-gpu-pr191` at `daaef07d` + (2026-10-18T18:29:19Z, `OMP_NUM_THREADS=8`, 1012 s), and by the U3 agent in its worktree with + `WW3GPU_OMP_THREADS=4` and the VA diagnostics: + + ```bash + WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON just ww3-gpu-matrix 4 serial + ``` + + `just ww3-gpu-matrix-list serial` with the same variables lists 7 entries, as the selftest + expects. +2. **OMPG, no MPI**, parts without `XYP3` and `KTP3`, which refuse a moving grid + (`w3kokkos_prop3md.F90:123-125` at `50a10730`): + + ```bash + WW3GPU_OMP_THREADS=4 WW3GPU_LAB_SWITCH=ON WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV,CPRT \ + WW3GPU_SWITCH_WORDS='ST4,SHRD,!PDLIB' just ww3-gpu-matrix 4 omp + ``` + + With `SHRD,!PDLIB` alone the list has 50 entries: `ww3_ts3` and `mww3_test_05`, each with five + propagation schemes (`PR1`, `PR2_UQ`, `PR2_UNO`, `PR3_UQ`, `PR3_UNO`) and five source-term sets + (`ST1`, `ST2`, `ST3`, `ST4`, `ST6`). The source-term sets differ only in the words + `ww3gpu_lab_switch` replaces (`ST`, `FLX`, `STAB`; v, a `diff` of the sorted switch words of + every `ST4` file against its four siblings, `regtests/cmake/ww3gpu_common.cmake:224-247`), so + under the lab switch the 40 others build the same 10 switches. `ST4` keeps one of each: the 10 + entries cover every propagation scheme on both tests. +3. **`ww3_multi`, `SHRD`**, agent A's commands of + [the `ww3_multi` log](2026-10-18-ww3-multi-resident-va.md): + + ```bash + WW3GPU_OMP_THREADS=4 WW3GPU_LAB_SWITCH=ON WW3GPU_KOKKOS=resident \ + WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,!PDLIB,!MGP,!MGG' just ww3-gpu-matrix 2 serial -R '^0(33|34|38|43|61):' + WW3GPU_OMP_THREADS=4 WW3GPU_LAB_SWITCH=ON WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV \ + WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,MGP,!PDLIB' just ww3-gpu-matrix 2 omp -R '^00(2|8):' + ``` + + then entry 61 alone (`-R '^061:'`, 1 job) to keep its `ww3_grid` output. +4. **`ww3_tnc1`**, entry 38 with `WW3GPU_SWITCH_WORDS='PR0;SHRD'`, 1 job, with + `WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV,CPRT` and then with `resident`. The first `resident` try ran + at the same time as the other set and its `setup:kokkos` failed ("No known features for CXX + compiler"), two configures of `kokkos/build/openmp-release` at once, as `regtests/README.md` + warns; it was run again alone. +5. **Restart pair.** `grep -n rstrt_b4b WW3/regtests/bin/matrix.base`: every restart-reproduction + entry is `MPI` (`ww3_tp2.17` `work_ma1`, `work_mc1`, `work_ice_restart`, `work_deep_restart` + with `PDLIB`; `ww3_ufs1.1` `work_d`, `work_b`; `matrix.base:2048-2067`, `:2171-2205` at + `50a10730`), where the resident preset stops (ADR-0017). No entry restarts a `SHRD` case. The + equivalent is written as [`scripts/restart_pair.sh`](../../scripts/restart_pair.sh) and was not + run: the owner stopped every new long run on 2026-10-18 to run them by hand. + +The VA counters are the `ww_kokkos: VA ...` lines of each entry's `logs/.K.log`, summed over +the processes; the parts line is the `Kokkos parts` line of `work_K_/log.*`. + +## Result + +The summaries, with their headers: +[serial](../data/ww3gpu_matrix_serial_pin-50a10730_2026-10-18.tsv), +[omp](../data/ww3gpu_matrix_omp_pin-50a10730_2026-10-18.tsv), +[multi](../data/ww3gpu_matrix_multi_pin-50a10730_2026-10-18.tsv), +[tnc1](../data/ww3gpu_matrix_tnc1_pin-50a10730_2026-10-18.tsv). `ww3_self` is `-` on every row: +no `test.comp` difference, so no second Fortran run was needed. "D2H" is the whole-`VA` +device-to-host copies of the Kokkos run (`WW_KOKKOS_VA_COUNT=1`, the U3 agent's runs). + +| Item | Entry | Parts | test.comp | nccmp_tol | judged | D2H | Spectra, `CPRT` calls | +|---|---|---|---|---|---|---|---| +| 1 | 1 `ww3_tp2.6` `ST4` | resident | n/a: FAIL, `ran` fortran-only | n/a | - | 0 | 0 | +| 1 | 2 `ww3_tp2.6` `ST4_ASCII` | resident | n/a: FAIL, `ran` fortran-only | n/a | - | 0 | 0 | +| 1 | 3 `ww3_tp2.15` `PR3_UQ` | resident | 0/23 | PASS | dir,dp,fp,hs | 0 | 6 900, 0 | +| 1 | 4 `ww3_tp2.15` `-g 5km` | resident | 0/23 | PASS | dir,dp,fp,hs | 0 | 137 000, 0 | +| 1 | 5 `ww3_tp2.15` `input_const` | resident | 0/23 | PASS | dir,dp,fp,hs | 0 | 6 624, 0 | +| 1 | 6 `ww3_tp2.15` `input_rho` | resident | 0/25 | PASS | dir,dp,fp,hs | 0 | 6 900, 0 | +| 1 | 7 `ww3_ts2` `ST4_PR3_UQ` | resident | 0/22 | n/a | - | 0 | 4 632, 0 | +| 2 | 1, 3, 5, 7, 9 `ww3_ts3` `ST4_{PR1,PR2_UQ,PR2_UNO,PR3_UQ,PR3_UNO}_OMP` | `SRCE,OUTG,IOPE,UPDV,CPRT` | 0/20 each | PASS | hs | 24 each | 42 025, 0 | +| 2 | 2, 4, 6, 8, 10 `mww3_test_05` `ST4_*_OMP`, the same five | `SRCE,OUTG,IOPE,UPDV,CPRT` | 0/27 each | PASS | hs | 153 each | 300 876, 39 | +| 3 | 33 `mww3_test_01` `PR3_UQ` | resident | 0/22 | PASS | hs | 12 | 0, 0 | +| 3 | 34 `mww3_test_02` `PR3_UQ` `grdset_a` | resident | 0/23 | PASS | dp,fp,hs | 0 | 0, 0 | +| 3 | 38 `mww3_test_03` `PR3_UQ` `grdset_a` | resident | 0/34 | PASS | dp,fp,hs | 3 | 0, 219 | +| 3 | 43 `mww3_test_04` `PR3_UQ` `grdset_a` | resident | 0/25 | PASS | dir,dp,fp,hs | 0 | 0, 0 | +| 3 | 61 `mww3_test_07` `default` | resident | n/a: FAIL, `ran` no | n/a | - | 0 | 0 | +| 3 | 2, 8 `mww3_test_05` `ST1`/`ST4_PR3_UQ_OMP` | `SRCE,OUTG,IOPE,UPDV` | 0/27 each | PASS | hs | 156 each | 300 876, 0 | +| 4 | 38 `ww3_tnc1` | `SRCE,OUTG,IOPE,UPDV,CPRT` | 0/30 | n/a | - | 0 | 405, 3 | +| 4 | 38 `ww3_tnc1` | resident | n/a: FAIL, `ran` fortran-only | n/a | - | 0 | 0 | + +- (v) Every entry whose Kokkos run ran gave `test.comp` 0 files different with the host copy of + `VA` poisoned, every asked part `=K` in the parts line, and `calls left on the Fortran ...: + SRCE=0 CPRT=0`. The owner's item 1 gave the same verdicts (rows 3 to 7: 0/23, 0/23, 0/23, 0/25, + 0/22). +- (v) Rows 1 and 2 of item 1 stop in `W3INIT`: "WW_KOKKOS_XYP3=1, BUT ITS KOKKOS TRANSLATION DOES + NOT COVER THIS SWITCH FILE" (`logs/1.K.log:160` of the owner's build), since `W3KOKKOS_PROP3_SETUP` + sets `XYP3` off on a grid that is neither rectilinear nor curvilinear + (`w3kokkos_prop3md.F90:122`, `:134` at `50a10730`); `ww3_tp2.6` reads the unstructured mesh + `limon_ll.msh`. The `ww3_ounf` error after it is a downstream effect. Refused by design, as + [the lab-switch log](2026-10-18-resident-regtest-lab-switch.md) (line 51) records. The owner + decided on 2026-10-18 to keep both as FAIL rows, with no change to the selection, the verdict or + the parts; issue #196 is the port of the unstructured path. +- (v) `ww3_tnc1` with `resident` stops in `W3INIT` the same way: without `PR3` and `UQ` the set-up + asks `KOKKOS_PART ( 'XYP3', .FALSE. )` (`w3kokkos_prop3md.F90:143`), whose message is + `w3kokkosmd.F90:1244-1245`. Refused by design (`PATCH.md` section 28). +- (v) `mww3_test_07` fails in the Fortran run too: its `ww3_grid` stops with "ERROR IN READING SDS4 + FROM FILE, IOSTAT = 5010" (`work_F_61/ww3_grid_refug.out`), the entry's namelist read against the + lab switch's `ST4`. A WW3 input failure of both runs under the lab switch, as agent A found; the + row stays FAIL. +- (v) The `selection:count` test of item 2 failed: `regtests/CMakeLists.txt` has no expected count + for the word list `ST4;SHRD;!PDLIB` ("gives 10 omp ANY ST4;SHRD;!PDLIB entries, not ;"), so the + check had nothing to compare. All 10 entries ran and passed. The check was not changed. +- (v) `mww3_test_01` to `_04` set `FLSOU` false in every `ww3_grid*.inp`, the sixth flag of its `FLDRY ... FLSOU` line + (`F T T T F F` and similar; `mww3_test_05`'s are `F T T T F T`), so `SRCE=K` there computes no spectrum: these entries test the propagation, the + outputs and the `ww3_multi` exchanges on the resident `VA`, not `W3SRCE`. +- (v) Whole-`VA` copies against agent A's after-run on `80644ba9`'s fork branch (the `ww3_multi` log): + `mww3_test_01` 12 → 12, `_02` 8 → 0, `_03` 216 → 3, `_04` 0 → 0, `_05` 156 → 156 without `CPRT` + and 153 with it. None went up. `_03` lost its `W3OUTG` fallback: with `CPRT` in the preset, + `W3OUTG`'s partition fields run on Kokkos (219 `W3CPRT` calls). The 24 per run of `ww3_ts3` and + the 153 to 156 of `mww3_test_05` are the Fortran propagation's host write per step, `XYP3` and + `KTP3` being off on a moving grid. +- ⚠ No wall time here is a timing: the machine was shared, and none enters a table. + +**The restart pair (gap 10), designed and not run.** `scripts/restart_pair.sh`, on the `full` case +(`kokkos/tests/fixtures/bench_small`) built with `switch_lab_shrd`: + +```bash +WW3GPU_SWITCH=switch_lab_shrd bash scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos resident --build-only +WW_KOKKOS_VA_POISON=1 OMP_NUM_THREADS=4 nix develop picasso/labs/pratico#ww3 --command bash scripts/restart_pair.sh +``` + +It runs, with one executable, `fortran/a` and `kokkos/a` from 00:00 to 12:00 (nine fields every +hour from 06:00, a restart at 06:00 and 12:00) and `fortran/b` and `kokkos/b` from 06:00, each +from its own `a` run's 06:00 restart copied to `restart.ww3`. It compares with `cmp` `out_grd.ww3` +and the 12:00 restart of `b` against `a` in each mode, and every file of `kokkos/x` against +`fortran/x`; a Kokkos difference that the Fortran pair shows too is printed as WW3's own. Checked: +`bash -n`, `shellcheck` (clean) and `--self-test` (the namelist edits), nothing more ⚠. + +## What it changes + +- Gap 2 under `SHRD` holds on the integrated pin: the six `ww3_multi` entries that run give the + Fortran's bytes with the host copy poisoned, now with `CPRT` in the preset. `mww3_test_08` (MPI) + stays unrun (ADR-0017). +- Gap 10 stays open: no matrix entry restarts a `SHRD` case, and the pair above waits for the owner. +- `kokkos/ports.tsv`: the `w3iosfmd:w3cprt` cell names this matrix record and stays "no": whether + lab-switch evidence moves a row to "yes" is the owner's step of plan 002. No other cell changed. +- `scripts/restart_pair.sh` is new and changes no gate; a person reviews it before it merges + (`AGENTS.md`, "Never cheat a regtest tool"). The `selection:count` gap for custom word lists is + for the owner. diff --git a/docs/log/2026-10-18-ww3-multi-resident-va.md b/docs/log/2026-10-18-ww3-multi-resident-va.md new file mode 100644 index 00000000..aa96e367 --- /dev/null +++ b/docs/log/2026-10-18-ww3-multi-resident-va.md @@ -0,0 +1,118 @@ +# `ww3_multi` and the resident `VA`: the defect reproduced, column syncs at the exchanges, no read at `W3WAVE`'s return + +2026-10-18, an agent for M. Hoffmann (goal `abbb0456-6043`); unit U1 + U7 of +[the production single-node plan](../compound/plans/2026-10-18-002-production-single-node-plan.md), +gaps 2 and 13 of [`PRODUCTION_READINESS_202610.md`](../PRODUCTION_READINESS_202610.md), +[WFIP-0005](../WFIPs/WFIP-0005-device-resident-step.md). The fork change is +[`PATCH.md` section 26](../../kokkos/src/fortran_iface/PATCH.md#26-ww3_multi-and-the-resident-va-column-syncs-no-read-at-w3waves-return-ww3-gpuu1-multi-va). + +## Question + +Does `ww3_multi` change a grid's `VA` on the host between two `W3WAVE` calls behind the resident +device copy, as gap 2 read from the code? If so, can the syncs sit where `ww3_multi` reads and +writes `VA`, moving only those columns, with the whole-array read at `W3WAVE`'s return removed, +and still give the Fortran's bytes? + +## What was done + +On the owner's Intel Core i9-14900 (32 hardware threads, shared with three other sessions; at +most 8 threads used: 2 matrix jobs of 4), in the pinned `ww3` shell (gfortran and g++ 16.2.0), +Kokkos OpenMP backend, never `--gpu`. + +1. Before, `WW3/` at `80644ba9`: the `ww3_multi` entries that are `SHRD` with `PR3 UQ`, with the + lab switch's source terms, the resident preset and the VA diagnostics in the environment: + + ```bash + WW_KOKKOS_VA_CHECK=1 WW_KOKKOS_VA_COUNT=1 WW3GPU_OMP_THREADS=4 WW3GPU_LAB_SWITCH=ON \ + WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,!PDLIB,!MGP,!MGG' \ + just ww3-gpu-matrix 2 serial -R '^0(33|34|38|43|61):' + ``` + + then the same without `WW_KOKKOS_VA_CHECK`, and the moving-grid entries of `mww3_test_05`, + where `XYP3` and `KTP3` refuse (`w3kokkos_prop3md.F90:123-133`): `WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV + WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,MGP,!PDLIB' just ww3-gpu-matrix 2 omp -R '^00(2|8):'`, with + and without the check. The task's subset `SRCE,KTP3,OUTG,IOPE,UPDV` stops in `W3INIT` there, + `KTP3` refusing the moving grid (v, `WW_KOKKOS_KTP3=1, BUT ITS KOKKOS TRANSLATION DOES NOT COVER + THIS SWITCH FILE`), so `KTP3` was dropped from it. The before runs used `kokkos/` with the new + entry point already added; `80644ba9` does not call it. +2. The change, on the fork branch `ww3-gpu/u1-multi-va` (`3bbd1557` to `f330dc53`), and the shim's + `ww_va_host_wrote_cols` here (`PATCH.md` section 26). A first plan (B1) marked the host newer at + `W3WAVE`'s return; the owner rejected it the same day, since on a GPU it adds a whole copy of + `VA` per grid per call, and asked that the read at the return go too. +3. After, `WW3/` at `8f6ba104`, whose tree equals `f330dc53` (`diff --stat` empty): the same + entries with `WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_CHECK=1` and plain, all with + `WW_KOKKOS_VA_COUNT=1` (the script `run_set.sh` of the session, which only sets these + variables around the two commands above). +4. `ww3_shel`, before and after: `WW_KOKKOS_VA_COUNT=1 OMP_NUM_THREADS=8 JOBS=8 bash + scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos resident`, and after also `smoke`, and both with + `WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_CHECK=1`. +5. Without `KOKKOS`: `gfortran -E -P` of `w3wavemd.F90`, `wminiomd.F90`, `wmwavemd.F90` of the + fork and of `WW3-noaa/` with every word of `switches/switch_lab_shrd`, then of + `mww3_test_08`'s `switch_ST4_PR3_UQ_MPI`, blank lines dropped. With `KOKKOS` and that MPI + switch: `cmake -S -B build/mpi_check/b -DSWITCH= -DWW_KOKKOS=ON + -DCMAKE_PREFIX_PATH=kokkos/build/openmp-release/install` and `cmake --build build/mpi_check/b + -j 8 --target ww3_multi`. +6. `just kokkos-test serial-debug`; `bash scripts/ww3-gpu.sh --self-test`; `shellcheck + scripts/ww3-gpu.sh`. + +The summaries are in +[`ww3gpu_matrix_multi_va_2026-10-18.tsv`](../data/ww3gpu_matrix_multi_va_2026-10-18.tsv). + +## Result + +- (v) Gap 2 is real. Before, with `WW_KOKKOS_VA_CHECK=1`, `mww3_test_02`, `_03` and `_04` + (`grdset_a`) stop: "2 columns of the host VA of grid 1 changed since the last sync, and no + W3KOKKOS_VA_HOST_WRITE declared it" at a kernel's sync (02), 110 columns of grid 1 at a host read + (03), 240 columns of grid 2 at a kernel's sync (04). Without the check they run and differ: + `test.comp` 5/23, 10/34 and 4/25 files different, `ww3_self` 0 each (a Fortran/Fortran run + gives the same bits), `nccmp-tol` FAIL; in 02 `out_grd.coarse`, `out_pnt.coarse`, the + spectra, track and field NetCDF. `mww3_test_01` (one grid) passes, 0/22. +- (v) On the moving grid of `mww3_test_05` (`SRCE,OUTG,IOPE,UPDV`), the check stops both entries + (138 columns of grid 2, at a host write), and without it both pass, 0/27: the host write before + the Fortran propagation declares the host newer before any kernel reads `d_va`, so the defect is + hidden there, as gap 2 said it might be. +- (v) After: all six entries give `test.comp` 0 files different, plain, with the check, and with + the host copy poisoned, every part shown as `=K` in the parts line. +- (v) Whole-`VA` copies device to host per run, before → after (`WW_KOKKOS_VA_COUNT=1`, plain runs): + `mww3_test_01` 24 → 12, `_02` 384 → 8, `_03` 216 → 216, `_04` 864 → 0, `_05` 156 → 156; none + went up. In their place the exchanges moved columns: in `_04`, 384 372 columns read + (737.99 MB) and 71 388 stored (137.07 MB) over the run, against 3 002.99 MB of whole copies + before. The 216 of `_03` are the `W3OUTG` fallback inside `W3WAVE` (0 Kokkos `W3OUTG` calls in + that run), and the 156 of `_05` the Fortran propagation's host write: both inside `W3WAVE`. +- (v) `ww3_shel` full, after, plain and poisoned: every model file byte-identical to the Fortran's, + every file but the NetCDF to `--ww3`'s, `nccmp-tol` PASS; whole-`VA` D2H 1 → 0 (21.21 MB less + per run), H2D 1 → 1 (the bind). Smoke: the same verdicts, D2H 37 → 37 (the output readers of + `PATCH.md` section 23). The run ends with the device copy newer; `scripts/ww3-gpu.sh` failed on + that line, as mutant A2 of the mutation audit had made it, and now prints it. +- ⚠ Wall time of full, one run each on a shared machine: 40.86 s before, 43.99 s and 32.35 s after + (the second run poisoned). The extra column copies of `ww3_multi` were not timed; no conclusion. +- (v) Without `KOKKOS`, the three files preprocess to `WW3-noaa`'s text with both switches (the + include path in one comment aside). With `KOKKOS` and the MPI switch, `ww3_multi` builds; it + was not run (ADR-0017). `just kokkos-test serial-debug`: 11/11; the self-test and shellcheck + pass. +- (v) `mww3_test_07` was not compared: with the lab switch its `ww3_grid` stops in the Fortran + run too, reading its `SDS4` namelist. + +**`mww3_test_08`** (`ww3_multi`, `MPI`, `SRCE,KTP3`), read and not run (ADR-0017). Its +`ST4_PR3_UQ_MPI` entry uses `input/ww3_multi_grdset_a.inp`: grids `outer` rank 1 and `inner` +rank 2 in one group, so the inner grid feeds the outer one (`WMIOHS` reads the inner `VA`, +`WMIOHG` writes the outer `VA`) and the outer grid sends boundary spectra to the inner one +(`WMIOBS`). Under `DIST MPI` the batched `W3SRCE` and `W3KTP3` run on the device copy and +`XYP3` does not (v, `docs/log/2026-10-18-resident-mpi-lab-switch.md`). At `80644ba9`: `W3WAVE` +of the outer grid returns with "both equal", `WMIOHG` writes its overlap columns on the host, and +in the next step `W3KTP3` on the device reads `d_va` without them; the host write before the +Fortran `W3XYP3` (`w3wavemd.F90:2037`) then copies `d_va` back over the host and the nesting +update is lost. The inner grid's boundary, taken from those outer columns by `WMIOBS`, differs. +This is the defect fixed here, and it matches the six differing files there, all of the inner +grid ⚠: why the outer grid's own outputs did not differ (its overlap points are masked by +`FLGHG1`/`FLGHG2`) was not traced, and the entry was not run with the fix. + +## What it changes + +- Gap 2's blocker for `ww3_multi` under `SHRD` is closed on these entries; the fork branch + `ww3-gpu/u1-multi-va` waits for the integrator's pin bump. `ww_kokkos` is 0.10.0. +- `mww3_test_08` is the regtest that would show the same fix under MPI, when the owner runs MPI. +- Left inside `W3WAVE`, for the output-reader units: the `W3OUTG` fallback's whole read per step + (`mww3_test_03`), and the Fortran propagation's host write per step when `XYP3` refuses. +- The change to `scripts/ww3-gpu.sh` turns a gate's failure into a printed line; a person reviews + it before it merges (`AGENTS.md`, "Never cheat a regtest tool"). diff --git a/docs/log/README.md b/docs/log/README.md index d5860e9d..62dc56a1 100644 --- a/docs/log/README.md +++ b/docs/log/README.md @@ -67,3 +67,8 @@ Date, author, issue/PR/WFIP. | 2026-10-18 | [The resident preset under MPI: refused by design; SRCE and KTP3 alone run, one entry differs](2026-10-18-resident-mpi-lab-switch.md) | #42, WFIP-0003 | | 2026-10-18 | [Where the Kokkos resident run of `full` spends its wall time on the CPU](2026-10-18-resident-profile.md) | #42, WFIP-0005 | | 2026-10-18 | [Five bit-neutral speed-ups of the resident preset: three built, two checked, one Fortran change tested](2026-10-18-resident-five-speedups.md) | #42, WFIP-0005 | +| 2026-10-18 | [The CUDA build of `ww_kokkos` and of the fork for the resident preset, build only: 51 nvcc warnings, none in a wired part](2026-10-18-cuda-build-only.md) | plan 2026-10-18-002 U5, ADR-0014 | +| 2026-10-18 | [The boundary and track outputs read only their columns of the resident `VA`](2026-10-18-resident-output-readers.md) | #42, WFIP-0005 | +| 2026-10-18 | [`ww3_multi` and the resident `VA`: the defect reproduced, column syncs at the exchanges, no read at `W3WAVE`'s return](2026-10-18-ww3-multi-resident-va.md) | plan 2026-10-18-002 U1 + U7, WFIP-0005 | +| 2026-10-18 | [`W3CPRT` on the resident `VA`: partitioning without copying the spectrum back](2026-10-18-cprt-resident.md) | plan 2026-10-18-002 U9, WFIP-0005 | +| 2026-10-18 | [U3 on the integrated pin: the resident serial, OMPG, `ww3_multi` and `ww3_tnc1` entries; the restart pair designed, not run](2026-10-18-u3-regtests-v2.md) | plan 2026-10-18-002 U3, #191 | diff --git a/kokkos/CMakeLists.txt b/kokkos/CMakeLists.txt index 69894043..0917357f 100644 --- a/kokkos/CMakeLists.txt +++ b/kokkos/CMakeLists.txt @@ -9,13 +9,26 @@ # # SPDX-License-Identifier: MIT cmake_minimum_required(VERSION 3.25) -project(ww3_gpu_kokkos VERSION 0.9.0 LANGUAGES C CXX Fortran) +# OFF is the install-only configure a WW3 build links against: ww_kokkos and its package +# config, with Kokkos as the one dependency searched (no GoogleTest, no NetCDF, no tools, no +# intro programs, no C or Fortran compiler). ON, the default, is the whole lab tree. +option(WW_KOKKOS_BUILD_TESTS "Build the intro programs, the GoogleTest suites and the NetCDF tools" ON) +if(WW_KOKKOS_BUILD_TESTS) + set(ww_languages C CXX Fortran) +else() + set(ww_languages CXX) +endif() +project(ww3_gpu_kokkos VERSION 0.10.0 LANGUAGES ${ww_languages}) set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED ON) set(CMAKE_CXX_EXTENSIONS OFF) option(WW_ENABLE_FORTRAN "Build the Fortran interface module (W3KOKKOSMD from the WW3/ submodule)" ON) +if(NOT WW_KOKKOS_BUILD_TESTS) + # WW3 compiles W3KOKKOSMD in its own build; without Fortran enabled there is nothing to build it with. + set(WW_ENABLE_FORTRAN OFF CACHE BOOL "" FORCE) +endif() option(WW_DETERMINISTIC "Force serial reductions for bit-reproducible validation builds" OFF) include(cmake/CompilerWarnings.cmake) @@ -30,17 +43,18 @@ set(WW_KOKKOS_ARCH "" CACHE STRING "Kokkos architecture the found Kokkos must ha if(WW_KOKKOS_ARCH) kokkos_check(ARCH ${WW_KOKKOS_ARCH}) endif() -find_package(GTest REQUIRED) -find_package(PkgConfig REQUIRED) -pkg_check_modules(NETCDF REQUIRED IMPORTED_TARGET netcdf) -if(WW_ENABLE_FORTRAN) - pkg_check_modules(NETCDFF REQUIRED IMPORTED_TARGET netcdf-fortran) +if(WW_KOKKOS_BUILD_TESTS) + find_package(GTest REQUIRED) + find_package(PkgConfig REQUIRED) + pkg_check_modules(NETCDF REQUIRED IMPORTED_TARGET netcdf) + if(WW_ENABLE_FORTRAN) + pkg_check_modules(NETCDFF REQUIRED IMPORTED_TARGET netcdf-fortran) + endif() + # enable_testing() must run before any directory that calls add_test(), so it + # comes before intro/ and tests/ rather than after them. + enable_testing() endif() -# enable_testing() must run before any directory that calls add_test(), so it -# comes before intro/ and tests/ rather than after them. -enable_testing() - # Environment every CTest case runs with. Kokkos' OpenMP backend prints a warning # unless OMP_PROC_BIND is set explicitly, and a fixed, small thread count keeps the # suite from fighting CI for cores and keeps its output reproducible. @@ -63,11 +77,13 @@ add_subdirectory(src/ww_kokkos) # Not guarded by WW_ENABLE_FORTRAN: the C ABI is C++ and belongs in ww_kokkos # either way. The directory guards the Fortran module itself. add_subdirectory(src/fortran_iface) -add_subdirectory(intro) -add_subdirectory(tools/nccmp-tol) -add_subdirectory(tools/bench_case) -add_subdirectory(tools/fetch_analyse) -add_subdirectory(tests) +if(WW_KOKKOS_BUILD_TESTS) + add_subdirectory(intro) + add_subdirectory(tools/nccmp-tol) + add_subdirectory(tools/bench_case) + add_subdirectory(tools/fetch_analyse) + add_subdirectory(tests) +endif() # The package WAVEWATCH III finds with `find_package(ww_kokkos)` when it is built with the # KOKKOS switch (-DWW_KOKKOS=ON, the h0ffmann/WW3 branch ww3-gpu/kokkos-st4): the static @@ -80,12 +96,12 @@ install(DIRECTORY src/ DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} FILES_MATCHING PATTERN "*.hpp") install(EXPORT ww_kokkosTargets NAMESPACE ww_kokkos:: DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ww_kokkos) -file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/ww_kokkosConfig.cmake - "include(CMakeFindDependencyMacro)\nfind_dependency(Kokkos)\n" - "include(\"\${CMAKE_CURRENT_LIST_DIR}/ww_kokkosTargets.cmake\")\n") +# The archive holds Kokkos code compiled against one Kokkos: its version, backends and +# architecture are written into the config, which refuses any other (cmake/ww_kokkosConfig.cmake.in). +configure_file(cmake/ww_kokkosConfig.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/ww_kokkosConfig.cmake @ONLY) # Entry points are only added, never removed or changed, so an install satisfies any -# request of the same major version up to its own (0.9.0 accepts find_package(ww_kokkos -# 0.3), 0.4, 0.5, 0.6, 0.7 and 0.8). Removing or changing an entry point needs a major bump (PATCH.md, "The fork today"). +# request of the same major version up to its own (0.10.0 accepts find_package(ww_kokkos +# 0.3), 0.4, 0.5, 0.6, 0.7, 0.8 and 0.9). Removing or changing an entry point needs a major bump (PATCH.md, "The fork today"). write_basic_package_version_file(${CMAKE_CURRENT_BINARY_DIR}/ww_kokkosConfigVersion.cmake COMPATIBILITY SameMajorVersion) install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ww_kokkosConfig.cmake diff --git a/kokkos/PORT_RANKING.md b/kokkos/PORT_RANKING.md index 7a42367d..8a8a481b 100644 --- a/kokkos/PORT_RANKING.md +++ b/kokkos/PORT_RANKING.md @@ -92,7 +92,7 @@ The record is the `regtested` column of `kokkos/ports.tsv`. | `w3uqckmd:w3qck3` | ported | `prop_qck3.hpp` | no: no regtest matrix yet; the CPU replay ran it inside W3XYP3: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS; the transparency branches with obstructions were not checked for coverage | | `w3uqckmd:w3qck1` | ported | `prop_qck1.hpp` | no: no regtest matrix yet; the CPU replay ran it inside W3KTP3 (refraction on): scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS | | `w3pro3md:w3ktp3` | ported | `prop_ktp3.hpp`, `prop3_shim.cpp`, `prop3_c.hpp` | no: no regtest matrix yet; the CPU replay ran it: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS, 441 792 point calls in 48 calls, refraction only: FLCK and FLCUR are false in bench_small, so the wavenumber shift (W3QCK2) never ran; on the resident VA (ww_ktp3_batch_va, fork 90631fad, WFIP-0005 unit 3b, PATCH.md section 13) the same replay byte-identical on 2026-10-17 on switch_lab_ompg and switch_lab_shrd, with and without WW_KOKKOS_VA_RESIDENT=1, VA traffic H2D 1 / D2H 13 per full run | -| `w3dispmd:wavnu1` | ported | `prop_wavnu1.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist | +| `w3dispmd:wavnu1` | ported | `prop_wavnu1.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md); only PTMEAN's call, not W3INIT's or W3ULEV's | | `w3uqckmd:w3qck2` | ported | `prop_qck2.hpp` | no: never run: W3KTP3 calls it only when FLCK is true, and the full and smoke cases have no current-induced wavenumber shift | | `w3wavemd:w3gath` | ported | `wave_gath_scat.hpp`, `wave_support_shim.cpp` | yes: scripts/ww3-gpu.sh --ww3-gpu full (bench_small) --cpu on 2026-10-16 (OpenMP backend, gfortran/g++ 16.2.0), with WW_KOKKOS_ALL=1 and with the W3WAVE support parts alone: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, the binary files byte-identical to --ww3, 13 824 calls (kokkos/src/fortran_iface/PATCH.md, section 8) | | `w3wavemd:w3scat` | ported | `wave_gath_scat.hpp`, `wave_support_shim.cpp` | yes: scripts/ww3-gpu.sh --ww3-gpu full (bench_small) --cpu on 2026-10-16 (OpenMP backend, gfortran/g++ 16.2.0), with WW_KOKKOS_ALL=1 and with the W3WAVE support parts alone: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, the binary files byte-identical to --ww3, 13 824 calls (kokkos/src/fortran_iface/PATCH.md, section 8) | @@ -126,12 +126,13 @@ The record is the `regtested` column of `kokkos/ports.tsv`. | `pdlib_w3profsmd:apply_boundary_condition` | ported | `pdlib_boundary.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim with the MPI hooks | | `pdlib_w3profsmd:pdlib_w3xypfsfct2` | ported | `pdlib_fsfct2.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); no regtest selects EXPFSFCT | | `pdlib_w3profsmd:pdlib_w3xypug` | ported | `pdlib_w3xypug.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3WAVE hunk | -| `w3partmd:ptnghb` | ported | `part_ptnghb.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk | -| `w3partmd:pt_fld` | ported | `part_fld.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk | -| `w3partmd:ptsort` | ported | `part_ptsort.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk | -| `w3partmd:ptmean` | ported | `part_ptmean.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs W3PART and a shim | +| `w3partmd:ptnghb` | ported | `part_ptnghb.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3partmd:pt_fld` | ported | `part_fld.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3partmd:ptsort` | ported | `part_ptsort.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3partmd:ptmean` | ported | `part_ptmean.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | | `w3snl3md:insnl3` | ported | `snl3_tables.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3IOGR hunk | -| `w3partmd:w3part` | ported | `part_w3part.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3IOSF hunk | +| `w3partmd:w3part` | ported | `part_w3part.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3iosfmd:w3cprt` | ported | `iosf_w3cprt.hpp`, `cprt_shim.cpp` | no: the owner decides whether lab-switch evidence counts (plan 2026-10-18-002); the WW3 regtest matrix ran it on 2026-10-18 at WW3 50a10730 with WW3GPU_LAB_SWITCH=ON and WW_KOKKOS_VA_POISON=1: ww3_tnc1 (SRCE,OUTG,IOPE,UPDV,CPRT) test.comp 0/30, 3 Kokkos calls; mww3_test_03 grdset_a (resident) 0/34, 219 calls; the five mww3_test_05 ST4 OMPG entries (SRCE,OUTG,IOPE,UPDV,CPRT) 0/27 each, 39 calls each (docs/log/2026-10-18-u3-regtests-v2.md, docs/data/ww3gpu_matrix_*_pin-50a10730_2026-10-18.tsv); before that the CPU replay ran it on 2026-10-18: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT on two partition variants of bench_small (switch_lab_shrd, and switch_lab_ompg for the fields variant), WW_KOKKOS_VA_POISON=1: every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 (docs/log/2026-10-18-cprt-resident.md) | | `w3snl3md:w3snl3` | ported | `snl3_gmd.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3SRCE hunk | | `w3src0md:w3spr0` | ported | `srce_st0_spr0.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests | | `w3src1md:w3spr1` | ported | `srce_st1_spr1.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests | @@ -259,9 +260,9 @@ The record is the `regtested` column of `kokkos/ports.tsv`. | `w3uostmd:uost_sourceterm_setgrid` | ported | `uost_uost_sourceterm_setgrid.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | | `w3uostmd:uost_srctrmcompute` | ported | `uost_uost_srctrmcompute.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | | `w3wavset:fd_wave_setup_apply_fct` | ported | `wavset_fd_wave_setup_apply_fct.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | -| `w3partmd:fifo_add` | ported | `part_fld.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | -| `w3partmd:fifo_first` | ported | `part_fld.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | -| `w3partmd:fifo_empty` | ported | `part_fld.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | +| `w3partmd:fifo_add` | ported | `part_fld.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3partmd:fifo_first` | ported | `part_fld.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3partmd:fifo_empty` | ported | `part_fld.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | | `w3wavset:differentiate_xydir` | ported | `wavset_differentiate_xydir.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | | `w3wavset:fd_wave_setup_scalar_prod` | ported | `wavset_fd_wave_setup_scalar_prod.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | | `w3wavset:trig_compute_diff` | ported | `wavset_trig_compute_diff.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | @@ -384,3 +385,5 @@ The record is the `regtested` column of `kokkos/ports.tsv`. | `w3pro2md:w3xyp2` | ported | `prop_xyp2.hpp`, `prop_params2.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist | | `w3pro2md:w3ktp2` | ported | `prop_ktp2.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist | | `w3profsmd:w3xypug` | ported | `prop_xypug.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once the four scheme hooks, a shim and the W3WAVE wiring exist | +| `wminiomd:wmiohg` | ported | `resident_shim.cpp` | yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), WW3GPU_LAB_SWITCH=ON matrix entries mww3_test_01 to _04 (resident) and mww3_test_05 (SRCE,OUTG,IOPE,UPDV), test.comp 0 files different, plain and with WW_KOKKOS_VA_POISON=1 (docs/log/2026-10-18-ww3-multi-resident-va.md) | +| `wminiomd:wmioeg` | ported | `resident_shim.cpp` | yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), the entries of WMIOHG, test.comp 0 files different; whether they reach 2.b was not checked | diff --git a/kokkos/PORT_STATUS.md b/kokkos/PORT_STATUS.md index 66fd9add..d44e49c7 100644 --- a/kokkos/PORT_STATUS.md +++ b/kokkos/PORT_STATUS.md @@ -45,11 +45,11 @@ Profile: full (bench_small, 120 x 80, 9 520 sea points, 12 h), Release -pg, swit | level | run time | lines | routines | files | what it counts (WW3@03adda91) | |---|---|---|---|---|---| -| translated | **99.3 %** | 67699 of 263375 (25.70 %) | 310 of 1023 (30.30 %) | 63 of 147 | a Kokkos translation exists in kokkos/src | -| ported | **99.3 %** | 17022 of 263375 (6.46 %) | 60 of 1023 (5.87 %) | 23 of 147 | WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag | -| regtested | **48.6 %** | 2677 of 263375 (1.02 %) | 10 of 1023 (0.98 %) | 5 of 147 | WW3's regtests ran with it on Kokkos and agreed with the Fortran | +| translated | **99.3 %** | 67947 of 263375 (25.80 %) | 313 of 1023 (30.60 %) | 65 of 147 | a Kokkos translation exists in kokkos/src | +| ported | **99.3 %** | 17270 of 263375 (6.56 %) | 63 of 1023 (6.16 %) | 25 of 147 | WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag | +| regtested | **48.6 %** | 2689 of 263375 (1.02 %) | 12 of 1023 (1.17 %) | 6 of 147 | WW3's regtests ran with it on Kokkos and agreed with the Fortran | -The port itself: 58497 lines in 347 files, the C++ under kokkos/src (every .cpp and .hpp line). +The port itself: 59511 lines in 350 files, the C++ under kokkos/src (every .cpp and .hpp line). On a GPU: none yet. The CUDA build links; the RTX 4090 run, `just smoke-ww3-gpu gpu`, is the owner's, supervised. diff --git a/kokkos/PORT_TABLE.tsv b/kokkos/PORT_TABLE.tsv index f06dee59..3953338c 100644 --- a/kokkos/PORT_TABLE.tsv +++ b/kokkos/PORT_TABLE.tsv @@ -26,78 +26,77 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi 18 w3initmd:w3mpip model/src/w3initmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — 19 w3iogomd:w3readflgrd model/src/w3iogomd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — 20 w3iopomd:nf90_err_check model/src/w3iopomd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -21 w3iosfmd:w3cprt model/src/w3iosfmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -22 w3nmlshelmd:read_domain_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -23 w3nmlshelmd:read_homogeneous_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -24 w3nmlshelmd:read_input_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -25 w3nmlshelmd:read_output_date_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -26 w3nmlshelmd:read_output_path_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -27 w3nmlshelmd:read_output_type_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -28 w3nmlshelmd:report_domain_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -29 w3nmlshelmd:report_homogeneous_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -30 w3nmlshelmd:report_input_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -31 w3nmlshelmd:report_output_date_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -32 w3nmlshelmd:report_output_path_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -33 w3nmlshelmd:report_output_type_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -34 w3nmlshelmd:w3nmlshel model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -35 w3odatmd:w3dmo2 model/src/w3odatmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -36 w3odatmd:w3dmo5 model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -37 w3odatmd:w3nout model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -38 w3odatmd:w3seto model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -39 w3parall:synchronize_ipgl_etc_array model/src/w3parall.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -40 w3servmd:itrace model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -41 w3servmd:nextln model/src/w3servmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -42 w3servmd:print_memcheck model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -43 w3servmd:str_to_upper model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -44 w3servmd:strsplit model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -45 w3servmd:wwdate model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -46 w3servmd:wwtime model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -47 w3timemd:d2j model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -48 w3timemd:dsec21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -49 w3timemd:stme21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -50 w3timemd:t2d model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -51 w3timemd:tdiff model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -52 w3timemd:tick21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -53 w3triamd:coordmax model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -54 w3triamd:fix_periodcity model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -55 w3triamd:is_in_ungrid model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -56 w3triamd:nvectri model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -57 w3triamd:spatial_grid model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -58 w3triamd:ug_gradients model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -59 w3triamd:w3nestug model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -60 w3updtmd:w3ubpt model/src/w3updtmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -61 w3updtmd:w3ucur model/src/w3updtmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -62 w3wdatmd:w3dimw model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -63 w3wdatmd:w3ndat model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -64 w3wdatmd:w3setw model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -65 yowelementpool:belongto model/src/yowelementpool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -66 yowexchangemodule:creatempitypes model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -67 yowexchangemodule:finalizeexchangemodule model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -68 yowexchangemodule:initnbrdomains model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -69 yowexchangemodule:pdlib_exchange1dreal model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -70 yowexchangemodule:pdlib_exchange2dreal_zero model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -71 yowexchangemodule:setdimsize model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -72 yowfunction:computelistnp_listnpa_listiplg model/src/yowfunction.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -73 yowfunction:computelistnp_listnpa_listiplg_kernel model/src/yowfunction.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -74 yownodepool:connnodes model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -75 yownodepool:insertconnnode model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -76 yownodepool:nodes model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -77 yowpdlibmain:assignmesh model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -78 yowpdlibmain:computeia_ja_posi_nnz model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -79 yowpdlibmain:computetria_ien_si_ccon model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -80 yowpdlibmain:exchangeghostids model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -81 yowpdlibmain:findconndomains model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -82 yowpdlibmain:findconnnodes model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -83 yowpdlibmain:findghostnodes model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -84 yowpdlibmain:initfromgriddim model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -85 yowpdlibmain:initmpi model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -86 yowpdlibmain:postpartition model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -87 yowpdlibmain:postpartition2 model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -88 yowpdlibmain:prepartition model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -89 yowpdlibmain:real_mpi_barrier_pdlib model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -90 yowpdlibmain:runparmetis model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -91 yowrankmodule:exchangeiplg model/src/yowrankmodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -92 yowrankmodule:initrankmodule model/src/yowrankmodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +21 w3nmlshelmd:read_domain_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +22 w3nmlshelmd:read_homogeneous_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +23 w3nmlshelmd:read_input_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +24 w3nmlshelmd:read_output_date_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +25 w3nmlshelmd:read_output_path_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +26 w3nmlshelmd:read_output_type_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +27 w3nmlshelmd:report_domain_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +28 w3nmlshelmd:report_homogeneous_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +29 w3nmlshelmd:report_input_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +30 w3nmlshelmd:report_output_date_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +31 w3nmlshelmd:report_output_path_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +32 w3nmlshelmd:report_output_type_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +33 w3nmlshelmd:w3nmlshel model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +34 w3odatmd:w3dmo2 model/src/w3odatmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +35 w3odatmd:w3dmo5 model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +36 w3odatmd:w3nout model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +37 w3odatmd:w3seto model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +38 w3parall:synchronize_ipgl_etc_array model/src/w3parall.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +39 w3servmd:itrace model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +40 w3servmd:nextln model/src/w3servmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +41 w3servmd:print_memcheck model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +42 w3servmd:str_to_upper model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +43 w3servmd:strsplit model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +44 w3servmd:wwdate model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +45 w3servmd:wwtime model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +46 w3timemd:d2j model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +47 w3timemd:dsec21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +48 w3timemd:stme21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +49 w3timemd:t2d model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +50 w3timemd:tdiff model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +51 w3timemd:tick21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +52 w3triamd:coordmax model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +53 w3triamd:fix_periodcity model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +54 w3triamd:is_in_ungrid model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +55 w3triamd:nvectri model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +56 w3triamd:spatial_grid model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +57 w3triamd:ug_gradients model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +58 w3triamd:w3nestug model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +59 w3updtmd:w3ubpt model/src/w3updtmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +60 w3updtmd:w3ucur model/src/w3updtmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +61 w3wdatmd:w3dimw model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +62 w3wdatmd:w3ndat model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +63 w3wdatmd:w3setw model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +64 yowelementpool:belongto model/src/yowelementpool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +65 yowexchangemodule:creatempitypes model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +66 yowexchangemodule:finalizeexchangemodule model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +67 yowexchangemodule:initnbrdomains model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +68 yowexchangemodule:pdlib_exchange1dreal model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +69 yowexchangemodule:pdlib_exchange2dreal_zero model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +70 yowexchangemodule:setdimsize model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +71 yowfunction:computelistnp_listnpa_listiplg model/src/yowfunction.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +72 yowfunction:computelistnp_listnpa_listiplg_kernel model/src/yowfunction.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +73 yownodepool:connnodes model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +74 yownodepool:insertconnnode model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +75 yownodepool:nodes model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +76 yowpdlibmain:assignmesh model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +77 yowpdlibmain:computeia_ja_posi_nnz model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +78 yowpdlibmain:computetria_ien_si_ccon model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +79 yowpdlibmain:exchangeghostids model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +80 yowpdlibmain:findconndomains model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +81 yowpdlibmain:findconnnodes model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +82 yowpdlibmain:findghostnodes model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +83 yowpdlibmain:initfromgriddim model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +84 yowpdlibmain:initmpi model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +85 yowpdlibmain:postpartition model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +86 yowpdlibmain:postpartition2 model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +87 yowpdlibmain:prepartition model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +88 yowpdlibmain:real_mpi_barrier_pdlib model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +89 yowpdlibmain:runparmetis model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +90 yowrankmodule:exchangeiplg model/src/yowrankmodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +91 yowrankmodule:initrankmodule model/src/yowrankmodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — — w3snl1md:w3snl1 model/src/w3snl1md.F90:115-473 48.35 full regtested — yes: kokkos/src/ww_kokkos/snl1_dia.cpp, kokkos/src/fortran_iface/snl1_shim.cpp yes: WW3@c9a6a0c6 (h0ffmann/WW3 ww3-gpu/kokkos-st4, which carries the hoff/kokkos-snl1 hunks) calls it under WW_KOKKOS_SNL1=1 passed yes byte-identical on all 100 (test.comp); byte-identical on all 25 (test.comp) nccmp-tol 71 PASS, 29 n/a, 0 FAIL; nccmp-tol 25 PASS none: ADR-0010 kokkos/port_report/W3SNL1_KOKKOS.md — pdlib_w3profsmd:pdlib_jacobi_gauss_seidel_block model/src/w3profsmd_pdlib.F90:5463-6406 37.54 pdlib translated — yes: kokkos/src/ww_kokkos/pdlib_jgs_block.hpp no: header-only; no shim and no W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/PDLIB_JACOBI_GAUSS_SEIDEL_BLOCK_KOKKOS.md — w3src4md:w3sds4 model/src/w3src4md.F90:2043-2659 23.23 full wired — yes: kokkos/src/ww_kokkos/st4_sds4.hpp,kokkos/src/ww_kokkos/st4_libmvec.hpp,kokkos/src/fortran_iface/st4_shim.cpp yes: WW3@c9a6a0c6 (h0ffmann/WW3 ww3-gpu/kokkos-st4) calls it under WW_KOKKOS_SDS4/SIN4/SPR4=1 pending no — — none: ADR-0010 kokkos/port_report/W3SDS4_KOKKOS.md @@ -119,15 +118,15 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3wavemd:w3wave model/src/w3wavemd.F90:229-2997 0.15 pdlib translated — yes: kokkos/src/ww_kokkos/wave_w3wave.hpp,kokkos/src/ww_kokkos/wave_w3wave_state.hpp,kokkos/src/ww_kokkos/wave_w3wave_hooks.hpp no: header-only host driver; no shim, no hook table filled, no W3WAVE hunk on the fork pending no — — none: ADR-0010 kokkos/port_report/W3WAVE_KOKKOS.md — w3sln1md:w3sln1 model/src/w3sln1md.F90:56-211 0.13 full wired — yes: kokkos/src/ww_kokkos/srce_ln1.hpp yes: inside W3SRCE/WW_KOKKOS_SRCE at 357f575c pending no — — none: ADR-0010 kokkos/port_report/W3SLN1_KOKKOS.md — w3wavemd:w3gath model/src/w3wavemd.F90:3024-3329 0.11 full regtested — yes: kokkos/src/ww_kokkos/wave_gath_scat.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_GATH=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/W3GATH_KOKKOS.md -— w3partmd:pt_fld model/src/w3partmd.F90:839-1156 0.05 pdlib translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no W3PART hunk yet pending no — — none: ADR-0010 kokkos/port_report/PT_FLD_KOKKOS.md +— w3partmd:pt_fld model/src/w3partmd.F90:839-1156 0.05 pdlib translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/PT_FLD_KOKKOS.md — w3iogomd:w3outg model/src/w3iogomd.F90:1201-2380 0.04 pdlib wired — yes: kokkos/src/ww_kokkos/iogo_w3outg.hpp,kokkos/src/fortran_iface/outg_shim.cpp,kokkos/src/fortran_iface/outg_c.hpp yes: WW_KOKKOS_OUTG (opt-in, not under WW_KOKKOS_ALL), in place of CALL W3OUTG at w3wavemd.F90 section 4.b (W3KOKKOS_OUTG in w3kokkos_outgmd.F90, ww_outg_va in kokkos/src/fortran_iface/outg_shim.cpp) at e0cc8ee9, on the resident VA; the Fortran W3OUTG when partitions, CALC_U3STOKES, SKEWNESS or CALC_WBT fields are requested pending no — — none: ADR-0010 kokkos/port_report/W3OUTG_KOKKOS.md -— w3partmd:ptmean model/src/w3partmd.F90:1175-1538 0.02 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptmean.hpp no: header-only; no shim and no W3PART/W3IOSF hunk yet pending no — — none: ADR-0010 kokkos/port_report/PTMEAN_KOKKOS.md -— w3partmd:ptsort model/src/w3partmd.F90:511-613 0.02 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptsort.hpp no: header-only; no shim and no W3PART hunk yet pending no — — none: ADR-0010 kokkos/port_report/PTSORT_KOKKOS.md +— w3partmd:ptmean model/src/w3partmd.F90:1175-1538 0.02 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptmean.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/PTMEAN_KOKKOS.md +— w3partmd:ptsort model/src/w3partmd.F90:511-613 0.02 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptsort.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/PTSORT_KOKKOS.md — pdlib_w3profsmd:apply_boundary_condition_va model/src/w3profsmd_pdlib.F90:5037-5148 0.01 pdlib translated — yes: kokkos/src/ww_kokkos/pdlib_boundary_va.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/APPLY_BOUNDARY_CONDITION_VA_KOKKOS.md — w3adatmd:w3dima model/src/w3adatmd.F90:847-1509 0.01 full translated — yes: kokkos/src/ww_kokkos/adat_w3dima.hpp,kokkos/src/ww_kokkos/adat_wadat.hpp no: host-side allocation routine; no shim and no W3INIT/W3IOGO hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3DIMA_KOKKOS.md — w3gdatmd:w3dims model/src/w3gdatmd.F90:1929-2154 0.01 full translated — yes: kokkos/src/ww_kokkos/gdat_w3dims.hpp,kokkos/src/ww_kokkos/gdat_sgrd.hpp no: host-only header; no shim and no W3IOGR hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3DIMS_KOKKOS.md — w3parall:init_get_isea model/src/w3parall.F90:1396-1504 0.01 pdlib translated — yes: kokkos/src/ww_kokkos/parall_init_get_isea.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/INIT_GET_ISEA_KOKKOS.md -— w3partmd:w3part model/src/w3partmd.F90:138-497 0.01 pdlib translated — yes: kokkos/src/ww_kokkos/part_w3part.hpp no: header-only; no shim and no W3IOSF hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3PART_KOKKOS.md +— w3partmd:w3part model/src/w3partmd.F90:138-497 0.01 pdlib translated — yes: kokkos/src/ww_kokkos/part_w3part.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/W3PART_KOKKOS.md — w3src4md:calc_ustar model/src/w3src4md.F90:1849-2011 0.01 pdlib wired — yes: kokkos/src/ww_kokkos/st4_calc_ustar.hpp yes: WW3@c9a6a0c6 (h0ffmann/WW3 ww3-gpu/kokkos-st4) calls it under WW_KOKKOS_SDS4/SIN4/SPR4=1 pending no — — none: ADR-0010 kokkos/port_report/CALC_USTAR_KOKKOS.md — w3updtmd:w3ulev model/src/w3updtmd.F90:2066-2614 0.01 pdlib translated — yes: kokkos/src/ww_kokkos/updt_w3ulev.hpp no: header-only launcher; no shim and no W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3ULEV_KOKKOS.md — w3updtmd:w3uwnd model/src/w3updtmd.F90:488-822 0.01 full regtested — yes: kokkos/src/ww_kokkos/updt_uwnd.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_UWND=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/W3UWND_KOKKOS.md @@ -149,7 +148,7 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3adatmd:w3xdma model/src/w3adatmd.F90:1522-2409 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/adat_w3xdma.hpp,kokkos/src/ww_kokkos/adat_w3xdma_wadat.hpp no: host-side allocation routine; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3XDMA_KOKKOS.md — w3adatmd:w3xeta model/src/w3adatmd.F90:3112-3300 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/adat_w3xeta.hpp no: host-side pointer selection; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3XETA_KOKKOS.md — w3dispmd:distab model/src/w3dispmd.F90:552-683 0.00 full regtested — yes: kokkos/src/ww_kokkos/disp_distab.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_DISTAB=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/DISTAB_KOKKOS.md -— w3dispmd:wavnu1 model/src/w3dispmd.F90:85-202 0.00 full translated — yes: kokkos/src/ww_kokkos/prop_wavnu1.hpp no: header-only device function; no shim and no W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/WAVNU1_KOKKOS.md +— w3dispmd:wavnu1 model/src/w3dispmd.F90:85-202 0.00 full translated — yes: kokkos/src/ww_kokkos/prop_wavnu1.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/WAVNU1_KOKKOS.md — w3dispmd:wavnu2 model/src/w3dispmd.F90:204-346 0.00 full regtested — yes: kokkos/src/ww_kokkos/disp_distab.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it inside DISTAB under WW_KOKKOS_DISTAB=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/WAVNU2_KOKKOS.md — w3dispmd:wavnu3 model/src/w3dispmd.F90:348-454 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/disp_wavnu3.hpp no: header-only device function; no shim yet pending no — — none: ADR-0010 kokkos/port_report/WAVNU3_KOKKOS.md — w3fldsmd:w3fldg model/src/w3fldsmd.F90:935-1474 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/flds_w3fldg.hpp no: host-staged field I/O routine; no shim and no ww3_shel hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3FLDG_KOKKOS.md @@ -169,10 +168,11 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3iopomd:w3iopo model/src/w3iopomd.F90:2131-2626 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/iopo_w3iopo.hpp,kokkos/src/ww_kokkos/iopo_state.hpp no: header-only host driver; no shim and no W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3IOPO_KOKKOS.md — w3iopomd:w3iopp model/src/w3iopomd.F90:242-859 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/iopo_w3iopp.hpp,kokkos/src/ww_kokkos/iopo_state.hpp no: header-only host driver; no shim and no W3INIT hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3IOPP_KOKKOS.md — w3iorsmd:w3iors model/src/w3iorsmd.F90:113-1600 0.00 full translated — yes: kokkos/src/ww_kokkos/iors_w3iors.hpp,kokkos/src/ww_kokkos/iors_stream.hpp no: header-only host driver; no shim and no W3INIT or W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3IORS_KOKKOS.md +— w3iosfmd:w3cprt model/src/w3iosfmd.F90:109-344 0.00 pdlib wired — yes: kokkos/src/ww_kokkos/iosf_w3cprt.hpp, kokkos/src/fortran_iface/cprt_shim.cpp yes: WW3/ pin a304018c, w3wavemd.F90:2692 (W3KOKKOS_CPRT under WW_KOKKOS_CPRT=1 and in the resident preset) pending no — — none: ADR-0010 kokkos/port_report/W3CPRT_KOKKOS.md — w3parall:get_jsea_ibelong model/src/w3parall.F90:1265-1384 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/parall_get_jsea_ibelong.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/GET_JSEA_IBELONG_KOKKOS.md — w3parall:init_get_jsea_isproc model/src/w3parall.F90:1152-1252 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/parall_init_get_jsea_isproc.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/INIT_GET_JSEA_ISPROC_KOKKOS.md — w3parall:set_up_nseal_nsealm model/src/w3parall.F90:1030-1139 0.00 full translated — yes: kokkos/src/ww_kokkos/parall_set_up_nseal_nsealm.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/SET_UP_NSEAL_NSEALM_KOKKOS.md -— w3partmd:ptnghb model/src/w3partmd.F90:630-821 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptnghb.hpp no: header-only; no shim and no W3PART hunk yet pending no — — none: ADR-0010 kokkos/port_report/PTNGHB_KOKKOS.md +— w3partmd:ptnghb model/src/w3partmd.F90:630-821 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptnghb.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/PTNGHB_KOKKOS.md — w3pro3md:w3map3 model/src/w3pro3md.F90:139-518 0.00 full regtested — yes: kokkos/src/ww_kokkos/prop_map3.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_MAP3=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/W3MAP3_KOKKOS.md — w3pro3md:w3mapt model/src/w3pro3md.F90:527-625 0.00 full regtested — yes: kokkos/src/ww_kokkos/prop_mapt.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_MAPT=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/W3MAPT_KOKKOS.md — w3sbt4md:insbt4 model/src/w3sbt4md.F90:145-231 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/srce_bt4.hpp no: header-only; no shim and no W3SRCE hunk yet pending no — — none: ADR-0010 kokkos/port_report/INSBT4_KOKKOS.md @@ -213,9 +213,9 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3iogomd:vplus_d model/src/w3iogomd.F90:4820-4860 — — translated — yes: kokkos/src/ww_kokkos/iogo_vplus_d.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/VPLUS_D_KOKKOS.md — w3parall:prop_freq_shift model/src/w3parall.F90:604-739 — — translated — yes: kokkos/src/ww_kokkos/prop_freq_shift.hpp no: header-only device function; no shim yet pending no — — none: ADR-0010 kokkos/port_report/PROP_FREQ_SHIFT_KOKKOS.md — w3parall:prop_refraction_pr1 model/src/w3parall.F90:285-431 — — translated — yes: kokkos/src/ww_kokkos/parall_prop_refraction_pr1.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/PROP_REFRACTION_PR1_KOKKOS.md -— w3partmd:fifo_add model/src/w3partmd.F90:1103-1113 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/FIFO_ADD_KOKKOS.md -— w3partmd:fifo_empty model/src/w3partmd.F90:1122-1133 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/FIFO_EMPTY_KOKKOS.md -— w3partmd:fifo_first model/src/w3partmd.F90:1142-1152 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/FIFO_FIRST_KOKKOS.md +— w3partmd:fifo_add model/src/w3partmd.F90:1103-1113 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/FIFO_ADD_KOKKOS.md +— w3partmd:fifo_empty model/src/w3partmd.F90:1122-1133 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/FIFO_EMPTY_KOKKOS.md +— w3partmd:fifo_first model/src/w3partmd.F90:1142-1152 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/FIFO_FIRST_KOKKOS.md — w3pro1md:w3ktp1 model/src/w3pro1md.F90:900-1230 — — translated — yes: kokkos/src/ww_kokkos/prop_ktp1.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_KTP1=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) pending no — — none: ADR-0010 kokkos/port_report/W3KTP1_KOKKOS.md — w3pro1md:w3xyp1 model/src/w3pro1md.F90:301-873 — — translated — yes: kokkos/src/ww_kokkos/prop_xyp1.hpp,kokkos/src/ww_kokkos/prop_params1.hpp,kokkos/src/ww_kokkos/prop_params.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_XYP1=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) pending no — — none: ADR-0010 kokkos/port_report/W3XYP1_KOKKOS.md — w3pro2md:w3ktp2 model/src/w3pro2md.F90:1306-1710 — — translated — yes: kokkos/src/ww_kokkos/prop_ktp2.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_KTP2=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) pending no — — none: ADR-0010 kokkos/port_report/W3KTP2_KOKKOS.md @@ -408,3 +408,5 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3wavset:trig_compute_lh_stress model/src/w3wavset.F90:448-584 — — translated — yes: kokkos/src/ww_kokkos/wavset_trig_compute_lh_stress.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/TRIG_COMPUTE_LH_STRESS_KOKKOS.md — w3wavset:trig_wave_setup_apply_fct model/src/w3wavset.F90:1020-1107 — — translated — yes: kokkos/src/ww_kokkos/wavset_trig_wave_setup_apply_fct.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/TRIG_WAVE_SETUP_APPLY_FCT_KOKKOS.md — w3wavset:trig_wave_setup_apply_precond model/src/w3wavset.F90:892-1005 — — translated — yes: kokkos/src/ww_kokkos/wavset_trig_wave_setup_apply_precond.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/TRIG_WAVE_SETUP_APPLY_PRECOND_KOKKOS.md +— wminiomd:wmioeg model/src/wminiomd.F90:3258-3262 — — regtested — yes: kokkos/src/fortran_iface/resident_shim.cpp yes: WW3/ pin a304018c, wminiomd.F90:3341-3363 and :3457-3463 (column read before 2.a, column write after 2.b, under the resident VA) passed yes not recorded — none: ADR-0010 kokkos/port_report/WMIOEG_KOKKOS.md +— wminiomd:wmiohg model/src/wminiomd.F90:2260-2266 — — regtested — yes: kokkos/src/fortran_iface/resident_shim.cpp yes: WW3/ pin a304018c, wminiomd.F90:2318-2327 (W3KOKKOS_VA_WRITE_COLS after the store loop, under the resident VA) passed yes not recorded — none: ADR-0010 kokkos/port_report/WMIOHG_KOKKOS.md diff --git a/kokkos/README.md b/kokkos/README.md index b44da58a..4bb7aa66 100644 --- a/kokkos/README.md +++ b/kokkos/README.md @@ -154,7 +154,7 @@ Trans. Parallel Distrib. Syst.* 33(4), 805–817, 2022, ``` CMakeLists.txt top level: options, find_package, the ww_kokkos target CMakePresets.json serial-debug, openmp-release, cuda-release, cuda-h100-release -cmake/ CompilerWarnings.cmake -- the one warning policy +cmake/ CompilerWarnings.cmake (the one warning policy), ww_kokkosConfig.cmake.in src/ww_kokkos/ real.hpp (float32 + helpers), spectrum_fixtures.hpp (JONSWAP, cos^2), snl1_{config,tables,dia} (the DIA port), fixture_io (fixture reader) src/fortran_iface/ ww_kokkos_c.hpp + snl1_shim.cpp (the C ABI, built into ww_kokkos), @@ -209,6 +209,11 @@ on the search path. `WW_KOKKOS_ARCH=HOPPER90` makes the configure call Kokkos' ` backends and/or architectures requested" (v, 2026-10-17) instead of building `sm_89` code under an H100 name. `scripts/ww3-gpu.sh --gpu` still builds `cuda-release`. +`just kokkos-cuda-test` ends with `ctest` on the CUDA default execution space, which is a device +run and the owner's. To build only: `cmake -S kokkos --preset cuda-release` and +`cmake --build kokkos/build/cuda-release` in the `#cuda` shell. The owner's supervised GPU run +follows [`docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md`](../docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md). + Two things the names do *not* promise: - **The backend comes from the shell, not from the preset.** OpenMP is the default @@ -227,6 +232,50 @@ Two things the names do *not* promise: |---|---|---| | `WW_ENABLE_FORTRAN` | `ON` | build the Fortran interface module `W3KOKKOSMD` and `intro_06` | | `WW_DETERMINISTIC` | `OFF` | pin reductions to a fixed summation order for bit-reproducible validation runs | +| `WW_KOKKOS_BUILD_TESTS` | `ON` | build the intro programs, the GoogleTest suites and the NetCDF tools; `OFF` is the install-only configure below | + +### Install-only build: `ww_kokkos` with Kokkos as its one dependency + +WW3 built with the `KOKKOS` switch links the installed `ww_kokkos` package +(`find_package(ww_kokkos 0.9 REQUIRED)`, v, `WW3/model/src/CMakeLists.txt:43` at `WW3@80644ba9`). +`-DWW_KOKKOS_BUILD_TESTS=OFF` configures only that package: the project enables C++ alone, and +GoogleTest, NetCDF, the intro programs, the tools and the Fortran module are left out +(`WW_ENABLE_FORTRAN` is forced off: WW3 compiles `W3KOKKOSMD` in its own build). The kernels and +their flags do not change: `-ffp-contract=off` and, on CUDA, `--fmad=false` stay `PRIVATE` to +`ww_kokkos` (`src/ww_kokkos/CMakeLists.txt`). + +```bash +# inside nix develop ./picasso/labs/pratico#ww3, from the repository root +cmake -S kokkos -B kokkos/build/install-only -G Ninja -DCMAKE_BUILD_TYPE=Release \ + "-DCMAKE_CXX_FLAGS=-O3 -march=x86-64-v3" -DWW_KOKKOS_BUILD_TESTS=OFF +cmake --build kokkos/build/install-only -j8 +cmake --install kokkos/build/install-only --prefix +# then WW3 against it (scripts/ww3-gpu.sh still builds openmp-release, for nccmp-tol): +WW3GPU_KOKKOS_PREFIX= scripts/ww3-gpu.sh --ww3-gpu smoke --cpu +``` + +The installed `ww_kokkosConfig.cmake` (from `cmake/ww_kokkosConfig.cmake.in`) records the +`Kokkos_VERSION`, `Kokkos_DEVICES` and `Kokkos_ARCH` of the Kokkos the archive was compiled +against, asks for `find_dependency(Kokkos EXACT)`, and sets `ww_kokkos_FOUND` false with +a message when the Kokkos found has other backends or another architecture. `EXACT` works because +Kokkos' `KokkosConfigVersion.cmake` is CMake's basic version file, which sets +`PACKAGE_VERSION_EXACT` when the requested version string equals its own (v, read in the +`kokkos-openmp-5.2.2` store path of the `#ww3` shell, 2026-10-18). + +What the configure searched, read from `cmake --trace-expand` (v, 2026-10-18): `find_package(Kokkos +REQUIRED)` in this tree, and `find_package(OpenMP REQUIRED COMPONENTS CXX)` from Kokkos' own +`find_dependency`; no other `find_package` or `pkg_check_modules` call ran, and `Kokkos_DIR` is the +only package directory in the cache. The `#ww3` shell's Kokkos is 5.2.2 with `OPENMP;SERIAL` and no +architecture. The 25 object files of `ww_kokkos` are byte-identical to the `openmp-release` +preset's (`diff -r -q` of the two `CMakeFiles/ww_kokkos.dir`, v, 2026-10-18): the option changes +what is configured around the library, not the library. + +The refusals, checked with a one-line consumer project (`find_package(ww_kokkos 0.9 REQUIRED)`) +on 2026-10-18 (v): with the installed config, it finds Kokkos 5.2.2; with `-DKokkos_DIR` pointed +at a Kokkos 5.2.0 it still takes 5.2.2 and ignores 5.2.0; with the recorded version edited to +5.2.1 it stops with "Could not find a configuration file for package "Kokkos" that exactly matches +requested version"; with the recorded backends edited to `CUDA;SERIAL` it stops with the +config's own message. ## Intro programs @@ -362,11 +411,18 @@ this is the short form. | partial | `WW_KOKKOS_=1` for each part wanted | `--kokkos SNL1,SDS4` | the listed parts | | full | `WW_KOKKOS_ALL=1` | `--kokkos all` | every wired part whose translation covers the switch file | | GPU preset | `WW_KOKKOS_ALL=0 WW_KOKKOS_SRCE=1` | `--kokkos gpu` (the default of `--gpu`) | `W3SRCE` with the source terms it calls (ADR-0013) | -| resident preset | `WW_KOKKOS_ALL=0`, `WW_KOKKOS_=1` for `SRCE`, `XYP3`, `KTP3`, `OUTG`, `IOPE`, `UPDV` | `--kokkos resident` | the parts that keep the spectrum `VA` on the device (WFIP-0005 units 1 to 5, #173, #179); not a default: none of its parts is regtested yet ([`ports.tsv`](ports.tsv)) | +| resident preset | `WW_KOKKOS_PRESET=resident`, or `WW_KOKKOS_ALL=0` and `WW_KOKKOS_=1` for `SRCE`, `XYP3`, `KTP3`, `OUTG`, `IOPE`, `UPDV`, `CPRT` (the line of [`presets.txt`](src/fortran_iface/presets.txt)) | `--kokkos resident` | the parts that keep the spectrum `VA` on the device (WFIP-0005 units 1 to 5, #173, #179); not a default: none of its parts is regtested yet ([`ports.tsv`](ports.tsv)) | An explicitly set `WW_KOKKOS_` wins over `WW_KOKKOS_ALL`; one C function, `ww_kokkos_flag` in `src/fortran_iface/part_flags.cpp`, resolves both for the Fortran and the -shims. `W3INIT` writes one line to `log.ww3` naming each part and where it ran. "Full" is 100 % +shims. `WW_KOKKOS_PRESET=` turns on the parts of one line of +[`src/fortran_iface/presets.txt`](src/fortran_iface/presets.txt) as if each were set to `1`; a set +`WW_KOKKOS_` still wins and an unknown name stops `W3INIT`. `W3INIT` writes one line to +`log.ww3` naming each part and where it ran, the effective set. Where a part asked on Kokkos would +run the Fortran (`WW_KOKKOS_ALL=1` on a switch file its translation does not cover, or a `W3SRCE` +step with a branch the kernel leaves to the Fortran), the model runs the Fortran and the shim +prints one line at exit with the count per part; `WW_KOKKOS_STRICT=1` stops the model there +instead, with a line naming the part and the reason (PATCH.md sections 24 and 25). "Full" is 100 % of the wired parts, not of WW3: their share of the profiled run time is in [`PORT_STATUS.md`](PORT_STATUS.md) (`python3 scripts/port_score.py --write`), and the I/O, set-up and the rest of the propagation stay Fortran. A part that is wired but whose `regtested` diff --git a/kokkos/cmake/ww_kokkosConfig.cmake.in b/kokkos/cmake/ww_kokkosConfig.cmake.in new file mode 100644 index 00000000..e5eb8f08 --- /dev/null +++ b/kokkos/cmake/ww_kokkosConfig.cmake.in @@ -0,0 +1,41 @@ +# ww_kokkosConfig.cmake, generated from kokkos/cmake/ww_kokkosConfig.cmake.in. +# libww_kokkos.a holds Kokkos templates instantiated against one Kokkos install. A consumer +# that found another one (another version, another backend set, another GPU architecture) +# would link code compiled for one Kokkos with the runtime of another, so the config asks +# for the same version EXACT and checks the backends and architecture after it. +# +# SPDX-License-Identifier: MIT +include(CMakeFindDependencyMacro) + +set(ww_kokkos_Kokkos_VERSION "@Kokkos_VERSION@") +set(ww_kokkos_Kokkos_DEVICES "@Kokkos_DEVICES@") +set(ww_kokkos_Kokkos_ARCH "@Kokkos_ARCH@") + +# KokkosConfigVersion.cmake sets PACKAGE_VERSION_EXACT when the requested version string +# equals its own, so EXACT accepts this Kokkos and no other. +find_dependency(Kokkos ${ww_kokkos_Kokkos_VERSION} EXACT) + +set(_ww_kokkos_have_devices ${Kokkos_DEVICES}) +set(_ww_kokkos_want_devices ${ww_kokkos_Kokkos_DEVICES}) +set(_ww_kokkos_have_arch ${Kokkos_ARCH}) +set(_ww_kokkos_want_arch ${ww_kokkos_Kokkos_ARCH}) +list(SORT _ww_kokkos_have_devices) +list(SORT _ww_kokkos_want_devices) +list(SORT _ww_kokkos_have_arch) +list(SORT _ww_kokkos_want_arch) +if(NOT "${_ww_kokkos_have_devices}" STREQUAL "${_ww_kokkos_want_devices}" + OR NOT "${_ww_kokkos_have_arch}" STREQUAL "${_ww_kokkos_want_arch}") + set(ww_kokkos_FOUND FALSE) + string(CONCAT ww_kokkos_NOT_FOUND_MESSAGE + "ww_kokkos was built against Kokkos ${ww_kokkos_Kokkos_VERSION} with backends " + "'${ww_kokkos_Kokkos_DEVICES}' and architecture '${ww_kokkos_Kokkos_ARCH}'; the Kokkos " + "found (${Kokkos_DIR}) has backends '${Kokkos_DEVICES}' and architecture '${Kokkos_ARCH}'. " + "Point CMAKE_PREFIX_PATH or Kokkos_ROOT at the Kokkos ww_kokkos was built with.") + return() +endif() +unset(_ww_kokkos_have_devices) +unset(_ww_kokkos_want_devices) +unset(_ww_kokkos_have_arch) +unset(_ww_kokkos_want_arch) + +include("${CMAKE_CURRENT_LIST_DIR}/ww_kokkosTargets.cmake") diff --git a/kokkos/port_report/W3CPRT_KOKKOS.md b/kokkos/port_report/W3CPRT_KOKKOS.md new file mode 100644 index 00000000..859581a4 --- /dev/null +++ b/kokkos/port_report/W3CPRT_KOKKOS.md @@ -0,0 +1,82 @@ +# W3CPRT_KOKKOS + +**Status:** ported, phase 2 (on the resident spectrum), wired on the fork branch +`ww3-gpu/u9-cprt` (PATCH.md section 27, 2026-10-18): `W3KOKKOS_CPRT` replaces `W3WAVE`'s host read +of `VA` and `CALL W3CPRT ( IMOD )` under `WW_KOKKOS_CPRT=1` (opt-in, not under `WW_KOKKOS_ALL`). +Not at the `WW3/` pin until the integrator merges the branch. Run byte-identical on the CPU +(below); a GPU never ran it ⚠. + +`W3CPRT` partitions the spectrum of every local sea point at an output time with partitioning on +(`FLPART`: gridded partition fields or the partition file, `w3wavemd.F90:2680-2682` at `80644ba9`). +Per point it turns the action spectrum `N(k,theta)` of `VA` into the energy spectrum +`E(f,theta) = N * 2 pi sigma / c_g` (`w3iosfmd.F90:262-267`), calls the watershed partitioning +`W3PART` (`w3partmd.F90:138-497`) and stores the `NP+1` columns `XP(1:15,0:NP)` it returns, the +whole spectrum first and then each partition, in `DTPRT`, one point after the other, with +`ICPRT(JSEA,1) = NP+1` and `ICPRT(JSEA,2)` the point's first column (`w3iosfmd.F90:284-326`). +`W3OUTG` section 6 and `W3IOSF` read only that table. On the Fortran path the whole `VA` came back +from the device at every such output time for it; here only the table does. + +## What was ported + +| | | +|---|---| +| WW3 routine | `W3CPRT`, `model/src/w3iosfmd.F90:109-344` at h0ffmann/WW3 `80644ba9` (the `WW3/` pin), identical at NOAA-EMC/WW3 `03adda91` (v, `git log 03adda91..80644ba9 -- model/src/w3iosfmd.F90 model/src/w3partmd.F90 model/src/w3dispmd.F90` empty, 2026-10-18) | +| Kokkos code | [`../src/ww_kokkos/iosf_w3cprt.hpp`](../src/ww_kokkos/iosf_w3cprt.hpp): `w3cprt_point` (sections 2-4, `:245-279`) and `w3cprt_store` (section 5, `:284-303`); the launches are in the shim | +| Callees | `W3PART` and what it calls, translated before and called unchanged: [`part_w3part.hpp`](../src/ww_kokkos/part_w3part.hpp) ([`W3PART_KOKKOS.md`](W3PART_KOKKOS.md)), `ptsort`, `ptnghb` (the `NEIGH` table, built on the host once per `NK`, `NTH` and copied in), `pt_fld`, `ptmean`, `wavnu1` | +| Not ported, by switch | `W3_T` (test output, `:313-317`), `W3_S` (`STRACE`). `W3_MPI`: `INIT_GET_ISEA` is taken as `ISEA = JSEA` (`W3_SHRD`, `w3parall.F90:1236`); `W3KOKKOS_CPRT_SETUP` refuses the flag under `MPI` and `T` | +| `PTMETH` | 1 to 5, all of `W3PART`'s branches; `DIMXP` is 2 for 4 and 5 and `((NK+1)/2)*((NTH-1)/2)` otherwise (`:213-219`) | +| Left to the Fortran | A sea point with a NaN depth (`:270-274`: two lines and `STOP`) and a spectrum `PTMEAN` cannot take (`IFPMAX = 0`, where the Fortran reads out of bounds, [`PTMEAN_KOKKOS.md`](PTMEAN_KOKKOS.md)): `W3KOKKOS_CPRT` then reads `VA` to the host and calls the Fortran `W3CPRT` whole | +| Structural changes | Points run in parallel, each one the Fortran's serial code for that point. The Fortran's `TMP` array grows by `TSFAC*NSEAL` columns and is copied into `DTPRT(:,1:FINSIZ)` (`:288-326`); every column `1..FINSIZ` is written once by `:303`, so `DTPRT` is the concatenation of the points' `XP(:,0:NP)` in `JSEA` order, which an integer prefix sum over `NP+1` places directly. `PTMEAN`'s format 1000 (`XP ARRAY TOO SMALL`) is counted per call on the device and written by `W3KOKKOS_CPRT` the same number of times, with the same text and value `DIMXP+1` | +| C shim | [`../src/fortran_iface/cprt_shim.cpp`](../src/fortran_iface/cprt_shim.cpp), `ww_cprt_va` and `ww_cprt_fetch` of [`cprt_c.hpp`](../src/fortran_iface/cprt_c.hpp) (ABI `ww_kokkos` 0.10); the Fortran side is `W3KOKKOS_CPRT` of `model/src/w3kokkos_cprtmd.F90` on `ww3-gpu/u9-cprt` | +| Partition fields | `ww_outg_part` of [`outg_c.hpp`](../src/fortran_iface/outg_c.hpp) runs `W3OUTG` section 6 (`w3iogomd.F90:2242-2340`, `ww::iogo::w3outg_part`, the code that was inside `w3outg`, moved unchanged) after `ww_outg_va`, so a gridded output with partition fields no longer makes `W3KOKKOS_OUTG` fall back to the Fortran `W3OUTG` and its host read of `VA` | + +## Arithmetic + +The reference is the `-O3` GIMPLE (`-fdump-tree-optimized-lineno`) of `w3iosfmd.F90` compiled with +the flags and defines of WW3's CMake Release build under `switch_lab_shrd` (gfortran 16.2.0, the +`pratico#ww3` shell), read beside the GIMPLE of `cprt_shim.cpp` with the `openmp-release` flags +(`-O3 -march=x86-64-v3 -std=c++20 -ffp-contract=off -fopenmp`, g++ 16.2.0), 2026-10-18. + +- (v) `UABS = U10*ASF` is one product in both (`_198 = _187 * _197`, `uabs_99 = _95 * _98`). +- (v) `UDIR = U10D*RADE`: both multiply by the same folded constant, `5.72957763671875e+1`. +- (v) `FACT = TPI*SIG(IK)/CG(IK,ISEA)`: both compute `SIG*6.283185482025146484375e+0` and divide by + `CG` (`fact_569 = _232 / _245`, `fact_125 = _115 / _124`); the product is commuted, which changes + no bit. +- (v) `E2(IK,ITH) = VA(ITH+(IK-1)*NTH,JSEA)*FACT`, one product per bin, commuted in the C++. +- (v) No FMA: contraction is off in the C++ (`-ffp-contract=off`), and WW3's Fortran flags have + no `-march` (`-O3 -g -fno-second-underscore ...`, `flags.make` of the build), so the x86-64 + baseline has no FMA instruction to contract into; the products and the quotient above stay + separate statements in both GIMPLE dumps. +- (v) The libm functions the shim's object imports are `cosf`, `logf`, `atan2f`, `sqrtf`, `fmodf` + and `roundf` (`nm -u`), those of `W3PART`'s translation; their parity with `w3partmd.o` + (`cosf`, `logf`, `atan2f`, `lroundf`, inline `sqrtss` and `fprem`) is argued in + [`W3PART_KOKKOS.md`](W3PART_KOKKOS.md) and [`PTMEAN_KOKKOS.md`](PTMEAN_KOKKOS.md). +- (v) `ICPRT` and the `DTPRT` columns are integers and copies: no arithmetic. +- ⚠ On a device `cosf`, `logf`, `atan2f`, `fmodf` and `roundf` are CUDA's, not glibc's (as for + `W3PART`). The CUDA build of the shim was not compiled. + +## Benchmarks + +None. No timing without parity (`AGENTS.md`). The wall times the runs below print are one run +each on a machine shared with other jobs ⚠ and are not quoted. + +## How it was regtested + +`VA` traffic, the checks and the commands are in +[`PATCH.md` section 27](../src/fortran_iface/PATCH.md) and +[`docs/log/2026-10-18-cprt-resident.md`](../../docs/log/2026-10-18-cprt-resident.md). In short, on +the CPU (OpenMP backend, gfortran/g++ 16.2.0, i9-14900 shared, never `--gpu`): + +- `scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT` on two + throwaway variants of `bench_small` with partitioning on, `WW_KOKKOS_VA_POISON=1`: every model + file and `partition.ww3` byte-identical to the Fortran of the same executable and to `--ww3` + (v); see the log for the five runs. +- `VA` device-to-host copies in those 12-hour runs (13 output times): 13 before, 1 after, the read + at `W3WAVE`'s return (v). +- WW3's regtest `ww3_tpt1.1`, the five partitioning methods `PTM = 1..5`, through the matrix + driver's `regtests/cmake/run_entry.cmake` with `WW3GPU_KOKKOS=CPRT`: `test.comp` found 0 of 22 + files different and `nccmp-tol` passed on each; 2 Kokkos `W3CPRT` calls on 18 sea points per + run, no fallback (v). The staged switch had `PR3 UQ` for the test's `PR0`, since the fork's + `KOKKOS` build needs `PR3` ⚠; `ww3_tpt1.1` is not in `matrix.base`, and the one serial + `matrix.base` entry with partition fields, `ww3_tnc1`, is `PR0` and does not build with + `KOKKOS`, so the matrix proper has not run this port ⚠. diff --git a/kokkos/port_report/W3OUTG_KOKKOS.md b/kokkos/port_report/W3OUTG_KOKKOS.md index faf070b0..890ad12c 100644 --- a/kokkos/port_report/W3OUTG_KOKKOS.md +++ b/kokkos/port_report/W3OUTG_KOKKOS.md @@ -35,7 +35,7 @@ before every gridded output. | Not ported, by switch | `W3_IG1` `HSIG` (`:1556-1558`), `W3_O9` negative Hs (`:1995-2003`), `W3_O8` filter (`:2058-2068`), `W3_T` prints (`:2219-2237`, `:2368-2375`), `W3_S` STRACE. Neither lab switch sets them; `W3KOKKOS_OUTG_SETUP` refuses `WW_KOKKOS_OUTG=1` under any of them and under `MPI`. The `W3_OMPG` directives are covered: `W3OutgScalars::ompg` selects the object's two choices that differ (below) | | Not computed | Locals that reach no output: `NKH` (`:1512-1513`), `FXPMC` (`:1388`), `ETMSSL`, `ETMSSCL` (`:1577-1589`), `ETUSCX`, `ETUSCY` (`:1617-1618`), `UBS` (`:1590`, `:1975`), `SCREST` (`:1809-1811`). The object keeps the arrays as stores that nothing reads (`NKH` with its `logf`) and folds the scalar `FXPMC` away; none of them has an effect a Release build can observe (no `errno` in Fortran, no trapping: `-ffpe-trap` is in WW3's Debug flags only) | | Structural changes | The Fortran runs each section as a loop over `JSEA`; here one thread runs sections 2-4 for one point, which reads only that point's data. The local `EBD(NK,NSEAL)` is a caller work view; `SKEWNESS`'s `F2`, `SIGHF`, `DFIMHF`, `FAK` are work views with one column per point. `FLOLOC` is computed on the host and passed as booleans | -| C shim | [`../src/fortran_iface/outg_shim.cpp`](../src/fortran_iface/outg_shim.cpp), `ww_outg_va` of [`outg_c.hpp`](../src/fortran_iface/outg_c.hpp) (ABI `ww_kokkos` 0.7): `A` is the resident `VA` (`ww::resident::device_va`), the fields are W3ADATMD's arrays in place on Serial and OpenMP, device buffers on a device with only the fields a host reader needs copied back; the Fortran side is `W3KOKKOS_OUTG` of `model/src/w3kokkos_outgmd.F90` on `ww3-gpu/kokkos-outg` ([`PATCH.md` section 15](../src/fortran_iface/PATCH.md#15-w3outg-on-the-resident-spectrum-ww3-gpukokkos-outg)). The Fortran `W3OUTG` runs instead when `FLPART` (section 6, which reads `W3CPRT`'s table), `CALC_U3STOKES` (`FLOLOC(6,8)`, `(6,12)`), `SKEWNESS` (`FLOLOC(8,7:9)`, translated, not wired) or `CALC_WBT` (`FLOLOC(2,17)`) is requested, or a requested 3-D field is not allocated | +| C shim | [`../src/fortran_iface/outg_shim.cpp`](../src/fortran_iface/outg_shim.cpp), `ww_outg_va` of [`outg_c.hpp`](../src/fortran_iface/outg_c.hpp) (ABI `ww_kokkos` 0.7): `A` is the resident `VA` (`ww::resident::device_va`), the fields are W3ADATMD's arrays in place on Serial and OpenMP, device buffers on a device with only the fields a host reader needs copied back; the Fortran side is `W3KOKKOS_OUTG` of `model/src/w3kokkos_outgmd.F90` on `ww3-gpu/kokkos-outg` ([`PATCH.md` section 15](../src/fortran_iface/PATCH.md#15-w3outg-on-the-resident-spectrum-ww3-gpukokkos-outg)). Since 2026-10-18 section 6 (`FLPART`, which reads `W3CPRT`'s table and no `A`) runs through `ww_outg_part` after `ww_outg_va` ([`PATCH.md` section 27](../src/fortran_iface/PATCH.md), [`W3CPRT_KOKKOS.md`](W3CPRT_KOKKOS.md)). The Fortran `W3OUTG` runs instead when `CALC_U3STOKES` (`FLOLOC(6,8)`, `(6,12)`), `SKEWNESS` (`FLOLOC(8,7:9)`, translated, not wired) or `CALC_WBT` (`FLOLOC(2,17)`) is requested, or a requested 3-D field is not allocated | | Plan | [`W3WAVE_SUPPORT_KOKKOS_PLAN_202610.md`](../../docs/W3WAVE_SUPPORT_KOKKOS_PLAN_202610.md), issue #42 | ## Arithmetic diff --git a/kokkos/port_report/WMIOEG_KOKKOS.md b/kokkos/port_report/WMIOEG_KOKKOS.md new file mode 100644 index 00000000..f6456693 --- /dev/null +++ b/kokkos/port_report/WMIOEG_KOKKOS.md @@ -0,0 +1,35 @@ +# WMIOEG_KOKKOS + +**Status:** ported, the stores of `VA` only, wired and regtested on the CPU (2026-10-18) on the +fork branch h0ffmann/WW3 `ww3-gpu/u1-multi-va` (`f330dc53`), under the resident `VA`; not at the +`WW3/` pin until the integrator moves it ⚠. + +`WMIOEG` updates grid `IMOD`'s `VA` from the grids of equal rank: it scales the native columns +(2.a) and adds the averaged, converted spectra of every other grid (2.b, or copies them on an SMC +grid). With the resident `VA` it is the same column path as +[`WMIOHG_KOKKOS.md`](WMIOHG_KOKKOS.md), with a read before, since 2.a and 2.b read the columns +they write. + +## What was ported + +| | | +|---|---| +| WW3 routine | `WMIOEG`, the stores of 2.a, `model/src/wminiomd.F90:3258-3262` at NOAA-EMC/WW3 `03adda91` (`VA(:,JSEA) = WGHT * VA(:,JSEA)`), and of 2.b (`:3278`, `:3326`, `:3330`) | +| Kokkos code | `ww_va_host_read_cols` (gather) and `ww_va_host_wrote_cols` (scatter) in [`../src/fortran_iface/resident_shim.cpp`](../src/fortran_iface/resident_shim.cpp) | +| Fortran side | `W3KOKKOS_VA_READ_COLS` before 2.a and `W3KOKKOS_VA_WRITE_COLS` after 2.b, with the union of `EQSTGE(IMOD,IMOD)%JSEA` and `EQSTGE(IMOD,J)%JSEA` ([`PATCH.md`](../src/fortran_iface/PATCH.md), section 26) | + +## Arithmetic + +None on the device; the Fortran statements run unchanged on the host columns, which the gather +made current first. Bits move, as in `WMIOHG_KOKKOS.md`. + +## Benchmarks + +None of its own: the regtest entries of `WMIOHG_KOKKOS.md` count both routines' columns +together, untimed ⚠. + +## How it was regtested + +The entries of [`WMIOHG_KOKKOS.md`](WMIOHG_KOKKOS.md). Whether those inputs reach `WMIOEG`'s 2.b +(grids of equal rank with overlap) was not checked ⚠: `WMIOEG` runs on every grid and its 2.a +loop runs when `EQSTGE(IMOD,IMOD)%NREC > 0`. diff --git a/kokkos/port_report/WMIOHG_KOKKOS.md b/kokkos/port_report/WMIOHG_KOKKOS.md new file mode 100644 index 00000000..5e7cde1f --- /dev/null +++ b/kokkos/port_report/WMIOHG_KOKKOS.md @@ -0,0 +1,47 @@ +# WMIOHG_KOKKOS + +**Status:** ported, the store of `VA` only, wired and regtested on the CPU (2026-10-18) on the +fork branch h0ffmann/WW3 `ww3-gpu/u1-multi-va` (`f330dc53`), under the resident `VA`; not at the +`WW3/` pin until the integrator moves it ⚠. + +`WMIOHG` gathers the spectra a higher-ranked grid sends to grid `IMOD` (two-way nesting), +averages and converts them in Fortran, and stores them into `VA(:,JSEA)` of grid `IMOD`. With the +resident `VA` the device copy must get the same columns. What runs on the device is the store's +data movement, nothing of the averaging or the conversion. + +## What was ported + +| | | +|---|---| +| WW3 routine | `WMIOHG`, the store loop 2.e, `model/src/wminiomd.F90:2260-2266` at NOAA-EMC/WW3 `03adda91`: `VA(I,JSEA) = SPEC2(I,IS) / SIG2(I) * CG(1+(I-1)/NTH,ISEA)`, `JSEA = HGSTGE(IMOD,J)%LJSEA(IS)`, every `I = 1..NSPEC` | +| Kokkos code | `ww_va_host_wrote_cols` in [`../src/fortran_iface/resident_shim.cpp`](../src/fortran_iface/resident_shim.cpp) (`host_wrote_cols`, kernel `ww_va.scatter_cols`), declared in [`../src/fortran_iface/resident_c.hpp`](../src/fortran_iface/resident_c.hpp) | +| Fortran side | `W3KOKKOS_VA_WRITE_COLS` of the fork's `w3kokkos_residentmd.F90`, called after the loop with `LJSEA(1:NTOT)` ([`PATCH.md`](../src/fortran_iface/PATCH.md), section 26) | +| Plan | [the production single-node plan](../../docs/compound/plans/2026-10-18-002-production-single-node-plan.md), unit U1 | + +## Arithmetic + +None on the device. The Fortran statement runs unchanged on the host array. The shim then copies +the host columns `LJSEA(1:NTOT)` into a buffer (`memcpy`), the buffer to the device +(`deep_copy`), and stores each element into `d_va(s, col)` (`ww_va.scatter_cols`). A copy of a +`float` moves its bits, so each device column holds the host column's bits (v, by reading the +code: no arithmetic in the kernel). The list is the JSEA of the Fortran loop itself, so no column +is missed or added; `WW_KOKKOS_VA_CHECK=1` would stop a run where another column changed, and +`WW_KOKKOS_VA_POISON=1` would put NaN into the output if a host reader missed a column. + +## Benchmarks + +| Case | Machine | Command | Result | +|---|---|---|---| +| `mww3_test_02`, `_03`, `_04` (`grdset_a`), resident preset, lab switch | i9-14900, OpenMP, 4 threads | `docs/log/2026-10-18-ww3-multi-resident-va.md`, step 3 | columns stored over the run: 18 432 (44.24 MB), 49 680 (95.39 MB), 71 388 (137.07 MB), with `WMIOEG`'s; not timed ⚠ | + +No timing: the matrix runs are not timed runs, and the machine was shared. + +## How it was regtested + +WW3's own regtests through the matrix driver (`WW3GPU_LAB_SWITCH=ON`, `WW3GPU_STRICT=ON`), the +Fortran and the Kokkos run of one executable compared by `test.comp`: `mww3_test_01` to `_04` +(`PR3_UQ`, resident preset) and `mww3_test_05` (`ST1`/`ST4_PR3_UQ_OMP`, `SRCE,OUTG,IOPE,UPDV`), +0 files different, plain and with the host copy poisoned (v, 2026-10-18; summaries in +[`ww3gpu_matrix_multi_va_2026-10-18.tsv`](../../docs/data/ww3gpu_matrix_multi_va_2026-10-18.tsv)). +Before the change `_02`, `_03`, `_04` differed in 5, 10 and 4 files. Under MPI it builds and was +not run (ADR-0017). diff --git a/kokkos/ports.tsv b/kokkos/ports.tsv index 8d1d1789..c84e17bb 100644 --- a/kokkos/ports.tsv +++ b/kokkos/ports.tsv @@ -41,7 +41,7 @@ w3pro3md:w3xyp3 model/src/w3pro3md.F90:641-1477 837 ported translated 2026-10-16 w3uqckmd:w3qck3 model/src/w3uqckmd.F90:925-1336 412 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run; wired 2026-10-17 (kokkos/src/fortran_iface/PATCH.md section 11) no: no regtest matrix yet; the CPU replay ran it inside W3XYP3: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS; the transparency branches with obstructions were not checked for coverage kokkos/src/ww_kokkos/prop_qck3.hpp yes: inside the Kokkos W3XYP3 (its two sweeps), under WW_KOKKOS_XYP3 at w3wavemd.F90:2022-2023 at fcdfabef w3uqckmd:w3qck1 model/src/w3uqckmd.F90:117-480 364 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run; wired 2026-10-17 (kokkos/src/fortran_iface/PATCH.md section 11) no: no regtest matrix yet; the CPU replay ran it inside W3KTP3 (refraction on): scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS kokkos/src/ww_kokkos/prop_qck1.hpp yes: inside the Kokkos W3KTP3 (the direction pass), under WW_KOKKOS_KTP3 at w3wavemd.F90:1864-1865 and 2224-2225 at fcdfabef w3pro3md:w3ktp3 model/src/w3pro3md.F90:1519-1961 443 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run; wired 2026-10-17 (kokkos/src/fortran_iface/PATCH.md section 11) no: no regtest matrix yet; the CPU replay ran it: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS, 441 792 point calls in 48 calls, refraction only: FLCK and FLCUR are false in bench_small, so the wavenumber shift (W3QCK2) never ran; on the resident VA (ww_ktp3_batch_va, fork 90631fad, WFIP-0005 unit 3b, PATCH.md section 13) the same replay byte-identical on 2026-10-17 on switch_lab_ompg and switch_lab_shrd, with and without WW_KOKKOS_VA_RESIDENT=1, VA traffic H2D 1 / D2H 13 per full run kokkos/src/ww_kokkos/prop_ktp3.hpp,kokkos/src/fortran_iface/prop3_shim.cpp,kokkos/src/fortran_iface/prop3_c.hpp yes: WW_KOKKOS_KTP3 (opt-in, not under WW_KOKKOS_ALL), every sea point of an intra-spectral part in one call at w3wavemd.F90:1864-1865 and 2224-2225 (W3KOKKOS_KTP3 in w3kokkos_prop3md.F90, ww_ktp3_batch in kokkos/src/fortran_iface/prop3_shim.cpp) at fcdfabef; on the resident VA (ww_ktp3_batch_va) at 90631fad -w3dispmd:wavnu1 model/src/w3dispmd.F90:85-202 118 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist kokkos/src/ww_kokkos/prop_wavnu1.hpp no: header-only device function; no shim and no W3WAVE hunk yet +w3dispmd:wavnu1 model/src/w3dispmd.F90:85-202 118 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md); only PTMEAN's call, not W3INIT's or W3ULEV's kokkos/src/ww_kokkos/prop_wavnu1.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it w3uqckmd:w3qck2 model/src/w3uqckmd.F90:517-891 375 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run; wired 2026-10-17 (kokkos/src/fortran_iface/PATCH.md section 11) no: never run: W3KTP3 calls it only when FLCK is true, and the full and smoke cases have no current-induced wavenumber shift kokkos/src/ww_kokkos/prop_qck2.hpp yes: inside the Kokkos W3KTP3 (the wavenumber pass), under WW_KOKKOS_KTP3 at w3wavemd.F90:1864-1865 and 2224-2225 at fcdfabef w3wavemd:w3gath model/src/w3wavemd.F90:3024-3329 306 ported translated 2026-10-16; static parity by -O3 GIMPLE and objdump comparison, a fresh reviewer (port report); instantiated by wave_support.cpp in the serial-debug, openmp-release and cuda-release builds (CUDA build only), never run yes: scripts/ww3-gpu.sh --ww3-gpu full (bench_small) --cpu on 2026-10-16 (OpenMP backend, gfortran/g++ 16.2.0), with WW_KOKKOS_ALL=1 and with the W3WAVE support parts alone: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, the binary files byte-identical to --ww3, 13 824 calls (kokkos/src/fortran_iface/PATCH.md, section 8) kokkos/src/ww_kokkos/wave_gath_scat.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_GATH=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp w3wavemd:w3scat model/src/w3wavemd.F90:3345-3644 300 ported translated 2026-10-16; static parity by -O3 GIMPLE and objdump comparison, a fresh reviewer (port report); instantiated by wave_support.cpp in the serial-debug, openmp-release and cuda-release builds (CUDA build only), never run yes: scripts/ww3-gpu.sh --ww3-gpu full (bench_small) --cpu on 2026-10-16 (OpenMP backend, gfortran/g++ 16.2.0), with WW_KOKKOS_ALL=1 and with the W3WAVE support parts alone: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, the binary files byte-identical to --ww3, 13 824 calls (kokkos/src/fortran_iface/PATCH.md, section 8) kokkos/src/ww_kokkos/wave_gath_scat.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_SCAT=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp @@ -75,12 +75,13 @@ pdlib_w3profsmd:pdlib_explicit_block model/src/w3profsmd_pdlib.F90:6408-6685 278 pdlib_w3profsmd:apply_boundary_condition model/src/w3profsmd_pdlib.F90:5150-5336 187 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB; a fresh review found no defect (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim with the MPI hooks kokkos/src/ww_kokkos/pdlib_boundary.hpp no: header-only; no shim yet pdlib_w3profsmd:pdlib_w3xypfsfct2 model/src/w3profsmd_pdlib.F90:1504-1877 374 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison with and without REF1; a fresh review found it matches wherever the Fortran is defined; WW3 runs the WII loop over NE, so the Fortran overflows the heap unless NE == npa, and the C++ bounds it to min(NE,npa), a departure awaiting the owner ⚠ (port report); no regtest reaches it (EXPFSFCT = F everywhere); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); no regtest selects EXPFSFCT kokkos/src/ww_kokkos/pdlib_fsfct2.hpp no: header-only; no shim yet pdlib_w3profsmd:pdlib_w3xypug model/src/w3profsmd_pdlib.F90:626-866 241 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB and REF1; a fresh review found no defect (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3WAVE hunk kokkos/src/ww_kokkos/pdlib_w3xypug.hpp no: header-only; no shim and no W3WAVE hunk yet -w3partmd:ptnghb model/src/w3partmd.F90:630-821 192 ported translated 2026-10-16; static parity by -O3 GIMPLE comparison under switch_NCEP_st4 (integer only, object identical to the CMake build); a fresh review found the translation exact in slot order, values, zero fill and rebuild rule and corrected a comment on the neighbour count, no code defect (port report); host only, no CUDA build needed no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk kokkos/src/ww_kokkos/part_ptnghb.hpp no: header-only; no shim and no W3PART hunk yet -w3partmd:pt_fld model/src/w3partmd.F90:839-1156 318 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4 (MAXVAL seed and maxss order, ABS and compare, FIFO ring of NSPEC kept), no WW3 defect found; a fresh review found no defect (FIFO ring, MAXVAL NaN and empty cases on host and in the CUDA SASS, the five sweeps, the NEIGH layout shared with part_ptnghb.hpp) (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no W3PART hunk yet -w3partmd:ptsort model/src/w3partmd.F90:511-613 103 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4: integer counting sort, the four loops in the Fortran's order and bounds, stable in I; a fresh review found no defect in the translation and fixed two report wording errors (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk kokkos/src/ww_kokkos/part_ptsort.hpp no: header-only; no shim and no W3PART hunk yet -w3partmd:ptmean model/src/w3partmd.F90:1175-1538 364 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4, every sum in the Fortran's order, the nine MAX/MIN sites in the object's operand order, SIGN folds, constants and the SIGP**(-5) power tree; a fresh review found no defect and corrected comments; WW3's out-of-bounds read when IFPMAX stays 0 (a NaN spectrum only) returns kPtmeanNoPeak so the shim calls the Fortran there (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs W3PART and a shim kokkos/src/ww_kokkos/part_ptmean.hpp no: header-only; no shim and no W3PART/W3IOSF hunk yet +w3partmd:ptnghb model/src/w3partmd.F90:630-821 192 ported translated 2026-10-16; static parity by -O3 GIMPLE comparison under switch_NCEP_st4 (integer only, object identical to the CMake build); a fresh review found the translation exact in slot order, values, zero fill and rebuild rule and corrected a comment on the neighbour count, no code defect (port report); host only, no CUDA build needed no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_ptnghb.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3partmd:pt_fld model/src/w3partmd.F90:839-1156 318 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4 (MAXVAL seed and maxss order, ABS and compare, FIFO ring of NSPEC kept), no WW3 defect found; a fresh review found no defect (FIFO ring, MAXVAL NaN and empty cases on host and in the CUDA SASS, the five sweeps, the NEIGH layout shared with part_ptnghb.hpp) (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3partmd:ptsort model/src/w3partmd.F90:511-613 103 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4: integer counting sort, the four loops in the Fortran's order and bounds, stable in I; a fresh review found no defect in the translation and fixed two report wording errors (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_ptsort.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3partmd:ptmean model/src/w3partmd.F90:1175-1538 364 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4, every sum in the Fortran's order, the nine MAX/MIN sites in the object's operand order, SIGN folds, constants and the SIGP**(-5) power tree; a fresh review found no defect and corrected comments; WW3's out-of-bounds read when IFPMAX stays 0 (a NaN spectrum only) returns kPtmeanNoPeak so the shim calls the Fortran there (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_ptmean.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it w3snl3md:insnl3 model/src/w3snl3md.F90:788-1649 862 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_glwu (NOGRB) and switch_UoM_nl3; the four LOG(AUXFR) of :1046 and :1273 follow the object's libmvec/scalar split; NKD is 1300 under ST4 by use association and 275 without, kept as a flag; a fresh review found no defect and corrected a call count in the report (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3IOGR hunk kokkos/src/ww_kokkos/snl3_tables.hpp no: header-only; no shim and no W3IOGR hunk yet -w3partmd:w3part model/src/w3partmd.F90:138-497 360 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4 (UPAR maxss order, MINVAL/MAXVAL/MAXLOC NaN seeds, NINT as lroundf with the IHMAX clamp, the wind-sea sort and combination in the Fortran's order); calls ptsort, pt_fld, ptmean and the PtnghbTable, none re-translated; a fresh review found no defect (NINT range-guarded and equal to lroundf for every float, the CUDA FFMA are libdevice and IEEE division only) and fixed the shim contract on kPtmeanNoPeak (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3IOSF hunk kokkos/src/ww_kokkos/part_w3part.hpp no: header-only; no shim and no W3IOSF hunk yet +w3partmd:w3part model/src/w3partmd.F90:138-497 360 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4 (UPAR maxss order, MINVAL/MAXVAL/MAXLOC NaN seeds, NINT as lroundf with the IHMAX clamp, the wind-sea sort and combination in the Fortran's order); calls ptsort, pt_fld, ptmean and the PtnghbTable, none re-translated; a fresh review found no defect (NINT range-guarded and equal to lroundf for every float, the CUDA FFMA are libdevice and IEEE division only) and fixed the shim contract on kPtmeanNoPeak (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_w3part.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3iosfmd:w3cprt model/src/w3iosfmd.F90:109-344 236 ported translated and wired 2026-10-18 on the resident VA (kokkos/src/fortran_iface/PATCH.md section 27); static parity by -O3 GIMPLE of w3iosfmd.F90 and of the shim under switch_lab_shrd: UABS, UDIR with the same folded RADE, FACT = (SIG*TPI)/CG, E2 = VA*FACT, no FMA, W3PART called unchanged, ICPRT and DTPRT integer placement (port report) no: the owner decides whether lab-switch evidence counts (plan 2026-10-18-002); the WW3 regtest matrix ran it on 2026-10-18 at WW3 50a10730 with WW3GPU_LAB_SWITCH=ON and WW_KOKKOS_VA_POISON=1: ww3_tnc1 (SRCE,OUTG,IOPE,UPDV,CPRT) test.comp 0/30, 3 Kokkos calls; mww3_test_03 grdset_a (resident) 0/34, 219 calls; the five mww3_test_05 ST4 OMPG entries (SRCE,OUTG,IOPE,UPDV,CPRT) 0/27 each, 39 calls each (docs/log/2026-10-18-u3-regtests-v2.md, docs/data/ww3gpu_matrix_*_pin-50a10730_2026-10-18.tsv); before that the CPU replay ran it on 2026-10-18: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT on two partition variants of bench_small (switch_lab_shrd, and switch_lab_ompg for the fields variant), WW_KOKKOS_VA_POISON=1: every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 (docs/log/2026-10-18-cprt-resident.md) kokkos/src/ww_kokkos/iosf_w3cprt.hpp, kokkos/src/fortran_iface/cprt_shim.cpp yes: WW3/ pin a304018c, w3wavemd.F90:2692 (W3KOKKOS_CPRT under WW_KOKKOS_CPRT=1 and in the resident preset) w3snl3md:w3snl3 model/src/w3snl3md.F90:181-777 597 ported translated 2026-10-16 (non-PDLIB WAVNU1 variant, with EXPAND and EXPND2; no NL3 switch has PDLIB); static parity by -O3 GIMPLE and object comparison under switch_NCEP_glwu (NCEP2 -> NOGRB, SCRIP/SCRIPNC dropped) and switch_UoM_nl3, including the libmvec powf grouping of the SCALE loop; a fresh review found no defect and corrected two statements in the report (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3SRCE hunk kokkos/src/ww_kokkos/snl3_gmd.hpp no: header-only; no shim and no W3SRCE hunk yet w3src0md:w3spr0 model/src/w3src0md.F90:81-231 151 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd with ST0 FLX2; a fresh review found no arithmetic defect and corrected where the report placed the reversed NaN-payload operands; compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests kokkos/src/ww_kokkos/srce_st0_spr0.hpp yes: WW_KOKKOS_SPR0, w3srcemd.F90:1086, :2201 at 357f575c w3src1md:w3spr1 model/src/w3src1md.F90:87-240 154 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd with ST1 FLX1; a fresh review found no defect; compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests kokkos/src/ww_kokkos/srce_st1_spr1.hpp yes: WW_KOKKOS_SPR1, w3srcemd.F90:1098, :2212 at 357f575c @@ -208,9 +209,9 @@ w3uostmd:uost_sourceterm_compute model/src/w3uostmd.F90:1056-1128 73 ported tran w3uostmd:uost_sourceterm_setgrid model/src/w3uostmd.F90:606-667 62 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_mixa (lab + TIDE UOST FLD2 IG1 IC3 IS2 BT8); a fresh review found the vector path taken at NTH = HUGE(0) where the object stays scalar (and an i+4 overflow), fixed and read again in the object no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/uost_uost_sourceterm_setgrid.hpp no: header-only; no shim and no caller hunk yet w3uostmd:uost_srctrmcompute model/src/w3uostmd.F90:298-343 46 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd+TIDE UOST FLD2 IG1 IC3 IS2 BT8 (mixa); a fresh review found no defect (forwarding call only; object and vtable slot read again) no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/uost_uost_srctrmcompute.hpp no: header-only; no shim and no caller hunk yet w3wavset:fd_wave_setup_apply_fct model/src/w3wavset.F90:1945-2026 82 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB+SETUP; a fresh review found no defect no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/wavset_fd_wave_setup_apply_fct.hpp no: header-only; no shim and no caller hunk yet -w3partmd:fifo_add model/src/w3partmd.F90:1103-1113 11 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd (inlined at all six PT_FLD sites; store, +1, signed > NSPEC wrap to 1 as cmovg in both objects); already the member Fifo::add of the PT_FLD translation, no code added; a fresh review found no defect (report wording on the NSPEC load corrected) no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet -w3partmd:fifo_first model/src/w3partmd.F90:1142-1152 11 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd; a fresh review found no defect no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet -w3partmd:fifo_empty model/src/w3partmd.F90:1122-1133 12 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd (inlined at three sites in PT_FLD as one integer IQ_START/IQ_END compare, same in the C++; already the member Fifo::empty of part_fld.hpp, translated with PT_FLD, not duplicated); a fresh review found no defect no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet +w3partmd:fifo_add model/src/w3partmd.F90:1103-1113 11 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd (inlined at all six PT_FLD sites; store, +1, signed > NSPEC wrap to 1 as cmovg in both objects); already the member Fifo::add of the PT_FLD translation, no code added; a fresh review found no defect (report wording on the NSPEC load corrected) no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3partmd:fifo_first model/src/w3partmd.F90:1142-1152 11 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd; a fresh review found no defect no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3partmd:fifo_empty model/src/w3partmd.F90:1122-1133 12 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd (inlined at three sites in PT_FLD as one integer IQ_START/IQ_END compare, same in the C++; already the member Fifo::empty of part_fld.hpp, translated with PT_FLD, not duplicated); a fresh review found no defect no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it w3wavset:differentiate_xydir model/src/w3wavset.F90:363-435 73 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB+SETUP (ww3_tp2.6, DIST MPI PDLIB SCOTCH); a fresh review found no arithmetic defect (both objects one tail jmp to the callee); fixed two off-by-one line citations no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/wavset_differentiate_xydir.hpp no: header-only; no shim and no caller hunk yet w3wavset:fd_wave_setup_scalar_prod model/src/w3wavset.F90:2655-2729 75 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB+SETUP; a fresh review found no defect (only a NaN-payload operand order of the multiply, which the Fortran object itself varies by path) no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/wavset_fd_wave_setup_scalar_prod.hpp no: header-only; no shim and no caller hunk yet w3wavset:trig_compute_diff model/src/w3wavset.F90:598-685 88 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB (ww3_tp2.6) + SETUP; a fresh review found no defect in the header; the report named the caller at :1712 wrongly, fixed no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/wavset_trig_compute_diff.hpp no: header-only; no shim and no caller hunk yet @@ -333,3 +334,5 @@ w3pro1md:w3ktp1 model/src/w3pro1md.F90:900-1230 331 ported translated 2026-10-16 w3pro2md:w3xyp2 model/src/w3pro2md.F90:508-1264 757 ported translated 2026-10-16 under switch_lab_shrd with PR2 for PR3; static parity by -O3 GIMPLE and assembly comparison (literals, MIN/MAX and the CGN tree, NINT as lroundf, W3QCK3 calls); a fresh reviewer found no code defect and fixed the report's pin; compiled on serial-debug, never run no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist kokkos/src/ww_kokkos/prop_xyp2.hpp,kokkos/src/ww_kokkos/prop_params2.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_XYP2=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) w3pro2md:w3ktp2 model/src/w3pro2md.F90:1306-1710 405 ported translated 2026-10-16 under switch_lab_shrd with PR2 for PR3; static parity by -O3 GIMPLE and assembly comparison (FDDMAX four-lane grouping, MIN/MAX operand order, W3QCK1/W3QCK2 arguments); a fresh reviewer found no arithmetic defect and fixed two citations; NaN-payload order (port report); compiled on serial-debug, never run no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist kokkos/src/ww_kokkos/prop_ktp2.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_KTP2=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) w3profsmd:w3xypug model/src/w3profsmd.F90:69-272 204 ported translated 2026-10-16 under the switch of ww3_tp2.6 (SHRD PR3, no PDLIB); static parity by -O3 GIMPLE and object comparison (final MAX as maxss with a +0 source, no FMA); the schemes W3XYPFSN2, W3XYPFSPSI2, W3XYPFSFCT2, W3XYPFSNIMP and DSEC21 are host hooks, SETDEPTH called ported; a fresh reviewer found no arithmetic defect and fixed one citation; compiled on serial-debug, never run no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once the four scheme hooks, a shim and the W3WAVE wiring exist kokkos/src/ww_kokkos/prop_xypug.hpp no: header-only host function; the four schemes are hooks; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_XYPUG=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) +wminiomd:wmiohg model/src/wminiomd.F90:2260-2266 7 ported the store loop 2.e repeated on the resident VA as a column scatter (ww_va_host_wrote_cols, kernel ww_va.scatter_cols), 2026-10-18; no arithmetic on the device, the Fortran statement runs unchanged on the host (PATCH.md section 26, port report) yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), WW3GPU_LAB_SWITCH=ON matrix entries mww3_test_01 to _04 (resident) and mww3_test_05 (SRCE,OUTG,IOPE,UPDV), test.comp 0 files different, plain and with WW_KOKKOS_VA_POISON=1 (docs/log/2026-10-18-ww3-multi-resident-va.md) kokkos/src/fortran_iface/resident_shim.cpp yes: WW3/ pin a304018c, wminiomd.F90:2318-2327 (W3KOKKOS_VA_WRITE_COLS after the store loop, under the resident VA) +wminiomd:wmioeg model/src/wminiomd.F90:3258-3262 5 ported the stores of 2.a (and of 2.b, :3278, :3326, :3330) repeated on the resident VA: column gather before, column scatter after (ww_va_host_read_cols, ww_va_host_wrote_cols), 2026-10-18; no arithmetic on the device (PATCH.md section 26, port report) yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), the entries of WMIOHG, test.comp 0 files different; whether they reach 2.b was not checked kokkos/src/fortran_iface/resident_shim.cpp yes: WW3/ pin a304018c, wminiomd.F90:3341-3363 and :3457-3463 (column read before 2.a, column write after 2.b, under the resident VA) diff --git a/kokkos/src/fortran_iface/CMakeLists.txt b/kokkos/src/fortran_iface/CMakeLists.txt index 82f9badf..9b2a5549 100644 --- a/kokkos/src/fortran_iface/CMakeLists.txt +++ b/kokkos/src/fortran_iface/CMakeLists.txt @@ -23,6 +23,26 @@ target_sources(ww_kokkos PRIVATE timers.hpp timers.cpp) +# presets.txt, the one list of WW_KOKKOS_PRESET's presets (scripts/ww3-gpu.sh and the regtest +# matrix read the same file), as the initializer part_flags.cpp includes. +set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS + "${CMAKE_CURRENT_SOURCE_DIR}/presets.txt") +file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/presets.txt" _ww_preset_lines) +set(_ww_presets "") +foreach(_l IN LISTS _ww_preset_lines) + if(_l MATCHES "^[ \t]*(#|$)") + continue() + elseif(_l MATCHES "^PRESET ([a-z][a-z0-9_]*) ([A-Z][A-Z0-9_]*(,[A-Z][A-Z0-9_]*)*)$") + string(APPEND _ww_presets " {\"${CMAKE_MATCH_1}\", \"${CMAKE_MATCH_2}\"},\n") + else() + message(FATAL_ERROR "presets.txt: unknown line '${_l}'") + endif() +endforeach() +file(CONFIGURE OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/ww_kokkos_presets.inc" + CONTENT "// Generated from presets.txt by fortran_iface/CMakeLists.txt; do not edit.\n${_ww_presets}" + @ONLY) +target_include_directories(ww_kokkos PRIVATE "${CMAKE_CURRENT_BINARY_DIR}") + set(WW_W3KOKKOSMD "${CMAKE_CURRENT_SOURCE_DIR}/../../../WW3/model/src/w3kokkosmd.F90" CACHE FILEPATH "WW3's w3kokkosmd.F90 (the WW3/ submodule, branch ww3-gpu/kokkos-wire-base)") diff --git a/kokkos/src/fortran_iface/PATCH.md b/kokkos/src/fortran_iface/PATCH.md index 33a002f0..7d095976 100644 --- a/kokkos/src/fortran_iface/PATCH.md +++ b/kokkos/src/fortran_iface/PATCH.md @@ -7,6 +7,8 @@ Read with `git -C WW3 fetch origin && git -C WW3 branch -r` and `git rev-list -- | h0ffmann/WW3 branch | Head | Against the `WW3/` pin | What it wires | |---|---|---|---| +| `ww3-gpu/integ-fixes` | `711afb80` | one commit on `ww3-gpu/integ-abbb0456` (`b733066b`, the merge of `u1-multi-va`, `u2-strict-preset`, `u8-out-readers` and `u9-cprt` onto `80644ba9`), pushed 2026-10-18 (v, `ls-remote`) | `W3KOKKOS_CPRT`'s two fallbacks to the Fortran `W3CPRT` report to `WW_KOKKOS_STRICT` ([section 28](#28-the-integration-fixes-ww3-gpuinteg-fixes)) | +| `ww3-gpu/u1-multi-va` | `f330dc53` | five commits on `80644ba9`, pushed 2026-10-18 (v, `ls-remote`) | `ww3_multi`'s exchanges sync only the columns of `VA` they read or write (`W3KOKKOS_VA_READ_COLS`, `W3KOKKOS_VA_WRITE_COLS`, `ww_va_host_wrote_cols`), no read at `W3WAVE`'s return, the ABI `0.10` ([section 26](#26-ww3_multi-and-the-resident-va-column-syncs-no-read-at-w3waves-return-ww3-gpuu1-multi-va)) | | `ww3-gpu/kokkos-outg` | `e0cc8ee9` | the `WW3/` pin of h0ffmann/ww3-gpu `goal/42-gpu-step-units45`; one commit on `ww3-gpu/kokkos-prop3dev`, pushed 2026-10-17 (v) | `W3OUTG`'s integrals on the device copy of `VA` (`ww_outg_va`, `W3KOKKOS_OUTGMD`), the host read of `VA` at an output time split per reader, the ABI `0.7` ([section 15](#15-w3outg-on-the-resident-spectrum-ww3-gpukokkos-outg)) | | `ww3-gpu/kokkos-prop3dev` | `90631fad` | the parent of `e0cc8ee9`, the `WW3/` pin of h0ffmann/ww3-gpu `goal/42-srce-prop3dev`; one commit on `ww3-gpu/kokkos-resident`, pushed 2026-10-17 (v) | `W3XYP3` and `W3KTP3` batched on the device copy of `VA` (`ww_xyp3_batch_va`, `ww_ktp3_batch_va`), their host syncs only on the Fortran path, the ABI `0.6` ([section 13](#13-the-propagation-on-the-resident-spectrum-ww3-gpukokkos-prop3dev)) | | `ww3-gpu/kokkos-resident` | `513954a6` | the parent of `90631fad`, the `WW3/` pin of h0ffmann/ww3-gpu `goal/42-srce-resident`; two commits on `ww3-gpu/kokkos-prop3`, pushed 2026-10-17 (v) | the resident spectrum: `VA` synced at W3WAVE's host statements (`W3KOKKOS_RESIDENTMD`) and `W3SRCE` batched on the device copy of `VA` (`ww_srce_batch_va`), the ABI `0.5` ([section 12](#12-the-resident-spectrum-on-ww3-gpukokkos-resident)) | @@ -1554,6 +1556,7 @@ section 12, not the whole `VA`. The ABI is `ww_kokkos` 0.8.0 (one entry point ad - **Not covered.** `W3IOTR` (track output) still reads the whole `VA`: the columns it reads come from the track positions it reads from its input file inside the routine (`w3iotrmd.F90:264`, `:819`), so listing them before the call means reproducing that reading; none of the cases writes tracks. + Section 23 covers it, and `W3IOBC ('WRITE')`, from inside `W3IOTR` once the track is read. `ww3_multi` (`wmwavemd.F90:1367` calls `W3IOPE` with no sync) is not supported by the resident `VA` at all (section 12). - **Memory contract**: phase 2. A device buffer of `NSPEC` floats per distinct column and a host one @@ -1833,3 +1836,416 @@ under its switch; without it the packet keeps 0 and the switch gate (`IS_LAB_SWI `W3SRCE` to the Fortran, as before. With the guard, `ww3_tp2.6` `ST4` and `ww3_ts2` `ST4_PR3_UQ` build, and their Kokkos runs stop at `WW_SRCE_INIT` code 5 because the preset asks for `SRCE` on a switch it does not translate (v, same matrix, entries 1 and 7). + +## 23. The boundary and track outputs read only their columns, `ww3-gpu/u8-out-readers` + +Two commits on `80644ba9` (section 22), pushed to h0ffmann/WW3 on 2026-10-18: `31a09522` (a +compile guard) and `791e8c34`. No ABI change: both use `ww_va_host_read_cols` of section 17, and +the fork still asks for `0.8`. They apply the resident-spectrum rule of `AGENTS.md` to the host +readers of `VA` in the output block of `W3WAVE` under `SHRD` (`w3wavemd.F90:2930-2990` at +`791e8c34`). The partial reads stay behind `WW_KOKKOS_IOPE=1`, opt-in, the flag of section 17. + +| Reader | What it reads of `VA` (at `80644ba9`) | Under `WW_KOKKOS_IOPE=1` now | +|---|---|---| +| `W3IOBC ('WRITE')`, boundary output | `VA(IS,ISEA)`, `ISEA = ISBPO(ISOUT)`, `ISOUT = 1..NBO2(NFBPO)`, in the `W3_SHRD` gather (`w3iobcmd.F90:660-668`); `ww3_grid` sets `ISBPO` to sea points only (`w3gridmd.F90:6005-6023`), read with the model definition | `W3KOKKOS_IOBC_READ` (`w3kokkos_pntmd.F90:90`) passes `ISBPO(1:NBO2(NFBPO))` in place of the full read (`w3wavemd.F90:2953-2958` at `791e8c34`) | +| `W3IOTR`, track output | `A(ITH,IK,JSEA)`, `JSEA = ISEA = MAPFS(IY,IX)`, for every point with `MASK1(IY,IX)` and `TSTSTR = 'SEA'`, which is `MAPSTA(IY,IX) > 0` (`w3iotrmd.F90:728-763`, `:774`, `:808-819`); its section 3 sets `MASK1` from the track file, inside the call (`:518-697`) | `W3WAVE` skips its read (`w3wavemd.F90:2936`); `W3IOTR` calls `W3KOKKOS_IOTR_READ` (`w3kokkos_pntmd.F90:107`) after section 3 and before section 4 (`w3iotrmd.F90:718-723` at `791e8c34`), with exactly those columns | +| `W3IOSF`, partitioned spectra | every `JSEA = 1..NSEAL` with `MAPSTA >= 0` (`w3iosfmd.F90:243-265`): every sea point that is not ice, dry or excluded | unchanged, the full read, which the rule allows for a reader of the whole field. A column read of the `MAPSTA >= 0` points would save only the ice and dry points; partitioning on the device belongs to the `W3CPRT` unit | +| The OASIS coupling (`W3_OASIS`, `J = 7`) | none: `SND_FIELDS_TO_ATMOS`, `_OCEAN` and `_ICE` read `W3ADATMD` fields and `ICEF`, and no `VA` (`w3agcmmd.F90:88-220`, `w3ogcmmd.F90:62-404`, `w3igcmmd.F90:76-167`, v by `grep`) | the sync is removed (`w3wavemd.F90:2978`); not compiled here, as there is no OASIS library ⚠ | + +- **Why inside `W3IOTR`.** The columns follow the track points that `W3IOTR` reads from + `track_i.ww3` in its section 3, with `BACKSPACE` and the `MASK2` of the previous call, so a list + made before the call would repeat that reading. The hunk is one `CALL` under + `W3_KOKKOS && W3_SHRD && !W3_MPI`; `W3KOKKOS_IOTR_READ` reads `MASK1` from `W3ODATMD`. It writes + the module `VA` while `W3IOTR` holds the same array as its `INTENT(IN)` dummy `A`. The `-O3` + code loads nothing of `A` before the call: in `gfortran -O3 -fdump-tree-optimized` of + `w3iotrmd.F90` with the entry's switch the call is at line 4477 of the dump and every use of + `a_1999(D)` is at lines 4847-4976 (v, 2026-10-18). The Fortran standard does not allow the actual + argument of a dummy to change through another name during the call, so this rests on the code + the compiler makes, not on the standard ⚠. +- **The resident state** stays `DEVICE` after each partial read, as in section 17: the next full + host read or write still copies the whole array, and the next kernel reads `d_va`. +- **The shim** (`resident_shim.cpp`) keeps up to four column lists per grid with their buffers, + the most recent first, in place of one: the point and the boundary outputs alternate at an output + time, and a single list would allocate the device buffers again at every call. +- **`31a09522`**: `W3KOKKOS_MAP3` and `W3KOKKOS_MAPT` named `W3ADATMD` variables that exist only + under `W3_PR3` (`w3adatmd.F90:520-525`, `:648-653`), so no KOKKOS switch without `PR3` compiled + (`ww3_tr1`, `SHRD RTD PR1`: "Symbol 'nmx0' referenced at (1) not found in module 'w3adatmd'", + v). The `USE` and the stores now sit under `W3KOKKOS_HAS_PR3`, as the calls did; without `PR3` + both parts stay off. +- **Without `KOKKOS`**, `gfortran -E -P` of `w3wavemd.F90` and `w3iotrmd.F90` with the entry's + switch is identical to the output from `WW3-noaa/`, blank lines dropped, apart from the include + path in one comment (v, 2026-10-18). `w3iotrmd.F90` is a fork file of the pnt unit (its `FILE` + line in `parts.d/pnt.txt`). + +The checkpoint on the CPU, 2026-10-18 (OpenMP backend, Kokkos 5.2.2, gfortran 16.2.0, Intel Core +i9-14900 shared with other sessions, `OMP_NUM_THREADS=4`; never `--gpu`). Each entry ran with +`WW3GPU_LAB_SWITCH=ON`, `WW3GPU_STRICT=ON`, `WW_KOKKOS_VA_POISON=1` and `WW_KOKKOS_VA_COUNT=1`, +before (`80644ba9`; for `ww3_tr1`, plus `31a09522`) and after (`791e8c34`). +[The log](../../../docs/log/2026-10-18-resident-output-readers.md) has the commands. + +| Entry, parts | Output it writes | test.comp, after | `VA` before | `VA` after | +|---|---|---|---|---| +| `ww3_tp2.2 -s PR3_UQ`, `resident` | track every 4 h (`track_o.ww3`, `ww3.202206_trck.nc`) | 0/24 files different (v) | D2H 4 (60.07 MB); host reads 8; partial reads 3, 2 of them copying 24 columns | D2H 4 (60.07 MB); host reads 4; partial reads 7, 5 copying 29 columns: the 3 track times after the first copy 5 columns, 4.4 kB | +| `ww3_tr1 -i input_std`, `SRCE,OUTG,IOPE,UPDV` | boundary every 1 h (`nest1.ww3` to `nest3.ww3`) | 0/23 files different (v) | D2H 7 (0.57 MB); host reads 8, writes 6; partial reads 7, 28 columns | D2H 7 (0.57 MB); host reads 1, writes 6; partial reads 14, 84 columns: 8 at each of the 7 boundary times, 70 kB in all | + +The host reads before these outputs are gone, and with the host copy poisoned the outputs still +have the Fortran's bits, so no column a reader needs was missed. The whole-spectrum copies per run +did not drop in these two entries. In `ww3_tp2.2` the track times are the wind times, where +`W3WAVE` returns and its exit read (`w3wavemd.F90:3160` at `791e8c34`) copies `VA` anyway; in +`ww3_tr1` (`PR1`) the host write before the Fortran propagation copies it at every step. What each +reader copied is now its columns, and the count per run drops only where no other full sync +follows in the same step. `bash scripts/ww3-gpu.sh --ww3-gpu smoke|full --cpu --kokkos resident`: +byte-identical to the Fortran and to `--ww3`, `nccmp-tol` PASS, D2H 37 (smoke) and 1 (full), as in +section 17 (v). The matrix driver drops the empty options of a first-kept entry +(`regtests/cmake/ww3gpu_common.cmake:337-343`: a `list(APPEND)` of an empty string onto an empty +list adds nothing), so `ww3_tr1` cannot be configured there; it ran through +`regtests/cmake/run_entry.cmake` directly, with a copy of the configured `ww3gpu_config.cmake` +whose parts are `SRCE,OUTG,IOPE,UPDV` ⚠. Nothing ran on a GPU ⚠. + +## 24. `WW_KOKKOS_STRICT`: a fallback to the Fortran stops or is counted, `ww3-gpu/u2-strict-preset` + +Fork commit `f0cbe70d` on `ww3-gpu/u2-strict-preset`, branched from `80644ba9` and pushed to +h0ffmann/WW3 on 2026-10-18; not at the pin until the integrator merges it into +`ww3-gpu/kokkos-pnt`. Gap 6 of +[`PRODUCTION_READINESS_202610.md`](../../../docs/PRODUCTION_READINESS_202610.md). Three ABI +entries in [`part_flags.cpp`](part_flags.cpp) (`ww_kokkos_strict`, `ww_kokkos_fallback_slot`, +`ww_kokkos_fallback_add`); `w3kokkosmd.F90` +85 lines, `w3initmd.F90` +6 -1, `w3wavemd.F90` 2 +changed. + +Four places ran the Fortran for a part asked on Kokkos and said so only in the parts line: +`KOKKOS_PART` under `WW_KOKKOS_ALL=1` on a switch file a translation does not cover (`GATH`, +`SCAT`, `UWND`, `UTRN`, `MAP3`, `MAPT`; the opt-in parts are never on under the master), +`W3KOKKOS_SRCE_SETUP` on a switch file that is not the lab switch's source terms, `W3INIT` +without `ST4` and `NL1`, and the per-step `W3SRCE` gate (`IQTPE <= 0`, Liu's ice dispersion, +`TAUWBUG = 0` with `IT > 0`). Each now calls `KOKKOS_FALLBACK ( PART, REASON )` +(`w3kokkosmd.F90:1257-1269` at `237ca0de`): + +- **`WW_KOKKOS_STRICT=1`**: `*** WAVEWATCH III ERROR IN W3KOKKOSMD : WW_KOKKOS_STRICT=1 AND PART + WOULD RUN ON THE FORTRAN: ` on `NDSE`, then `EXTCDE(1)`, on every rank. +- **Unset (the default)**: the Fortran, as before, and the shim counts the call. At exit each + process prints one line to `stderr`, as the `*_COUNT` lines do: + `ww_kokkos: calls left on the Fortran (WW_KOKKOS_STRICT=1 stops instead): SRCE=0`. A set-up + fallback counts 1 per `W3INIT`; `SRCE` counts the time steps whose batch ran the Fortran. The + line appears only when some part could fall back. + +**The hot path.** The per-step gate is a new function, `W3KOKKOS_SRCE_STEP ( IT )` +(`w3kokkosmd.F90:1912-1917`), which `W3WAVE` calls once per step in place of +`W3KOKKOS_SRCE_ON` before the batch (`w3wavemd.F90:2407`), outside the `!$OMP PARALLEL` loop over +sea points. It returns `W3KOKKOS_SRCE_ON ( IT )` and adds one test of two `LOGICAL`s; only on +the fallback branch does it call `SRCE_FALLBACK`, which under the default is one call and one +increment of a plain counter (`ww_kokkos_fallback_add`; no lock, the caller is serial). The +counter's slot is made once in `W3KOKKOS_SRCE_SETUP`. `WW_KOKKOS_STRICT` is read once by the shim +(a function-local static) and cached in `KOKKOS_STRICT` by `W3KOKKOS_SETUP`. The per-point +`W3KOKKOS_SRCE_ON` of the Fortran loop (`w3wavemd.F90:2485`) is unchanged: that loop runs only +when the step's gate was false, so counting the step is complete without a counter in the loop. + +**The matrix.** `ww3gpu_kokkos_env` (`regtests/cmake/ww3gpu_common.cmake`) sets +`WW_KOKKOS_STRICT=1` in the Kokkos run of every pair and unsets it in the Fortran runs (`F`, +`G`), so a pair can no longer compare the Fortran with itself unseen: an entry whose Kokkos run +would fall back now fails at run time instead. This changes how a comparison is invoked and is +reviewed by a person before it merges (AGENTS.md, "Never cheat a regtest tool"). + +**What it was checked on** (2026-10-18, i9-14900, `OMP_NUM_THREADS=8`, the machine shared with +three other jobs, so every wall time is one run and ⚠). The runs re-ran `ww3_grid`, `ww3_strt`, +`ww3_shel` and `ww3_ounf` in a copy of the smoke case's Kokkos directory with every `WW_KOKKOS_*` +unset and only the variables named: + +| Run | Result | +|---|---| +| `bash scripts/ww3-gpu.sh --ww3-gpu smoke --cpu --kokkos resident`, before (`80644ba9`) and after | every model file byte-identical to the Fortran's, and the after run's four files identical (`cmp`) to the before run's (v); `ww3_shel` with Kokkos 3.96 s before, 3.65 s and 10.93 s after (load average 17 during the second) ⚠ | +| the same smoke, `WW_KOKKOS_ALL=1` | `SRCE=0` in the line at exit (v) | +| smoke with `TAUWBUG = 0` in `&SIN4`, `WW_KOKKOS_ALL=1` | `ww_kokkos: calls left on the Fortran (WW_KOKKOS_STRICT=1 stops instead): SRCE=2160` (v) | +| the same, `WW_KOKKOS_STRICT=1` | stops: `... WW_KOKKOS_STRICT=1 AND PART SRCE WOULD RUN ON THE FORTRAN: TAUWBUG = 0 (SINTAILPAR(4) <= 0.5) WITH IT > 0, WHICH NO TRANSLATION CAN REPRODUCE` (v) | +| `switch_lab_ompg` with `WNT2` for `WNT1` (a scratch copy, not committed), `--kokkos all` | `UWND=1 SRCE=0` at exit; the pair byte-identical (v) | +| the same build, `WW_KOKKOS_ALL=1 WW_KOKKOS_STRICT=1` | stops: `... PART UWND WOULD RUN ON THE FORTRAN: WW_KOKKOS_ALL=1 ASKS FOR IT, BUT ITS KOKKOS TRANSLATION DOES NOT COVER THIS SWITCH FILE` (v) | + +## 25. `WW_KOKKOS_PRESET`: the resident part list in one table, `ww3-gpu/u2-strict-preset` + +Fork commit `237ca0de` on `ww3-gpu/u2-strict-preset` (the child of `f0cbe70d`); not at the pin +until the integrator merges it. Gap 7 of +[`PRODUCTION_READINESS_202610.md`](../../../docs/PRODUCTION_READINESS_202610.md), smallest form. +One ABI entry, `ww_kokkos_preset_status`; `w3kokkosmd.F90` +24 -1 lines. + +[`presets.txt`](presets.txt) holds one line per preset, `PRESET `; today only +`PRESET resident SRCE,XYP3,KTP3,OUTG,IOPE,UPDV`, the list that was a line of +`scripts/ww3-gpu.sh`. Three readers, one file: + +- **The shim.** `CMakeLists.txt` here turns the file into `ww_kokkos_presets.inc` at configure + time, and `part_flags.cpp` includes it. `ww_kokkos_flag` reads `WW_KOKKOS_PRESET` once: a set + `WW_KOKKOS_` still wins, then a part of the preset is on as if set to `1` (so a part the + switch file does not cover stops the model, as under `--kokkos resident`), then the opt-in + list and `WW_KOKKOS_ALL` decide as before. +- **`scripts/ww3-gpu.sh`**: `kokkos_preset` reads the `PRESET` lines, so `--kokkos resident` + sets the same six `WW_KOKKOS_=1`. Both runs of a pair unset `WW_KOKKOS_PRESET`, and the + Fortran run unsets `WW_KOKKOS_STRICT`. +- **The matrix**: `ww3gpu_kokkos_preset` reads it for `WW3GPU_KOKKOS=`; `selftest.cmake` + checks a fixture table and that the real `resident` names only parts of `parts.d`. + +`W3KOKKOS_SETUP` (`w3kokkosmd.F90:1182-1188` at `237ca0de`) stops on a name the file lacks: +`*** WAVEWATCH III ERROR IN W3KOKKOSMD : WW_KOKKOS_PRESET= IS NOT A PRESET OF +kokkos/src/fortran_iface/presets.txt`. `W3KOKKOS_REPORT` writes ` Kokkos preset +(WW_KOKKOS_PRESET): resident` on the line before the parts line, which stays the effective set, +once per `W3INIT`. No namelist: the next step of gap 7. + +Checked the same way as section 24 (2026-10-18): `WW_KOKKOS_PRESET=resident` alone gives the +parts line of `--kokkos resident` character for character and the same four model files (`cmp`, +v); `WW_KOKKOS_PRESET=bogus` stops with the line above (v). +## 26. `ww3_multi` and the resident `VA`: column syncs, no read at `W3WAVE`'s return, `ww3-gpu/u1-multi-va` + +Five commits on `80644ba9` (section 22), pushed to h0ffmann/WW3 on 2026-10-18: `3bbd1557` +(`W3KOKKOS_RESIDENTMD` and the ABI), `bb5c516e` (`wminiomd.F90`), `f4df761a` (`wmwavemd.F90`), +`5ef94512` (`w3wavemd.F90`) and `f330dc53` (the remark of gap 13 of +[`PRODUCTION_READINESS_202610.md`](../../../docs/PRODUCTION_READINESS_202610.md)). Unit U1 + U7 +of [the production single-node plan](../../../docs/compound/plans/2026-10-18-002-production-single-node-plan.md), +under the resident-spectrum rule of `AGENTS.md`. The ABI is `ww_kokkos` 0.10.0 (one entry point +added, `ww_va_host_wrote_cols`); the fork asks for `0.10`. + +**The defect.** `ww3_multi` reads and writes the `VA` of a grid on the host between two `W3WAVE` +calls of that grid. Before this change `W3WAVE` ended with a whole-array read +(`w3wavemd.F90:3151-3153` at `80644ba9`), which left both copies "equal"; the writes below then +changed the host copy with no sync, and the next kernel on the device copy (`W3XYP3`, `W3KTP3` or +the batched `W3SRCE`) read it without the nesting update. `WW_KOKKOS_VA_CHECK=1` stops such a run; +without it the outputs differ (the table below). + +**Every host use of `VA` outside `W3WAVE`** in the drivers at `80644ba9`, read with +`grep -n "VA" ww3_shel.F90 wm*.F90` and every routine those files call that takes `VA` (v, +2026-10-18): + +| Site (line at `80644ba9`) | Use | Under `W3_KOKKOS` now | +|---|---|---| +| `WMIOBS`, `wminiomd.F90:374` | reads `VA(:,JSEA)` at the boundary points it sends, `JSEA` from `NBI2S(IOFF+I,2)` (`SHRD`) or `INIT_GET_JSEA_ISPROC` (`DIST`, local points only) | `W3KOKKOS_VA_READ_COLS` with that list, per receiving grid (`wminiomd.F90:365-384` at `f330dc53`) | +| `WMIOHS`, `:1636`, `:1640` | reads `VA(:,JSEA)`, `JSEA = HGSTGE(J,IMOD)%ISEND(I,1)`, `I = 1..NR` | `W3KOKKOS_VA_READ_COLS` with `ISEND(1:NR,1)` (`:1651-1659`) | +| `WMIOHG`, `:2264` | writes every element of `VA(:,JSEA)`, `JSEA = HGSTGE(IMOD,J)%LJSEA(IS)`, `IS = 1..NTOT`, reads none | `W3KOKKOS_VA_WRITE_COLS` with `LJSEA(1:NTOT)` after the loop (`:2318-2327`) | +| `WMIOES`, `:2726`, `:2732`, `:2736` | reads `VA(:,JSEA)`, `JSEA = EQSTGE(J,IMOD)%SJS(I)`, `I = 1..NR` | `W3KOKKOS_VA_READ_COLS` with `SJS(1:NR)` (`:2772-2780`) | +| `WMIOEG`, `:3261`, `:3278`, `:3326`, `:3330` | reads and writes `VA(:,JSEA)`: `JSEA = EQSTGE(IMOD,IMOD)%JSEA(I)` (2.a) and `EQSTGE(IMOD,J)%JSEA(I)` for every other grid with `NREC > 0` (2.b, SMC included) | the union of those lists, read before 2.a and stored after 2.b (`:3341-3363`, `:3457-3463`) | +| `WMWAVE`, `wmwavemd.F90:1367` | `W3IOPE ( VA )`, the unified point output | as before `W3WAVE`'s own `W3IOPE`: `W3KOKKOS_IOPE_READ` under `WW_KOKKOS_IOPE=1`, otherwise `W3KOKKOS_VA_HOST_READ`; under `MPI` a host read before the `W3MPIP` requests start (`wmwavemd.F90:1365-1370`, `:1379-1387`) | +| `ww3_shel.F90` | none: `VA` is named only in commented-out lines (`:1950-2003`) | nothing | +| `WMINIT` (`wminitmd.F90:2993-3006`) | calls `WMIOBS` before any kernel ran | covered by the `WMIOBS` row | + +`W3IOBC`, `W3IORS`, `W3IOSF`, `W3IOTR` and `W3OUTG` read `VA` only when `W3WAVE` calls them, +and the sites of sections 12, 15, 17 and 23 sync them. `wmupdtmd.F90:1407-1844` names a local +scalar `VA`, not the spectrum. `WMIOBG` writes `BBPI0`/`BBPIN`, not `VA`. `PDLIB_exchange2Dreal_zero +( VA )` after `WMIOHG` and `WMIOEG` runs only under `PDLIB`, where no resident part is allowed +(sections 13, 15, 17), so the state stays `HOST` there. + +- **`W3KOKKOS_VA_READ_COLS ( N, COLS )`**, `W3KOKKOS_VA_WRITE_COLS ( N, COLS )` + (`w3kokkos_residentmd.F90:86-132` at `f330dc53`): `COLS` are `JSEA`, `0..NSEALM`, of the + current grid (`IGRID`, `VA` of `W3SETG`/`W3SETW`); `N < 1` does nothing; both stop the model + inside an OpenMP parallel region, as every sync. The read is `ww_va_host_read_cols` (section 17). +- **`ww_va_host_wrote_cols`** (`resident_shim.cpp`, declared in `resident_c.hpp`): with the state + `DEVICE` or `BOTH`, the listed host columns are packed into the list's host buffer, copied once + to the device buffer and scattered into `d_va` by one `parallel_for` (`ww_va.scatter_cols`); with + `HOST`, nothing. The state is not changed. It shares the column lists of the partial read + (section 23), now up to eight per grid (`kMaxColSets`): `ww3_multi` uses one list per + neighbouring grid and direction at every exchange. Bits move and nothing is computed. Under + `WW_KOKKOS_VA_CHECK=1` the listed columns are taken as written and every other column must still + hold the bits of the last sync; under `WW_KOKKOS_VA_RESIDENT=1` `d_va` is then copied back + whole, so a column written and not listed shows in the output. +- **`W3WAVE`'s return** no longer reads `VA` back (section 12's `:3083` row, `:3151-3153` at + `80644ba9`). A run of `ww3_shel` may end with the device copy newer; `resident_shim.cpp` + still says so at finalize, and `scripts/ww3-gpu.sh` now prints that line instead of failing on + it (the check of mutant A2 in section 21 guarded the read this change removes). The check that + a host reader missed its sync is `WW_KOKKOS_VA_POISON=1`. +- **Why `wminiomd.F90` and `wmwavemd.F90` join the fork's diff.** The plan's first choice (B1) + marked the host newer at `W3WAVE`'s return, inside the files the fork already changed; the owner + rejected it on 2026-10-18, because on a GPU it costs a whole host-to-device copy of `VA` per + grid per call. The syncs belong where `ww3_multi` reads and writes, and they move only the + columns exchanged. Both files are `FILE` lines of `parts.d/resident.txt`, so + `scripts/ww3-gpu.sh --ww3` still accepts the fork. +- **Without `KOKKOS`**, `gfortran -E -P` of `w3wavemd.F90`, `wminiomd.F90` and `wmwavemd.F90` with + `switches/switch_lab_shrd` and with `mww3_test_08`'s `switch_ST4_PR3_UQ_MPI` is identical to + `WW3-noaa/`'s, blank lines and the include path in one comment aside (v, 2026-10-18). With + `KOKKOS` and that MPI switch, `ww3_multi` builds (v, `cmake -S -DSWITCH= + -DWW_KOKKOS=ON`, `cmake --build ... --target ww3_multi`); it was not run (ADR-0017). + +The checkpoint on the CPU, 2026-10-18 (OpenMP backend, gfortran 16.2.0, Intel Core i9-14900 +shared with other sessions, 2 matrix jobs of 4 threads; never `--gpu`), with +`WW3GPU_LAB_SWITCH=ON` and `WW3GPU_STRICT=ON`, before (`80644ba9`) and after (`f330dc53`'s tree). +[The log](../../../docs/log/2026-10-18-ww3-multi-resident-va.md) has the commands; the summaries +are [`ww3gpu_matrix_multi_va_2026-10-18.tsv`](../../../docs/data/ww3gpu_matrix_multi_va_2026-10-18.tsv). + +| Entry, parts | Before, `VA_CHECK=1` | Before, plain: test.comp | After, poisoned and plain: test.comp | Whole `VA` D2H, before → after | Columns after (read; stored) | +|---|---|---|---|---|---| +| 33 `mww3_test_01` `PR3_UQ`, `resident` | runs | 0/22 | 0/22 | 24 (24.13 MB) → 12 (12.07 MB) | 1 440 (3.46 MB); 0 | +| 34 `mww3_test_02` `PR3_UQ` `grdset_a`, `resident` | stops: 2 columns of grid 1 changed | 5/23, `ww3_self` 0/23 | 0/23 | 384 (853.40 MB) → 8 (32.29 MB) | 28 032 (67.28 MB); 18 432 (44.24 MB) | +| 38 `mww3_test_03` `PR3_UQ` `grdset_a`, `resident` | stops: 110 columns of grid 1 | 10/34, `ww3_self` 0/34 | 0/34 | 216 (441.68 MB) → 216 (441.68 MB) | 100 224 (192.43 MB); 49 680 (95.39 MB) | +| 43 `mww3_test_04` `PR3_UQ` `grdset_a`, `resident` | stops: 240 columns of grid 2 | 4/25, `ww3_self` 0/25 | 0/25 | 864 (3 002.99 MB) → 0 | 384 372 (737.99 MB); 71 388 (137.07 MB) | +| 2, 8 `mww3_test_05` `ST1`/`ST4_PR3_UQ_OMP` (moving grid), `SRCE,OUTG,IOPE,UPDV` | stops: 138 columns of grid 2 | 0/27 | 0/27 | 156 (790.79 MB) → 156 (790.79 MB) | 66 120 (158.69 MB); 6 312 (15.15 MB) | + +- Every after run is byte-identical with the host copy poisoned, so no host reader of `VA` outside + `W3WAVE` on these entries reads a column it did not sync (v). +- The whole-array copies went down or stayed; none went up (v, `WW_KOKKOS_VA_COUNT=1`). In 38 the + 216 that stay are one per step of the `W3OUTG` fallback inside `W3WAVE` (`W3OUTG Kokkos calls: 0` + in that run), and in `mww3_test_05` the 156 each way are the host write of the Fortran + propagation (`XYP3` and `KTP3` refuse a moving grid): the output block and the Fortran + propagation, not the exchanges. +- `mww3_test_05` passes before without the diagnostic because, with `XYP3` and `KTP3` off, a + host write of the Fortran propagation (`w3wavemd.F90:1885` or `:2037`) comes before any kernel + in each step and declares the host newer, as gap 2 predicted. +- `ww3_shel`: `bash scripts/ww3-gpu.sh --ww3-gpu full|smoke --cpu --kokkos resident`, plain and + with `WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_CHECK=1`: every model file byte-identical to the + Fortran's and every file but the NetCDF to `--ww3`'s, `nccmp-tol` PASS; whole-`VA` D2H on full + 1 → 0 (one copy of 21.21 MB per run), on smoke 37 → 37 (the output readers of section 23) (v). + Wall time of full, one run each on the shared machine ⚠: 40.86 s before, 43.99 s and 32.35 s + after; no difference can be read from these. +- `mww3_test_07` (`default`) was not compared: with the lab switch its `ww3_grid` stops in the + Fortran run too (`ERROR IN READING SDS4`, the entry's namelist against `ST4`'s). + +## 27. `W3CPRT` on the resident `VA`, and `W3OUTG`'s partition fields without it, `ww3-gpu/u9-cprt` + +One commit on `80644ba9` (section 22), `4f126d85`, pushed to h0ffmann/WW3 on 2026-10-18; not at the +pin until the integrator merges it. ABI `ww_kokkos` 0.10 (three entry points added: `ww_cprt_va`, +`ww_cprt_fetch` of [`cprt_c.hpp`](cprt_c.hpp), `ww_outg_part` of [`outg_c.hpp`](outg_c.hpp)). +Unit U9 of the production plan: at every output time with partitioning on, `W3WAVE` read the whole +`VA` back from the device for `W3CPRT` (`w3wavemd.F90:2688-2693` at `80644ba9`), again before +`W3IOSF` (`:2963-2966`), and `W3KOKKOS_OUTG` fell back to the Fortran `W3OUTG`, with its own host +read, whenever a partition field was asked for (`FLPART`). GFS-Wave writes partitions. + +- **The fork.** A unit file, `model/src/w3kokkos_cprtmd.F90` (`W3KOKKOS_CPRTMD`): the flag + `KOKKOS_CPRT` from `WW_KOKKOS_CPRT` (opt-in, `kOptInOnly` of [`part_flags.cpp`](part_flags.cpp)), + refused outside `SHRD` and under `MPI` or `T`; `W3KOKKOS_CPRT ( IMOD )` packs what `W3CPRT` + reads, calls `ww_cprt_va`, allocates `ICPRT` and `DTPRT` as `W3CPRT` does (`w3iosfmd.F90:221-229`, + `:320-326`), writes `PTMEAN`'s format 1000 the number of times the kernel counted, and calls + `ww_cprt_fetch`. A sea point with a NaN depth, or a spectrum `PTMEAN` cannot take, makes it read + `VA` to the host and call the Fortran `W3CPRT` whole instead (the first stops the model there, + `:270-274`; the second reads out of bounds in the Fortran, + [`PTMEAN_KOKKOS.md`](../../port_report/PTMEAN_KOKKOS.md)). In `w3wavemd.F90`, two hunks: at + `:2688` the host read and `CALL W3CPRT` become `IF ( LOCAL .AND. FLPART .AND. KOKKOS_CPRT ) CALL + W3KOKKOS_CPRT ( IMOD ) ELSE` the old lines; before `W3IOSF` the host read becomes + `IF ( .NOT. KOKKOS_CPRT )`. `W3IOSF` reads `ICPRT`, `DTPRT`, `DW`, `U10`, `U10D`, `CX`, `CY`, + `ASF`, `TIME` and the grid, and no `VA` (its `USE` lists, `w3iosfmd.F90:429-453`, v), so it needs + no host copy once `W3CPRT` is on Kokkos. In `w3kokkos_outgmd.F90`, `FLPART` no longer forces the + Fortran `W3OUTG`: after `ww_outg_va` (run with `FLPART` false, as before), `W3KOKKOS_OUTG` calls + `ww_outg_part` for section 6, which reads the table and no `VA` (`w3iogomd.F90:2242-2340`) and + whose fields no other section reads. That holds whichever of `W3CPRT` or `W3KOKKOS_CPRT` built + the table. `model/src/CMakeLists.txt` asks for `ww_kokkos 0.10`. +- **The kernel** ([`../ww_kokkos/iosf_w3cprt.hpp`](../ww_kokkos/iosf_w3cprt.hpp), the shim + [`cprt_shim.cpp`](cprt_shim.cpp)). The sea points go in chunks (`WW_KOKKOS_CPRT_CHUNK`, by default + four per thread on a host backend, 1024 on a device), one work slot per point. Per chunk: a launch + runs `W3CPRT` sections 2-4 and `W3PART` (unchanged, + [`W3PART_KOKKOS.md`](../../port_report/W3PART_KOKKOS.md)) per point in the Fortran's serial + order; an integer reduction counts format-1000 messages and no-peak points; an integer prefix sum + over `NP+1` gives each point its first `DTPRT` column and `ICPRT(JSEA+1,2)`; a launch copies + `XP(:,0:NP)` there. Integer sums are exact in any order, so the table is in the Fortran's `JSEA` + order by construction. The static parity argument is in + [`W3CPRT_KOKKOS.md`](../../port_report/W3CPRT_KOKKOS.md). +- **Memory contract**: phase 2. `VA` is read in `d_va` (`ww::resident::device_va`). On a device the + inputs (`MAPSF`, `MAPSTA`, `U10`, `U10D`, `ASF`, `DW`, `CG`, `WN`, the frequency and direction + tables, `EWN1`, `ECG1`) are copied in per call, `CG` and `WN` being most of it; `ICPRT` and the + `FINSIZ` used columns of `DTPRT` come back. The work slots, 288 kB per point for `NK = NTH = 24` + and `IHMAX = 50` (`w3cprt_nreal` and `w3cprt_nint`) (`PTMEAN`'s sums are sized for `NSPEC+1` partitions), live for the run. `ww_outg_part` + copies `ICPRT` and `DTPRT` in on a device and only the partition fields `W3IOGO` writes back + (`FLOGRD(4,k)` and `FLOGR2(4,k)`, `w3iogomd.F90:3283-3367`). No Unified Memory. +- **Counters**: `WW_KOKKOS_CPRT_COUNT=1` (calls, sea points, fallbacks, bytes in and back); + `WW_KOKKOS_OUTG_COUNT=1` adds the partition-field calls and bytes. **Timers**: `cprt.call`, + `cprt.h2d`, `cprt.kernel`, `cprt.d2h`, `outg.part`. +- **Not covered.** `W3_MPI` (the `DIST` runs keep the single host read of section 11); the + `WW_KOKKOS_STRICT` count of section 24 (the two fallbacks above are not reported to it: this + branch is on `80644ba9`, without `KOKKOS_FALLBACK`); the `resident` preset of section 25 does not + list `CPRT`, so the runs below name the parts. + +The checkpoint on the CPU, 2026-10-18 (OpenMP backend, Kokkos 5.2.2, gfortran/g++ 16.2.0, i9-14900 +shared with other agents' jobs, `OMP_NUM_THREADS=4`; never `--gpu`); the commands, the two +throwaway `bench_small` variants with partitioning on and the numbers are in +[`docs/log/2026-10-18-cprt-resident.md`](../../../docs/log/2026-10-18-cprt-resident.md): + +- `bash scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV[,CPRT]` with + `WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1`, five runs on `switch_lab_shrd` and + `switch_lab_ompg`: every model file and `partition.ww3` byte-identical to the Fortran of the + same executable and to `--ww3` (v). `VA` device-to-host copies in 12 hours with 13 output times: + 13 before, 1 after (the read at `W3WAVE`'s return, which section 26 removes), with only the + partition file and with all 17 partition fields (v). Per output time a device would move back + 1.14 MB of table and, with every partition field requested, 3.39 MB of fields, against the + 21.21 MB of `VA`, and in 2.21 MB of inputs (projected on the host backend). +- WW3's `ww3_tpt1.1`, the five partitioning methods (`-N -g PTM1` .. `PTM5`), through + `regtests/cmake/run_entry.cmake` with `WW3GPU_KOKKOS=CPRT`: `test.comp` 0 of 22 files different + and `nccmp-tol` PASS on each, 2 Kokkos calls on 18 sea points each (v), with `PR3 UQ` in the + staged switch for its `PR0` ⚠, because the fork's `KOKKOS` build needs `PR3`. No `matrix.base` + entry with partitioning builds with `KOKKOS` (`ww3_tnc1` is `PR0`), so the matrix proper did not + run this part ⚠. + +## 28. The integration fixes, `ww3-gpu/integ-fixes` + +One fork commit, `711afb80`, on `b733066b` of `ww3-gpu/integ-abbb0456` (sections 23 to 27 merged +onto `80644ba9`), pushed to h0ffmann/WW3 on 2026-10-18; not at the pin until the integrator moves +it. No ABI change. + +- **The `CPRT` fallbacks report to strict mode.** Section 27 left them out of section 24's count. + `W3KOKKOS_CPRT` now calls `KOKKOS_FALLBACK ( 'CPRT', REASON )` on both branches that run the + Fortran `W3CPRT` (`w3kokkos_cprtmd.F90:167-175` at `711afb80`): a sea point with a NaN depth, and + a spectrum `PTMEAN` cannot take (the kernel's `FALLBACK` flag). With `WW_KOKKOS_STRICT=1` the + model stops on `... PART CPRT WOULD RUN ON THE FORTRAN: `; without it the call is counted, + one per output time that fell back. `W3KOKKOS_CPRT_SETUP` makes the counter when the part is on + (`:78-81`), so the line at exit shows `CPRT=0` when nothing fell back. The path that runs the + kernel adds one `LOGICAL` store per call and nothing per point. Neither fallback was made to + happen in a run: no case here has a NaN depth or a spectrum that trips `PTMEAN` ⚠. +- **`CPRT` joins the `resident` preset** ([`presets.txt`](presets.txt): + `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT`). Its consumers changed with it: the shim compiles the file + (`just kokkos-test serial-debug` and `openmp-release`), `scripts/ww3-gpu.sh --self-test` checks + the seven flags, and `regtests/cmake/selftest.cmake` has `CPRT` in its fixture `parts.d` and in + the expected list. The selection words stay `ST4;NL1;PR3;UQ;SHRD;!PDLIB;!MGP;!MGG`: under a set + flag `KOKKOS_PART` stops on a switch the part does not cover, and `CPRT` refuses a switch without + `SHRD` or with `MPI` or `T` (`w3kokkos_cprtmd.F90:70-76`), a subset of what `OUTG`, already in + the preset, refuses (`SHRD` missing, `MPI`, `IG1`, `O8`, `O9`, `T`, `w3kokkos_outgmd.F90:74-80`). + So the preset stops on no switch it ran before (v, by reading both set-ups). +- **The `KOKKOS` build without `PR3`.** The `PR3`-only `USE` at `w3kokkosmd.F90:3183` of + `80644ba9` is the one `31a09522` (section 23) put under `W3KOKKOS_HAS_PR3`, and `b733066b` + contains that commit (`git merge-base --is-ancestor`, v). Section 27's note that the fork's + `KOKKOS` build needs `PR3` was written on `u9-cprt`, which branched before it. No fork change + was needed: `ww3_tnc1` (`SHRD PR0`) builds with `KOKKOS` at `b733066b` and at `711afb80` (v, + below). + +`ww3_tnc1`, matrix entry 38, the serial `matrix.base` entry that writes partition fields (`PHS +PTP`, `FIELD%PARTITION = '0 1 2'`), on 2026-10-18 (OpenMP backend, gfortran 16.2.0, i9-14900 +shared with other agents, `OMP_NUM_THREADS=4`; never `--gpu`), with `WW3GPU_LAB_SWITCH=ON`, +`WW3GPU_STRICT=ON`, `WW3GPU_SWITCH_WORDS=PR0;SHRD`, `WW_KOKKOS_VA_POISON=1` and +`WW_KOKKOS_VA_COUNT=1`: + +| `WW3GPU_KOKKOS` | Result | +|---|---| +| `resident` | builds; `CPRT=K OUTG=K IOPE=K UPDV=K SRCE=K`, 3 `W3CPRT` Kokkos calls on 243 sea points and 3 partition-field calls of `W3OUTG`; test.comp 0/30 files different; the entry fails its parts-line check, `XYP3,KTP3` not on Kokkos (v) | +| `SRCE,OUTG,IOPE,UPDV,CPRT` | passes: test.comp 0/30, `CPRT` fallbacks 0, whole-`VA` D2H 0, `calls left on the Fortran ...: SRCE=0 CPRT=0` (v) | + +The first row shows a gap of the preset that `CPRT` does not cause: without `PR3` and `UQ`, +`W3KOKKOS_PROP3_SETUP` compiles its body out (`w3kokkos_prop3md.F90:111-138`), so `XYP3` and `KTP3` +never reach `KOKKOS_PART`, the run neither stops nor counts them, and only the matrix's parts-line +check reports it. The selection words keep every `PR0` entry out of a `resident` matrix run. + +Closed by a second fork commit, `a304018c` on `711afb80` (the integrator, 2026-10-18): +`W3KOKKOS_PROP3_SETUP` gains an `#else` branch for a switch without `PR3` and `UQ` that asks +`KOKKOS_PART ( 'XYP3', .FALSE. )` and `( 'KTP3', .FALSE. )` and registers both. A set flag or a +preset now stops, and `WW_KOKKOS_ALL=1` counts or, under `WW_KOKKOS_STRICT=1`, stops. On +`switch_lab_shrd` and `switch_lab_ompg` the preprocessed module (`gfortran -E -P`, `#ww3` shell) is +the same as before apart from blank lines (v). `ww3_tnc1` with `WW3GPU_KOKKOS=resident`, as above, +at `a304018c`: the Kokkos run stops in `W3INIT` with `WW_KOKKOS_XYP3=1, BUT ITS KOKKOS TRANSLATION +DOES NOT COVER THIS SWITCH FILE`, the row reads `ran fortran-only` (v), where at `711afb80` it ran +with both parts silently on the Fortran. + +`bash scripts/ww3-gpu.sh --ww3-gpu smoke|full --cpu --kokkos resident` with `WW_KOKKOS_VA_POISON=1 +WW_KOKKOS_VA_COUNT=1`, at `711afb80` (2026-10-18, same machine, one run each): every model file +byte-identical to the Fortran of the same executable and every file but the NetCDF to `--ww3`'s, +`nccmp-tol` PASS; whole-`VA` D2H 37 on smoke and 0 on full, as in section 26; `CPRT=K` with 0 calls, +since neither case partitions (v). + +## 29. The ESMF cap reads the resident `VA` before its exports, `ww3-gpu/integ-fixes` + +Fork commit `50a10730` on `a304018c`, 2026-10-18. Section 26 removed the read at `W3WAVE`'s return +and synced every host use of `VA` in `ww3_shel` and `ww3_multi`. `wmesmfmd.F90`, the ESMF cap of +`ww3_multi` (built only with `MULTI_ESMF`, `model/src/CMakeLists.txt:193-199`), reads the export +grid's `VA` on the host after `WMWAVE` in `SetExport` (`wmesmfmd.F90:2824-3046` at `a304018c`): +`W3OUTG` under `USE_W3OUTG_FOR_EXPORT`, `CalcCharnk` and `CalcRoughl` (`W3SPR3`/`W3SPR4` on +`va(:,jsea)`), `CalcStokes3D`, `CalcPStokes`, `CalcBotcur`, `CalcRadstr2D`. Every one reads the +whole field, so one `W3KOKKOS_VA_HOST_READ` after the grid's `w3set*` calls covers them; the +"Resident spectrum" rule allows a whole-field read before an output that needs every point. Its +`w3uini ( va )` at `:2180` runs in `InitializeP3` before the resident `VA` is bound (the C++ side +binds it on the first sync, `w3kokkos_residentmd.F90:68-70`), so it needs no sync. + +- Without `KOKKOS`, `gfortran -E -P` with `switch_lab_shrd` gives `WW3-noaa/`'s file apart from the + `__LINE__` values `ESMF_LOGERR_PASSTHRU` writes into error messages and the include-path comment + (v); with `KOKKOS`, only the `USE` and the read are added (v). +- Not compiled and not run: there is no ESMF library here ⚠. No regtest of this repository builds + `MULTI_ESMF`, so the matrix evidence of section 26 and of U3 holds at this commit. +- Not covered: the NUOPC cap of the UFS (`wav_comp_nuopc.F90`, `wav_import_export.F90`, + `wav_restart_mod.F90`), which is not in the fork at `a304018c`; `production/GFS.v17` has it, with + 16 and 13 uses of `VA` in the last two (v, `git show`). The pin move to that branch + ([plan](../../../docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md)) lists and syncs them. diff --git a/kokkos/src/fortran_iface/cprt_c.hpp b/kokkos/src/fortran_iface/cprt_c.hpp new file mode 100644 index 00000000..1cccbd57 --- /dev/null +++ b/kokkos/src/fortran_iface/cprt_c.hpp @@ -0,0 +1,69 @@ +// kokkos/src/fortran_iface/cprt_c.hpp +// The C ABI of the cprt wiring unit: W3CPRT's spectral partitioning on the resident VA (PATCH.md +// section 27). Declarations only; cprt_shim.cpp implements them, and model/src/w3kokkos_cprtmd.F90 +// on the fork (h0ffmann/WW3 ww3-gpu/u9-cprt) mirrors them. The rules of ww_kokkos_c.hpp hold: +// Fortran order, caller-owned arrays, float is WW3's REAL, nothing throws. +// +// SPDX-License-Identifier: MIT +#pragma once + +extern "C" { + +/// The packed integers of ww_cprt_va, in this order (1-based in the Fortran). +enum WwCprtIpar { + WW_CPRT_NK = 0, + WW_CPRT_NTH, + WW_CPRT_NSEAL, + WW_CPRT_NSEA, ///< rows of MAPSF, U10, U10D; columns 0:NSEA of CG, WN and DW + WW_CPRT_NX, ///< MAPSTA(NY,NX) + WW_CPRT_NY, + WW_CPRT_NASF, ///< SIZE(ASF), at least NSEA + WW_CPRT_PTMETH, ///< PTMETH, IHMAX, FLCOMB (1 or 0) of W3ODATMD for the current output grid + WW_CPRT_IHMAX, + WW_CPRT_FLCOMB, + WW_CPRT_DIMP, ///< DIMP of W3ODATMD, 15 + WW_CPRT_NAR1D, ///< NAR1D and N1MAX of W3DISPMD + WW_CPRT_N1MAX, + WW_CPRT_NIPAR +}; + +/// The packed reals of ww_cprt_va, in this order. +enum WwCprtRpar { + WW_CPRT_DTH = 0, ///< DTH, XFR, FACHFE, FTE of W3GDATMD + WW_CPRT_XFR, + WW_CPRT_FACHFE, + WW_CPRT_FTE, + WW_CPRT_HSPMIN, ///< HSPMIN, WSMULT, WSCUT, PTFCUT of W3ODATMD + WW_CPRT_WSMULT, + WW_CPRT_WSCUT, + WW_CPRT_PTFCUT, + WW_CPRT_UNDEF, ///< UNDEF of CONSTANTS (a variable, constants.F90:85) + WW_CPRT_DSIE, ///< DSIE of W3DISPMD + WW_CPRT_NRPAR +}; + +/// Sections 1-5 of W3CPRT (w3iosfmd.F90:109-344) on d_va of grid `igrid` +/// (ww::resident::device_va: VA crosses only when the host holds the newer bits). The partition +/// table is kept in the shim's buffers until ww_cprt_fetch. Out: *finsiz, FINSIZ = ICPRT(NSEAL+1,2)-1 +/// (:312); *nxpfull, the number of PTMEAN calls that met a full XP array, each a format 1000 line +/// the caller writes on NDSE when IAPROC == NAPERR (w3partmd.F90:1416); *fallback, 1 when a point +/// met PTMEAN's IFPMAX = 0 (a NaN spectrum, part_ptmean.hpp), where the Fortran reads out of +/// bounds: the caller then discards this result, reads VA to the host and calls the Fortran +/// W3CPRT. The caller has stopped already on a NaN depth (:270-274). va is VA(1,0), nspec = +/// SIZE(VA,1), nsealm = UBOUND(VA,2); mapsf (NSEA,3), mapsta (NY,NX), u10, u10d (NSEA), asf +/// (NASF), dw (0:NSEA), cg, wn (0:NK+1,0:NSEA), sig, dsip (0:NK+1), dsii (NK), th, ecos, esin (NTH), +/// ewn1, ecg1 (0:NAR1D). Returns a WwKokkosStatus. +int ww_cprt_va(int igrid, const int* ipar, int nipar, const float* rpar, int nrpar, float* va, + int nspec, int nsealm, const int* mapsf, const int* mapsta, const float* u10, + const float* u10d, const float* asf, const float* dw, const float* cg, + const float* wn, const float* sig, const float* dsip, const float* dsii, + const float* th, const float* ecos, const float* esin, const float* ewn1, + const float* ecg1, int* finsiz, int* nxpfull, int* fallback); + +/// The table of the last ww_cprt_va of grid `igrid` into WW3's arrays: icprt is +/// ICPRT(NSEALM+1,2), every entry written; dtprt DTPRT(DIMP,MAX(1,FINSIZ)), columns 1..finsiz +/// written (the caller zeroes DTPRT when finsiz is 0, :325). Returns a WwKokkosStatus, +/// WW_KOKKOS_ERR_BAD_SHAPE when igrid, nsealm or finsiz differ from that call's. +int ww_cprt_fetch(int igrid, int nsealm, int* icprt, int finsiz, float* dtprt); + +} // extern "C" diff --git a/kokkos/src/fortran_iface/cprt_shim.cpp b/kokkos/src/fortran_iface/cprt_shim.cpp new file mode 100644 index 00000000..1d6ea05d --- /dev/null +++ b/kokkos/src/fortran_iface/cprt_shim.cpp @@ -0,0 +1,424 @@ +// kokkos/src/fortran_iface/cprt_shim.cpp +// The bind(C) shim of the cprt wiring unit (PATCH.md section 27): W3CPRT's spectral partitioning +// on the resident VA. Marshalling and the order of the launches only; the arithmetic is the +// translated headers iosf_w3cprt.hpp and part_w3part.hpp, called unchanged. +// +// The caller is W3WAVE (w3wavemd.F90) through W3KOKKOS_CPRT of w3kokkos_cprtmd.F90 on the fork, +// at an output time with FLPART, outside any OpenMP region, in place of CALL W3CPRT ( IMOD ) and +// of the host read of VA before it. +// +// Order. The local sea points go in chunks of `chunk` points (WW_KOKKOS_CPRT_CHUNK, read once; +// by default four per thread of the execution space on a host backend, 1024 on a device), each +// point with its own work slot. Per chunk: one launch runs sections 2-4 (W3PART) for every point; +// a reduction counts XP-full messages and the points PTMEAN cannot take; an integer prefix sum +// gives each point its first DTPRT column and ICPRT(JSEA+1,2); a last launch copies XP(:,0:NP) to +// those columns. Integer sums are exact in any order, so DTPRT and ICPRT are the Fortran's JSEA +// order by construction, and each point's floating-point arithmetic is W3PART's serial code. +// +// Memory contract (phase 2). VA is read in d_va (ww::resident::device_va), so it crosses only +// when the host holds the newer bits, and W3WAVE makes no host read of VA for this call. On a +// backend whose execution space reads host memory (Serial, OpenMP) the inputs are WW3's own +// arrays, read in place. On a device the inputs (MAPSF, MAPSTA, U10, U10D, ASF, DW, CG, WN, the +// frequency and direction tables, DISTAB's EWN1 and ECG1) are copied in per call, which is per +// output time. ICPRT (NSEALM+1 by 2 ints) and the FINSIZ columns of DTPRT (15 Reals each) are +// built in the shim's buffers and copied back by ww_cprt_fetch; nothing else comes back. The work +// slots are device buffers of `chunk` slots that live for the run. No Unified Memory. +// +// WW_KOKKOS_CPRT_COUNT=1 prints, at exit, the calls, the sea points, the fallbacks to the Fortran +// and the bytes copied in and back (on a host backend the copies a device would make, labelled as +// such: none is made). WW_KOKKOS_TIMERS=1 times cprt.call, cprt.h2d, cprt.kernel and cprt.d2h. +// +// SPDX-License-Identifier: MIT + +#include "fortran_iface/cprt_c.hpp" +#include "fortran_iface/timers.hpp" +#include "fortran_iface/ww_kokkos_c.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ww_kokkos/iosf_w3cprt.hpp" +#include "ww_kokkos/part_ptnghb.hpp" +#include "ww_kokkos/part_w3part.hpp" +#include "ww_kokkos/real.hpp" +#include "ww_kokkos/resident_state.hpp" + +#if defined(_OPENMP) +#include +#endif + +namespace ww::cprtshim { +namespace { + +using ww::Real; +using Exec = Kokkos::DefaultExecutionSpace; +using Mem = Exec::memory_space; +static_assert(std::is_same_v, "the C ABI passes REAL as float"); +constexpr bool kHostReadable = Kokkos::SpaceAccessibility::accessible; + +template +using H1 = Kokkos::View>; +template +using H2 = Kokkos::View>; +template +using D1 = Kokkos::View; +template +using D2 = Kokkos::View; +template +using U1 = Kokkos::View>; +template +using U2 = Kokkos::View>; + +std::atomic g_calls{0}, g_points{0}, g_fallbacks{0}, g_h2d_bytes{0}, g_d2h_bytes{0}; + +void print_counts() { + const char* proj = kHostReadable ? " (projected for a device, none made)" : ""; + std::fprintf(stderr, "ww_kokkos: W3CPRT Kokkos calls: %lld\n", g_calls.load()); + std::fprintf(stderr, "ww_kokkos: W3CPRT sea points: %lld\n", g_points.load()); + std::fprintf(stderr, "ww_kokkos: W3CPRT fallbacks to the Fortran (no peak): %lld\n", g_fallbacks.load()); + std::fprintf(stderr, "ww_kokkos: W3CPRT input bytes copied H2D%s: %lld\n", proj, g_h2d_bytes.load()); + std::fprintf(stderr, "ww_kokkos: W3CPRT table bytes copied back D2H%s: %lld\n", proj, g_d2h_bytes.load()); +} + +/// WW_KOKKOS_CPRT_COUNT=1 registers the exit print when the library loads, so a run in which +/// the entry point was never called still says 0. +[[maybe_unused]] const bool g_count_on = [] { + const char* v = std::getenv("WW_KOKKOS_CPRT_COUNT"); + const bool c = v != nullptr && std::strcmp(v, "1") == 0; + if (c) std::atexit(print_counts); + return c; +}(); + +int fail(int code, const char* what) { + std::fprintf(stderr, "ww_kokkos: cprt: %s (code %d)\n", what, code); + return code; +} + +/// Points per chunk: WW_KOKKOS_CPRT_CHUNK when a positive integer, else the default above. +int chunk_size() { + static const int n = [] { + if (const char* v = std::getenv("WW_KOKKOS_CPRT_CHUNK")) { + char* end = nullptr; + const long c = std::strtol(v, &end, 10); + if (end != v && *end == '\0' && c > 0 && c <= (1L << 20)) return static_cast(c); + std::fprintf(stderr, "ww_kokkos: cprt: WW_KOKKOS_CPRT_CHUNK='%s' ignored\n", v); + } + return kHostReadable ? 4 * Exec().concurrency() : 1024; + }(); + return n; +} + +/// The shim's buffers, kept between calls. +struct Bufs { + // Inputs on a device (unused on a host backend). + D2 mapsf, mapsta; + D1 u10, u10d, asf, dw, sig, dsip, dsii, th, ecos, esin, ewn1, ecg1; + D2 cg, wn; + D1 neigh; + ww::part::PtnghbTable table; + // Work slots and per-point results of a chunk. + D1 wr; + D1 wi, np, nxp, rc, first; + // The table, until ww_cprt_fetch. + D2 icprt; + D2 dtprt; + int igrid = -1, nsealm = -1, finsiz = -1; +}; +Bufs& bufs() { + static auto* b = new Bufs(); // never destroyed, as every shim's state + static bool hooked = false; + if (!hooked && Kokkos::is_initialized()) { + hooked = true; + Kokkos::push_finalize_hook([] { *b = Bufs(); }); + } + return *b; +} + +template +void ensure(V& v, const char* name, size_t n0, size_t n1 = 0) { + if constexpr (V::rank == 1) { + if (v.extent(0) != n0) v = V(Kokkos::view_alloc(Kokkos::WithoutInitializing, std::string(name)), n0); + } else { + if (v.extent(0) != n0 || v.extent(1) != n1) + v = V(Kokkos::view_alloc(Kokkos::WithoutInitializing, std::string(name)), n0, n1); + } +} + +/// A read-only input: WW3's array itself on a host backend; else its device buffer, refreshed. +template +U1 in1(D1& buf, const char* name, const T* p, size_t n) { + if (kHostReadable) return U1(p, n); + ensure(buf, name, n); + Kokkos::deep_copy(buf, H1(p, n)); + g_h2d_bytes.fetch_add(static_cast(n * sizeof(T)), std::memory_order_relaxed); + return U1(buf.data(), n); +} +template +U2 in2(D2& buf, const char* name, const T* p, size_t n0, size_t n1) { + if (kHostReadable) return U2(p, n0, n1); + ensure(buf, name, n0, n1); + Kokkos::deep_copy(buf, H2(p, n0, n1)); + g_h2d_bytes.fetch_add(static_cast(n0 * n1 * sizeof(T)), std::memory_order_relaxed); + return U2(buf.data(), n0, n1); +} +/// What a device would copy in, counted on a host backend too. +void count_in(size_t bytes) { + if (kHostReadable) g_h2d_bytes.fetch_add(static_cast(bytes), std::memory_order_relaxed); +} + +bool nested_in_openmp() { +#if defined(_OPENMP) + return omp_get_level() > 0; +#else + return false; +#endif +} + +int run(int igrid, const int* ipar, const float* rpar, float* va, int nspec, int nsealm, const int* mapsf, + const int* mapsta, const float* u10, const float* u10d, const float* asf, const float* dw, + const float* cg, const float* wn, const float* sig, const float* dsip, const float* dsii, + const float* th, const float* ecos, const float* esin, const float* ewn1, const float* ecg1, + int* finsiz_out, int* nxpfull_out, int* fallback_out) { + const int nk = ipar[WW_CPRT_NK], nth = ipar[WW_CPRT_NTH], nseal = ipar[WW_CPRT_NSEAL]; + const int nsea = ipar[WW_CPRT_NSEA], nx = ipar[WW_CPRT_NX], ny = ipar[WW_CPRT_NY]; + const int nasf = ipar[WW_CPRT_NASF], ptmeth = ipar[WW_CPRT_PTMETH], ihmax = ipar[WW_CPRT_IHMAX]; + const int nar1d = ipar[WW_CPRT_NAR1D], n1max = ipar[WW_CPRT_N1MAX]; + // Under W3_SHRD NSEAL = NSEA = NSEALM; JSEA = ISEA is what the kernel assumes. + if (nk < 1 || nth < 1 || nspec != nk * nth || nseal < 0 || nseal != nsea || nsealm < nseal || nx < 1 || + ny < 1 || nasf < nsea || ihmax < 1 || ipar[WW_CPRT_DIMP] != ww::part::kDimp || nar1d < 2 || n1max < 0 || + n1max > nar1d - 1) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_va: bad sizes"); + const int dimxp = ww::iosf::w3cprt_dimxp(nk, nth, ptmeth); + if (dimxp < 1) return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_va: DIMXP < 1"); + + ww::timers::Region tcall("cprt.call"); + Bufs& b = bufs(); + b.finsiz = -1; + ww::resident::DevVa dva; + const int st = ww::resident::device_va(igrid, va, nspec, nsealm, dva); + if (st != WW_KOKKOS_OK) return fail(st, "the resident VA refused this call"); + if (dva.stride(1) != static_cast(nspec)) return fail(WW_KOKKOS_ERR_BAD_SHAPE, "the resident VA is padded"); + + ww::iosf::W3cprtArgs a; + ww::part::PtmeanGrid g{}; + { + ww::timers::Region t("cprt.h2d"); + const size_t nk2 = static_cast(nk) + 2, ns1 = static_cast(nsea) + 1; + const size_t ns = static_cast(nsea), nt = static_cast(nth); + a.va = U2(dva.data(), static_cast(nspec), static_cast(nsealm) + 1); + a.mapsf = in2(b.mapsf, "ww_cprt.mapsf", mapsf, ns, 3); + a.mapsta = in2(b.mapsta, "ww_cprt.mapsta", mapsta, static_cast(ny), static_cast(nx)); + a.u10 = in1(b.u10, "ww_cprt.u10", u10, ns); + a.u10d = in1(b.u10d, "ww_cprt.u10d", u10d, ns); + a.asf = in1(b.asf, "ww_cprt.asf", asf, ns); + a.dw = in1(b.dw, "ww_cprt.dw", dw, ns1); + a.cg = in2(b.cg, "ww_cprt.cg", cg, nk2, ns1); + a.wn = in2(b.wn, "ww_cprt.wn", wn, nk2, ns1); + count_in((ns * 3 + static_cast(nx) * static_cast(ny)) * sizeof(int) + + (3 * ns + ns1 + 2 * nk2 * ns1) * sizeof(Real)); + const auto s_sig = in1(b.sig, "ww_cprt.sig", sig, nk2); + const auto s_dsip = in1(b.dsip, "ww_cprt.dsip", dsip, nk2); + const auto s_dsii = in1(b.dsii, "ww_cprt.dsii", dsii, static_cast(nk)); + const auto s_th = in1(b.th, "ww_cprt.th", th, nt); + const auto s_ecos = in1(b.ecos, "ww_cprt.ecos", ecos, nt); + const auto s_esin = in1(b.esin, "ww_cprt.esin", esin, nt); + const size_t na = static_cast(nar1d) + 1; + const auto s_ewn1 = in1(b.ewn1, "ww_cprt.ewn1", ewn1, na); + const auto s_ecg1 = in1(b.ecg1, "ww_cprt.ecg1", ecg1, na); + count_in((2 * nk2 + static_cast(nk) + 3 * nt + 2 * na) * sizeof(Real)); + // PTNGHB's table (w3partmd.F90:630-821): a function of NK and NTH, rebuilt when they change. + if (b.table.update(nk, nth) || b.neigh.extent(0) != b.table.neigh.size()) { + ensure(b.neigh, "ww_cprt.neigh", b.table.neigh.size()); + Kokkos::deep_copy(b.neigh, H1(b.table.neigh.data(), b.table.neigh.size())); + } + g.nk = nk; + g.nth = nth; + g.dth = rpar[WW_CPRT_DTH]; + g.xfr = rpar[WW_CPRT_XFR]; + g.fachfe = rpar[WW_CPRT_FACHFE]; + g.fte = rpar[WW_CPRT_FTE]; + g.sig = s_sig.data(); + g.dsip = s_dsip.data(); + g.dsii = s_dsii.data(); + g.th = s_th.data(); + g.ecos = s_ecos.data(); + g.esin = s_esin.data(); + g.disp.ewn1 = s_ewn1.data(); + g.disp.ecg1 = s_ecg1.data(); + g.disp.dsie = rpar[WW_CPRT_DSIE]; + g.disp.n1max = n1max; + } + const ww::part::PtmeanOpts o{rpar[WW_CPRT_HSPMIN], rpar[WW_CPRT_WSMULT], ptmeth, rpar[WW_CPRT_UNDEF]}; + const ww::part::W3partOpts po{ihmax, rpar[WW_CPRT_WSCUT], rpar[WW_CPRT_PTFCUT], ipar[WW_CPRT_FLCOMB] != 0}; + + const int chunk = nseal < chunk_size() ? (nseal > 0 ? nseal : 1) : chunk_size(); + const size_t nreal = static_cast(ww::iosf::w3cprt_nreal(nk, nth, dimxp)); + const size_t nint = static_cast(ww::iosf::w3cprt_nint(nk, nth, dimxp, ihmax)); + const size_t nxp = static_cast(ww::part::kDimp) * (static_cast(dimxp) + 1); + const size_t nspec_s = static_cast(nspec); + if (b.wr.extent(0) < nreal * chunk) b.wr = D1(Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_cprt.wr"), nreal * chunk); + if (b.wi.extent(0) < nint * chunk) b.wi = D1(Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_cprt.wi"), nint * chunk); + for (D1* v : {&b.np, &b.nxp, &b.rc, &b.first}) + if (v->extent(0) < static_cast(chunk)) *v = D1(Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_cprt.pt"), chunk); + // ICPRT = 0, ICPRT(1,2) = 1 (:230-231). + b.icprt = D2("ww_cprt.icprt", static_cast(nsealm) + 1, 2); + Kokkos::deep_copy(Kokkos::subview(b.icprt, 0, 1), 1); + if (b.dtprt.extent(0) != static_cast(ww::part::kDimp) || b.dtprt.extent(1) == 0) + b.dtprt = D2(Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_cprt.dtprt"), ww::part::kDimp, + static_cast(nseal > 0 ? nseal : 1)); + + const auto wr = b.wr; + const auto wi = b.wi; + const auto vnp = b.np; + const auto vnxp = b.nxp; + const auto vrc = b.rc; + const auto vfirst = b.first; + const auto icprt = b.icprt; + const int* neigh = b.neigh.data(); + int base = 0, nxpfull = 0; + { + ww::timers::Region t("cprt.kernel"); + for (int c0 = 0; c0 < nseal; c0 += chunk) { + const int n = (nseal - c0 < chunk) ? nseal - c0 : chunk; + // Sections 2-4, one point per index. + Kokkos::parallel_for( + "ww::iosf::w3cprt_point", Kokkos::RangePolicy(0, n), KOKKOS_LAMBDA(const int i) { + Real* r = wr.data() + nreal * static_cast(i); + int* ib = wi.data() + nint * static_cast(i); + Real* e2 = r; + Real* xp = e2 + nspec_s; + const ww::part::W3partScratch w = ww::part::w3part_carve(xp + nxp, ib, nk, nth, dimxp, ihmax); + int np = 0, nfull = 0; + vrc(i) = ww::iosf::w3cprt_point(a, g, o, po, neigh, dimxp, c0 + i + 1, e2, xp, w, np, nfull); + vnp(i) = np; + vnxp(i) = nfull; + }); + int nopeak = 0, nfull = 0; + Kokkos::parallel_reduce( + "ww::iosf::w3cprt_status", Kokkos::RangePolicy(0, n), + KOKKOS_LAMBDA(const int i, int& sx, int& sp) { + sx += vnxp(i); + if (vrc(i) == ww::part::kPtmeanNoPeak) sp += 1; + }, + nfull, nopeak); + if (nopeak > 0) { + *finsiz_out = 0; + *nxpfull_out = 0; + *fallback_out = 1; + g_fallbacks.fetch_add(1, std::memory_order_relaxed); + return WW_KOKKOS_OK; + } + nxpfull += nfull; + // ICPRT(JSEA,2) = 1 + the columns of the points before JSEA (:251, :286). + int total = 0; + const int b0 = base; + Kokkos::parallel_scan( + "ww::iosf::w3cprt_scan", Kokkos::RangePolicy(0, n), + KOKKOS_LAMBDA(const int i, int& acc, const bool final) { + const int np = vnp(i); + const int cols = np >= 0 ? np + 1 : 0; + if (final) { + vfirst(i) = b0 + 1 + acc; + icprt(c0 + i + 1, 1) = b0 + 1 + acc + cols; // ICPRT(JSEA+1,2) + } + acc += cols; + }, + total); + if (static_cast(base) + static_cast(total) > b.dtprt.extent(1)) { + size_t cap = 2 * b.dtprt.extent(1); + if (cap < static_cast(base) + static_cast(total)) + cap = static_cast(base) + static_cast(total); + Kokkos::resize(b.dtprt, static_cast(ww::part::kDimp), cap); + } + const auto dtprt = b.dtprt; + Kokkos::parallel_for( + "ww::iosf::w3cprt_store", Kokkos::RangePolicy(0, n), KOKKOS_LAMBDA(const int i) { + const Real* xp = wr.data() + nreal * static_cast(i) + nspec_s; + ww::iosf::w3cprt_store(c0 + i + 1, vnp(i), vfirst(i), xp, icprt, dtprt); + }); + base += total; + } + Exec().fence("ww_cprt_va"); + } + b.igrid = igrid; + b.nsealm = nsealm; + b.finsiz = base; // FINSIZ = ICPRT(NSEAL+1,2) - 1 (:312) + *finsiz_out = base; + *nxpfull_out = nxpfull; + *fallback_out = 0; + g_calls.fetch_add(1, std::memory_order_relaxed); + g_points.fetch_add(nseal, std::memory_order_relaxed); + return WW_KOKKOS_OK; +} + +} // namespace +} // namespace ww::cprtshim + +extern "C" int ww_cprt_va(int igrid, const int* ipar, int nipar, const float* rpar, int nrpar, float* va, + int nspec, int nsealm, const int* mapsf, const int* mapsta, const float* u10, + const float* u10d, const float* asf, const float* dw, const float* cg, + const float* wn, const float* sig, const float* dsip, const float* dsii, + const float* th, const float* ecos, const float* esin, const float* ewn1, + const float* ecg1, int* finsiz, int* nxpfull, int* fallback) { + using namespace ww::cprtshim; + if (ipar == nullptr || rpar == nullptr || nipar != WW_CPRT_NIPAR || nrpar != WW_CPRT_NRPAR) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_va: bad packet"); + if (va == nullptr || mapsf == nullptr || mapsta == nullptr || u10 == nullptr || u10d == nullptr || + asf == nullptr || dw == nullptr || cg == nullptr || wn == nullptr || sig == nullptr || dsip == nullptr || + dsii == nullptr || th == nullptr || ecos == nullptr || esin == nullptr || ewn1 == nullptr || + ecg1 == nullptr || finsiz == nullptr || nxpfull == nullptr || fallback == nullptr) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_va: a null pointer"); + try { + if (!Kokkos::is_initialized()) return fail(WW_KOKKOS_ERR_NOT_INITIALISED, "before ww_kokkos_init"); + if (nested_in_openmp()) return fail(WW_KOKKOS_ERR_UNSUPPORTED, "inside an OpenMP region"); + // W3CPRT reads VA and never writes it: d_va keeps the state device_va left (BOTH or DEVICE). + return run(igrid, ipar, rpar, va, nspec, nsealm, mapsf, mapsta, u10, u10d, asf, dw, cg, wn, sig, dsip, + dsii, th, ecos, esin, ewn1, ecg1, finsiz, nxpfull, fallback); + } catch (const std::bad_alloc& e) { + return fail(WW_KOKKOS_ERR_RUNTIME, e.what()); + } catch (const std::exception& e) { + return fail(WW_KOKKOS_ERR_KERNEL, e.what()); + } catch (...) { + return fail(WW_KOKKOS_ERR_KERNEL, "unknown exception"); + } +} + +extern "C" int ww_cprt_fetch(int igrid, int nsealm, int* icprt, int finsiz, float* dtprt) { + using namespace ww::cprtshim; + if (icprt == nullptr || dtprt == nullptr) return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_fetch: a null pointer"); + try { + if (!Kokkos::is_initialized()) return fail(WW_KOKKOS_ERR_NOT_INITIALISED, "before ww_kokkos_init"); + Bufs& b = bufs(); + if (b.finsiz < 0 || igrid != b.igrid || nsealm != b.nsealm || finsiz != b.finsiz) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_fetch: no table of this grid and size"); + ww::timers::Region t("cprt.d2h"); + const size_t nrow = static_cast(nsealm) + 1; + Kokkos::deep_copy(H2(icprt, nrow, 2), b.icprt); + long long bytes = static_cast(nrow * 2 * sizeof(int)); + if (finsiz > 0) { + const size_t nc = static_cast(finsiz); + Kokkos::deep_copy(H2(dtprt, static_cast(ww::part::kDimp), nc), + Kokkos::subview(b.dtprt, Kokkos::ALL, std::make_pair(size_t{0}, nc))); + bytes += static_cast(nc * ww::part::kDimp * sizeof(Real)); + } + Exec().fence("ww_cprt_fetch"); + g_d2h_bytes.fetch_add(bytes, std::memory_order_relaxed); + b.finsiz = -1; + return WW_KOKKOS_OK; + } catch (const std::bad_alloc& e) { + return fail(WW_KOKKOS_ERR_RUNTIME, e.what()); + } catch (const std::exception& e) { + return fail(WW_KOKKOS_ERR_KERNEL, e.what()); + } catch (...) { + return fail(WW_KOKKOS_ERR_KERNEL, "unknown exception"); + } +} diff --git a/kokkos/src/fortran_iface/outg_c.hpp b/kokkos/src/fortran_iface/outg_c.hpp index 298f83d6..a4975dc9 100644 --- a/kokkos/src/fortran_iface/outg_c.hpp +++ b/kokkos/src/fortran_iface/outg_c.hpp @@ -81,4 +81,47 @@ int ww_outg_va(int igrid, const int* ipar, int nipar, const float* rpar, int nrp const float* ec2, const float* es2, const float* esc, const int* mapsf, const int* mapsta, float* const* fields, int nfields); +/// The packed integers of ww_outg_part, in this order. +enum WwOutgPartIpar { + WW_OUTGP_NSEAL = 0, + WW_OUTGP_NSEA, ///< rows of MAPSF + WW_OUTGP_NX, ///< MAPSTA(NY,NX) + WW_OUTGP_NY, + WW_OUTGP_NROW, ///< SIZE(PWST) = NSEALM: the rows of every partition field + WW_OUTGP_NOSWLL, ///< NOSWLL: the partition fields are (NROW, 0:NOSWLL) + WW_OUTGP_NICP, ///< SIZE(ICPRT,1) = NSEALM+1 + WW_OUTGP_NDTP, ///< SIZE(DTPRT,2), the columns of DTPRT(DIMP,:) + WW_OUTGP_DIMP, ///< DIMP, 15 + WW_OUTGP_NOGRP, ///< FLOGRD(NOGRP,NGRPP) + WW_OUTGP_NGRPP, + WW_OUTGP_FLOUTG, ///< 1 or 0 + WW_OUTGP_FLOUTG2, ///< 1 or 0 + WW_OUTGP_NIPAR +}; + +/// The packed reals of ww_outg_part. +enum WwOutgPartRpar { WW_OUTGP_UNDEF = 0, WW_OUTGP_WSCUT, WW_OUTGP_NRPAR }; + +/// The partition fields of ww_outg_part, the order of `fields` (W3ADATMD arrays): the first +/// fifteen are (NROW, 0:NOSWLL), PWST and PNR (NROW). FLOGRD(4,k) names field k+1. +enum WwOutgPartField { + WW_OUTGP_PHS = 0, WW_OUTGP_PTP, WW_OUTGP_PLP, WW_OUTGP_PDIR, WW_OUTGP_PSI, WW_OUTGP_PWS, + WW_OUTGP_PTHP0, WW_OUTGP_PQP, WW_OUTGP_PPE, WW_OUTGP_PGW, WW_OUTGP_PSW, WW_OUTGP_PTM1, + WW_OUTGP_PT1, WW_OUTGP_PT2, WW_OUTGP_PEP, + WW_OUTGP_N2D, + WW_OUTGP_PWST = WW_OUTGP_N2D, WW_OUTGP_PNR, + WW_OUTGP_NFIELD +}; + +/// Section 6 of W3OUTG (w3iogomd.F90:2242-2340), the partition fields from W3CPRT's table, alone +/// (ww::iogo::w3outg_part). It reads no VA. W3KOKKOS_OUTG calls it after ww_outg_va when FLPART +/// is the only reason the Fortran W3OUTG would run (PATCH.md section 27). On a backend that reads +/// host memory every field is written in place; on a device ICPRT and DTPRT are copied in and only +/// the fields W3IOGO ('WRITE') writes (FLOGRD(4,k) under FLOUTG, FLOGR2(4,k) under FLOUTG2, +/// w3iogomd.F90:3283-3367) come back. mapsf (NSEA,3), mapsta (NY,NX), icprt (NICP,2), dtprt +/// (DIMP,NDTP), flogrd, flogr2 LOGICAL(4) (NOGRP,NGRPP). Returns a WwKokkosStatus. +int ww_outg_part(const int* ipar, int nipar, const float* rpar, int nrpar, const int* flogrd, + const int* flogr2, const int* mapsf, const int* mapsta, const int* icprt, + const float* dtprt, float* const* fields, int nfields); + } // extern "C" diff --git a/kokkos/src/fortran_iface/outg_shim.cpp b/kokkos/src/fortran_iface/outg_shim.cpp index 524c8d7f..f4d94b8a 100644 --- a/kokkos/src/fortran_iface/outg_shim.cpp +++ b/kokkos/src/fortran_iface/outg_shim.cpp @@ -5,9 +5,10 @@ // The caller is W3WAVE (w3wavemd.F90) through W3KOKKOS_OUTG of w3kokkos_outgmd.F90 on the fork, // at a field-output time, outside any OpenMP region, in place of CALL W3OUTG (VA, FLPFLD, FLOUTG, // FLOUTG2). W3KOKKOS_OUTG calls the Fortran W3OUTG instead when a field this unit does not compute -// is requested: the partitions of section 6 (FLPFLD; they read W3CPRT's table, built on the host -// from VA), CALC_U3STOKES (FLOLOC(6,8), (6,12)), SKEWNESS (FLOLOC(8,7..9)) and CALC_WBT -// (FLOLOC(2,17)). So ww::iogo::w3outg runs here with FLPART false and no hook requested. +// is requested: CALC_U3STOKES (FLOLOC(6,8), (6,12)), SKEWNESS (FLOLOC(8,7..9)) and CALC_WBT +// (FLOLOC(2,17)). So ww::iogo::w3outg runs here with FLPART false and no hook requested; the +// partition fields of section 6 (FLPFLD), which read W3CPRT's table and no VA, are ww_outg_part's, +// called after it (PATCH.md section 27). // // Memory contract (phase 2). A reads d_va of the grid (ww::resident::device_va), so VA crosses // only when the host holds the newer bits, and W3WAVE makes no host read of VA for this call. On a @@ -376,9 +377,143 @@ int run(int igrid, const int* ipar, const float* rpar, const int* flogrd, const return WW_KOKKOS_OK; } +std::atomic g_part_calls{0}, g_part_back{0}, g_part_back_bytes{0}; + +void print_part_counts() { + const char* proj = kHostReadable ? " (projected for a device, none made)" : ""; + std::fprintf(stderr, "ww_kokkos: W3OUTG partition-field Kokkos calls: %lld\n", g_part_calls.load()); + std::fprintf(stderr, "ww_kokkos: W3OUTG partition fields copied back D2H%s: %lld\n", proj, g_part_back.load()); + std::fprintf(stderr, "ww_kokkos: W3OUTG partition bytes copied back D2H%s: %lld\n", proj, + g_part_back_bytes.load()); +} +[[maybe_unused]] const bool g_part_count_on = [] { + const char* v = std::getenv("WW_KOKKOS_OUTG_COUNT"); + const bool c = v != nullptr && std::strcmp(v, "1") == 0; + if (c) std::atexit(print_part_counts); + return c; +}(); + +/// The device buffers of section 6, kept between calls (unused on a host backend). +struct PartBufs { + D2 mapsf, mapsta, icprt; + D2 dtprt; + D2 f2[WW_OUTGP_N2D]; + D1 f1[WW_OUTGP_NFIELD - WW_OUTGP_N2D]; +}; +PartBufs& part_bufs() { + static auto* b = new PartBufs(); // never destroyed, as every shim's state + static bool hooked = false; + if (!hooked && Kokkos::is_initialized()) { + hooked = true; + Kokkos::push_finalize_hook([] { *b = PartBufs(); }); + } + return *b; +} + +int run_part(const int* ipar, const float* rpar, const int* flogrd, const int* flogr2, const int* mapsf, + const int* mapsta, const int* icprt, const float* dtprt, float* const* fields) { + const int nseal = ipar[WW_OUTGP_NSEAL], nsea = ipar[WW_OUTGP_NSEA], nx = ipar[WW_OUTGP_NX]; + const int ny = ipar[WW_OUTGP_NY], nrow = ipar[WW_OUTGP_NROW], noswll = ipar[WW_OUTGP_NOSWLL]; + const int nicp = ipar[WW_OUTGP_NICP], ndtp = ipar[WW_OUTGP_NDTP], nogrp = ipar[WW_OUTGP_NOGRP]; + const int ngrpp = ipar[WW_OUTGP_NGRPP]; + if (nseal < 0 || nsea < nseal || nx < 1 || ny < 1 || nrow < nseal || noswll < 0 || nicp < nseal + 1 || + ndtp < 1 || ipar[WW_OUTGP_DIMP] != 15 || nogrp < 4 || ngrpp < 17) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_outg_part: bad sizes"); + for (int k = 0; k < WW_OUTGP_NFIELD; ++k) + if (fields[k] == nullptr) return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_outg_part: a null field"); + const bool floutg = ipar[WW_OUTGP_FLOUTG] != 0, floutg2 = ipar[WW_OUTGP_FLOUTG2] != 0; + const auto want = [&](int j) { + const int k = 3 + (j - 1) * nogrp; // FLOGRD(4,j) + return (floutg && flogrd[k] != 0) || (floutg2 && flogr2[k] != 0); + }; + + ww::timers::Region tcall("outg.part"); + PartBufs& b = part_bufs(); + ww::iogo::W3OutgScalars s{}; + s.nseal = nseal; + s.noswll = noswll; + s.undef = rpar[WW_OUTGP_UNDEF]; + s.wscut = rpar[WW_OUTGP_WSCUT]; + Args g; + const size_t nr = static_cast(nrow), nc = static_cast(noswll) + 1; + const auto pin2 = [&](D2& buf, const char* name, const int* p, size_t n0, size_t n1) { + if (in_place()) return U2(p, n0, n1); + ensure(buf, name, n0, n1); + Kokkos::deep_copy(buf, H2(p, n0, n1)); + return U2(buf.data(), n0, n1); + }; + g.mapsf = pin2(b.mapsf, "ww_outg.part.mapsf", mapsf, static_cast(nsea), 3); + g.mapsta = pin2(b.mapsta, "ww_outg.part.mapsta", mapsta, static_cast(ny), static_cast(nx)); + g.icprt = pin2(b.icprt, "ww_outg.part.icprt", icprt, static_cast(nicp), 2); + if (in_place()) { + g.dtprt = U2(dtprt, 15, static_cast(ndtp)); + } else { + ensure(b.dtprt, "ww_outg.part.dtprt", 15, static_cast(ndtp)); + Kokkos::deep_copy(b.dtprt, H2(dtprt, 15, static_cast(ndtp))); + g.dtprt = U2(b.dtprt.data(), 15, static_cast(ndtp)); + } + Args::V2* o2[WW_OUTGP_N2D] = {&g.phs, &g.ptp, &g.plp, &g.pdir, &g.psi, &g.pws, &g.pthp0, &g.pqp, + &g.ppe, &g.pgw, &g.psw, &g.ptm1, &g.pt1, &g.pt2, &g.pep}; + for (int k = 0; k < WW_OUTGP_N2D; ++k) { + if (in_place()) { + *o2[k] = U2(fields[k], nr, nc); + } else { + ensure(b.f2[k], "ww_outg.part.field2", nr, nc); + *o2[k] = U2(b.f2[k].data(), nr, nc); + } + } + Args::V1* o1[WW_OUTGP_NFIELD - WW_OUTGP_N2D] = {&g.pwst, &g.pnr}; + for (int k = 0; k < WW_OUTGP_NFIELD - WW_OUTGP_N2D; ++k) { + if (in_place()) { + *o1[k] = U1(fields[WW_OUTGP_N2D + k], nr); + } else { + ensure(b.f1[k], "ww_outg.part.field1", nr); + *o1[k] = U1(b.f1[k].data(), nr); + } + } + ww::iogo::w3outg_part(Exec(), s, g); + Exec().fence("ww_outg_part"); + for (int k = 0; k < WW_OUTGP_NFIELD; ++k) { + if (!want(k + 1)) continue; + const size_t n = k < WW_OUTGP_N2D ? nr * nc : nr; + if (!in_place()) { + if (k < WW_OUTGP_N2D) + Kokkos::deep_copy(H2(fields[k], nr, nc), b.f2[k]); + else + Kokkos::deep_copy(H1(fields[k], nr), b.f1[k - WW_OUTGP_N2D]); + } + g_part_back.fetch_add(1, std::memory_order_relaxed); + g_part_back_bytes.fetch_add(static_cast(n * sizeof(float)), std::memory_order_relaxed); + } + if (!in_place()) Exec().fence("ww_outg_part d2h"); + g_part_calls.fetch_add(1, std::memory_order_relaxed); + return WW_KOKKOS_OK; +} + } // namespace } // namespace ww::outgshim +extern "C" int ww_outg_part(const int* ipar, int nipar, const float* rpar, int nrpar, const int* flogrd, + const int* flogr2, const int* mapsf, const int* mapsta, const int* icprt, + const float* dtprt, float* const* fields, int nfields) { + using namespace ww::outgshim; + if (ipar == nullptr || rpar == nullptr || nipar != WW_OUTGP_NIPAR || nrpar != WW_OUTGP_NRPAR || + nfields != WW_OUTGP_NFIELD || fields == nullptr || flogrd == nullptr || flogr2 == nullptr || + mapsf == nullptr || mapsta == nullptr || icprt == nullptr || dtprt == nullptr) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_outg_part: bad packet"); + try { + if (!Kokkos::is_initialized()) return fail(WW_KOKKOS_ERR_NOT_INITIALISED, "before ww_kokkos_init"); + if (nested_in_openmp()) return fail(WW_KOKKOS_ERR_UNSUPPORTED, "inside an OpenMP region"); + return run_part(ipar, rpar, flogrd, flogr2, mapsf, mapsta, icprt, dtprt, fields); + } catch (const std::bad_alloc& e) { + return fail(WW_KOKKOS_ERR_RUNTIME, e.what()); + } catch (const std::exception& e) { + return fail(WW_KOKKOS_ERR_KERNEL, e.what()); + } catch (...) { + return fail(WW_KOKKOS_ERR_KERNEL, "unknown exception"); + } +} + extern "C" int ww_outg_va(int igrid, const int* ipar, int nipar, const float* rpar, int nrpar, const int* flogrd, const int* flogr2, const int* flogrr, float* va, int nspec, int nsealm, const float* cg, const float* wn, const float* dw, const float* sig, diff --git a/kokkos/src/fortran_iface/part_flags.cpp b/kokkos/src/fortran_iface/part_flags.cpp index f5c2d640..d17c8d04 100644 --- a/kokkos/src/fortran_iface/part_flags.cpp +++ b/kokkos/src/fortran_iface/part_flags.cpp @@ -15,7 +15,8 @@ // Fortran unless their own flag is "1", so WW_KOKKOS_ALL=1 cannot route them to Kokkos. The PR3 // propagation of the prop3 unit (W3XYP3, W3KTP3, PATCH.md section 11), W3OUTG of the outg unit // (PATCH.md section 15), the partial read of VA before W3IOPE of the pnt unit (section 16) and -// W3UICE's zeroing of VA repeated on the device by the updv unit (section 17) are opt-in the same +// W3UICE's zeroing of VA repeated on the device by the updv unit (section 17) and W3CPRT of the +// cprt unit (section 27) are opt-in the same // way until their rows are regtested. ST4_TABLES is opt-in as well (the owner, 2026-10-17): the // C++ rebuild of the ST4 tables gives the Fortran's values ("0 values differ", the // ST4_TABLES row of PATCH.md) but cost about 8 s serial per run (coast, SRCE 18.3-18.6 s against 26.6-26.7 s with ST4_TABLES, @@ -23,25 +24,112 @@ // while without it the shim reads the tables the Fortran built, the same bits by // construction. // +// WW_KOKKOS_PRESET= (PATCH.md section 25) turns on the parts of one line of presets.txt as if +// each WW_KOKKOS_ were "1", so a part the switch file does not cover stops the model as it +// does under --kokkos resident; a WW_KOKKOS_ that is set still wins, and the master decides +// the parts the preset does not name. WW_KOKKOS_STRICT=1 (section 24) is read here once as well, +// and the per-part count of calls left on the Fortran that it would have stopped is kept here and +// printed once at exit. Every variable is read once per process (a function-local static): no +// getenv on a per-call path. +// // SPDX-License-Identifier: MIT #include "fortran_iface/ww_kokkos_c.hpp" +#include #include #include namespace { +struct Preset { + const char* name; + const char* parts; // PART,PART,... without the WW_KOKKOS_ prefix +}; +constexpr Preset kPresets[] = { +#include "ww_kokkos_presets.inc" +}; +constexpr Preset kUnknown{"", ""}; + +// WW_KOKKOS_PRESET as read once: nullptr unset or empty, &kUnknown for a name presets.txt lacks. +const Preset* preset() { + static const Preset* const p = []() -> const Preset* { + const char* v = std::getenv("WW_KOKKOS_PRESET"); + if (v == nullptr || *v == '\0') return nullptr; + for (const Preset& q : kPresets) + if (std::strcmp(v, q.name) == 0) return &q; + return &kUnknown; + }(); + return p; +} + +// Whether `part` is one item of the comma list `list`. +bool in_list(const char* part, const char* list) { + const std::size_t n = std::strlen(part); + for (const char* s = list; *s != '\0';) { + const char* e = std::strchr(s, ','); + const std::size_t m = e ? static_cast(e - s) : std::strlen(s); + if (m == n && std::strncmp(s, part, n) == 0) return true; + if (e == nullptr) break; + s = e + 1; + } + return false; +} + +// The parts that can fall back to the Fortran, each with its count; filled at set-up, serially. +constexpr int kMaxSlots = 64; +char g_slot_name[kMaxSlots][25] = {}; +long long g_slot_count[kMaxSlots] = {}; +int g_nslots = 0; + +void print_fallbacks() { + std::fprintf(stderr, "ww_kokkos: calls left on the Fortran (WW_KOKKOS_STRICT=1 stops instead):"); + for (int i = 0; i < g_nslots; ++i) std::fprintf(stderr, " %s=%lld", g_slot_name[i], g_slot_count[i]); + std::fprintf(stderr, "\n"); +} + constexpr const char* kOptInOnly[] = {"WW_KOKKOS_XYP1", "WW_KOKKOS_KTP1", "WW_KOKKOS_XYP2", "WW_KOKKOS_KTP2", "WW_KOKKOS_XYPUG", "WW_KOKKOS_XYP3", "WW_KOKKOS_KTP3", "WW_KOKKOS_OUTG", "WW_KOKKOS_IOPE", - "WW_KOKKOS_UPDV", "WW_KOKKOS_ST4_TABLES"}; + "WW_KOKKOS_UPDV", "WW_KOKKOS_CPRT", + "WW_KOKKOS_ST4_TABLES"}; } // namespace extern "C" int ww_kokkos_flag(const char* name) { if (name == nullptr) return 0; if (const char* v = std::getenv(name)) return std::strcmp(v, "1") == 0 ? 1 : 0; + constexpr const char kPrefix[] = "WW_KOKKOS_"; + if (const Preset* p = preset(); p != nullptr && std::strncmp(name, kPrefix, sizeof kPrefix - 1) == 0 && + in_list(name + sizeof kPrefix - 1, p->parts)) + return 1; for (const char* p : kOptInOnly) if (std::strcmp(name, p) == 0) return 0; const char* all = std::getenv("WW_KOKKOS_ALL"); return (all != nullptr && std::strcmp(all, "1") == 0) ? 2 : 0; } + +extern "C" int ww_kokkos_preset_status() { + const Preset* p = preset(); + return p == nullptr ? 0 : (p == &kUnknown ? -1 : 1); +} + +extern "C" int ww_kokkos_strict() { + static const int strict = [] { + const char* v = std::getenv("WW_KOKKOS_STRICT"); + return (v != nullptr && std::strcmp(v, "1") == 0) ? 1 : 0; + }(); + return strict; +} + +extern "C" int ww_kokkos_fallback_slot(const char* part) { + if (part == nullptr || std::strlen(part) >= sizeof g_slot_name[0]) return -1; + for (int i = 0; i < g_nslots; ++i) + if (std::strcmp(g_slot_name[i], part) == 0) return i; + if (g_nslots == kMaxSlots) return -1; + if (g_nslots == 0) std::atexit(print_fallbacks); + std::strcpy(g_slot_name[g_nslots], part); + return g_nslots++; +} + +extern "C" void ww_kokkos_fallback_add(int slot) { + if (slot >= 0 && slot < g_nslots) ++g_slot_count[slot]; +} diff --git a/kokkos/src/fortran_iface/parts.d/cprt.txt b/kokkos/src/fortran_iface/parts.d/cprt.txt new file mode 100644 index 00000000..47b15ccb --- /dev/null +++ b/kokkos/src/fortran_iface/parts.d/cprt.txt @@ -0,0 +1,4 @@ +# W3CPRT's spectral partitioning on the resident VA, PATCH.md section 27 (cprt_shim.cpp); opt-in, not under WW_KOKKOS_ALL +# Read by scripts/ww3-gpu.sh: PART (WW_KOKKOS_), COUNT , FILE ; PATCH.md section 9. +PART CPRT +COUNT WW_KOKKOS_CPRT_COUNT diff --git a/kokkos/src/fortran_iface/parts.d/pnt.txt b/kokkos/src/fortran_iface/parts.d/pnt.txt index 07e602f9..073723d7 100644 --- a/kokkos/src/fortran_iface/parts.d/pnt.txt +++ b/kokkos/src/fortran_iface/parts.d/pnt.txt @@ -1,3 +1,4 @@ -# The point output's spectra copied back alone from the resident VA, PATCH.md section 16 (resident_shim.cpp); opt-in, not under WW_KOKKOS_ALL +# The point, boundary and track outputs' spectra copied back alone from the resident VA, PATCH.md sections 17 and 23 (resident_shim.cpp); opt-in, not under WW_KOKKOS_ALL # Read by scripts/ww3-gpu.sh: PART (WW_KOKKOS_), COUNT , FILE ; PATCH.md section 9. PART IOPE +FILE model/src/w3iotrmd.F90 diff --git a/kokkos/src/fortran_iface/parts.d/resident.txt b/kokkos/src/fortran_iface/parts.d/resident.txt index 33c6669a..630abb49 100644 --- a/kokkos/src/fortran_iface/parts.d/resident.txt +++ b/kokkos/src/fortran_iface/parts.d/resident.txt @@ -1,3 +1,6 @@ -# The resident spectrum VA, WFIP-0005 units 1 and 2 (resident_shim.cpp): no part of its own +# The resident spectrum VA, WFIP-0005 units 1 and 2 (resident_shim.cpp): no part of its own; its column syncs in ww3_multi (PATCH.md section 26) # Read by scripts/ww3-gpu.sh: PART (WW_KOKKOS_), COUNT , FILE ; PATCH.md section 9. COUNT WW_KOKKOS_VA_COUNT +FILE model/src/wminiomd.F90 +FILE model/src/wmwavemd.F90 +FILE model/src/wmesmfmd.F90 diff --git a/kokkos/src/fortran_iface/presets.txt b/kokkos/src/fortran_iface/presets.txt new file mode 100644 index 00000000..4cf81d79 --- /dev/null +++ b/kokkos/src/fortran_iface/presets.txt @@ -0,0 +1,9 @@ +# The run-time presets of WW_KOKKOS_PRESET= (part_flags.cpp, PATCH.md section 25), one line each: +# PRESET . The one list: CMake compiles it into part_flags.cpp, and +# scripts/ww3-gpu.sh --kokkos and the matrix's WW3GPU_KOKKOS= read it, so the three +# cannot drift. Every part is a PART of parts.d. +# +# resident: W3SRCE batched and the PR3 propagation, W3OUTG, the point output's read, W3UICE's +# write of VA and W3CPRT's partitioning on the resident spectrum (WFIP-0005; PATCH.md sections 11, +# 15 to 18, 27 and 28). CPRT refuses no switch OUTG accepts: both need SHRD and refuse MPI and T. +PRESET resident SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT diff --git a/kokkos/src/fortran_iface/resident_c.hpp b/kokkos/src/fortran_iface/resident_c.hpp index e9a7d747..a3a4eb6b 100644 --- a/kokkos/src/fortran_iface/resident_c.hpp +++ b/kokkos/src/fortran_iface/resident_c.hpp @@ -1,7 +1,7 @@ // kokkos/src/fortran_iface/resident_c.hpp // The C ABI of the resident spectrum (WFIP-0005 units 1 and 2, the partial read of the pnt -// unit, PATCH.md section 16, and the zeroed columns of the updv unit, section 17): declarations -// only. +// unit, PATCH.md section 16, the zeroed columns of the updv unit, section 17, and the partial +// write of ww3_multi's exchanges, section 26): declarations only. // resident_shim.cpp implements the ww_va_* calls, srce_shim.cpp ww_srce_batch_va; // model/src/w3kokkos_residentmd.F90 and w3kokkosmd.F90 on the fork // (h0ffmann/WW3 ww3-gpu/kokkos-resident) mirror them. The rules of ww_kokkos_c.hpp hold: @@ -33,7 +33,7 @@ // The last two, and the byte comparison of every model file in scripts/ww3-gpu.sh, came out of // the mutation audit of docs/log/2026-10-17-mutation-audit.md. // Under WW_KOKKOS_VA_RESIDENT=1 the partial read stays partial (a HOST array is pushed first, as -// every host call does), so a column W3IOPE reads and the list misses keeps stale bits there. +// every host call does), so a column W3IOPE, W3IOBC or W3IOTR reads and the list misses keeps stale bits there. // // SPDX-License-Identifier: MIT #pragma once @@ -48,14 +48,25 @@ int ww_va_host_read(int igrid, float* va, int nspec, int nsealm); /// Before a host write of VA: ww_va_host_read, then state HOST. int ww_va_host_write(int igrid, float* va, int nspec, int nsealm); -/// Before W3IOPE (point output) reads VA on the host, with WW_KOKKOS_IOPE=1 (the pnt unit, -/// PATCH.md section 16): when the device holds the newer bits (state DEVICE), copy back only the +/// Before W3IOPE (point output), W3IOBC ('WRITE', boundary output) or section 4 of W3IOTR +/// (track output) reads VA on the host, with WW_KOKKOS_IOPE=1 (the pnt unit, PATCH.md sections +/// 17 and 23): when the device holds the newer bits (state DEVICE), copy back only the /// columns cols[0..ncol) of VA, W3WAVE's JSEA in 0..nsealm (duplicates allowed), gathered on the /// device; otherwise (HOST, BOTH) nothing. The state is NOT changed: every other column of the /// host VA stays stale, so the next ww_va_host_read still copies the whole array. A column /// outside 0..nsealm is BAD_SHAPE. va, nspec, nsealm as for ww_va_host_read. int ww_va_host_read_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const int* cols); +/// After host statements that wrote the columns cols[0..ncol) of VA and no other (ww3_multi's +/// WMIOHG and WMIOEG between W3WAVE calls, PATCH.md section 26; a column they also read needs +/// ww_va_host_read_cols first): with state DEVICE or BOTH those host columns are copied into +/// d_va, packed into one buffer (only their bytes cross); with HOST nothing. The state is NOT +/// changed. Duplicates allowed; a column outside 0..nsealm is BAD_SHAPE. Under +/// WW_KOKKOS_VA_CHECK=1 the listed columns are taken as written and every other column must +/// still hold the bits of the last sync. Under WW_KOKKOS_VA_RESIDENT=1 d_va is then copied back +/// whole, so a column written and not listed shows in the output. +int ww_va_host_wrote_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const int* cols); + /// Before a host statement whose only writes of VA set whole columns to 0, which /// ww_va_device_zero_cols then repeats on d_va (W3UICE under WW_KOKKOS_UPDV=1, the updv unit, /// PATCH.md section 17): no copy and no change of state. Under WW_KOKKOS_VA_RESIDENT=1 it is a diff --git a/kokkos/src/fortran_iface/resident_shim.cpp b/kokkos/src/fortran_iface/resident_shim.cpp index 7e961c57..19b42720 100644 --- a/kokkos/src/fortran_iface/resident_shim.cpp +++ b/kokkos/src/fortran_iface/resident_shim.cpp @@ -5,7 +5,7 @@ // // The Fortran owns VA; this file only copies it whole, host to device or back, when the // other side holds the newer bits, or (ww_va_host_read_cols, the pnt unit) copies back the few -// columns W3IOPE reads without changing which side is newer, or (ww_va_device_zero_cols, the updv +// columns W3IOPE, W3IOBC ('WRITE') or W3IOTR reads without changing which side is newer, or (ww_va_device_zero_cols, the updv // unit) repeats on d_va the zeroing of whole columns that W3UICE made on the host. A copy moves bits and computes nothing. Every call // comes from outside an OpenMP parallel region (the Fortran wrappers refuse one), so the // contexts and counters need no lock. Views die in Kokkos' finalize hook. @@ -27,26 +27,33 @@ #include #include #include +#include #include namespace ww::resident { namespace { +// One column list of the partial read and its buffers. Kept for the run: W3IOPE and W3IOBC ask +// for the same columns at every output time, so each list is allocated once. +struct ColSet { + std::vector cols; + Kokkos::View d_cols; + Kokkos::View d_buf; + Kokkos::View h_buf; +}; + struct Ctx { float* host = nullptr; int nspec = 0, nsealm = -1; DevVa d; Side side = Side::Host; - // The partial read: the sorted column list of the last call and its device copies, kept - // because W3IOPE asks for the same columns at every point-output time. - std::vector cols; + // The partial read: every sorted column list asked for, the most recent first. W3IOTR's list + // changes with the track; past kMaxColSets lists the oldest is dropped. + std::vector sets; // WW_KOKKOS_VA_CHECK=1: the host bits of VA when the two copies were last made equal, kept up to // date by the declared host writes of whole columns (the partial read, the zeroed columns). std::vector snap; bool snap_ok = false; - Kokkos::View d_cols; - Kokkos::View d_buf; - Kokkos::View h_buf; size_t count() const { return static_cast(nspec) * static_cast(nsealm + 1); } }; @@ -56,12 +63,19 @@ std::map& grids() { return *m; } +// The column lists a grid keeps buffers for: W3IOPE's and W3IOBC's, which repeat, a few of +// W3IOTR's, which follow the track, and ww3_multi's exchanges (PATCH.md section 26), one list per +// neighbouring grid and direction, read and written at each exchange. +constexpr size_t kMaxColSets = 8; + struct Counts { long long h2d = 0, d2h = 0, host_reads = 0, host_writes = 0, device_calls = 0; long long part_calls = 0, part_d2h = 0, part_cols = 0; long long zero_calls = 0, zero_launches = 0, zero_cols = 0, zero_checks = 0; + long long wrote_calls = 0, wrote_h2d = 0, wrote_cols = 0; long long va_checks = 0, poisons = 0; double h2d_bytes = 0.0, d2h_bytes = 0.0, spectrum_bytes = 0.0, part_bytes = 0.0; + double wrote_bytes = 0.0; }; Counts g_n; bool g_hooks = false; @@ -95,9 +109,7 @@ void release() { std::fprintf(stderr, "ww_kokkos: VA of grid %d was still newer on the device at finalize\n", igrid); c.d = DevVa(); - c.d_cols = {}; - c.d_buf = {}; - c.h_buf = {}; + c.sets.clear(); } grids().clear(); } @@ -238,6 +250,33 @@ int host_sync(int igrid, float* va, int nspec, int nsealm, bool write) { return WW_KOKKOS_OK; } +/// The buffers of the sorted column list `want` (n > 0 columns), moved to the front of the +/// grid's lists; allocated on its first use, dropping the oldest list past kMaxColSets. The +/// partial reads and the partial writes share the lists. +ColSet& col_set(Ctx& c, const std::vector& want) { + const int n = static_cast(want.size()); + const auto hit = + std::find_if(c.sets.begin(), c.sets.end(), [&](const ColSet& s) { return s.cols == want; }); + if (hit != c.sets.end()) { + std::rotate(c.sets.begin(), hit, hit + 1); + } else { + ColSet s; + s.cols = want; + s.d_cols = Kokkos::View( + Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols"), n); + Kokkos::deep_copy(s.d_cols, Kokkos::View>( + s.cols.data(), n)); + s.d_buf = Kokkos::View( + Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols_buf"), c.nspec, n); + s.h_buf = Kokkos::View( + Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols_host"), c.nspec, n); + if (c.sets.size() >= kMaxColSets) c.sets.pop_back(); + c.sets.insert(c.sets.begin(), std::move(s)); + } + return c.sets.front(); +} + /// The partial read: d_va(:, cols) gathered into d_buf on the device, one copy of d_buf back, /// and its columns stored into the Fortran array. Bits move and nothing is computed, so each /// host column copied holds what ww_va_host_read would have put there. The state is left as it @@ -263,23 +302,12 @@ int host_read_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const } if (want.empty()) return WW_KOKKOS_OK; const int n = static_cast(want.size()); - if (want != c->cols) { - c->cols = want; - c->d_cols = Kokkos::View( - Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols"), n); - Kokkos::deep_copy(c->d_cols, Kokkos::View>( - c->cols.data(), n)); - c->d_buf = Kokkos::View( - Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols_buf"), nspec, n); - c->h_buf = Kokkos::View( - Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols_host"), nspec, n); - } + ColSet& cs = col_set(*c, want); { ww::timers::Region tc("va.d2h.cols"); const DevVa d = c->d; - const auto dc = c->d_cols; - const auto buf = c->d_buf; + const auto dc = cs.d_cols; + const auto buf = cs.d_buf; const size_t ns = static_cast(nspec); Kokkos::parallel_for( "ww_va.gather_cols", Kokkos::RangePolicy<>(0, ns * static_cast(n)), @@ -287,17 +315,17 @@ int host_read_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const const size_t k = i / ns, s = i % ns; buf(s, k) = d(s, dc(k)); }); - Kokkos::deep_copy(c->h_buf, buf); + Kokkos::deep_copy(cs.h_buf, buf); Kokkos::fence("ww_va.d2h_cols"); } for (int k = 0; k < n; ++k) - std::memcpy(c->host + static_cast(c->cols[k]) * static_cast(nspec), - &c->h_buf(0, k), static_cast(nspec) * sizeof(float)); + std::memcpy(c->host + static_cast(cs.cols[k]) * static_cast(nspec), + &cs.h_buf(0, k), static_cast(nspec) * sizeof(float)); // The host columns just copied are part of what the host holds now. if (c->snap_ok) for (int k = 0; k < n; ++k) - std::memcpy(c->snap.data() + static_cast(c->cols[k]) * static_cast(nspec), - &c->h_buf(0, k), static_cast(nspec) * sizeof(float)); + std::memcpy(c->snap.data() + static_cast(cs.cols[k]) * static_cast(nspec), + &cs.h_buf(0, k), static_cast(nspec) * sizeof(float)); ++g_n.part_d2h; g_n.part_cols += n; g_n.part_bytes += static_cast(static_cast(n) * static_cast(nspec) * @@ -305,6 +333,66 @@ int host_read_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const return WW_KOKKOS_OK; } +/// The partial write: after host statements that wrote the columns cols of VA and no other +/// (ww3_multi's nesting and equal-rank updates, PATCH.md section 26), those host columns packed +/// into the list's h_buf, one copy of it to d_buf, and a scatter into d_va. Bits move and nothing is +/// computed, so each device column holds the host column's bits. The state is left as it was: +/// with DEVICE the other device columns are still the newer ones, with BOTH both copies are equal +/// again, and with HOST d_va is stale anyway and nothing moves. +int host_wrote_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const int* cols) { + ww::timers::Region t("va.host.wcols"); + Ctx* c = bind(igrid, va, nspec, nsealm); + if (c == nullptr) return WW_KOKKOS_ERR_BAD_SHAPE; + if (ncol < 0 || (ncol > 0 && cols == nullptr)) return WW_KOKKOS_ERR_BAD_SHAPE; + std::vector want(cols, cols + ncol); + std::sort(want.begin(), want.end()); + want.erase(std::unique(want.begin(), want.end()), want.end()); + if (!want.empty() && (want.front() < 0 || want.back() > nsealm)) { + say("ww_va_host_wrote_cols: a column outside 0..NSEALM"); + return WW_KOKKOS_ERR_BAD_SHAPE; + } + const size_t ns = static_cast(nspec); + // The listed columns are declared written: the snapshot takes their new host bits first, so + // the check below still catches a column the host wrote and the list misses. + if (c->snap_ok) + for (const int j : want) + std::memcpy(c->snap.data() + static_cast(j) * ns, + c->host + static_cast(j) * ns, ns * sizeof(float)); + if (const int rc = check(*c, igrid, "a partial host write"); rc != WW_KOKKOS_OK) return rc; + if (c->side == Side::Host || want.empty()) { + if (resident_diag() && c->side == Side::Host) { + if (!Kokkos::is_initialized()) return WW_KOKKOS_ERR_NOT_INITIALISED; + h2d(*c); + } + return WW_KOKKOS_OK; + } + if (!Kokkos::is_initialized()) return WW_KOKKOS_ERR_NOT_INITIALISED; + const int n = static_cast(want.size()); + ColSet& cs = col_set(*c, want); + for (int k = 0; k < n; ++k) + std::memcpy(&cs.h_buf(0, k), c->host + static_cast(cs.cols[k]) * ns, + ns * sizeof(float)); + { + ww::timers::Region tc("va.h2d.cols"); + Kokkos::deep_copy(cs.d_buf, cs.h_buf); + const DevVa d = c->d; + const auto dc = cs.d_cols; + const auto buf = cs.d_buf; + Kokkos::parallel_for( + "ww_va.scatter_cols", Kokkos::RangePolicy<>(0, ns * static_cast(n)), + KOKKOS_LAMBDA(const size_t i) { + const size_t k = i / ns, s = i % ns; + d(s, dc(k)) = buf(s, k); + }); + Kokkos::fence("ww_va.h2d_cols"); + } + ++g_n.wrote_h2d; + g_n.wrote_cols += n; + g_n.wrote_bytes += static_cast(static_cast(n) * ns * sizeof(float)); + if (resident_diag()) d2h(*c); // a column the host wrote and the list missed is overwritten + return WW_KOKKOS_OK; +} + /// The device side of a host statement that set whole columns of VA to 0 and wrote nothing else: /// the same statement, VA(:,JSEA) = 0., on d_va. A store of +0.0f, so the device column holds /// the bits of the host column. HOST: the host write is the newer one and d_va is stale anyway. @@ -435,6 +523,12 @@ int ww_va_host_read_cols(int igrid, float* va, int nspec, int nsealm, int ncol, return guarded([&]() { return host_read_cols(igrid, va, nspec, nsealm, ncol, cols); }); } +int ww_va_host_wrote_cols(int igrid, float* va, int nspec, int nsealm, int ncol, + const int* cols) { + ++g_n.wrote_calls; + return guarded([&]() { return host_wrote_cols(igrid, va, nspec, nsealm, ncol, cols); }); +} + int ww_va_host_update(int igrid, float* va, int nspec, int nsealm) { ++g_n.zero_calls; return guarded([&]() { @@ -487,6 +581,10 @@ void ww_va_report(void) { std::fprintf(stderr, "ww_kokkos: VA partial host reads: %lld\n", g_n.part_calls); std::fprintf(stderr, "ww_kokkos: VA partial device-to-host copies: %lld\n", g_n.part_d2h); std::fprintf(stderr, "ww_kokkos: VA partial columns copied: %lld\n", g_n.part_cols); + std::fprintf(stderr, "ww_kokkos: VA partial host writes: %lld\n", g_n.wrote_calls); + std::fprintf(stderr, "ww_kokkos: VA partial host-to-device copies: %lld\n", g_n.wrote_h2d); + std::fprintf(stderr, "ww_kokkos: VA partial columns stored: %lld (%.4f MB)\n", g_n.wrote_cols, + g_n.wrote_bytes / 1.0e6); std::fprintf(stderr, "ww_kokkos: VA host updates repeated on the device: %lld\n", g_n.zero_calls); std::fprintf(stderr, "ww_kokkos: VA device zeroing launches: %lld\n", g_n.zero_launches); std::fprintf(stderr, "ww_kokkos: VA device columns zeroed: %lld\n", g_n.zero_cols); diff --git a/kokkos/src/fortran_iface/ww_kokkos_c.hpp b/kokkos/src/fortran_iface/ww_kokkos_c.hpp index facba817..844c9f3e 100644 --- a/kokkos/src/fortran_iface/ww_kokkos_c.hpp +++ b/kokkos/src/fortran_iface/ww_kokkos_c.hpp @@ -874,6 +874,22 @@ int ww_st6_last_error(void); /// variable other than "1" is off whatever WW_KOKKOS_ALL says. int ww_kokkos_flag(const char* name); +/// WW_KOKKOS_PRESET (presets.txt): 0 unset or empty, 1 a preset of presets.txt, -1 any other +/// name, which W3KOKKOS_SETUP turns into a stop. Its parts are on in ww_kokkos_flag as if set to 1. +int ww_kokkos_preset_status(void); + +/// 1 when WW_KOKKOS_STRICT is "1": a part asked on Kokkos that would run the Fortran stops the +/// model instead (W3KOKKOSMD, KOKKOS_FALLBACK). Read once per process. +int ww_kokkos_strict(void); + +/// The counter of Fortran fallbacks of `part` (at most 24 characters), created on first use; -1 +/// when the name is too long or the table is full. Call at set-up, serially; the first call +/// registers one line at exit that prints every counter. +int ww_kokkos_fallback_slot(const char* part); + +/// One more call of the slot's part on the Fortran: one increment, no lock (callers are serial). +void ww_kokkos_fallback_add(int slot); + // --------------------------------------------------------------------------- // W3WAVE's per-step support routines (wave_support_shim.cpp): W3GATH, W3SCAT // (W3_SHRD), W3UWND (WNT1, RWND), W3UTRN (IC0), W3DZXY, W3MAP3, W3MAPT (PR3), and diff --git a/kokkos/src/ww_kokkos/CMakeLists.txt b/kokkos/src/ww_kokkos/CMakeLists.txt index c3a4fd4c..35d6d9f7 100644 --- a/kokkos/src/ww_kokkos/CMakeLists.txt +++ b/kokkos/src/ww_kokkos/CMakeLists.txt @@ -73,6 +73,7 @@ target_sources(ww_kokkos PRIVATE part_fld.hpp part_ptmean.hpp part_w3part.hpp + iosf_w3cprt.hpp pdlib_params.hpp pdlib_fsn2.hpp pdlib_fspsi2.hpp diff --git a/kokkos/src/ww_kokkos/iogo_w3outg.hpp b/kokkos/src/ww_kokkos/iogo_w3outg.hpp index d6ecc50d..ed268bac 100644 --- a/kokkos/src/ww_kokkos/iogo_w3outg.hpp +++ b/kokkos/src/ww_kokkos/iogo_w3outg.hpp @@ -660,6 +660,31 @@ KOKKOS_INLINE_FUNCTION void w3outg_part_point(const W3OutgArgs& g, co } } +/// Section 6 (:2242-2340): 6.a the UNDEF fill of the partition fields over their NSEALM rows, +/// 6.b the copy from ICPRT and DTPRT per local sea point. It reads no A: of g it uses mapsf, +/// mapsta, icprt, dtprt and the partition fields only, and s.nseal, s.noswll, s.undef, s.wscut. +/// No other section reads what it writes, so it may run apart from the rest of W3OUTG. +template +void w3outg_part(const ExecSpace& exec, const W3OutgScalars& s, const W3OutgArgs& g) { + const Real undef = s.undef; + const int nsealm = static_cast(g.pwst.extent(0)); + const int ncol = s.noswll + 1; + Kokkos::parallel_for( + "ww::iogo::w3outg_part_init", Kokkos::RangePolicy(exec, 0, nsealm), KOKKOS_LAMBDA(const int j) { + for (int c = 0; c < ncol; ++c) { + g.phs(j, c) = undef; g.ptp(j, c) = undef; g.plp(j, c) = undef; g.pdir(j, c) = undef; + g.psi(j, c) = undef; g.pws(j, c) = undef; g.pthp0(j, c) = undef; g.pqp(j, c) = undef; + g.ppe(j, c) = undef; g.pgw(j, c) = undef; g.psw(j, c) = undef; g.ptm1(j, c) = undef; + g.pt1(j, c) = undef; g.pt2(j, c) = undef; g.pep(j, c) = undef; + } + g.pwst(j) = undef; + g.pnr(j) = undef; + }); + Kokkos::parallel_for( + "ww::iogo::w3outg_part", Kokkos::RangePolicy(exec, 1, s.nseal + 1), + KOKKOS_LAMBDA(const int jsea) { w3outg_part_point(g, s, jsea); }); +} + /// W3OUTG(A, FLPART, FLOUTG, FLOUTG2), :1201-2380, as switch_lab_shrd (s.ompg false) or /// switch_lab_ompg (s.ompg true) compiles it. template @@ -700,23 +725,7 @@ void w3outg(const ExecSpace& exec, const W3OutgScalars& s, const W3OutgFlags& f, KOKKOS_LAMBDA(const int jsea) { w3outg_point(g, s, l, jsea); }); // 6. (:2242-2340) - if (f.flpart) { - const int ncol = s.noswll + 1; - Kokkos::parallel_for( - "ww::iogo::w3outg_part_init", Kokkos::RangePolicy(exec, 0, nsealm), KOKKOS_LAMBDA(const int j) { - for (int c = 0; c < ncol; ++c) { - g.phs(j, c) = undef; g.ptp(j, c) = undef; g.plp(j, c) = undef; g.pdir(j, c) = undef; - g.psi(j, c) = undef; g.pws(j, c) = undef; g.pthp0(j, c) = undef; g.pqp(j, c) = undef; - g.ppe(j, c) = undef; g.pgw(j, c) = undef; g.psw(j, c) = undef; g.ptm1(j, c) = undef; - g.pt1(j, c) = undef; g.pt2(j, c) = undef; g.pep(j, c) = undef; - } - g.pwst(j) = undef; - g.pnr(j) = undef; - }); - Kokkos::parallel_for( - "ww::iogo::w3outg_part", Kokkos::RangePolicy(exec, 1, s.nseal + 1), - KOKKOS_LAMBDA(const int jsea) { w3outg_part_point(g, s, jsea); }); - } + if (f.flpart) w3outg_part(exec, s, g); // :2347-2362 const bool u3s1 = floloc(6, 8); diff --git a/kokkos/src/ww_kokkos/iosf_w3cprt.hpp b/kokkos/src/ww_kokkos/iosf_w3cprt.hpp new file mode 100644 index 00000000..756b54ef --- /dev/null +++ b/kokkos/src/ww_kokkos/iosf_w3cprt.hpp @@ -0,0 +1,121 @@ +// kokkos/src/ww_kokkos/iosf_w3cprt.hpp +// W3CPRT: the spectral partitioning of every local sea point at an output time, ported. Per point +// it converts the action spectrum N(k,theta) of VA to the energy spectrum E(f,theta) = N*TPI*SIG/CG, +// calls W3PART (part_w3part.hpp, unchanged) and stores NP+1 columns XP(1:15,0:NP) in DTPRT one point +// after the other, with ICPRT(JSEA,1) = NP+1 and ICPRT(JSEA,2) the first column of point JSEA. +// Phase 2: it reads the resident VA (ww::resident::device_va), so an output time with partitioning +// copies the partition table back to the host instead of the spectrum. +// +// WW3 heritage: SUBROUTINE W3CPRT, WW3-noaa/model/src/w3iosfmd.F90:109-344 at NOAA-EMC/WW3 03adda91 +// (identical in the h0ffmann/WW3 fork at the WW3/ pin 80644ba9), author H. L. Tolman, with C. Bunney's +// DIMXP of 25-Jul-2018. Only what W3_SHRD compiles is ported: INIT_GET_ISEA gives ISEA = JSEA +// (w3parall.F90:1236, NAPROC = 1). Not ported: W3_T (test output), W3_S (STRACE). The NaN depth stop +// (:270-274) is the caller's (W3KOKKOS_CPRT runs it before the call, which changes no output: the +// Fortran stops before it writes anything a later reader sees). +// +// Two parts, both in the Fortran's order: +// w3cprt_point sections 2-4 (:245-279) for one JSEA: the MAPSTA test, UABS, UDIR, DEPTH, E2 +// and the W3PART call. Each point reads only its own column, so points run in +// parallel and each one is the Fortran's serial code. +// w3cprt_store section 5 (:284-303) for one JSEA after an integer prefix sum: the counters +// and the copy of XP(:,0:NP) to its columns. The Fortran's TMP array grows by +// TSFAC*NSEAL columns and its new columns are zeroed (:288-299), but every column +// 1..FINSIZ is written once by :303 before DTPRT = TMP(:,1:FINSIZ) (:320-326), so +// DTPRT is the concatenation of the XP(:,0:NP) of the points in JSEA order, which +// is what the prefix sum places. Integer sums are exact in any order. +// +// Arithmetic, from the -O3 GIMPLE of w3iosfmd.F90 (gfortran 16.2.0, WW3's Release flags, the +// lab switch; port report kokkos/port_report/W3CPRT_KOKKOS.md): +// - UABS = U10*ASF, UDIR = U10D*RADE, one mulss each. +// - FACT = (TPI*SIG(IK))/CG(IK,ISEA): the product with the folded REAL TPI first, then divss. +// - E2(IK,ITH) = VA(ITH+(IK-1)*NTH,JSEA)*FACT, one mulss. No FMA (contraction off). +// +// SPDX-License-Identifier: LGPL-3.0-or-later +#pragma once + +#include + +#include "part_w3part.hpp" +#include "real.hpp" +#include "st4_params.hpp" + +namespace ww::iosf { + +using ww::Real; + +/// W3PART's result for a point the MAPSTA test of :253 skips (its ICPRT(JSEA,1) stays 0). +inline constexpr int kCprtSkipped = -2; + +/// DIMXP of :213-219. +KOKKOS_INLINE_FUNCTION int w3cprt_dimxp(int nk, int nth, int ptmeth) { + if (ptmeth == 4 || ptmeth == 5) return 2; + return ((nk + 1) / 2) * ((nth - 1) / 2); +} + +/// The Reals and ints of one point's work: E2(NK,NTH), XP(DIMP,0:DIMXP) and W3PART's scratch. +KOKKOS_INLINE_FUNCTION int w3cprt_nreal(int nk, int nth, int dimxp) { + return nk * nth + ww::part::kDimp * (dimxp + 1) + ww::part::w3part_nreal(nk, nth, dimxp); +} +KOKKOS_INLINE_FUNCTION int w3cprt_nint(int nk, int nth, int dimxp, int ihmax) { + return ww::part::w3part_nint(nk, nth, dimxp, ihmax); +} + +/// The arrays of one call, index 0 at the Fortran's lower bound, LayoutLeft: +/// va VA(NSPEC,0:NSEALM) va(isp-1, jsea) +/// mapsf MAPSF(NSEA,3) mapsf(isea-1, 0) = IX; mapsta MAPSTA(NY,NX) +/// u10, u10d, asf (1:NSEA) u10(isea-1) +/// dw DW(0:NSEA) dw(isea); cg, wn CG, WN(0:NK+1,0:NSEA), cg(ik, isea) +template +struct W3cprtArgs { + template + using V1 = Kokkos::View>; + template + using V2 = Kokkos::View>; + V2 va; + V2 mapsf, mapsta; + V1 u10, u10d, asf, dw; + V2 cg, wn; +}; + +/// Sections 2-4 (:245-279) for JSEA = jsea (1-based). Returns W3PART's status (kPtmeanOk, +/// kPtmeanXpFull, kPtmeanNoPeak) and sets np (kCprtSkipped when MAPSTA < 0), xp as W3PART's +/// XP(J,N) = xp[(J-1) + 15*N], nxpfull W3PART's count. e2 holds NK*NTH Reals; w is carved. +template +KOKKOS_INLINE_FUNCTION int w3cprt_point(const W3cprtArgs& a, const ww::part::PtmeanGrid& g, + const ww::part::PtmeanOpts& o, const ww::part::W3partOpts& po, + const int* neigh, int dimxp, int jsea, Real* e2, Real* xp, + const ww::part::W3partScratch& w, int& np, int& nxpfull) { + const int nk = g.nk, nth = g.nth; + const int isea = jsea; // INIT_GET_ISEA under W3_SHRD + const int ix = a.mapsf(isea - 1, 0); + const int iy = a.mapsf(isea - 1, 1); + nxpfull = 0; + if (a.mapsta(iy - 1, ix - 1) < 0) { // :253 + np = kCprtSkipped; + return ww::part::kPtmeanOk; + } + const Real uabs = a.u10(isea - 1) * a.asf(isea - 1); // :258-260 + const Real udir = a.u10d(isea - 1) * ww::st4::kRade; + const Real depth = a.dw(isea); + for (int ik = 1; ik <= nk; ++ik) { // :262-267 + const Real fact = ww::st4::kTpi * g.sig[ik] / a.cg(ik, isea); + for (int ith = 1; ith <= nth; ++ith) + e2[(ik - 1) + (ith - 1) * nk] = a.va(ith + (ik - 1) * nth - 1, jsea) * fact; + } + return ww::part::w3part(g, o, po, neigh, e2, uabs, udir, depth, &a.wn(1, isea), np, xp, dimxp, w, + nxpfull); // :278-279 +} + +/// Section 5 (:284-303) for one point: c is NP+1 when NP >= 0 and 0 otherwise, first its +/// ICPRT(JSEA,2) (1-based column of DTPRT). icprt is ICPRT(NSEALM+1,2), dtprt DTPRT(DIMP,:). +template +KOKKOS_INLINE_FUNCTION void w3cprt_store(int jsea, int np, int first, const Real* xp, const V2i& icprt, + const V2r& dtprt) { + // ICPRT(JSEA+1,2) = ICPRT(JSEA,2) [+ NP + 1] (:251, :286) is the caller's prefix sum. + if (np < 0) return; + icprt(jsea - 1, 0) = np + 1; + for (int n = 0; n <= np; ++n) + for (int j = 0; j < ww::part::kDimp; ++j) dtprt(j, first - 1 + n) = xp[j + ww::part::kDimp * n]; +} + +} // namespace ww::iosf diff --git a/regtests/CMakeLists.txt b/regtests/CMakeLists.txt index fa4e9c43..53c7faec 100644 --- a/regtests/CMakeLists.txt +++ b/regtests/CMakeLists.txt @@ -23,7 +23,8 @@ # flag to 0. ADR-0011's ST4 checkpoint runs one at a time. # WW3GPU_KOKKOS empty (the default): the Kokkos run sets WW3GPU_ROUTINES as above. Otherwise # the flags of scripts/ww3-gpu.sh --kokkos, so a regtest and a timed run share -# them: resident (SRCE,XYP3,KTP3,OUTG,IOPE,UPDV), all (WW_KOKKOS_ALL=1), none, +# them: a preset of kokkos/src/fortran_iface/presets.txt (resident: +# SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT), all (WW_KOKKOS_ALL=1), none, # or a comma list of the parts of kokkos/src/fortran_iface/parts.d; the # selection is then WW3GPU_SWITCH_WORDS alone, and the Kokkos run must show # each part as =K in W3INIT's parts line @@ -231,7 +232,7 @@ set_tests_properties(record:summary PROPERTIES FIXTURES_CLEANUP ww3-gpu LABELS " set(_n 0) foreach(test IN LISTS E_TESTS) - list(GET E_OPTS ${_n} opts) + ww3gpu_entry_opts(opts "${E_OPTS}" ${_n}) math(EXPR _n "${_n} + 1") ww3gpu_words(o "${opts}") ww3gpu_opt_of(sw -s ${o}) diff --git a/regtests/cmake/run_entry.cmake b/regtests/cmake/run_entry.cmake index 07f769a3..36c0b856 100644 --- a/regtests/cmake/run_entry.cmake +++ b/regtests/cmake/run_entry.cmake @@ -14,7 +14,9 @@ # WW3GPU_KOKKOS the Kokkos run takes that preset's flags instead (ww3gpu_kokkos_env), kokkos_log # is W3INIT's parts line showing each part on Kokkos, and kokkos_spectra lists every # "W3 spectra|points computed" counter. Every run sets WW_KOKKOS_ALL and unsets the parts it -# does not turn on, so no flag is inherited from the shell. +# does not turn on, so no flag is inherited from the shell. The Kokkos run also sets +# WW_KOKKOS_STRICT=1: a part it asks for that would run the Fortran stops it, so a pair never +# compares the Fortran with itself unseen (kokkos/src/fortran_iface/PATCH.md section 24). cmake_minimum_required(VERSION 3.21) include("${CONFIG}") include("${CMAKE_CURRENT_LIST_DIR}/ww3gpu_common.cmake") diff --git a/regtests/cmake/selftest.cmake b/regtests/cmake/selftest.cmake index 5b241c48..10e681e8 100644 --- a/regtests/cmake/selftest.cmake +++ b/regtests/cmake/selftest.cmake @@ -77,6 +77,26 @@ expect("${M_OPTS}" "-s MP -w work_MP;-s MH -w work_MH -f -p $mpi -n $nr -t $nth; ww3gpu_select_entries(A "${t}/rt/bin/matrix.base" "${t}/rt" all) list(LENGTH A_TESTS na) expect("${na}" "12" "all selection, the three together") +# An entry with no options at all, first in the selection: its options must stay an item of +# _OPTS, or every later entry reads the options of the next one (and the last is out of range). +file(MAKE_DIRECTORY "${t}/rt0/c/input" "${t}/rt0/bin") +file(WRITE "${t}/rt0/c/input/switch" "F90 SHRD ST4 NL1\n") +file(WRITE "${t}/rt0/c/input/switch_K1" "F90 SHRD ST4 NL1\n") +file(WRITE "${t}/rt0/bin/matrix.base" [=[ + echo "$rtst $ww3 c" >> matrix.body + echo "$rtst -s K1 -w work_K1 $ww3 c" >> matrix.body + echo "$rtst $ww3 c" >> matrix.body +]=]) +ww3gpu_select_entries(Z "${t}/rt0/bin/matrix.base" "${t}/rt0" serial) +list(LENGTH Z_TESTS nzt) +list(LENGTH Z_OPTS nzo) +expect("${nzt}|${nzo}" "3|3" "an entry with no options first: one options item per test") +set(zo "") +foreach(i 0 1 2) + ww3gpu_entry_opts(o "${Z_OPTS}" ${i}) + list(APPEND zo "[${o}]") +endforeach() +expect("${zo}" "[];[-s K1 -w work_K1];[]" "an entry with no options first: the options, read back") ww3gpu_select_entries(T "${t}/rt/bin/matrix.base" "${t}/rt" serial ST4) expect("${T_TESTS}" "a;a;a;a;b;b;a;b" "serial selection by ST4, tests (NL3 and NL1X are in)") ww3gpu_physics(w SDS4 SNL1 SPR4 SIN4) @@ -144,11 +164,11 @@ endforeach() # --- WW3GPU_KOKKOS: parts.d, the preset, the run environment, the parts line, the words ------ file(MAKE_DIRECTORY "${t}/parts.d") file(WRITE "${t}/parts.d/a.txt" "# a unit\nPART SRCE\nCOUNT WW_KOKKOS_SRCE_COUNT\n\nFILE model/src/w3srcemd.F90\n") -file(WRITE "${t}/parts.d/b.txt" "PART XYP3\nPART KTP3\nPART OUTG\nPART IOPE\nPART UPDV\nCOUNT WW_KOKKOS_PROP3_COUNT\n") +file(WRITE "${t}/parts.d/b.txt" "PART XYP3\nPART KTP3\nPART OUTG\nPART IOPE\nPART UPDV\nPART CPRT\nCOUNT WW_KOKKOS_PROP3_COUNT\n") ww3gpu_parts_d(pp pc "${t}/parts.d") -expect("${pp}|${pc}" "SRCE;XYP3;KTP3;OUTG;IOPE;UPDV|WW_KOKKOS_SRCE_COUNT;WW_KOKKOS_PROP3_COUNT" "parts.d") +expect("${pp}|${pc}" "SRCE;XYP3;KTP3;OUTG;IOPE;UPDV;CPRT|WW_KOKKOS_SRCE_COUNT;WW_KOKKOS_PROP3_COUNT" "parts.d") ww3gpu_kokkos_preset(f resident "${pp}") -expect("${f}" "SRCE;XYP3;KTP3;OUTG;IOPE;UPDV" "preset resident") +expect("${f}" "SRCE;XYP3;KTP3;OUTG;IOPE;UPDV;CPRT" "preset resident") ww3gpu_kokkos_preset(f all "${pp}") expect("${f}" "ALL" "preset all") ww3gpu_kokkos_preset(f none "${pp}") @@ -159,14 +179,27 @@ ww3gpu_kokkos_preset(f "SRCE,BOGUS" "${pp}") if(NOT f MATCHES "^ERROR:'BOGUS'") fail("preset with an unknown part: ${f}") endif() +# The presets come from presets.txt: a fixture table with another preset, and a name it lacks. +set(_real_presets "${WW3GPU_PRESETS_TXT}") +set(WW3GPU_PRESETS_TXT "${t}/presets.txt") +file(WRITE "${WW3GPU_PRESETS_TXT}" "# a table\nPRESET resident SRCE,XYP3\nPRESET lean SRCE\n") +ww3gpu_kokkos_preset(f resident "${pp}") +expect("${f}" "SRCE;XYP3" "preset resident read from presets.txt") +ww3gpu_kokkos_preset(f lean "${pp}") +expect("${f}" "SRCE" "a second preset of presets.txt") +ww3gpu_kokkos_preset(f bogus "${pp}") +if(NOT f MATCHES "^ERROR:'bogus' is not a preset") + fail("an unknown preset: ${f}") +endif() +set(WW3GPU_PRESETS_TXT "${_real_presets}") ww3gpu_kokkos_env(e K "SRCE" "SRCE;XYP3" "WW_KOKKOS_SRCE_COUNT") -expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_SRCE=1;WW_KOKKOS_XYP3=;WW_KOKKOS_SRCE_COUNT=1" "env, Kokkos run of a part list") +expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_STRICT=1;WW_KOKKOS_PRESET=;WW_KOKKOS_SRCE=1;WW_KOKKOS_XYP3=;WW_KOKKOS_SRCE_COUNT=1" "env, Kokkos run of a part list") ww3gpu_kokkos_env(e F "SRCE" "SRCE;XYP3" "WW_KOKKOS_SRCE_COUNT") -expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_SRCE=;WW_KOKKOS_XYP3=;WW_KOKKOS_SRCE_COUNT=" "env, Fortran run: ALL=0, nothing inherited") +expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_STRICT=;WW_KOKKOS_PRESET=;WW_KOKKOS_SRCE=;WW_KOKKOS_XYP3=;WW_KOKKOS_SRCE_COUNT=" "env, Fortran run: ALL=0, nothing inherited") ww3gpu_kokkos_env(e K "ALL" "SRCE" "") -expect("${e}" "WW_KOKKOS_ALL=1;WW_KOKKOS_SRCE=" "env, Kokkos run of all") +expect("${e}" "WW_KOKKOS_ALL=1;WW_KOKKOS_STRICT=1;WW_KOKKOS_PRESET=;WW_KOKKOS_SRCE=" "env, Kokkos run of all") ww3gpu_kokkos_env(e G "ALL" "SRCE" "") -expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_SRCE=" "env, the repeated Fortran run of all") +expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_STRICT=;WW_KOKKOS_PRESET=;WW_KOKKOS_SRCE=" "env, the repeated Fortran run of all") set(pl " Kokkos parts (K Kokkos, F Fortran): SNL1=F SRCE=K GATH=F XYP3=K KTP3=K (3 of 5 on Kokkos)") ww3gpu_parts_line_check(m "${pl}" "SRCE;XYP3;KTP3") expect("${m}" "" "parts line, every part on Kokkos") diff --git a/regtests/cmake/stage.cmake b/regtests/cmake/stage.cmake index 2fdba263..234abdb8 100644 --- a/regtests/cmake/stage.cmake +++ b/regtests/cmake/stage.cmake @@ -25,7 +25,7 @@ if(WW3GPU_LAB_SWITCH) endif() set(i 0) foreach(test IN LISTS E_TESTS) - list(GET E_OPTS ${i} opts) + ww3gpu_entry_opts(opts "${E_OPTS}" ${i}) math(EXPR i "${i} + 1") ww3gpu_words(o "${opts}") ww3gpu_opt_of(inp -i ${o}) diff --git a/regtests/cmake/ww3gpu_common.cmake b/regtests/cmake/ww3gpu_common.cmake index 54574422..7f6e0bbd 100644 --- a/regtests/cmake/ww3gpu_common.cmake +++ b/regtests/cmake/ww3gpu_common.cmake @@ -83,10 +83,32 @@ function(ww3gpu_parts_d parts counts dir) set(${counts} "${c}" PARENT_SCOPE) endfunction() +# The presets of WW_KOKKOS_PRESET (PATCH.md section 25), the file the shim compiles and +# scripts/ww3-gpu.sh reads; WW3GPU_PRESETS_TXT may point elsewhere (selftest.cmake). +if(NOT DEFINED WW3GPU_PRESETS_TXT) + set(WW3GPU_PRESETS_TXT "${CMAKE_CURRENT_LIST_DIR}/../../kokkos/src/fortran_iface/presets.txt") +endif() + +# ww3gpu_preset_parts( ): the PART,... list of PRESET in WW3GPU_PRESETS_TXT, or +# empty when the file has no such preset. +function(ww3gpu_preset_parts out name) + set(r "") + if(EXISTS "${WW3GPU_PRESETS_TXT}") + file(STRINGS "${WW3GPU_PRESETS_TXT}" lines REGEX "^PRESET ") + foreach(l IN LISTS lines) + if(l MATCHES "^PRESET ([a-z][a-z0-9_]*) ([A-Z][A-Z0-9_]*(,[A-Z][A-Z0-9_]*)*)$" AND CMAKE_MATCH_1 STREQUAL name) + set(r "${CMAKE_MATCH_2}") + break() + endif() + endforeach() + endif() + set(${out} "${r}" PARENT_SCOPE) +endfunction() + # ww3gpu_kokkos_preset( ): the flags the Kokkos run of WW3GPU_KOKKOS sets, as -# scripts/ww3-gpu.sh --kokkos does: "all" is ALL (WW_KOKKOS_ALL=1), "resident" the part list of -# that preset (kokkos_preset in ww3-gpu.sh), "none" nothing, and a comma list its parts, each of -# ; is ERROR: for anything else. +# scripts/ww3-gpu.sh --kokkos does: "all" is ALL (WW_KOKKOS_ALL=1), a preset of presets.txt +# (resident) its part list, "none" nothing, and a comma list its parts, each of ; is +# ERROR: for anything else. function(ww3gpu_kokkos_preset out spec parts) if(spec STREQUAL "all") set(${out} ALL PARENT_SCOPE) @@ -94,8 +116,13 @@ function(ww3gpu_kokkos_preset out spec parts) elseif(spec STREQUAL "none") set(${out} "" PARENT_SCOPE) return() - elseif(spec STREQUAL "resident") - set(spec "SRCE,XYP3,KTP3,OUTG,IOPE,UPDV") + elseif(spec MATCHES "^[a-z][a-z0-9_]*$") + ww3gpu_preset_parts(pl "${spec}") + if(pl STREQUAL "") + set(${out} "ERROR:'${spec}' is not a preset of ${WW3GPU_PRESETS_TXT}" PARENT_SCOPE) + return() + endif() + set(spec "${pl}") endif() string(REPLACE "," ";" l "${spec}") if(l STREQUAL "") @@ -114,7 +141,10 @@ endfunction() # ww3gpu_kokkos_env( ): the environment of one run, a list of # NAME=VALUE, NAME= meaning unset. Every WW_KOKKOS_ of is unset unless the Kokkos run # turns it on, WW_KOKKOS_ALL is 0 except in a Kokkos run whose has ALL, and the Kokkos run -# sets every counter of . The Fortran runs never inherit a flag from the shell. +# sets every counter of . The Kokkos run sets WW_KOKKOS_STRICT=1, so a part it asks for +# that would run the Fortran stops the run instead of comparing the Fortran with itself (PATCH.md +# section 24); the Fortran runs unset it. No run takes WW_KOKKOS_PRESET from the shell. The +# Fortran runs never inherit a flag from the shell. function(ww3gpu_kokkos_env out k flags parts counts) set(e "") if(k STREQUAL "K" AND "ALL" IN_LIST flags) @@ -122,6 +152,12 @@ function(ww3gpu_kokkos_env out k flags parts counts) else() list(APPEND e "WW_KOKKOS_ALL=0") endif() + if(k STREQUAL "K") + list(APPEND e "WW_KOKKOS_STRICT=1") + else() + list(APPEND e "WW_KOKKOS_STRICT=") + endif() + list(APPEND e "WW_KOKKOS_PRESET=") foreach(p IN LISTS parts) if(k STREQUAL "K" AND p IN_LIST flags) list(APPEND e "WW_KOKKOS_${p}=1") @@ -260,11 +296,25 @@ endfunction() # ww3gpu_select_entries( [ # []]): every kept entry, in order, as two parallel lists _TESTS and _OPTS -# (the entry's options, $ww3 and the test dropped, space-joined). is a list of switch +# (the entry's options, $ww3 and the test dropped, space-joined; an entry with none is +# WW3GPU_NO_OPTS, since a CMake list cannot hold an empty first item, and ww3gpu_entry_opts reads +# it back as empty). is a list of switch # words, NL1 by default, ANY for no physics filter; every entry whose switch has one of them and # passes ww3gpu_switch_words on is in exactly one of serial (none of MPI, OMPG, OMPH), omp # (OMPG or OMPH, no MPI) and mpi (MPI); all is the three. A line whose switch file is missing is # dropped, and so is a switch with both ST4 and FLX5, which KOKKOS refuses. +set(WW3GPU_NO_OPTS "") + +# ww3gpu_entry_opts( ): the options of entry of a _OPTS list of +# ww3gpu_select_entries, empty for an entry that has none. +function(ww3gpu_entry_opts out optl i) + list(GET optl ${i} o) + if(o STREQUAL "${WW3GPU_NO_OPTS}") + set(o "") + endif() + set(${out} "${o}" PARENT_SCOPE) +endfunction() + function(ww3gpu_select_entries prefix base rt kind) set(phys "${ARGV4}") if(phys STREQUAL "") @@ -336,6 +386,9 @@ function(ww3gpu_select_entries prefix base rt kind) continue() endif() list(JOIN tok " " opts) + if(opts STREQUAL "") + set(opts "${WW3GPU_NO_OPTS}") + endif() list(APPEND tests "${test}") list(APPEND optl "${opts}") endforeach() diff --git a/scripts/restart_pair.sh b/scripts/restart_pair.sh new file mode 100755 index 00000000..c04018e1 --- /dev/null +++ b/scripts/restart_pair.sh @@ -0,0 +1,133 @@ +#!/usr/bin/env bash +# restart_pair.sh: does a run restarted from its own restart file give the bytes of the run it +# was cut from, with the resident Kokkos preset, on a SHRD case? Gap 10 of +# docs/PRODUCTION_READINESS_202610.md. Every restart-reproduction entry of WW3's matrix.base at +# the WW3/ pin (rstrt_b4b: ww3_tp2.17, ww3_ufs1.1) is MPI, where no resident part runs +# (ADR-0017), so this is the smallest equivalent on the full case of scripts/ww3-gpu.sh. +# +# scripts/restart_pair.sh run the four runs below and compare +# scripts/restart_pair.sh --self-test check the namelist edits on a scratch copy; runs no model +# +# It needs the executable that `WW3GPU_SWITCH=switch_lab_shrd bash scripts/ww3-gpu.sh --ww3-gpu +# full --cpu --kokkos resident --build-only` builds in build/ww3-gpu-openmp/ww3/bin, and stops +# when that build's switch is not SHRD with KOKKOS. Inside the pinned ww3 shell (just ww3). +# +# Case: kokkos/tests/fixtures/bench_small (120 x 80, 9 520 sea points, 12 hours), with nine +# fields every hour from 06:00 to 12:00 and a restart at 06:00 and at 12:00. Runs, one executable: +# fortran/a, kokkos/a 00:00 to 12:00 from the cold start +# fortran/b, kokkos/b 06:00 to 12:00 from /a's 06:00 restart, copied to restart.ww3 +# The Fortran runs set WW_KOKKOS_ALL=0; the Kokkos runs WW_KOKKOS_PRESET=resident, +# WW_KOKKOS_STRICT=1 and every *_COUNT=1. Both keep WW_KOKKOS_VA_POISON and WW_KOKKOS_VA_CHECK +# from the caller. Compared with cmp: out_grd.ww3 and the 12:00 restart of b against a (restart +# reproduction), and every file of kokkos/x against fortran/x (bit for bit). +# +# Exit codes: 0 every comparison equal; 1 a run failed or a file differs; 3 no SHRD KOKKOS build; +# 64 a bad argument. A Kokkos difference that the Fortran pair shows too is printed as WW3's own. +set -euo pipefail + +root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +case_in="$root/kokkos/tests/fixtures/bench_small" +bdir="${WW3GPU_RESTART_BUILD:-$root/build/ww3-gpu-openmp}" +runs="${WW3GPU_RUNS:-$root/build/runs}/restart-pair" +fields="HS LM T02 T0M1 T01 FP DIR SPR DP" + +# edit_case : the fixture's ww3_shel.nml with , the fields and the restarts. +edit_case() { + local d="$1" start="$2" + cp "$case_in"/* "$d"/ + sed -i -e "s#^ DOMAIN%START = .*# DOMAIN%START = '20200101 $start'#" \ + -e "s#^ TYPE%FIELD%LIST = .*# TYPE%FIELD%LIST = '$fields'#" \ + -e "s#^ DATE%FIELD = .*# DATE%FIELD = '20200101 060000' '3600' '20200101 120000'\n DATE%RESTART = '20200101 060000' '21600' '20200101 120000'#" \ + "$d/ww3_shel.nml" +} + +if [ "${1:-}" = --self-test ]; then + t="$(mktemp -d)" + trap 'rm -rf -- "$t"' EXIT + edit_case "$t" 060000 + for want in " DOMAIN%START = '20200101 060000'" " TYPE%FIELD%LIST = '$fields'" \ + " DATE%FIELD = '20200101 060000' '3600' '20200101 120000'" \ + " DATE%RESTART = '20200101 060000' '21600' '20200101 120000'" " DOMAIN%STOP = '20200101 120000'"; do + grep -qxF "$want" "$t/ww3_shel.nml" || { echo "restart_pair: self-test: no line '$want'" >&2; exit 1; } + done + [ "$(grep -c 'DATE%' "$t/ww3_shel.nml")" = 2 ] || { echo "restart_pair: self-test: DATE lines" >&2; exit 1; } + echo "restart_pair: self-test passed" + exit 0 +fi +[ $# = 0 ] || { sed -n '2,/^set -euo/{/^set -euo/!p}' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 64; } + +bin="$bdir/ww3/bin" +sw="$(cat "$bdir/switch" 2>/dev/null || true)" +[ -x "$bin/ww3_shel" ] && [[ " $sw " == *" SHRD "* ]] && [[ " $sw " == *" KOKKOS "* ]] \ + && [[ " $sw " != *" OMPG "* ]] && [[ " $sw " != *" MPI "* ]] || { + echo "restart_pair: $bin is not a SHRD KOKKOS build; run WW3GPU_SWITCH=switch_lab_shrd bash scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos resident --build-only" >&2 + exit 3 +} + +unset_parts=(-u WW_KOKKOS_PRESET -u WW_KOKKOS_STRICT -u WW_KOKKOS_ALL) +counts=() +for f in "$root"/kokkos/src/fortran_iface/parts.d/*.txt; do + while read -r key val _; do + case "$key" in PART) unset_parts+=(-u "WW_KOKKOS_$val") ;; COUNT) counts+=("$val=1") ;; esac + done < "$f" +done +preset="$(awk '$1 == "PRESET" && $2 == "resident" { print $3 }' "$root/kokkos/src/fortran_iface/presets.txt")" +[ -n "$preset" ] || { echo "restart_pair: no resident preset in presets.txt" >&2; exit 1; } + +run() { # + local d="$runs/$1/$2" start=000000 + local -a flags=(WW_KOKKOS_ALL=0) + [ "$1" = fortran ] || flags=(WW_KOKKOS_PRESET=resident WW_KOKKOS_STRICT=1 "${counts[@]}") + mkdir -p "$d" + if [ "$2" = b ]; then + start=060000 + cp "$runs/$1/a/restart001.ww3" "$d/restart.ww3" + fi + edit_case "$d" "$start" + (cd "$d" && "$bin/ww3_grid" > ww3_grid.out 2>&1) || { tail -20 "$d/ww3_grid.out" >&2; return 1; } + (cd "$d" && env "${unset_parts[@]}" OMP_PROC_BIND="${OMP_PROC_BIND:-false}" "${flags[@]}" \ + "$bin/ww3_shel" > ww3_shel.out 2>&1) || { tail -20 "$d/ww3_shel.out" >&2; return 1; } + grep -q "End of program" "$d/ww3_shel.out" || { echo "restart_pair: ww3_shel did not finish in $d" >&2; return 1; } + if [ "$1" = kokkos ]; then + local p line + line="$(grep -h "Kokkos parts" "$d/ww3_shel.out" | head -1)" + for p in ${preset//,/ }; do + [[ "$line" == *"$p=K"* ]] || { echo "restart_pair: $d: $p is not K in '$line'" >&2; return 1; } + done + fi + echo ">> $1/$2: ww3_shel from 20200101 $start finished; $(cd "$d" && find . -maxdepth 1 -name 'restart0*.ww3' -printf '%f ' | sort)" +} + +same() { #