From 4c3d87faaca6e4bbfb6bca06e0ca7034470bd743 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 11:37:32 -0300 Subject: [PATCH 01/27] docs/compound: plan for the production single-node Kokkos runtime Goal abbb0456-6043, from the owner's brief of 2026-10-18 (session e664b977-8f49-44ff-a914-0cf9abbb0456): seven units after #189 and #190, the ww3_multi resident VA first, then strict mode, wider CPU regtests, a preset W3INIT reads, GPU build-only readiness, packaging and a stale remark. Records the brainstorm decisions B1 to B6. Tested: nix shell nixpkgs#vale -c scripts/vale.sh on the plan, 0 errors Co-Authored-By: Claude Opus 5.5 --- ...6-10-18-002-production-single-node-plan.md | 153 ++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 docs/compound/plans/2026-10-18-002-production-single-node-plan.md diff --git a/docs/compound/plans/2026-10-18-002-production-single-node-plan.md b/docs/compound/plans/2026-10-18-002-production-single-node-plan.md new file mode 100644 index 00000000..9b708c88 --- /dev/null +++ b/docs/compound/plans/2026-10-18-002-production-single-node-plan.md @@ -0,0 +1,153 @@ +--- +title: Production-ready single-node Kokkos runtime, CPU, GPU-prepared - Plan +type: feat +date: 2026-10-18 +artifact_contract: ce-unified-plan/v1 +product_contract_source: owner brief, session of 2026-10-18 +execution: code +--- + +# Production-ready single-node Kokkos runtime - Plan + +## Goal Capsule + +- **Objective:** the shared-memory (single-node) Kokkos runtime of the `h0ffmann/WW3` fork gives + the Fortran's bits on every WW3 driver it accepts, says when it falls back, is configured + without the lab script, builds for CUDA and installs as one optional dependency. +- **Means:** seven units (U1 to U7 below, the owner's numbering), fork changes as one commit + series on `ww3-gpu/kokkos-pnt` and one pin bump, WW3's own regtests under + `WW3GPU_LAB_SWITCH=ON` as the only tests (ADR-0010). +- **Authority:** AGENTS.md over this plan. ADR-0016 (this is a validation phase the owner + called, 2026-10-18), ADR-0017 (MPI stays Fortran: no MPI port), ADR-0014 (a GPU run is the + owner's, supervised; agents only `--build-only`). +- **Stop conditions:** a `test.comp` difference that a Fortran/Fortran run does not reproduce + stops the unit and is reported; no tolerance is widened, no entry is dropped or marked + expected-to-fail, no `--gpu` run. +- **Execution profile:** large loop; branch `goal/abbb0456-6043-production-single-node` from + `origin/main` (`5b48e08`); independent units as sub-agents in worktrees, their commits + cherry-picked onto the branch; one PR through `just pr`. + +--- + +## Brainstorm outcome + +Decisions taken before planning, each with the reason; the owner can reverse any of them in +review. + +- **B1. The `ww3_multi` fix is at `W3WAVE`'s exit, not in `wm*.F90`.** Gap 2's next step: + `W3WAVE` ends with a host read (`w3wavemd.F90:3151-3153` at `80644ba9`) that leaves both + copies "equal". Marking the host newer on return costs one host-to-device copy of `VA` at the + next call per grid and needs no edit of the `ww3_multi` sources, which keeps the fork's diff + inside the files it already changes. Cheaper alternatives (a sync call in each `WMIO*G`) add + hunks to three more WW3 files; rejected for the reviewer of gap 12. Measure the copy's cost on + `full`; if it is above 2 % of the resident run's wall time ⚠, record it and leave a narrower + fix for a later PR. +- **B2. Strict mode is one environment flag, `WW_KOKKOS_STRICT=1`**, read where the parts are + resolved: `KOKKOS_PART` under `WW_KOKKOS_ALL=1` and the per-call `W3SRCE` gate stop with + `EXTCDE` and a `NDSE` line naming the part and the reason, instead of falling back. Without + it, behaviour is today's, plus one line at the end of the run with the per-part count of + Fortran fallback calls. The matrix's Kokkos run sets `WW_KOKKOS_STRICT=1`; its Fortran run + does not. +- **B3. Configuration, smallest useful form: `WW_KOKKOS_PRESET=`**, read by `W3INIT` + through the shim, with `resident` the only preset (its part list moved from + `scripts/ww3-gpu.sh:190` into one table in `kokkos/src/fortran_iface/`, which the script then + reads, so the two cannot drift). A per-part variable still overrides the preset. The effective + set is the existing parts line, printed once on rank 0. No namelist: the brief says one only if + needed, and a `&KOKKOS_NML` group adds a WW3 input-file format for one user; recorded as the + next step of gap 7. +- **B4. Packaging: `WW_KOKKOS_BUILD_TESTS` (default ON here)**; OFF drops GoogleTest, NetCDF, + the tools and the Fortran language from the configure. `ww_kokkosConfig.cmake` records + `Kokkos_VERSION` and the enabled backends, and `find_dependency(Kokkos EXACT)`. + WW3's `find_package(ww_kokkos 0.9 REQUIRED)` is unchanged; the check lives in the package. +- **B5. The ADR on where `ww_kokkos` lives** is drafted Proposed with three options (external + package WW3 finds; submodule of WW3; inside `model/src`), no recommendation beyond what the + evidence of gap 12 supports, and the licence question marked ⚠. +- **B6. GPU readiness is a build and a checklist.** `--build-only` of the fork on the CUDA + backend with the resident preset's parts, warnings read and listed; the checklist is a file the + owner follows during a supervised run. + +## Requirements + +- R1. A SHRD `ww3_multi` entry under the lab switch with the resident preset and + `WW_KOKKOS_VA_CHECK=1` passes `test.comp` against the Fortran, before and after the fix is + shown (the reproduction fails first, or is shown not to fail with the reason). +- R2. With `WW_KOKKOS_STRICT=1` no run can complete on the Fortran for a part it asked on + Kokkos; the matrix's Kokkos runs use it. +- R3. The 7 resident serial entries, the OMPG entries with the part subset, and one restart + pair pass under the lab switch, each recorded in `docs/data/` and `docs/log/`. +- R4. `WW_KOKKOS_PRESET=resident` alone gives the same parts line as `--kokkos resident`. +- R5. The fork and `ww_kokkos` build on Kokkos' CUDA backend, `--build-only`; a checklist for + the owner's supervised run exists. +- R6. `ww_kokkos` configures and installs with Kokkos as its one dependency; the ADR is Proposed. +- R7. The stale remark of gap 13 is corrected. + +## Scope boundaries + +- Out: MPI (ADR-0017), `PDLIB`, any GPU run, any timing in a table (the owner's, median of + three on a quiet machine), the profile's speed proposals (each its own PR with L2 evidence). +- A change to `regtests/` or to how a comparison is invoked (U2) is reviewed by a human before + merge; the PR body says so. + +## Implementation units + +Fork work: each unit that changes the fork works on its own fork branch from `80644ba9` +(`ww3-gpu/u1-multi-va`, `ww3-gpu/u2-strict-preset`, `ww3-gpu/u6-package`), and the integrator +merges them, in that order, into `ww3-gpu/kokkos-pnt`, then bumps the pin once. Every fork change +gets a section in `kokkos/src/fortran_iface/PATCH.md`. + +### U1 + U7. `ww3_multi` and the resident `VA`; the stale remark (agent A) + +- **Files:** fork `model/src/w3wavemd.F90`, `w3kokkosmd.F90:99-104`; `PATCH.md`; possibly + `kokkos/src/fortran_iface/resident_shim.cpp`. +- **Approach:** reproduce first: the `mww3_test_0x` entries that are SHRD with + `WW3GPU_LAB_SWITCH=ON`, the resident preset, `WW_KOKKOS_VA_CHECK=1`; and a part subset + where `XYP3` refuses (`SRCE,KTP3,OUTG,IOPE,UPDV`). Then B1, then the same entries + byte-identical. Say whether this explains `mww3_test_08` (MPI): read its path, do not port MPI. +- **Done:** a `docs/log/` entry with the commands, before and after; the fork commits pushed. + +### U2 + U4. Strict mode and the preset (agent B) + +- **Files:** fork `w3kokkosmd.F90` (`KOKKOS_PART`, `W3KOKKOS_SRCE_ON`), `w3initmd.F90`; + `kokkos/src/fortran_iface/part_flags.cpp`, a preset table; `scripts/ww3-gpu.sh`; + `regtests/cmake/run_entry.cmake` (Kokkos run sets `WW_KOKKOS_STRICT=1`) and + `regtests/cmake/selftest.cmake`; `PATCH.md`; `docs/GLOSSARY.md`. +- **Done:** `just ww3-gpu-matrix-selftest`, `bash scripts/ww3-gpu.sh --self-test`, shellcheck, + `just kokkos-test serial-debug` and `openmp-release`; `just smoke-ww3-gpu` with the preset + from `WW_KOKKOS_PRESET` only, parts line identical to `--kokkos resident`. + +### U3. Wider CPU regtest coverage (agent E, after integration) + +- **Approach:** on the integrated pin, `WW3GPU_LAB_SWITCH=ON`: the resident serial selection; + the OMPG selection with the part subset; the restart-reproduction pair of a SHRD case (gap 10). + Failures go through U4a of `2026-10-18-001-cpu-resident-validation-plan.md`. +- **Done:** summaries in `docs/data/`, one `docs/log/` entry, `kokkos/ports.tsv` `regtested` + only where the record supports it (`port_score.py --write`, `--check`). + +### U5. GPU readiness, build-only (agent D; the final build again after integration) + +- **Files:** `docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md`; the build log summary in `docs/log/`. +- **Done:** `scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --build-only` with the resident preset and + `just kokkos-test cuda-release` built (no run on the device), warnings listed; the checklist + names `scripts/libm_bits.sh --device`, `just smoke-ww3-gpu gpu`, `just full-ww3-gpu gpu`, + `just ww3-gpu-matrix-cuda` on the entries U3 passed, and what to record for each (ADR-0014). + +### U6. Packaging and the ADR (agent C) + +- **Files:** `kokkos/CMakeLists.txt`, `kokkos/cmake/ww_kokkosConfig.cmake.in`, `kokkos/README.md`, + `docs/ADRs/ADR-0018-*.md` (Proposed), the ADR index. +- **Done:** an install-only configure and install in a scratch prefix with only Kokkos found; + the installed config records the Kokkos version and backends; WW3 builds against that prefix + (`just smoke-ww3-gpu` on it); `just kokkos-test serial-debug` still passes. + +## Integration and gates (the integrator) + +Merge the fork branches, push `ww3-gpu/kokkos-pnt`, bump `WW3/`, cherry-pick the units' commits, +update `docs/PRODUCTION_READINESS_202610.md` (each gap's state), then the gates of AGENTS.md for +what changed, `security-reviewer` (scripts/ changes), a report-only code review, `just pr`. + +## Owner steps (listed in the PR body) + +- U5 timing: median of three on a quiet machine, then the `bench/results/` record. +- U6: whether the lab-switch evidence lets `kokkos/ports.tsv` rows move to `regtested` "yes". +- Any GPU run of U5. +- Whether to act on the profile's speed proposals. From f8ca5973d3fb3b0b4896de6ee1d5471df6dff9f9 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 11:39:11 -0300 Subject: [PATCH 02/27] docs/compound: B1 marks the host newer only where ww3_multi writes VA The owner rejected a host-to-device copy of VA per grid per step: the fix must not cost speed. The plan now syncs after the WMIOHG and WMIOEG host writes only, so ww3_shel pays nothing. Tested: nix shell nixpkgs#vale -c scripts/vale.sh on the plan, 0 errors Co-Authored-By: Claude Opus 5.5 --- ...026-10-18-002-production-single-node-plan.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/compound/plans/2026-10-18-002-production-single-node-plan.md b/docs/compound/plans/2026-10-18-002-production-single-node-plan.md index 9b708c88..e2edff98 100644 --- a/docs/compound/plans/2026-10-18-002-production-single-node-plan.md +++ b/docs/compound/plans/2026-10-18-002-production-single-node-plan.md @@ -34,14 +34,15 @@ execution: code Decisions taken before planning, each with the reason; the owner can reverse any of them in review. -- **B1. The `ww3_multi` fix is at `W3WAVE`'s exit, not in `wm*.F90`.** Gap 2's next step: - `W3WAVE` ends with a host read (`w3wavemd.F90:3151-3153` at `80644ba9`) that leaves both - copies "equal". Marking the host newer on return costs one host-to-device copy of `VA` at the - next call per grid and needs no edit of the `ww3_multi` sources, which keeps the fork's diff - inside the files it already changes. Cheaper alternatives (a sync call in each `WMIO*G`) add - hunks to three more WW3 files; rejected for the reviewer of gap 12. Measure the copy's cost on - `full`; if it is above 2 % of the resident run's wall time ⚠, record it and leave a narrower - fix for a later PR. +- **B1. The `ww3_multi` fix marks the host newer only where `ww3_multi` writes `VA`** (the + owner, 2026-10-18: the fix must not cost speed). `W3WAVE` keeps its exit read + (`w3wavemd.F90:3151-3153` at `80644ba9`); the host writes of `WMIOHG` (`wminiomd.F90:2264`) + and `WMIOEG` (`:3261`, `:3278`, `:3326`, `:3330`) are each followed by + `W3KOKKOS_VA_HOST_WRITE` behind `W3_KOKKOS`. `ww3_shel` pays no copy, `ww3_multi` pays one + only on a step that writes. Rejected: marking the host newer at `W3WAVE`'s exit, one + host-to-device copy of `VA` per grid per step, which on a GPU is a bus transfer the resident + design exists to avoid. The cost is one more WW3 file in the fork diff (gap 12). A partial + update of the columns written is a further option, taken only if small and clearly correct. - **B2. Strict mode is one environment flag, `WW_KOKKOS_STRICT=1`**, read where the parts are resolved: `KOKKOS_PART` under `WW_KOKKOS_ALL=1` and the per-call `W3SRCE` gate stop with `EXTCDE` and a `NDSE` line naming the part and the reason, instead of falling back. Without From 8eab96d6e2ab16114e218fd1958487edcf2e6c97 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 11:41:10 -0300 Subject: [PATCH 03/27] AGENTS.md: resident spectrum hard rule, no whole-spectrum copy to sync The owner, 18 October 2026: a defect of the resident VA is never fixed by copying the whole spectrum between host and device per step, per grid or per W3WAVE call; sync what the host wrote, where it wrote it. Stated in AGENTS.md (invariant and section), CONTRIBUTING.md and the ww3-port skill. Tested: nix shell nixpkgs#vale -c scripts/vale.sh AGENTS.md CONTRIBUTING.md, 0 errors; python3 .claude/hooks/check_agent_frontmatter.py, ok Co-Authored-By: Claude Opus 5.5 --- .claude/skills/ww3-port/SKILL.md | 4 ++++ AGENTS.md | 25 +++++++++++++++++++++++++ CONTRIBUTING.md | 7 +++++++ 3 files changed, 36 insertions(+) diff --git a/.claude/skills/ww3-port/SKILL.md b/.claude/skills/ww3-port/SKILL.md index c5ae44a0..d023db00 100644 --- a/.claude/skills/ww3-port/SKILL.md +++ b/.claude/skills/ww3-port/SKILL.md @@ -46,6 +46,10 @@ compile the module with the flags in `/model/src/CMakeFiles/ww3_lib.dir/f (`w3gdatmd.F90:2278`). A shim mirrors that with one context per grid: initialised with W3INIT's `IMOD`, selected at the call by `IGRID` (`snl1_shim.cpp`, `contexts()`). One file-static context silently breaks `ww3_multi`. +- **Resident `VA`: sync what the host wrote, never the whole spectrum.** A host write between + device kernels (`ww3_multi`'s `WMIOHG`, `WMIOEG`) needs a sync at that write, of what it wrote; + marking the host newer at every `W3WAVE` return copies the whole spectrum per call and is + refused (`AGENTS.md`, "Resident spectrum"). Count with `WW_KOKKOS_VA_COUNT=1`. - **Initialisation follows the Fortran's condition.** `INSNL1` runs only when `IQTPE > 0` (`w3iogrmd.F90:1801`); with `IQTYPE <= 0` an `NL1` switch computes GQM and never reaches the port. diff --git a/AGENTS.md b/AGENTS.md index 512aa06d..3d67b069 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,6 +30,10 @@ looser. routine it replaces, inside the same WW3 executable; `test.comp` finds no file different on any regtest entry that exercises it, and a tolerance is a diagnostic, never the verdict ([Bit for bit](#bit-for-bit-with-the-fortran-hard-rule)). +- **No full-spectrum copy to keep the copies in sync**: once `VA` is resident, no code path + copies the whole spectrum between host and device per step, per grid or per `W3WAVE` call to + keep the two copies consistent; a host write is followed by a sync of what it wrote, where it + writes it ([Resident spectrum](#resident-spectrum-hard-rule)). - **Speed is the goal, parity is its condition**: the repository exists to make WW3 run faster with Kokkos and CUDA without changing the Fortran's arithmetic; a speed figure carries the machine, the command and the median of three runs, and says "regtest pending ⚠" until WW3's @@ -285,6 +289,27 @@ broken most often: keeps the spectrum resident. Never use Unified Memory to avoid thinking about transfers, and never read the CUDA kernel column as "the model will be this much faster". +## Resident spectrum (hard rule) + +The owner, 18 October 2026: a defect of the resident spectrum is never fixed by copying all of +it. A whole-spectrum copy is not the optimal solution, and the resident design exists to remove +those transfers. + +- **What it covers.** `VA` and any other array the resident preset binds on the device + (`kokkos/src/fortran_iface/resident_shim.cpp`), inside `W3WAVE`'s time loop and between + `W3WAVE` calls of `ww3_shel` and `ww3_multi`. +- **What to do instead.** Sync at the host write that made the copies differ, and only the part + written when the shim can (it already moves columns, `resident_shim.cpp:270`, `:290`). Read the + device copy to the host only where the host reads it (an output, a restart). A whole-array copy + is never chosen because it is simpler. +- **What is allowed.** The copy when `VA` is bound and the one at the end of the run; the reads + before an output or a restart write that needs the whole field; `WW_KOKKOS_VA_CHECK`, a + diagnostic, off by default; the per-point parts of phase 1, which copy per call by design and + are not resident. +- **How it is checked.** A change to the resident sync reports `WW_KOKKOS_VA_COUNT=1` (bytes host + to device and device to host) on `full` before and after; the whole-spectrum copies per step + must not grow, or the change is refused. + ## Bit for bit with the Fortran (hard rule) The owner, 15 October 2026: every ported code of WW3 is bit-to-bit equal to the behaviour of the diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 619a2c9b..b08ca89d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -140,6 +140,13 @@ A mistake in a figure is as welcome as one in the text: the gallery links an iss WW3's own regtest run with the Fortran routine and with the Kokkos routine; CTest is the runner, not a test of the port's own. A change to `regtests/` runs `just ww3-gpu-matrix-selftest` and `just ww3-gpu-matrix-list`. Anything tried that did not become a number in a table, failures included, goes in a dated entry in `docs/log/`. +- The resident spectrum is never kept consistent by copying all of it (`AGENTS.md`, "Resident + spectrum"). When the host writes `VA` between two device kernels (`ww3_multi`'s boundary and + nesting updates, for one), sync what was written, where it is written; read the device copy to + the host only before an output or a restart that needs it. A whole-spectrum copy per step, per + grid or per `W3WAVE` call is refused, even when it is the simpler fix: on a GPU it is a bus + transfer every step. A change to the sync reports `WW_KOKKOS_VA_COUNT=1` on `full` before and + after. - `codemeta.json` is generated from `CITATION.cff` and `.zenodo.json` (`just codemeta`); never edit it by hand. - Run `just kokkos-test serial-debug` after touching `kokkos/`, and `bash -n` plus From 5899594994bb199c1c81e5e2a568c016e5b5f89e Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 11:46:35 -0300 Subject: [PATCH 04/27] docs/compound: plan units U8 and U9, no whole-spectrum copy in readers The owner, 2026-10-18: the resident spectrum is the design, with no copies. U8 turns the output readers' whole-array reads into column reads, U9 ports partitioning onto the resident VA, U1 moves the ww3_multi exchange to the device and drops W3WAVE's exit read. Tested: nix shell nixpkgs#vale -c scripts/vale.sh on the plan, 0 errors Co-Authored-By: Claude Opus 5.5 --- ...6-10-18-002-production-single-node-plan.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/compound/plans/2026-10-18-002-production-single-node-plan.md b/docs/compound/plans/2026-10-18-002-production-single-node-plan.md index e2edff98..243450c1 100644 --- a/docs/compound/plans/2026-10-18-002-production-single-node-plan.md +++ b/docs/compound/plans/2026-10-18-002-production-single-node-plan.md @@ -140,6 +140,34 @@ gets a section in `kokkos/src/fortran_iface/PATCH.md`. the installed config records the Kokkos version and backends; WW3 builds against that prefix (`just smoke-ww3-gpu` on it); `just kokkos-test serial-debug` still passes. +### U8. No whole-spectrum copy in the output readers (agent F) + +The owner, 2026-10-18: the resident spectrum is the design, with no copies; where code still +needs one, port what removes it. `W3WAVE` stays Fortran and calls the Kokkos parts, so every +host statement that reads `VA` needs a current host copy. U8 replaces the whole-array reads +before `W3IOTR`, `W3IOBC 'WRITE'` and the OASIS send (`w3wavemd.F90:2934-2975` at `80644ba9`) +with column reads of the points each one reads, as `W3KOKKOS_IOPE_READ` does. The restart +reads stay (the rule allows them). Every run uses `WW_KOKKOS_VA_POISON=1`: `WW_KOKKOS_VA_CHECK` +catches an unsynced host write, not a stale host read. + +### U9. Partitioning on the device (agent G) + +`W3CPRT` reads every point's spectrum at each output time with partitioning on +(`w3wavemd.F90:2691`). U9 translates it per point onto the resident `VA` and copies only the +partition fields back, or stops before code with the obstacle for the owner. + +### U1, extended. The `ww3_multi` exchange on the device (agent A) + +The gather and scatter of the exchanged points of `WMIOBG`, `WMIOHG` and `WMIOEG` run as a +kernel on the resident `VA`, so only those points' spectra cross. The exit read of `W3WAVE` +(`w3wavemd.F90:3151-3153`) is removed. + +### Next, not in this PR: zero copies on a host backend + +On Kokkos' OpenMP backend the device copy of `VA` is a second host array. An unmanaged View over +the Fortran `VA` would remove the copies there entirely ⚠ (the measured copy cost on `full` is +0.002 s, `docs/log/2026-10-18-resident-profile.md`, so this is about memory, not time). + ## Integration and gates (the integrator) Merge the fork branches, push `ww3-gpu/kokkos-pnt`, bump `WW3/`, cherry-pick the units' commits, From af194e743da2afaf12ca156f37a9dbd572b5b34f Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 11:52:01 -0300 Subject: [PATCH 05/27] kokkos: install-only ww_kokkos package pinned to its Kokkos; ADR-0018 drafted Unit U6 of the production single-node plan (goal abbb0456-6043). WW_KOKKOS_BUILD_TESTS (default ON, today's tree) set OFF configures ww_kokkos with C++ only and Kokkos as the one package searched: no GoogleTest, NetCDF, tools, intro programs or Fortran module. The kernels and their PRIVATE contraction flags are unchanged; the 25 objects equal openmp-release's. The installed ww_kokkosConfig.cmake (from cmake/ww_kokkosConfig.cmake.in) records Kokkos_VERSION, Kokkos_DEVICES and Kokkos_ARCH, asks for find_dependency(Kokkos EXACT), which Kokkos' basic version file honours, and refuses other backends or architecture. No fork change: WW3's find_package(ww_kokkos 0.9 REQUIRED) is untouched. scripts/ww3-gpu.sh gains WW3GPU_KOKKOS_PREFIX, so --ww3-gpu builds WW3 against an installed package elsewhere (a changed prefix reconfigures WW3). ADR-0018 (Proposed) sets out where ww_kokkos could live for upstream: an external package, a submodule of WW3, or inside model/src, with no option chosen; its licence count corrects gap 12, which called the C++ MIT (249 of 347 files are LGPL-3.0-or-later, 22 MIT, 76 without a line, legal reading pending). Gaps 5 and 12 of PRODUCTION_READINESS_202610.md, kokkos/README.md and the glossary updated. Tested: install-only configure traced with cmake --trace-expand in the #ww3 shell: only find_package(Kokkos) and Kokkos' own find_package(OpenMP) ran, Kokkos_DIR the only package dir in the cache; build + install into kokkos/build/install-only.prefix: ok; a consumer find_package(ww_kokkos 0.9) finds Kokkos 5.2.2, stops on a recorded 5.2.1 (EXACT) and on recorded backends CUDA;SERIAL; WW3GPU_KOKKOS_PREFIX= scripts/ww3-gpu.sh --ww3-gpu smoke --cpu: Kokkos/Fortran byte-identical, every binary file byte-identical to --ww3, nccmp-tol PASS with 0 difference; just kokkos-test serial-debug and openmp-release: 11/11 each; scripts/ww3-gpu.sh --self-test ok; shellcheck and bash -n ok; vale 0 errors on the 5 Markdown files; python3 -m unittest discover tests: 88 OK; port_score --check current. Co-Authored-By: Claude Opus 5.5 --- ...0018-where-ww-kokkos-lives-for-upstream.md | 131 ++++++++++++++++++ docs/ADRs/README.md | 1 + docs/GLOSSARY.md | 2 + docs/PRODUCTION_READINESS_202610.md | 47 ++++--- kokkos/CMakeLists.txt | 52 ++++--- kokkos/README.md | 46 +++++- kokkos/cmake/ww_kokkosConfig.cmake.in | 41 ++++++ scripts/ww3-gpu.sh | 18 ++- 8 files changed, 301 insertions(+), 37 deletions(-) create mode 100644 docs/ADRs/ADR-0018-where-ww-kokkos-lives-for-upstream.md create mode 100644 kokkos/cmake/ww_kokkosConfig.cmake.in diff --git a/docs/ADRs/ADR-0018-where-ww-kokkos-lives-for-upstream.md b/docs/ADRs/ADR-0018-where-ww-kokkos-lives-for-upstream.md new file mode 100644 index 00000000..1116e880 --- /dev/null +++ b/docs/ADRs/ADR-0018-where-ww-kokkos-lives-for-upstream.md @@ -0,0 +1,131 @@ +# ADR-0018: Where `ww_kokkos` lives if WW3 upstream takes the Kokkos runtime + +| | | +|---|---| +| **Status** | Proposed | +| **Date** | 2026-10-18 | +| **Deciders** | Hoffmann | +| **Written by** | an agent, from gap 12 of [`PRODUCTION_READINESS_202610.md`](../PRODUCTION_READINESS_202610.md) and unit U6 of the [production single-node plan](../compound/plans/2026-10-18-002-production-single-node-plan.md); no option is chosen | +| **Scope** | Where the C++ library `ww_kokkos` and its C ABI would live for a WW3 upstream that takes the `KOKKOS` switch: an external package WW3 finds, a submodule of WW3, or sources inside `model/src`. Not whether upstream wants it, not the Fortran half (it is in WW3 in every option), not the single-node scope ([ADR-0017](ADR-0017-kokkos-port-is-single-node-mpi-stays-fortran.md)) | +| **Related** | [ADR-0010](ADR-0010-kokkos-validated-by-ww3-regtests-only.md); [ADR-0016](ADR-0016-speed-first-regtests-postponed-one-person-loop.md); [ADR-0017](ADR-0017-kokkos-port-is-single-node-mpi-stays-fortran.md); [`kokkos/README.md`](../../kokkos/README.md), "Install-only build"; [`kokkos/src/fortran_iface/PATCH.md`](../../kokkos/src/fortran_iface/PATCH.md) | + +## Context + +The port has two halves. The Fortran half is on the fork (`h0ffmann/WW3`, pinned at +`WW3@80644ba9`); the C++ half is `kokkos/` in this repository. `diff -rq WW3-noaa/model WW3/model` +(v, 2026-10-18) lists what the fork adds to NOAA's WW3 at the commit it branched from: + +| Fork change | Size (v, 2026-10-18) | +|---|---| +| Six new modules, `model/src/w3kokkos*md.F90` | 4 082 lines | +| `w3initmd.F90`, `w3srcemd.F90`, `w3wavemd.F90` | 114, 456 and 249 changed lines (`diff \| grep -c '^[<>]'`) | +| `model/src/CMakeLists.txt` | 64 changed lines, the `WW_KOKKOS` block | +| `model/src/cmake/switches.json` | the `kokkos` category | + +The C++ that the `KOKKOS` switch links is larger (v, `cat kokkos/src/*/*.{cpp,hpp} | wc -l`, +2026-10-18): 48 571 lines in `kokkos/src/ww_kokkos/` (the kernels) and 9 647 in +`kokkos/src/fortran_iface/` (the C ABI and shims), 347 files. + +How WW3 reaches it today: `find_package(ww_kokkos 0.9 REQUIRED)` and `target_link_libraries(ww3_lib +PUBLIC ww_kokkos::ww_kokkos)` (v, `model/src/CMakeLists.txt:43-44` at `WW3@80644ba9`). Since this +change the package can be configured with Kokkos as its one dependency +(`-DWW_KOKKOS_BUILD_TESTS=OFF`), and its config asks for the exact Kokkos version, backends and +architecture it was built with (v, `kokkos/README.md`, "Install-only build", configure traced +2026-10-18). Its own version contract is `SameMajorVersion` at 0.9.0 with entry points only added +(v, `kokkos/CMakeLists.txt`); nothing checks the "only added" ⚠. + +Licences, as the files state them (v, `grep SPDX-License-Identifier`, 2026-10-18): + +- WW3 is under the GNU Lesser General Public License v3 (v, `WW3/LICENSE.md`, first lines). The + fork's new modules carry `LGPL-3.0-or-later` (v, `model/src/w3kokkosmd.F90:33`). +- Of the 347 C++ files under `kokkos/src/`, 249 carry `LGPL-3.0-or-later`, all in `ww_kokkos/`: + the kernels translated from WW3's Fortran. 22 carry `MIT`: the shims, the timers, the fixture + reader and `real.hpp`. 76 carry no line (71 in `ww_kokkos/`, 5 in `fortran_iface/`); the + repository is MIT (v, `LICENSE`), but whether a file translated from WW3 without the line is + MIT or a derived work under the LGPL is not settled here ⚠. +- Gap 12 of `PRODUCTION_READINESS_202610.md` read "the C++ it calls is MIT"; the count above + shows most of the C++ is LGPL-3.0-or-later, as `README.md`, "licensing", already says of the + translated kernels. + +⚠ No legal reading was made for any statement on what a licence permits in this ADR. Each option +names the licence question it raises; a person qualified to answer it has to. + +## Decision + +None yet. This ADR records the three options with the evidence above, for the owner and, if it +comes to that, for WW3's maintainers to choose from. What the evidence supports today: + +- Option (a) is what exists and was built end to end on 2026-10-18: install-only configure, + install, WW3 built against the prefix, `smoke` byte-identical pair to pair + (`PRODUCTION_READINESS_202610.md`, gap 5). +- Options (b) and (c) were not built. Their costs below are read from the files, not measured ⚠. + +## Alternatives considered + +### (a) An external package WW3 finds (today's layout) + +WW3 keeps `find_package(ww_kokkos REQUIRED)` behind `-DWW_KOKKOS=ON`; `ww_kokkos` is +released and installed on its own, like NetCDF or ESMF. + +- The upstream diff is the Fortran half only: 4 082 new and 819 changed lines of Fortran, 64 lines + of CMake, one `switches.json` category. +- The C++ keeps its own repository, licence files, CMake minimum (3.25; WW3 asks for 3.19, v, + `WW3/CMakeLists.txt:4`) and C++20 requirement, and its own release cadence. +- Two version contracts must move together: WW3's `find_package` minimum and the ABI version of + `ww_kokkos`. Today a WW3 commit that needs a new entry point raises the minimum by hand. +- An operational centre installs one more library, built against the same Kokkos as its WW3; the + `EXACT` check makes a mismatch a configure error instead of a link-time surprise. +- WW3's exported targets name `ww_kokkos::ww_kokkos` (`ww3_lib` links it `PUBLIC`), so a consumer + of an installed WW3 must find the package too ⚠ (no consumer built). +- Licence: the LGPL-derived kernels are distributed from a repository whose top licence is MIT, + each file with its own line. Whether that is enough is the question marked ⚠ above. + +### (b) A submodule of WW3 + +WW3 adds `ww_kokkos` as a git submodule at a pinned commit and builds it with `add_subdirectory` +or `ExternalProject_Add` when `KOKKOS` is set. + +- WW3 has no submodule at the pin (v, no `.gitmodules` in `WW3/`); this would be its first, with + `git clone --recurse-submodules` in every build recipe that uses the switch. +- The gitlink pins the C++ exactly, so the two-version problem of (a) becomes one commit; the + `find_package` minimum is no longer needed. +- `add_subdirectory` brings this tree's CMake 3.25 minimum and C++20 into WW3's configure, and the + install-only option is what keeps GoogleTest and NetCDF out of it. `ExternalProject_Add` keeps + the trees apart at the cost of a second configure step ⚠ (neither built). +- The C++ stays in its own repository with its own licence files, as in (a). +- Who maintains the submodule's repository, and who may move the gitlink, is a governance + question WW3's maintainers answer, not this repository. + +### (c) Sources inside `model/src` + +The C++ is copied into WW3, for instance `model/src/kokkos/`, and built by WW3's CMake. + +- The upstream diff grows by about 58 000 lines of C++ in 347 files, about 14 times the 4 082 new lines of the Fortran half. +- One repository, one version, one licence file: the LGPL-derived kernels would sit in an LGPL + project. The 22 MIT files would sit in it too ⚠ (whether WW3 accepts MIT files, or relicences + them, is for its maintainers and a legal reading). +- WW3's CMake would need C++ in its languages, the contraction flags (`-ffp-contract=off`, + `--fmad=false`) on the C++ sources only, and the CUDA `libm` table generator; today those live + in `kokkos/src/ww_kokkos/CMakeLists.txt`. +- The lab's tools and intro programs stay here; the port's validation is WW3's regtests in every + option (ADR-0010), so no test suite moves. +- Every change to the C++ becomes a WW3 pull request; this repository would then track WW3 for + the C++ as it does for the Fortran. + +## Consequences + +Whatever is chosen: + +- The Fortran half is upstream's in every option; its 819 changed lines are what a reviewer reads, + and with `KOKKOS` off they compile to the original (`PRODUCTION_READINESS_202610.md`, gap 12). +- The licence of the files without an `SPDX` line has to be settled before any option ships ⚠. +- The `EXACT` Kokkos check stays: in (b) and (c) the Kokkos is still an external package, and a + build against another Kokkos still links code compiled for one Kokkos with another's runtime. +- Gap 12 stays a Blocker for target (b) of `PRODUCTION_READINESS_202610.md` until an option is + accepted. Gap 5's configure dependencies are closed by the install-only option; its other + items (a static archive only, C++20, CMake 3.25) remain. + +## Revisit when + +The owner opens a conversation with WW3's maintainers about taking the runtime, or a legal reading +of the licence question is available, or WW3 adds submodules or C++ for another reason. diff --git a/docs/ADRs/README.md b/docs/ADRs/README.md index 4ff3f9f0..d2e360fa 100644 --- a/docs/ADRs/README.md +++ b/docs/ADRs/README.md @@ -35,3 +35,4 @@ fills the `/adrs/` page of the book's website on every build. | [ADR-0015](ADR-0015-triton-kernels-allowed-under-three-conditions.md) | Triton GPU kernels are allowed as an exception to "no Python in lab code" when better fitted than the Kokkos kernel, integrated with the Fortran/C++ code behind a run-time flag, and regtested by WW3's regtests | Accepted | 2026-10-17 | | [ADR-0016](ADR-0016-speed-first-regtests-postponed-one-person-loop.md) | The goal is WW3's run time on Kokkos and CUDA; WW3's regtests move to a validation phase the owner calls, speed figures before it are marked "regtest pending", and engineering work follows Compound Engineering's loop sized for one person (supersedes ADR-0011) | Accepted | 2026-10-17 | | [ADR-0017](ADR-0017-kokkos-port-is-single-node-mpi-stays-fortran.md) | The Kokkos port is single node (`SHRD`, with OpenMP, on the CPU and one GPU); an MPI build of WW3 keeps the Fortran path, and no MPI path is translated, wired or regtested while it stands | Accepted | 2026-10-18 | +| [ADR-0018](ADR-0018-where-ww-kokkos-lives-for-upstream.md) | Where `ww_kokkos` lives if WW3 upstream takes the Kokkos runtime: an external package WW3 finds (today's layout, built end to end), a submodule of WW3, or sources inside `model/src`; no option chosen, the licence of the translated C++ marked ⚠ | Proposed | 2026-10-18 | diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index feee1a45..be1cc1eb 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -721,6 +721,7 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | `WW_KOKKOS_DEVICE_ID` | environment variable selecting the GPU (default 0), read instead of `MPI_Comm_rank()` because phase 1 links no MPI `(v)` [kokkos/README.md](../kokkos/README.md) | Set per rank by the launcher in an MPI job. | [kokkos/src/fortran_iface/ww_kokkos_c.hpp](../kokkos/src/fortran_iface/ww_kokkos_c.hpp) | | `WW_DETERMINISTIC` | CMake option: "force serial reductions for bit-reproducible validation builds" `(v)` [kokkos/CMakeLists.txt](../kokkos/CMakeLists.txt) | On in `serial-debug`; changes nothing for the DIA, which has no reduction. | [course/11-kokkos-and-modern-cpp.md](../course/11-kokkos-and-modern-cpp.md) | | `WW_ENABLE_FORTRAN` / `WW_WW3_BUILD_DIR` | build the Fortran module, reference and drivers (default ON) / a configured WW3 build (`libww3.a` + `mod/`) that enables the `ww3_lib` cross-check `(v)` [kokkos/README.md](../kokkos/README.md) | The second is what `just l1-crosscheck` sets. | [kokkos/README.md](../kokkos/README.md) | +| `WW_KOKKOS_BUILD_TESTS` / `WW3GPU_KOKKOS_PREFIX` | CMake option of `kokkos/` (default ON): OFF is the install-only configure, `ww_kokkos` and its package config with Kokkos the one dependency searched, no GoogleTest, NetCDF, tools or Fortran / the variable that makes `scripts/ww3-gpu.sh --ww3-gpu` build WW3 against a `ww_kokkos` package installed elsewhere `(v)` [kokkos/CMakeLists.txt](../kokkos/CMakeLists.txt), [scripts/ww3-gpu.sh](../scripts/ww3-gpu.sh) | The installed `ww_kokkosConfig.cmake` asks for the Kokkos version it was built with `EXACT` and refuses other backends or architecture. | [kokkos/README.md](../kokkos/README.md) | | `WW_FIXTURE_DIR` / `WW_TEST_ENVIRONMENT` | compile definition pointing tests at `tests/fixtures/` / the CTest environment `OMP_PROC_BIND=false;OMP_NUM_THREADS=2` `(v)` [kokkos/CMakeLists.txt](../kokkos/CMakeLists.txt) | Test plumbing. | [kokkos/tests/CMakeLists.txt](../kokkos/tests/CMakeLists.txt) | | `serial-debug` / `openmp-release` / `cuda-release` | the three presets: Debug + sanitizers + `WW_DETERMINISTIC` / Release `-O3 -march=x86-64-v3` / Release with `nvcc_wrapper`, arch 89 `(v)` [kokkos/CMakePresets.json](../kokkos/CMakePresets.json) | The name says build type, not backend. | [course/11-kokkos-and-modern-cpp.md](../course/11-kokkos-and-modern-cpp.md) | | `cuda-h100-release` / `WW_KOKKOS_ARCH` | the H100 preset: `cuda-release` with `CMAKE_CUDA_ARCHITECTURES=90` and `WW_KOKKOS_ARCH=HOPPER90`; `WW_KOKKOS_ARCH`, a CMake cache variable of `kokkos/`, names a Kokkos architecture the found Kokkos must have, checked with Kokkos' `kokkos_check` at configure time `(v)` [kokkos/CMakePresets.json](../kokkos/CMakePresets.json), [kokkos/CMakeLists.txt](../kokkos/CMakeLists.txt) | Not the default: the `#cuda` shell's Kokkos is ADA89, so the preset stops at configure there (v, 2026-10-17); `scripts/ww3-gpu.sh --gpu` builds `cuda-release`. | [kokkos/README.md](../kokkos/README.md) | @@ -2226,6 +2227,7 @@ Every term above, alphabetically, with the section it lives in. - `ww_kokkos`: [This repository's own names](#this-repositorys-own-names) - `ww_kokkos_c.hpp`: [This repository's own names](#this-repositorys-own-names) - `ww_kokkos_c.hpp` / `snl1_shim.cpp` / `w3kokkosmd.F90`: [This repository's own names](#this-repositorys-own-names) +- `WW_KOKKOS_BUILD_TESTS` / `WW3GPU_KOKKOS_PREFIX`: [This repository's own names](#this-repositorys-own-names) - `WW_KOKKOS_DEVICE_ID`: [This repository's own names](#this-repositorys-own-names) - `WW_KOKKOS_TIMERS` / `--timers` / `ww::timers::Region`: [This repository's own names](#this-repositorys-own-names) - `WW_KOKKOS_ERR_RUNTIME`: [This repository's own names](#this-repositorys-own-names) diff --git a/docs/PRODUCTION_READINESS_202610.md b/docs/PRODUCTION_READINESS_202610.md index dea60643..b2834f88 100644 --- a/docs/PRODUCTION_READINESS_202610.md +++ b/docs/PRODUCTION_READINESS_202610.md @@ -41,14 +41,14 @@ regtest matrix yet"). 10 of the 310 rows of `kokkos/ports.tsv` are regtested and | 2 | `ww3_multi` writes `VA` on the host between `W3WAVE` calls with no sync | Blocker | Blocker | Mark the host newer when `W3WAVE` returns; run `mww3_test_0x` with `WW_KOKKOS_VA_CHECK=1` | | 3 | MPI: four of the six resident parts are `SHRD` only, the preset stops on `DIST MPI`, the communicator is ignored, the device id is not per rank | Blocker | Major | A resident preset without `XYP3`, `OUTG`, `IOPE`, `UPDV`; one run on `switch_lab_mpi` | | 4 | Bit parity measured with gfortran and g++ only; the operational compiler and the GPU are not measured | Blocker | Major | Static argument for the Intel build's math calls; one smoke with it | -| 5 | Build and packaging: a separate static library whose install needs GoogleTest and NetCDF, an unversioned Kokkos, C++20 and CMake 3.25 | Major | Blocker | An install-only configure with Kokkos as the one dependency, its version recorded in `ww_kokkosConfig.cmake` | +| 5 | Build and packaging: a separate static library, C++20 and CMake 3.25; the install-only configure and the exact Kokkos check are done (U6) | Major | Major | A shared-library build and a recipe for the module systems of an operational centre | | 6 | `WW_KOKKOS_ALL=1` and `W3KOKKOS_SRCE_ON` fall back to the Fortran without a word | Major | Minor | A strict flag that turns every fallback into a stop, and a per-run fallback count | | 7 | Configuration is 61 environment variables, no namelist; the preset lives in a lab script | Major | Major | A namelist group read by `W3INIT`, the environment as override, the effective set printed once | | 8 | Observability: log lines on every rank, counters at exit to `stderr`, no provenance in the output files | Major | Minor | Gate the lines by `NAPLOG`; one NetCDF global attribute | | 9 | OpenMP inside MPI: the per-point parts are serialised by `!$OMP CRITICAL`; the thread budget per rank is not set by the shim | Major | Minor | Measure the hybrid entries of `ww3_ufs1.1` before claiming a speed for them | | 10 | Restart with the resident spectrum: covered by the sync calls, never regtested | Major | Minor | The restart-reproduction pair of `ww3_ufs1.1` (`work_c` against `work_d`) under the lab switch | | 11 | `PDLIB` and unstructured grids: translated, not wired, gated out | Minor ⚠ | Major | None for GFS-Wave v16; a wiring unit for `PDLIB` when the owner asks | -| 12 | The fork patch: 4 082 lines of new Fortran, 819 changed lines in three WW3 files, the C++ in another repository under another licence | Minor | Blocker | An ADR on where `ww_kokkos` lives for upstream | +| 12 | The fork patch: 4 082 lines of new Fortran, 819 changed lines in three WW3 files, 58 218 lines of C++ in another repository, most of it LGPL-3.0-or-later, some MIT, 76 files with no licence line | Minor | Blocker | ADR-0018 (Proposed) names the three places `ww_kokkos` could live; the owner chooses, after a legal reading ⚠ | | 13 | A stale contract remark: "one spectral grid per process" | Minor | Minor | Edit the remark in the fork | The sections below give the evidence of each gap. The ranked top five closes the page. @@ -214,9 +214,9 @@ What a WW3 build needs today: |---|---|---| | A separately built, installed `ww_kokkos` | `find_package(ww_kokkos 0.9 REQUIRED)` (v, `WW3 model/src/CMakeLists.txt:43`) | Upstream has no place to fetch it from | | A static archive only | `add_library(ww_kokkos STATIC)` (v, `kokkos/CMakeLists.txt:52`) | No shared build for a coupled system that links WW3 as a library | -| GoogleTest, NetCDF C and Fortran, a Fortran compiler to configure the library | `find_package(GTest REQUIRED)`, `pkg_check_modules(NETCDF ...)`, `NETCDFF` (v, `kokkos/CMakeLists.txt:12`, `:33-37`) | The install needs the test and tool dependencies | +| GoogleTest, NetCDF C and Fortran, a Fortran compiler to configure the library | `find_package(GTest REQUIRED)`, `pkg_check_modules(NETCDF ...)`, `NETCDFF` under `WW_KOKKOS_BUILD_TESTS` (default ON), and C and Fortran enabled only then (v, `kokkos/CMakeLists.txt`) | Closed (U6): `-DWW_KOKKOS_BUILD_TESTS=OFF` configures with C++ only and searches Kokkos only (see below) | | C++20, CMake 3.25 | v, `kokkos/CMakeLists.txt:11`, `:14` | WW3 asks for CMake 3.19 (v, `WW3/CMakeLists.txt:4`) | -| Kokkos, unversioned | `find_package(Kokkos REQUIRED)` and `find_dependency(Kokkos)` (v, `kokkos/CMakeLists.txt:26`, `:83-85`) | WW3's configure may find a Kokkos other than the one the archive was built with; only `WW_KOKKOS_ARCH`, optional, checks the architecture (v, `:27-32`) | +| Kokkos, pinned by the package | `kokkos/cmake/ww_kokkosConfig.cmake.in`: `find_dependency(Kokkos EXACT)`, then `Kokkos_DEVICES` and `Kokkos_ARCH` compared with the values recorded at build time (v) | Closed (U6): another Kokkos is a configure error of WW3's `find_package(ww_kokkos 0.9 REQUIRED)`, unchanged in the fork | | A version contract | `SameMajorVersion` at 0.9.0, entry points only added (v, `kokkos/CMakeLists.txt:86-90`) | Sound for a 0.x series, but no symbol check enforces "only added" ⚠ | | Contraction off in the kernels | `-ffp-contract=off`, `--fmad=false`, `PRIVATE` to `ww_kokkos` (v, `kokkos/src/ww_kokkos/CMakeLists.txt:297-303`) | Correct where it is; WW3's side does not check it was built this way | | `ww3_lib` links `ww_kokkos::ww_kokkos` `PUBLIC` | v, `WW3 model/src/CMakeLists.txt:44` | WW3's exported targets (v, `:285-299`) then name a package a consumer must also find ⚠ (no consumer built) | @@ -226,9 +226,19 @@ What a WW3 build needs today: - **(a) Major.** NCEP can build it by hand; it cannot pin it as one more library in its stack without the items above. - **(b) Blocker.** An upstream optional dependency must configure with its own dependencies only. -- **Next step.** An option (`WW_BUILD_TESTS`, default on here) that skips GoogleTest, NetCDF, - the tools and the Fortran language for an install-only configure; write `Kokkos_VERSION` and - the enabled backends into `ww_kokkosConfig.cmake` and check them in WW3's `find_package`. +- **State (U6, 2026-10-18).** `WW_KOKKOS_BUILD_TESTS=OFF` is the install-only configure + (`kokkos/README.md`, "Install-only build"). Traced with `cmake --trace-expand` in the `#ww3` + shell, it ran `find_package(Kokkos REQUIRED)` and, inside Kokkos' config, + `find_package(OpenMP REQUIRED COMPONENTS CXX)`, and no other `find_package` or + `pkg_check_modules`; `Kokkos_DIR` is the only package directory in its cache (v). The + installed config records Kokkos 5.2.2, `OPENMP;SERIAL`, no architecture (v). WW3 built + against that prefix with `WW3GPU_KOKKOS_PREFIX= scripts/ww3-gpu.sh --ww3-gpu smoke + --cpu` (v, 2026-10-18): WW3's cache names that prefix as `ww_kokkos_DIR`; the Kokkos and the + Fortran runs of the same executable wrote byte-identical files (`out_grd.ww3`, `out_pnt.ww3`, + `restart.ww3`, the NetCDF), and against `--ww3` every binary model file was byte-identical and + `nccmp-tol` found 0 difference in every NetCDF field. No fork change was needed. +- **Next step.** A shared build of `ww_kokkos` for a coupled system, and the CMake and C++ + minimums against what an operational centre's module system provides ⚠. ## 6. Failure behaviour @@ -342,16 +352,22 @@ batched `W3SRCE` is compiled only `!defined(W3_PDLIB)` (v, `w3wavemd.F90:452`, ` `diff -r` of the fork against `WW3-noaa/` (v, today): six new modules, 4 082 lines (`w3kokkos*md.F90`), and 114, 456 and 249 changed lines in `w3initmd.F90`, `w3srcemd.F90`, `w3wavemd.F90`, plus a `kokkos` category in `switches.json` and the CMake block of section 5. The -Fortran is LGPL-3.0-or-later (v, `w3kokkosmd.F90:33`); the C++ it calls is MIT (v, the 12 -`SPDX` lines of `kokkos/src/fortran_iface/*.cpp`) and lives here, not in WW3. The `#ifdef +Fortran is LGPL-3.0-or-later (v, `w3kokkosmd.F90:33`). The C++ it calls lives here, not in WW3: +58 218 lines in 347 files under `kokkos/src/`, of which 249 files carry `LGPL-3.0-or-later` (the +kernels translated from WW3, all in `ww_kokkos/`), 22 carry `MIT` (the shims, timers, fixture +reader) and 76 carry no line (v, `grep SPDX-License-Identifier`, 2026-10-18). An earlier version +of this section called the C++ MIT; the count corrects it. The `#ifdef W3_KOKKOS` hunks inside `W3WAVE`'s time loop are what a reviewer reads; with the `KOKKOS` switch off they compile to the original. - **(a) Minor.** Operations can run the fork. - **(b) Blocker.** Upstream needs to know where the C++ comes from, who maintains it, and how its version is pinned. -- **Next step.** An ADR: `ww_kokkos` as an external package WW3 finds, as a submodule of WW3, or - inside `model/src`; the licences permit each ⚠ (no legal reading). +- **State (U6, 2026-10-18).** [ADR-0018](ADRs/ADR-0018-where-ww-kokkos-lives-for-upstream.md), + Proposed, sets out the three options (an external package WW3 finds, a submodule of WW3, + sources inside `model/src`) with the size and licence of each; only the first was built. +- **Next step.** The owner's choice among them, after a legal reading of the licence of the + translated C++ and of the files with no licence line ⚠. ## 13. A stale remark in the contract @@ -381,11 +397,10 @@ Ranked by what blocks both targets first, then by how much of GFS-Wave's run it (`model/src/CMakeLists.txt:113-115`) calls another math library; no GPU run compared the device libm. Next: the static argument for the Intel build's transcendental calls, then one smoke. -5. **The library cannot be installed as a plain dependency.** Its configure needs GoogleTest, - NetCDF and Fortran, C++20 and CMake 3.25, and records no Kokkos version - (`kokkos/CMakeLists.txt:11-37`, `:83-85`); the C++ lives outside WW3 under another licence. - Next: an install-only option, the Kokkos version in the package config, and an ADR on where - `ww_kokkos` lives. +5. **Where the library lives for upstream is open.** Since U6 it installs with Kokkos as its one + dependency and pins that Kokkos exactly (`kokkos/README.md`, "Install-only build"); it still + needs C++20 and CMake 3.25, and its C++ lives outside WW3, mostly under WW3's licence. + Next: the owner's choice among the options of ADR-0018, after a legal reading. ## How the checks were made diff --git a/kokkos/CMakeLists.txt b/kokkos/CMakeLists.txt index 69894043..c1906757 100644 --- a/kokkos/CMakeLists.txt +++ b/kokkos/CMakeLists.txt @@ -9,13 +9,26 @@ # # SPDX-License-Identifier: MIT cmake_minimum_required(VERSION 3.25) -project(ww3_gpu_kokkos VERSION 0.9.0 LANGUAGES C CXX Fortran) +# OFF is the install-only configure a WW3 build links against: ww_kokkos and its package +# config, with Kokkos as the one dependency searched (no GoogleTest, no NetCDF, no tools, no +# intro programs, no C or Fortran compiler). ON, the default, is the whole lab tree. +option(WW_KOKKOS_BUILD_TESTS "Build the intro programs, the GoogleTest suites and the NetCDF tools" ON) +if(WW_KOKKOS_BUILD_TESTS) + set(ww_languages C CXX Fortran) +else() + set(ww_languages CXX) +endif() +project(ww3_gpu_kokkos VERSION 0.9.0 LANGUAGES ${ww_languages}) set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED ON) set(CMAKE_CXX_EXTENSIONS OFF) option(WW_ENABLE_FORTRAN "Build the Fortran interface module (W3KOKKOSMD from the WW3/ submodule)" ON) +if(NOT WW_KOKKOS_BUILD_TESTS) + # WW3 compiles W3KOKKOSMD in its own build; without Fortran enabled there is nothing to build it with. + set(WW_ENABLE_FORTRAN OFF CACHE BOOL "" FORCE) +endif() option(WW_DETERMINISTIC "Force serial reductions for bit-reproducible validation builds" OFF) include(cmake/CompilerWarnings.cmake) @@ -30,17 +43,18 @@ set(WW_KOKKOS_ARCH "" CACHE STRING "Kokkos architecture the found Kokkos must ha if(WW_KOKKOS_ARCH) kokkos_check(ARCH ${WW_KOKKOS_ARCH}) endif() -find_package(GTest REQUIRED) -find_package(PkgConfig REQUIRED) -pkg_check_modules(NETCDF REQUIRED IMPORTED_TARGET netcdf) -if(WW_ENABLE_FORTRAN) - pkg_check_modules(NETCDFF REQUIRED IMPORTED_TARGET netcdf-fortran) +if(WW_KOKKOS_BUILD_TESTS) + find_package(GTest REQUIRED) + find_package(PkgConfig REQUIRED) + pkg_check_modules(NETCDF REQUIRED IMPORTED_TARGET netcdf) + if(WW_ENABLE_FORTRAN) + pkg_check_modules(NETCDFF REQUIRED IMPORTED_TARGET netcdf-fortran) + endif() + # enable_testing() must run before any directory that calls add_test(), so it + # comes before intro/ and tests/ rather than after them. + enable_testing() endif() -# enable_testing() must run before any directory that calls add_test(), so it -# comes before intro/ and tests/ rather than after them. -enable_testing() - # Environment every CTest case runs with. Kokkos' OpenMP backend prints a warning # unless OMP_PROC_BIND is set explicitly, and a fixed, small thread count keeps the # suite from fighting CI for cores and keeps its output reproducible. @@ -63,11 +77,13 @@ add_subdirectory(src/ww_kokkos) # Not guarded by WW_ENABLE_FORTRAN: the C ABI is C++ and belongs in ww_kokkos # either way. The directory guards the Fortran module itself. add_subdirectory(src/fortran_iface) -add_subdirectory(intro) -add_subdirectory(tools/nccmp-tol) -add_subdirectory(tools/bench_case) -add_subdirectory(tools/fetch_analyse) -add_subdirectory(tests) +if(WW_KOKKOS_BUILD_TESTS) + add_subdirectory(intro) + add_subdirectory(tools/nccmp-tol) + add_subdirectory(tools/bench_case) + add_subdirectory(tools/fetch_analyse) + add_subdirectory(tests) +endif() # The package WAVEWATCH III finds with `find_package(ww_kokkos)` when it is built with the # KOKKOS switch (-DWW_KOKKOS=ON, the h0ffmann/WW3 branch ww3-gpu/kokkos-st4): the static @@ -80,9 +96,9 @@ install(DIRECTORY src/ DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} FILES_MATCHING PATTERN "*.hpp") install(EXPORT ww_kokkosTargets NAMESPACE ww_kokkos:: DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ww_kokkos) -file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/ww_kokkosConfig.cmake - "include(CMakeFindDependencyMacro)\nfind_dependency(Kokkos)\n" - "include(\"\${CMAKE_CURRENT_LIST_DIR}/ww_kokkosTargets.cmake\")\n") +# The archive holds Kokkos code compiled against one Kokkos: its version, backends and +# architecture are written into the config, which refuses any other (cmake/ww_kokkosConfig.cmake.in). +configure_file(cmake/ww_kokkosConfig.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/ww_kokkosConfig.cmake @ONLY) # Entry points are only added, never removed or changed, so an install satisfies any # request of the same major version up to its own (0.9.0 accepts find_package(ww_kokkos # 0.3), 0.4, 0.5, 0.6, 0.7 and 0.8). Removing or changing an entry point needs a major bump (PATCH.md, "The fork today"). diff --git a/kokkos/README.md b/kokkos/README.md index b44da58a..d4a1e651 100644 --- a/kokkos/README.md +++ b/kokkos/README.md @@ -154,7 +154,7 @@ Trans. Parallel Distrib. Syst.* 33(4), 805–817, 2022, ``` CMakeLists.txt top level: options, find_package, the ww_kokkos target CMakePresets.json serial-debug, openmp-release, cuda-release, cuda-h100-release -cmake/ CompilerWarnings.cmake -- the one warning policy +cmake/ CompilerWarnings.cmake (the one warning policy), ww_kokkosConfig.cmake.in src/ww_kokkos/ real.hpp (float32 + helpers), spectrum_fixtures.hpp (JONSWAP, cos^2), snl1_{config,tables,dia} (the DIA port), fixture_io (fixture reader) src/fortran_iface/ ww_kokkos_c.hpp + snl1_shim.cpp (the C ABI, built into ww_kokkos), @@ -227,6 +227,50 @@ Two things the names do *not* promise: |---|---|---| | `WW_ENABLE_FORTRAN` | `ON` | build the Fortran interface module `W3KOKKOSMD` and `intro_06` | | `WW_DETERMINISTIC` | `OFF` | pin reductions to a fixed summation order for bit-reproducible validation runs | +| `WW_KOKKOS_BUILD_TESTS` | `ON` | build the intro programs, the GoogleTest suites and the NetCDF tools; `OFF` is the install-only configure below | + +### Install-only build: `ww_kokkos` with Kokkos as its one dependency + +WW3 built with the `KOKKOS` switch links the installed `ww_kokkos` package +(`find_package(ww_kokkos 0.9 REQUIRED)`, v, `WW3/model/src/CMakeLists.txt:43` at `WW3@80644ba9`). +`-DWW_KOKKOS_BUILD_TESTS=OFF` configures only that package: the project enables C++ alone, and +GoogleTest, NetCDF, the intro programs, the tools and the Fortran module are left out +(`WW_ENABLE_FORTRAN` is forced off: WW3 compiles `W3KOKKOSMD` in its own build). The kernels and +their flags do not change: `-ffp-contract=off` and, on CUDA, `--fmad=false` stay `PRIVATE` to +`ww_kokkos` (`src/ww_kokkos/CMakeLists.txt`). + +```bash +# inside nix develop ./picasso/labs/pratico#ww3, from the repository root +cmake -S kokkos -B kokkos/build/install-only -G Ninja -DCMAKE_BUILD_TYPE=Release \ + "-DCMAKE_CXX_FLAGS=-O3 -march=x86-64-v3" -DWW_KOKKOS_BUILD_TESTS=OFF +cmake --build kokkos/build/install-only -j8 +cmake --install kokkos/build/install-only --prefix +# then WW3 against it (scripts/ww3-gpu.sh still builds openmp-release, for nccmp-tol): +WW3GPU_KOKKOS_PREFIX= scripts/ww3-gpu.sh --ww3-gpu smoke --cpu +``` + +The installed `ww_kokkosConfig.cmake` (from `cmake/ww_kokkosConfig.cmake.in`) records the +`Kokkos_VERSION`, `Kokkos_DEVICES` and `Kokkos_ARCH` of the Kokkos the archive was compiled +against, asks for `find_dependency(Kokkos EXACT)`, and sets `ww_kokkos_FOUND` false with +a message when the Kokkos found has other backends or another architecture. `EXACT` works because +Kokkos' `KokkosConfigVersion.cmake` is CMake's basic version file, which sets +`PACKAGE_VERSION_EXACT` when the requested version string equals its own (v, read in the +`kokkos-openmp-5.2.2` store path of the `#ww3` shell, 2026-10-18). + +What the configure searched, read from `cmake --trace-expand` (v, 2026-10-18): `find_package(Kokkos +REQUIRED)` in this tree, and `find_package(OpenMP REQUIRED COMPONENTS CXX)` from Kokkos' own +`find_dependency`; no other `find_package` or `pkg_check_modules` call ran, and `Kokkos_DIR` is the +only package directory in the cache. The `#ww3` shell's Kokkos is 5.2.2 with `OPENMP;SERIAL` and no +architecture. The 25 object files of `ww_kokkos` are byte-identical to the `openmp-release` +preset's (`diff -r -q` of the two `CMakeFiles/ww_kokkos.dir`, v, 2026-10-18): the option changes +what is configured around the library, not the library. + +The refusals, checked with a one-line consumer project (`find_package(ww_kokkos 0.9 REQUIRED)`) +on 2026-10-18 (v): with the installed config, it finds Kokkos 5.2.2; with `-DKokkos_DIR` pointed +at a Kokkos 5.2.0 it still takes 5.2.2 and ignores 5.2.0; with the recorded version edited to +5.2.1 it stops with "Could not find a configuration file for package "Kokkos" that exactly matches +requested version"; with the recorded backends edited to `CUDA;SERIAL` it stops with the +config's own message. ## Intro programs diff --git a/kokkos/cmake/ww_kokkosConfig.cmake.in b/kokkos/cmake/ww_kokkosConfig.cmake.in new file mode 100644 index 00000000..e5eb8f08 --- /dev/null +++ b/kokkos/cmake/ww_kokkosConfig.cmake.in @@ -0,0 +1,41 @@ +# ww_kokkosConfig.cmake, generated from kokkos/cmake/ww_kokkosConfig.cmake.in. +# libww_kokkos.a holds Kokkos templates instantiated against one Kokkos install. A consumer +# that found another one (another version, another backend set, another GPU architecture) +# would link code compiled for one Kokkos with the runtime of another, so the config asks +# for the same version EXACT and checks the backends and architecture after it. +# +# SPDX-License-Identifier: MIT +include(CMakeFindDependencyMacro) + +set(ww_kokkos_Kokkos_VERSION "@Kokkos_VERSION@") +set(ww_kokkos_Kokkos_DEVICES "@Kokkos_DEVICES@") +set(ww_kokkos_Kokkos_ARCH "@Kokkos_ARCH@") + +# KokkosConfigVersion.cmake sets PACKAGE_VERSION_EXACT when the requested version string +# equals its own, so EXACT accepts this Kokkos and no other. +find_dependency(Kokkos ${ww_kokkos_Kokkos_VERSION} EXACT) + +set(_ww_kokkos_have_devices ${Kokkos_DEVICES}) +set(_ww_kokkos_want_devices ${ww_kokkos_Kokkos_DEVICES}) +set(_ww_kokkos_have_arch ${Kokkos_ARCH}) +set(_ww_kokkos_want_arch ${ww_kokkos_Kokkos_ARCH}) +list(SORT _ww_kokkos_have_devices) +list(SORT _ww_kokkos_want_devices) +list(SORT _ww_kokkos_have_arch) +list(SORT _ww_kokkos_want_arch) +if(NOT "${_ww_kokkos_have_devices}" STREQUAL "${_ww_kokkos_want_devices}" + OR NOT "${_ww_kokkos_have_arch}" STREQUAL "${_ww_kokkos_want_arch}") + set(ww_kokkos_FOUND FALSE) + string(CONCAT ww_kokkos_NOT_FOUND_MESSAGE + "ww_kokkos was built against Kokkos ${ww_kokkos_Kokkos_VERSION} with backends " + "'${ww_kokkos_Kokkos_DEVICES}' and architecture '${ww_kokkos_Kokkos_ARCH}'; the Kokkos " + "found (${Kokkos_DIR}) has backends '${Kokkos_DEVICES}' and architecture '${Kokkos_ARCH}'. " + "Point CMAKE_PREFIX_PATH or Kokkos_ROOT at the Kokkos ww_kokkos was built with.") + return() +endif() +unset(_ww_kokkos_have_devices) +unset(_ww_kokkos_want_devices) +unset(_ww_kokkos_have_arch) +unset(_ww_kokkos_want_arch) + +include("${CMAKE_CURRENT_LIST_DIR}/ww_kokkosTargets.cmake") diff --git a/scripts/ww3-gpu.sh b/scripts/ww3-gpu.sh index dbbb9548..60509c13 100755 --- a/scripts/ww3-gpu.sh +++ b/scripts/ww3-gpu.sh @@ -85,6 +85,10 @@ # itself (picasso/labs/pratico: #ww3, or #cuda for --gpu). # Builds go to build//, runs to build/runs/-[-BACKEND]/ ($WW3GPU_RUNS). # +# WW3GPU_KOKKOS_PREFIX= (--ww3-gpu) builds WW3 against the ww_kokkos package installed in +# , for instance by an install-only configure (kokkos/README.md), in place of the one this +# script installs from kokkos/build/-release; that build still runs, for nccmp-tol. +# # WW3GPU_PREBUILT= runs programs built beforehand and builds nothing, with no Nix: the # Docker image (docker/Dockerfile) sets it. holds ww3/bin and ww3-gpu-openmp/bin (WW3's # programs), bin/nccmp-tol, cases/ (ww3_ts1, bench_small, ww4/), and ww4/ (ww4_standalone @@ -490,14 +494,19 @@ else preset="${backend}-release" kbuild="$root/kokkos/build/$preset" kprefix="$kbuild/install" + if [ -n "${WW3GPU_KOKKOS_PREFIX:-}" ]; then + kprefix="$(cd "$WW3GPU_KOKKOS_PREFIX" 2>/dev/null && pwd)" \ + || { echo "ww3-gpu: WW3GPU_KOKKOS_PREFIX=$WW3GPU_KOKKOS_PREFIX is not a directory" >&2; exit 64; } + fi if [ "$mode" = ww3-gpu ]; then fresh_for_toolchain "$kbuild" [ -f "$kbuild/CMakeCache.txt" ] || cmake -S "$root/kokkos" --preset "$preset" >/dev/null # nvcc writes its errors to stdout: keep them in a log and show them, or the build fails silently. cmake --build "$kbuild" -j"$jobs" > "$kbuild/build.log" 2>&1 \ || { grep -E -A8 "FAILED|error" "$kbuild/build.log" | head -60; echo "ww3-gpu: the Kokkos build ($preset) failed; $kbuild/build.log" >&2; exit 1; } - cmake --install "$kbuild" --prefix "$kprefix" >/dev/null - echo ">> ww_kokkos ($preset) installed in $kprefix" + cmake --install "$kbuild" --prefix "$kbuild/install" >/dev/null + echo ">> ww_kokkos ($preset) installed in $kbuild/install" + [ -z "${WW3GPU_KOKKOS_PREFIX:-}" ] || echo ">> WW3 links the ww_kokkos package in $kprefix (WW3GPU_KOKKOS_PREFIX)" fi # --- WW3 ------------------------------------------------------------------------------------ @@ -516,6 +525,11 @@ else printf '%s\n' "$switch_now" > "$switch_file" [ -d "$bdir/ww3" ] && [[ "$bdir/ww3" == "$root/build/"* ]] && rm -rf -- "$bdir/ww3" fi + # Another ww_kokkos package needs a fresh configure: WW3's cache holds the ww_kokkos_DIR it found. + if [ "$mode" = ww3-gpu ] && [ "$(cat "$bdir/kprefix" 2>/dev/null || echo "$kbuild/install")" != "$kprefix" ]; then + [ -d "$bdir/ww3" ] && [[ "$bdir/ww3" == "$root/build/"* ]] && rm -rf -- "$bdir/ww3" + fi + [ "$mode" != ww3-gpu ] || printf '%s\n' "$kprefix" > "$bdir/kprefix" if [ ! -f "$bdir/ww3/CMakeCache.txt" ]; then args=(-S "$ww3" -B "$bdir/ww3" -DSWITCH="$switch_file" -DCMAKE_BUILD_TYPE=Release) if [ "$mode" = ww3-gpu ]; then From d3504805cf07bf0cd94388506c92689c249ffec0 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 11:52:51 -0300 Subject: [PATCH 06/27] docs: GPU supervised-run checklist and the CUDA build-only record (U5) Goal abbb0456-6043, unit U5 of docs/compound/plans/2026-10-18-002-production-single-node-plan.md. ww_kokkos (cuda-release) and the fork for the resident preset were built on Kokkos' CUDA backend, build only, with no device run; the 51 nvcc warnings are listed in a dated log entry, none in a wired part (48 host-only lambdas in the unwired DO_OUTPUT_EXCHANGES, the GCC novector pragma nvcc does not know and still forwards, one dead store). The checklist gives the owner the order of the supervised run (libm_bits --device, smoke, full, the CUDA matrix on the entries U3 passed, filled by the integrator) and what ADR-0014 asks each step to record. It notes that just ww3-gpu-matrix-cuda forwards neither WW3GPU_KOKKOS nor WW3GPU_LAB_SWITCH, left for review. Tested: nix develop picasso/labs/pratico#cuda --command cmake -S kokkos --preset cuda-release and cmake --build kokkos/build/cuda-release -j8: pass; JOBS=8 scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident --build-only: pass (21 programs); nix shell nixpkgs#vale -c scripts/vale.sh on the 7 changed files: 0 errors, 0 warnings; python3 scripts/figures.py check: pass; relative links of the changed pages resolve. No GPU run. Co-Authored-By: Claude Opus 5.5 --- docs/GLOSSARY.md | 4 + docs/GPU_LIBM_BITS_202610.md | 4 +- docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md | 176 ++++++++++++++++++++ docs/PRODUCTION_READINESS_202610.md | 4 + docs/log/2026-10-18-cuda-build-only.md | 91 ++++++++++ docs/log/README.md | 1 + kokkos/README.md | 5 + 7 files changed, 284 insertions(+), 1 deletion(-) create mode 100644 docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md create mode 100644 docs/log/2026-10-18-cuda-build-only.md diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index be1cc1eb..e71fecb0 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -520,6 +520,8 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | NVMe | ⚠ Non-Volatile Memory Express (not expanded in the repo) | Fast local storage named among today's lab-machine features. | [pubs/proposal/pt/05-justification.md](../pubs/proposal/pt/05-justification.md) | | FP32 / FP64 / float32 | single / double precision; WW3's default `REAL` is 4 bytes, `ww::Real` is `float` `(v)` [kokkos/src/ww_kokkos/real.hpp](../kokkos/src/ww_kokkos/real.hpp) | The port keeps `float` state and accumulates in `double`. | [course/11-kokkos-and-modern-cpp.md](../course/11-kokkos-and-modern-cpp.md) | | FMA / `-ffp-contract` / `--fmad` | fused multiply-add and the GCC / nvcc flags that control contraction `(v)` [kokkos/README.md](../kokkos/README.md) | One rounding where the Fortran does two; `ww_kokkos` builds with `-ffp-contract=off` and `--fmad=false` for bit parity. | [course/12-porting-a-kernel-w3snl1.md](../course/12-porting-a-kernel-w3snl1.md), [kokkos/PORT_STATUS.md](../kokkos/PORT_STATUS.md) | +| `--expt-relaxed-constexpr` / nvcc `#20013-D` | nvcc's flag that lets device code call a `constexpr` host function, and the warning it prints without it `(v)` the CUDA build log, [docs/log/2026-10-18-cuda-build-only.md](log/2026-10-18-cuda-build-only.md) | Not set in the lab's CUDA Kokkos; the 48 warnings of 2026-10-18 are the host-only lambdas of the translated, unwired `DO_OUTPUT_EXCHANGES`. | [docs/log/2026-10-18-cuda-build-only.md](log/2026-10-18-cuda-build-only.md) | +| `#pragma GCC novector` | a GCC 14+ pragma that stops the vectoriser on the next loop `(v)` `g++ -fopt-info-vec-all` ("dont-vectorize loop") | Keeps the Fortran's operand order where g++'s vector body swaps an add; nvcc warns `#1675-D` and still passes it to the host g++. | [kokkos/src/ww_kokkos/uost_uost_sourceterm_compute.hpp](../kokkos/src/ww_kokkos/uost_uost_sourceterm_compute.hpp) | | ULP | ⚠ unit in the last place (not expanded in the repo) | The granularity of float rounding; the budget of a bit-parity test. | [kokkos/src/ww_kokkos/snl1_config.hpp](../kokkos/src/ww_kokkos/snl1_config.hpp) | | `powi()` / `pow11()` | integer-power helpers that reproduce gfortran's multiplication chain for `x**n` `(v)` [kokkos/README.md](../kokkos/README.md) | Why the port is bit-for-bit rather than "close": `x**n` is not `std::pow`. | [course/12-porting-a-kernel-w3snl1.md](../course/12-porting-a-kernel-w3snl1.md) | | Kokkos | ⚠ the Sandia-led C++ performance-portability library (not expanded; 5.2.0 pinned `(v)` [kokkos/README.md](../kokkos/README.md)) | One kernel source runs on Serial, OpenMP or CUDA backends chosen at build time; the abstraction WW4 proposes too. | [course/11-kokkos-and-modern-cpp.md](../course/11-kokkos-and-modern-cpp.md) | @@ -1054,6 +1056,7 @@ Every term above, alphabetically, with the section it lives in. - EXPAND: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - EXPND2: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - EXTCDE: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) +- `--expt-relaxed-constexpr` / nvcc `#20013-D`: [HPC and software](#hpc-and-software) - `extern "C"`: [HPC and software](#hpc-and-software) - EXTIOF / EXTOPN: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - F, E: [Wave physics and the spectrum](#wave-physics-and-the-spectrum) @@ -1127,6 +1130,7 @@ Every term above, alphabetically, with the section it lives in. - F_ZHAO_V1: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - GAULEG / INSNLGQM: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - GAULEG: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) +- `#pragma GCC novector`: [HPC and software](#hpc-and-software) - gdb stack sampling: [HPC and software](#hpc-and-software) - GEBCO: [HPC and software](#hpc-and-software) - GEFS-Wave: [Models, projects and institutions](#models-projects-and-institutions) diff --git a/docs/GPU_LIBM_BITS_202610.md b/docs/GPU_LIBM_BITS_202610.md index 3602e2a6..f4a64183 100644 --- a/docs/GPU_LIBM_BITS_202610.md +++ b/docs/GPU_LIBM_BITS_202610.md @@ -280,7 +280,9 @@ reference, not of the port. It is not needed now. ## What is not shown -- No device run: that the device's bits equal the host's is ⚠ until `--device` runs. +- No device run: that the device's bits equal the host's is ⚠ until `--device` runs. The + owner's supervised run does it first, and records it here: + [`GPU_SUPERVISED_RUN_CHECKLIST_202610.md`](GPU_SUPERVISED_RUN_CHECKLIST_202610.md), step 1. - No kernel of `kokkos/src` calls these headers yet; [WFIP-0006](WFIPs/WFIP-0006-device-libm-bits.md) is the design, and until it is done ADR-0014's verdict by tolerance stands. - `mvec_powf` was checked against one CPU's libmvec; a second vendor's `rcpps` was not seen ⚠. diff --git a/docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md b/docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md new file mode 100644 index 00000000..432f04f4 --- /dev/null +++ b/docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md @@ -0,0 +1,176 @@ +# Checklist for the owner's supervised GPU run, October 2026 + +The steps, in order, for the first runs of the Kokkos port on Kokkos' CUDA backend, and what each +one must leave on record. Only the owner runs them, at a terminal, watching the run (AGENTS.md, +"Running on the GPU"); every command below asks for a typed "yes" and exits 7 without one. An +agent never runs them, never sets `WW3GPU_IN_SHELL` or `WW3GPU_GPU_CONFIRMED`, and never pipes +the answer. + +A GPU run is judged by `nccmp-tol` with `kokkos/tools/nccmp-tol/tolerances.txt`, against the +Fortran of the same executable and against `--ww3`, and a tolerance is never widened to make it +pass ([ADR-0014](ADRs/ADR-0014-gpu-runs-judged-by-tolerance.md)). Why the bits can differ, site +by site, is [`GPU_LIBM_BITS_202610.md`](GPU_LIBM_BITS_202610.md). This page is unit U5 of +[`compound/plans/2026-10-18-002-production-single-node-plan.md`](compound/plans/2026-10-18-002-production-single-node-plan.md) +and the GPU half of gap 4 of [`PRODUCTION_READINESS_202610.md`](PRODUCTION_READINESS_202610.md). + +Nothing on this page has run on a GPU. The build of step 0 was compiled on 2026-10-18 without a +run ([`log/2026-10-18-cuda-build-only.md`](log/2026-10-18-cuda-build-only.md)); every statement +about a device result is ⚠ until the step that answers it has run. + +## 0. Before the first command + +**The build.** Both commands compile only; neither touches the device. + +```bash +git submodule update --init WW3 WW3-noaa && just submodule-init +nix develop ./picasso/labs/pratico#cuda --command cmake -S kokkos --preset cuda-release +nix develop ./picasso/labs/pratico#cuda --command cmake --build kokkos/build/cuda-release -j8 +scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident --build-only +``` + +Do not use `just kokkos-cuda-test` for this step: its last line runs `ctest` with CUDA as the +default execution space, which is a device run. On 2026-10-18 both builds passed with 51 nvcc +warnings, none in a part the fork calls, and 8 gfortran warnings in WW3's own files (v, the log +entry above). A new warning in the build of the run's commit is read before step 1. + +**Write down, once, in the log entry of the run:** + +| What | Command | +|---|---| +| ww3-gpu commit, and whether the tree is clean | `git rev-parse --short HEAD`, `git status --porcelain` | +| `WW3/` pin (fork), `WW3-noaa/` pin, `picasso/` pin | `git submodule status WW3 WW3-noaa picasso` | +| CPU model and hardware threads | `grep -m1 'model name' /proc/cpuinfo`, `nproc --all` | +| GPU model and driver version | `nvidia-smi --query-gpu=name,driver_version --format=csv,noheader` | +| the CUDA version the driver supports | the header of `nvidia-smi` | +| nvcc, Kokkos, the host compilers | `nvcc --version`, the `kokkos-cuda-` store path in `kokkos/build/cuda-release/CMakeCache.txt`, `gfortran --version`, and the "CXX compiler identification" line of the configure (nvcc_wrapper's host g++, 14.4.0 on 2026-10-18, while the `#ww3` shell's is 16.2.0) | +| whether the GPU drives a display | `nvidia-smi --query-gpu=display_active --format=csv,noheader` | +| the machine's load before and after | `cut -d' ' -f1 /proc/loadavg` | + +A GPU that drives a display has a watchdog that ends a long kernel with +`cudaErrorLaunchTimeout`; `WW_KOKKOS_SRCE_BATCH_MIB` (default 128 MiB on a device) keeps a +`W3SRCE` launch short for that reason, and its limit was never measured +(`kokkos/src/fortran_iface/PATCH.md`, the `WW_KOKKOS_SRCE_BATCH_MIB` paragraph). Record the value +used when it is set. A launch timeout is a stop, not a reason to change the case. + +**Stop rules for every step.** A step that fails stops the checklist. The failure is recorded as +it is, with its output, in the log entry. A tolerance, a tolerance file, a comparison script and +the way it is called stay as they are; a change to any of them is its own pull request reviewed +by a person (AGENTS.md, "Never cheat a regtest tool"). + +## 1. The device math: `scripts/libm_bits.sh --device` + +```bash +scripts/libm_bits.sh --device +``` + +It runs `device_check` on GPU 0: every transcribed function (`expf`, `logf`, `sincosf`, `powf`, +the libmvec copies, `tanhf`, `log10f`, `atan2f`, `coshf`, `sinhf`, `exp`, `tanh`), every input +of each against the host's glibc, with CUDA's own functions printed beside them. Exit +0: no difference; 1: a difference; 2: a CUDA error; 7: not confirmed. + +Record: + +- the full output, per function: inputs checked, inputs different, the first differing input; +- the exit code; +- in [`GPU_LIBM_BITS_202610.md`](GPU_LIBM_BITS_202610.md), a dated "Results on the device" + section beside "Results on the host", and the "What is not shown" line about the device updated + only by what the run showed. + +A difference here does not stop steps 2 to 4 (ADR-0014 already judges them by tolerance), but it +says which `⚠` of the libm page stays. + +## 2. The smoke case on the GPU + +```bash +just smoke-ww3-gpu gpu +scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident +``` + +The first is `--kokkos gpu` (`SRCE` alone, ADR-0013), the default of `--gpu`; the second is the +resident preset (`SRCE,XYP3,KTP3,OUTG,IOPE,UPDV`). Both run `ww3_ts1` twice in one executable +(the Fortran, then Kokkos), then `--ww3`, pure NOAA WW3 from `WW3-noaa/` on the CPU. + +Record, for each of the two commands: + +- the `nccmp-tol` table against the Fortran of the same executable, and the one against `--ww3`, + every field, listed or not: `max|d|`, rms, largest relative difference, and the verdict of the + judged ones (`hs` on the smoke: "1 judged, 25 unlisted", ADR-0014, "Consequences"); +- which model files `cmp` found different, against each reference (on CUDA the script reports + them and goes on); +- the parts line the Kokkos run printed, which shows the parts the preset asked for ran on + Kokkos; +- the exit code. + +The tables go into `docs/data/ww3gpu_cuda_smoke_.md`, the command and the verdict +into the log entry. A failed `nccmp-tol` stops the checklist. + +## 3. The full case on the GPU + +```bash +just full-ww3-gpu gpu +scripts/ww3-gpu.sh --ww3-gpu full --gpu --kokkos resident +``` + +`bench_small`: 120 x 80, 9 520 sea points, NK = NTH = 24, 12 hours. Record the same items as in +step 2, in `docs/data/ww3gpu_cuda_full_.md`; the full case judges `hs` of five +listed fields (ADR-0014). The wall times these runs print are one run each: a check that the +pieces work, never a number for a table. + +**The timing, only after step 3 passed on the same commit:** + +```bash +just perf-ww3-gpu full 3 resident +``` + +Three runs of each, alternated, every output compared with the first run's, and the medians; the +script writes `perf.tsv` only after its own `nccmp-tol` comparisons passed. On a quiet machine +(load average recorded before and after). Then `python3 scripts/results.py record ` +writes the `bench/results/` record, whose `regtest` says "GPU: judged by tolerance, not bits" +(ADR-0014, point 4), and `just results check`. A figure quoted anywhere names the machine, the +command and the median of three, and says "regtest pending ⚠" until the matrix of step 4 has +passed on its commit (ADR-0016). + +## 4. WW3's regtest matrix on the GPU: `just ww3-gpu-matrix-cuda` + +On the entries the CPU regtests of this pull request passed: + +> **Entries: filled by U3 of this PR.** The integrator writes here the CTest names +> (`NNN::`) of the entries that passed on the CPU with the resident preset and the +> lab switch, and the record they came from in `docs/data/`. + +```bash +just ww3-gpu-matrix-cuda 2 serial -R '' +``` + +Two things about the recipe as it is on 2026-10-18 (v, `justfile`, the `ww3-gpu-matrix-cuda` +recipe, and `regtests/CMakeLists.txt`): + +- It forwards `WW3GPU_ROUTINES` only. `WW3GPU_KOKKOS=resident` and `WW3GPU_LAB_SWITCH=ON`, which + the CPU entries of U3 used, are not passed to its CMake configure, so the recipe runs each + entry's own `matrix.base` switch with the routines of `WW3GPU_ROUTINES` (`SNL1` by default), + not the resident preset. Running the U3 entries as U3 ran them needs those two variables + forwarded, a change to how a regtest driver is called and so reviewed by a person first. + Until then, record which configuration ran. +- `WW3GPU_STRICT` defaults to ON, so an entry fails when `test.comp` finds a file different. On + CUDA that verdict is expected and is not the ADR-0014 verdict: read the `nccmp_tol` and + `judged` columns of `summary.tsv` for that, and record both. + +Record: + +- `build/ww3-gpu-matrix-cuda-[...]/summary.tsv`, copied to + `docs/data/ww3gpu_matrix_cuda__.tsv` with its header lines (date, + commits, machine, command, switch, Kokkos flags); +- per entry: `test_comp`, `nccmp_tol`, `judged`, `kokkos_spectra` (an entry with 0 did not + exercise the port and is reported as such), `ww3_self`; +- the log entry: which entries passed `nccmp-tol`, which differ and in which field. + +## After the run + +- One dated entry in `docs/log/` with every record above, indexed in + [`log/README.md`](log/README.md), and linked from the pull request that commits it. +- [`PRODUCTION_READINESS_202610.md`](PRODUCTION_READINESS_202610.md), gap 4: the "GPU" paragraph + updated by what was measured. +- ADR-0014's "Revisit when" names the first supervised run: the owner reads its `nccmp-tol` + tables, every field, and decides whether the ADR stands. +- `kokkos/ports.tsv` changes only where the record supports it, then + `python3 scripts/port_score.py --write` and `--check`. diff --git a/docs/PRODUCTION_READINESS_202610.md b/docs/PRODUCTION_READINESS_202610.md index b2834f88..c0be3219 100644 --- a/docs/PRODUCTION_READINESS_202610.md +++ b/docs/PRODUCTION_READINESS_202610.md @@ -195,6 +195,10 @@ WFIP-0006) with the host, and every flag's parity note says "GPU ⚠ never run" (ADR-0014). WCOSS2 has no GPUs (the `WCOSS2` row of [`GLOSSARY.md`](GLOSSARY.md)), so for (a) today the GPU is not the path; GFS-Wave's spectral grid, 50 x 36, also needs the DIA's level-1 scratch on the RTX 4090 (v, `kokkos/PORT_STATUS.md:63`), a speed matter, not a parity one. +The CUDA build of the fork for the resident preset compiles, build only, with no warning in a +wired part (v, [`log/2026-10-18-cuda-build-only.md`](log/2026-10-18-cuda-build-only.md)); the +owner's supervised run follows +[`GPU_SUPERVISED_RUN_CHECKLIST_202610.md`](GPU_SUPERVISED_RUN_CHECKLIST_202610.md). **Real kind.** The interface is `REAL(C_FLOAT)`; "a WW3 built with an 8-byte default REAL needs a different ABI" (v, `w3kokkosmd.F90:30-31`). diff --git a/docs/log/2026-10-18-cuda-build-only.md b/docs/log/2026-10-18-cuda-build-only.md new file mode 100644 index 00000000..57527d09 --- /dev/null +++ b/docs/log/2026-10-18-cuda-build-only.md @@ -0,0 +1,91 @@ +# The CUDA build of `ww_kokkos` and of the fork for the resident preset, build only: 51 nvcc warnings, none in a wired part + +2026-10-18, an agent for M. Hoffmann; unit U5 of +[`2026-10-18-002-production-single-node-plan.md`](../compound/plans/2026-10-18-002-production-single-node-plan.md), +[ADR-0014](../ADRs/ADR-0014-gpu-runs-judged-by-tolerance.md). No device run. + +## Question + +Do `ww_kokkos` and the `h0ffmann/WW3` fork build on Kokkos' CUDA backend for the resident preset +(`--kokkos resident`, `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV`), and does any warning of nvcc, the Kokkos +wrapper or gfortran touch the bits: contraction, a conversion, a math call, a host function +called from device code? The answer is the precondition of the owner's supervised run, +[`GPU_SUPERVISED_RUN_CHECKLIST_202610.md`](../GPU_SUPERVISED_RUN_CHECKLIST_202610.md). + +## What was done + +On branch `goal/abbb0456-6043-u5-gpu-ready` at `4f87fec` (from +`goal/abbb0456-6043-production-single-node`), `WW3/` at `80644ba9` (`ww3-gpu/kokkos-pnt`), +`WW3-noaa/` at `03adda91`, `picasso/` at `be76132`. Machine: Intel Core i9-14900, 32 threads, +shared with three other agents (at most 8 build jobs); NVIDIA GeForce RTX 4090, driver 615.71.09 +(`nvidia-smi --query-gpu=name,driver_version`, a driver query, no kernel launched). + +Toolchain, all from the `#cuda` shell of `picasso/labs/pratico` (v, the configure log): nvcc +12.9.86 (`cuda_12.9.r12.9/compiler.36037853_0`), Kokkos 5.2.2 (`kokkos-cuda-5.2.2`, devices +`OPENMP;SERIAL;CUDA`, `-arch=sm_89`), nvcc_wrapper's host compiler g++ 14.4.0, gfortran and gcc +16.2.0, CMake 4.4.3, glibc 2.44, `WW_KOKKOS_LIBM=ON`. + +```bash +nix develop ./picasso/labs/pratico#cuda --command cmake -S kokkos --preset cuda-release +nix develop ./picasso/labs/pratico#cuda --command cmake --build kokkos/build/cuda-release -j8 -- -v +JOBS=8 scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident --build-only +``` + +`just kokkos-cuda-test` was not used: its last line runs `ctest` with CUDA as the default +execution space, a device run. `just kokkos-test cuda-release` was not used either: it builds in +the `#ww3` shell and also runs `ctest`. The `--build-only` command rebuilt `cuda-release` (the +script's toolchain stamp differs from a hand configure) and built WW3 with the switch +`switch_lab_ompg` plus `KOKKOS`, `-DCMAKE_CXX_COMPILER=nvcc_wrapper`, Fortran `-O3`. The logs +are in the worktree's `build/u5/` and `build/ww3-gpu-cuda/{configure,build}.log` (git-ignored). + +## Result + +Both builds passed (v): the 73 steps of `ww_kokkos` with its tools and intro programs, and the 21 +WW3 programs (`ww3_shel`, `ww3_multi`, ...) in `build/ww3-gpu-cuda/ww3/bin`. The second build of +`cuda-release` printed the same 53 lines matching `warning` as the first (v, `grep -c`). + +**Contraction and math flags** (v, the `-v` build log and `nvcc_wrapper -dryrun` on a test file): +every one of the 25 translation units of `libww_kokkos.a` is compiled with `-ffp-contract=off +--fmad=false`; nvcc_wrapper passes `-ffp-contract=off` to the host g++, ptxas gets `--fmad +false`, and cicc `-ftz=0 -prec_div=1 -prec_sqrt=1 -fmad=0`. No `--use_fast_math` anywhere. + +**nvcc warnings, 51 in the Kokkos code, grouped:** + +| File:line | Warning | Count | Can it touch the bits | +|---|---|---|---| +| `kokkos/src/ww_kokkos/pdlib_do_output_exchanges.hpp`:151, 153, 156, 159, 217-220, 247, 248, 265-267, 271, 272, 291 | `#20013-D` calling a `constexpr __host__` function (`operator()`) from a `__host__ __device__` function (`walk_columns`, or the packing lambda of `do_output_exchanges`); "the experimental flag `--expt-relaxed-constexpr` can be used to allow this" | 48 (16 sites, 3 each) | The lambdas `one`, `e3`, `swell`, `cols` of `walk_columns` are host-only to nvcc, and the packing kernel calls them on the device. It compiled; whether the device code calls them correctly is ⚠ (never run). No floating-point arithmetic in this routine (`kokkos/ports.tsv`, row `pdlib_field_vec:do_output_exchanges`), so a defect would misplace values, not round them. The routine is translated only: header-only, no shim, no `W3WAVE` hunk (v, `ports.tsv`), instantiated only by `wave_support.cpp:305` for the compile check. Not in the resident preset and not called by the fork. | +| `kokkos/src/ww_kokkos/uost_uost_sourceterm_compute.hpp`:54, 56 | `#1675-D` unrecognized GCC pragma (`#pragma GCC novector`) | 2 | nvcc's front end does not know the pragma, and passes it on: on a test file the host g++ 14.4.0 under nvcc_wrapper reports "splitting region at dont-vectorize loop" (v, `-Xcompiler -fopt-info-vec-all`). The pragma keeps the Fortran's operand order of `S_LD + S_SE` when both are NaN (the header's comment, lines 14-16); it is off in device code (`!defined(__CUDA_ARCH__)`). `UOST_SOURCETERM` is translated only, no shim (v, `ports.tsv`). | +| `kokkos/src/ww_kokkos/wavset_fd_compute_lh_stress.hpp`:45 | `#550-D` variable `syy_x` was set but never used | 1 | No: a dead store in a translation of `W3WAVSET`, translated only (v, `ports.tsv`). Not changed: translate, don't improve. | + +No warning about a double-to-float conversion, an unsupported math function, a host function +called from a kernel of a wired part, or contraction, in either build (v, `grep -i warning`). + +**CMake, one warning:** Kokkos' package "does not support CXX extensions. Forcing +`-DCMAKE_CXX_EXTENSIONS=Off`" at WW3's `find_package(ww_kokkos)`. No effect on the arithmetic ⚠ +(it selects `-std=c++20` over `-std=gnu++20`). + +**gfortran, 8 warnings, all in WW3's own files:** "Legacy Extension: Missing comma in FORMAT +string" at `w3dispmd.F90:913`, `w3iorsmd.F90:490`, `w3iogomd.F90:316`, `w3iogomd.F90:2734`, +`w3iopomd.F90:2415`, `ww3_gspl.F90:3029`, `ww3_ounp.F90:1344`, `ww3_prnc.F90:2430`. A FORMAT +string formats text output; it does not touch the model's arithmetic. None in the fork's +`w3kokkos*md.F90` modules (v, `grep`). + +**One thing that is not a warning and may touch the bits:** the CUDA build's host C++ is +compiled by g++ 14.4.0 (nvcc_wrapper's host compiler), while the CPU build and every static +parity argument of `kokkos/ports.tsv` use g++ 16.2.0. A host branch of a kernel in the CUDA +build (a fallback that runs on the host, or the `Kokkos::Serial` path a shim takes) was never +compared with the Fortran under g++ 14 ⚠. On the device the question is the libm one of +[`GPU_LIBM_BITS_202610.md`](../GPU_LIBM_BITS_202610.md). + +Nothing ran on the GPU: no `ctest`, no `ww3_shel`, no `libm_bits --device`. + +## What it changes + +- The checklist [`GPU_SUPERVISED_RUN_CHECKLIST_202610.md`](../GPU_SUPERVISED_RUN_CHECKLIST_202610.md) + starts from this build. +- `just ww3-gpu-matrix-cuda` forwards `WW3GPU_ROUTINES` only, not `WW3GPU_KOKKOS` or + `WW3GPU_LAB_SWITCH`, so it cannot run the resident-preset entries of the CPU matrix as they ran + (v, `justfile`). Left for the integrator and a person's review: it changes how a regtest + driver is called. +- The `#20013-D` sites matter once `DO_OUTPUT_EXCHANGES` is wired on a device; the wiring unit + reads them first. Not fixed here. diff --git a/docs/log/README.md b/docs/log/README.md index d5860e9d..39f78ba1 100644 --- a/docs/log/README.md +++ b/docs/log/README.md @@ -67,3 +67,4 @@ Date, author, issue/PR/WFIP. | 2026-10-18 | [The resident preset under MPI: refused by design; SRCE and KTP3 alone run, one entry differs](2026-10-18-resident-mpi-lab-switch.md) | #42, WFIP-0003 | | 2026-10-18 | [Where the Kokkos resident run of `full` spends its wall time on the CPU](2026-10-18-resident-profile.md) | #42, WFIP-0005 | | 2026-10-18 | [Five bit-neutral speed-ups of the resident preset: three built, two checked, one Fortran change tested](2026-10-18-resident-five-speedups.md) | #42, WFIP-0005 | +| 2026-10-18 | [The CUDA build of `ww_kokkos` and of the fork for the resident preset, build only: 51 nvcc warnings, none in a wired part](2026-10-18-cuda-build-only.md) | plan 2026-10-18-002 U5, ADR-0014 | diff --git a/kokkos/README.md b/kokkos/README.md index d4a1e651..7b8432e7 100644 --- a/kokkos/README.md +++ b/kokkos/README.md @@ -209,6 +209,11 @@ on the search path. `WW_KOKKOS_ARCH=HOPPER90` makes the configure call Kokkos' ` backends and/or architectures requested" (v, 2026-10-17) instead of building `sm_89` code under an H100 name. `scripts/ww3-gpu.sh --gpu` still builds `cuda-release`. +`just kokkos-cuda-test` ends with `ctest` on the CUDA default execution space, which is a device +run and the owner's. To build only: `cmake -S kokkos --preset cuda-release` and +`cmake --build kokkos/build/cuda-release` in the `#cuda` shell. The owner's supervised GPU run +follows [`docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md`](../docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md). + Two things the names do *not* promise: - **The backend comes from the shell, not from the preset.** OpenMP is the default From c0f4feccf925e4f5758a5e4060b7b59d5bb1f6b3 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 12:00:16 -0300 Subject: [PATCH 07/27] docs/compound: plan for the first supervised GPU run and the five plans after it The owner asked whether WW3 with its step resident on the GPU can beat the CPU-only resident run, and for the next five plans for speed or correctness, consolidated with NOAA's GFS-Wave use case. Plan 1, the first supervised GPU run, is detailed for the next session: agent preparation with no device run (tool fixes for human review, builds, CPU reference, static occupancy), the owner's ordered session with a stop rule per step, and the decision table for plans 2 and 4. The traps come from a read of ww3-gpu.sh, nccmp-tol, results.py, libm_bits.sh and the CUDA build-only record. Tested: nix shell nixpkgs#vale -c scripts/vale.sh on the plan, 0 errors Co-Authored-By: Claude Opus 5.5 --- ...10-18-003-gpu-first-supervised-run-plan.md | 302 ++++++++++++++++++ 1 file changed, 302 insertions(+) create mode 100644 docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md diff --git a/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md new file mode 100644 index 00000000..cdc7465e --- /dev/null +++ b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md @@ -0,0 +1,302 @@ +--- +title: The first supervised GPU run of the resident preset, and the five plans after it - Plan +type: feat +date: 2026-10-18 +artifact_contract: ce-unified-plan/v1 +product_contract_source: owner, session of 2026-10-18 (goal abbb0456-6043, PR #191) +execution: code +--- + +# The first supervised GPU run, and the five plans after it - Plan + +## Goal Capsule + +- **Objective:** answer, with one measured number, whether WW3 with its time step resident on + the GPU beats the same WW3 resident on the CPU, and how far the GPU's output is from the + Fortran's; then pick the next plan from that answer. +- **Means:** plan 1 below: agent preparation with no device run (phase A), one session the owner + supervises at the terminal (phase B), records and a decision (phase C). +- **Authority:** AGENTS.md over this plan; "Running on the GPU (hard rule)" (no `--gpu` run + without the owner), ADR-0014 (a GPU run is judged by `nccmp-tol`'s tolerances, never widened), + ADR-0017 (single node), "Resident spectrum (hard rule)". +- **Stop conditions:** any `nccmp-tol` failure, any NaN in the GPU output where the Fortran has a + value, a watchdog kill that a smaller batch does not cure, or a run that is not reproducible run + to run: record it and stop that step; never change a tolerance or a gate during the session. +- **Execution profile:** large. Phase A in the next session, on a branch from `main` after PR + #191 merges; phase B when the owner says so; phase C in the same PR as phase A's records or a + follow-up. + +--- + +## Context: NOAA's most common use of WW3 + +GFS-Wave, and GEFS-Wave built the same way, is a `ww3_multi` mosaic of three grids (global 10 +arcmin, Southern Ocean 15 arcmin, Arctic 9 km), 50 x 36 spectral components, ST4 with the DIA, +to 384 h, run with MPI on WCOSS2, whose nodes are AMD EPYC 7742 with no GPU, moving to Google +Cloud by December 2027 (v, `docs/GLOSSARY.md` rows GFS-Wave and WCOSS2, sources read +2026-10-16). + +Against that use, the Kokkos runtime has three gaps after PR #191 is written: + +- `ww3_multi`: the resident path gave wrong numbers between `W3WAVE` calls (gap 2 of + `docs/PRODUCTION_READINESS_202610.md`); U1 of PR #191 fixes it, and its regtests decide. +- MPI: out of scope (ADR-0017). +- GFS-Wave's own switch file was never read. If it has `SEED` or `FLD2`, the Kokkos `W3SRCE` + does not run there (gap 1) ⚠. + +## Could WW3 entirely on the GPU beat ww3-gpu on the CPU? + +On `full` (`bench_small`, 9 520 sea points, 24 x 24 spectral components, 12 h), very likely, by +an order of magnitude ⚠. No WW3 run on a GPU exists yet (`kokkos/PORT_STATUS.md`); this is a +projection from these measurements: + +| Fact | Number | Source | +|---|---|---| +| CPU resident, 32 threads, `full` | 14.36 s wall; `W3SRCE` kernel 12.45 s (87 %); outside Kokkos 0.22 s (1.5 %) | `docs/log/2026-10-18-resident-profile.md`, one run, load 3.5 ⚠ | +| One thread, port against Fortran | 174.78 s against 174.27 s | same log, one run each ⚠ | +| `W3SRCE` kernel, 1 to 32 threads | 12.6x faster (the i9-14900 has 24 cores) | same log | +| DIA kernel alone, 1000 points, OpenMP 32 threads against RTX 4090 | 5.02 ms against 0.047 ms, 107x | `bench/results/snl1-i9-14900-rtx4090-2026-09-15.json`, median of three (v) | +| The same call with the per-call copies of phase 1 | 5.95 ms against 0.75 ms, 7.9x | same record (v) | + +The second row says the CPU gain is all in spreading work over threads: the per-point code costs +what the Fortran costs. The last two rows bracket the GPU: with the spectrum resident (the design +since WFIP-0005, and with no copies after PR #191), the end-to-end ratio moves from 7.9x toward +the kernel's 107x. If the whole step ran 20 to 50 times faster on the GPU, `full` would take 0.5 +to 1 s against 14.4 s ⚠. Three assumptions carry that figure, and plan 1 tests each: + +1. **The whole `W3SRCE` kernel scales like the DIA alone.** It does not, as written: the body on + `main` is `w3srce_team`, `TeamPolicy(n, AUTO)`, 168 registers, most of the work in + `Kokkos::single`, so by ptxas' estimate about three threads per SM do the non-DIA work + (`origin/goal/42-gpu-step-srceteam`, `438ab8b`, `12685ff`, not merged). It also runs with + `--fmad=false` and the device copies of glibc's functions (`WW_KOKKOS_LIBM`), both slower than + CUDA's defaults. +2. **Launches are cheap.** Each step launches several kernels; on 9 520 points their latency may + dominate. Smoke (one point, 2 161 launches) is slower on a GPU by construction. +3. **The host remainder stays small.** 1.5 % on the CPU becomes the floor once kernels shrink: + 0.22 s against a 0.5 s target. + +**"Without any CPU" is not possible.** Set-up, namelists, I/O and the time-loop driver stay on the +host; a GPU is driven from a CPU. The target is the whole time step on the device, the host only +orchestrating and writing outputs: plan 4. + +**More CPU threads do not change the picture.** At 12.6x on 32 threads of a 24-core part, the +i9 is near its ceiling. Beyond one node is MPI (ADR-0017). A 128-core EPYC node may scale further +⚠ (not measured). Inside a node, `W3XYP3`'s threads are idle in 57 % of its samples, which is +1.1 s at most. + +**Correctness costs.** The GPU is not yet bit for bit: a GPU run is judged by tolerance +(ADR-0014) until `scripts/libm_bits.sh --device` shows the device copies of glibc's functions +give glibc's bits, and the host C++ of the CUDA build is compiled by g++ 14.4.0, not the 16.2.0 of +every parity argument so far (`docs/log/2026-10-18-cuda-build-only.md`) ⚠. + +## The five plans, ranked + +| # | Plan | Attacks | Why this rank | Owner decision needed | +|---|---|---|---|---| +| 1 | The first supervised GPU run (this file, below) | the unknown: GPU against CPU, and the GPU's distance from the Fortran | cheapest, most information; decides whether 2 and 4 aim at the CPU or the GPU | the run itself | +| 2 | Several sea points per vector lane in `W3SRCE` | DIA, `W3SDS4`, `W3SIN4`, the integration: 12.0 of 14.4 s on the CPU | the only CPU plan that attacks the per-point cost, now all of it; on a GPU the same layout gives coalesced loads | yes: it changes loop nesting (AGENTS.md, "Porting"); its own PR with the per-lane parity argument | +| 3 | GFS-Wave's own configuration | correctness on what NOAA runs | without it every speed-up is shown on a configuration NOAA does not run | which tag to read | +| 4 | The whole time step resident on the GPU | the host remainder, launch count, readers left after PR #191 | what makes plan 1's number hold on cases with more output than `full` | lane Q's merge | +| 5 | Thread efficiency in one node, and a many-core baseline | `W3XYP3` imbalance, pinning, scaling on a quiet machine and a WCOSS2-like or cloud EPYC node | the honest CPU baseline a GPU result must beat for NOAA's move to the cloud | access to such a node | + +Not ranked: skipping `W3SDS4`'s zero-weight terms (12 of 23 products, profile log) changes the +result for an infinite or NaN input, so it is the owner's decision and its own PR. + +### Plan 2 in short + +A team of eight points, one per AVX2 lane, each lane its own point's arithmetic in the +Fortran's order, spectra point-minor in team scratch (profile log, proposal 1). No sum crosses a +lane, so the parity argument is the per-lane GIMPLE. Covers ranks 1 to 4 of the profile if built +as a layer for the whole `W3SRCE` team body. Trap: the five libmvec sites of ST4 +(`st4_libmvec.hpp`) keep the Fortran's groups of four directions per call, so those loops stay. +Gain ⚠: the profile does not separate gathers from arithmetic. + +### Plan 3 in short + +Read the switch of the GFS-Wave release tag and record it in gap 1; if it has `SEED` or `FLD2`, +translate their `W3SRCE` branches (ADR-0002 order permitting); build a three-grid `ww3_multi` +regtest case with that switch from WW3's inputs, and add it to the matrix under +`WW3GPU_LAB_SWITCH=OFF`. + +### Plan 4 in short + +After PR #191 (readers U8, U9, the `ww3_multi` exchange of U1): review and land lane Q +(`w3srce_split`, 96 threads per point; three surviving mutants in `12685ff`'s message to resolve +first); move the forcing updates left on the host; cut launches per step (merge the small +kernels, or CUDA graphs if Kokkos' graph API covers them ⚠); count with `WW_KOKKOS_VA_COUNT=1` +and the timers. + +### Plan 5 in short + +`W3XYP3` component scheduling (dynamic over components, or a team size from the component +count), `OMP_PROC_BIND`/`OMP_PLACES` on a quiet machine, scaling at 8, 16, 24, 32 threads, median +of three; one run on a many-core EPYC node if the owner has one. + +--- + +## Plan 1 in detail: the first supervised GPU run + +### What the run must answer + +- Q1. Wall time of `ww3_shel`, `full`, `--kokkos resident`, on the RTX 4090 against the same + preset on the CPU (32 threads) and against `--ww3`, median of three, same commit. +- Q2. Per-field distance of the GPU output from the Fortran of the same executable and from + `--ww3` (`max|d|`, rms, largest relative difference), and whether it passes + `kokkos/tools/nccmp-tol/tolerances.txt` unchanged. +- Q3. Where the GPU wall time goes (timers): `W3SRCE`, propagation, outputs, host remainder, + copies. +- Q4. Whether the device copies of glibc's functions give glibc's bits (`libm_bits --device`). +- Q5. Whether the run is reproducible run to run on the GPU. + +### Facts the plan is built on (each checked in a file, 2026-10-18) + +- `scripts/ww3-gpu.sh --gpu` asks for a typed "yes" at a terminal (`:359-370` at `fe3cbc8`), exit + 7 without one, exit 4 without a driver; it then re-executes inside + `nix develop picasso/labs/pratico#cuda`. `--build-only` skips the prompt and is the agents'. +- `--gpu` alone means the `gpu` preset, `SRCE` only (`:296-298`); `resident` is + `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV` (`:191-197`). `just smoke-ww3-gpu gpu`, `just full-ww3-gpu gpu` + and `just perf-ww3-gpu full 3` run `gpu`, not `resident`; `perf-ww3-gpu` takes the parts as + its third argument (`justfile:215`). +- `--timers` fences the device (a breakdown, not a timing) and cannot combine with `--repeat`. + `--repeat N` alternates `--ww3` and Kokkos runs and requires every Kokkos NetCDF to equal run + 1's (`:709-721`). +- Comparison: `nccmp-tol` on the first `ww3.*.nc` only (`:673-677`), against the Fortran of the + same executable then against `--ww3` (`:695`); `cmp` of every model file is reported, not fatal, + on CUDA (`:682`, `:698`). `full` writes only `HS` (`bench_small/ww3_ounf.nml:10`); `coast` + writes every `OUTG` field. +- `nccmp-tol` counts a NaN or fill in the test where the reference has a value as "dropped", says + so on `stderr` and still passes the variable (`kokkos/tools/nccmp-tol/nccmp_tol.cpp:52-56`). + `compare.cpp` has no 0/360 degree wrap for `dir` and `dp`. +- `counts_resident.txt` expects `W3SRCE batch inputs unchanged, not copied: 422` on `coast`. At + the device's 128 MiB batch cap `coast` takes two launches per step, so no shadow is kept and + the line differs: the script exits 1 before `nccmp-tol`. With `WW_KOKKOS_SRCE_BATCH_MIB` set the + check is skipped (`:651-662`). +- The 4090 drives the display; the driver's watchdog ends a long kernel + (`cudaErrorLaunchTimeout`); the batch cap's estimate assumes the GPU runs at least a sixth of the + CPU's speed (`kokkos/src/fortran_iface/PATCH.md` section 19). +- `just ww3-gpu-matrix-cuda` forwards `WW3GPU_ROUTINES` only, not `WW3GPU_KOKKOS` or + `WW3GPU_LAB_SWITCH`, and `WW3GPU_STRICT` is on, so every entry fails on any byte difference + (`justfile:143-159`). +- `results.py record` needs `--repeat` of at least 3, writes the ADR-0014 wording for CUDA, and + refuses a second study `ww3-vs-ww3gpu--cuda-` on the same day (`:121-171`); it keeps + no per-field statistics. +- `libm_bits.sh --device` asks its own "yes", builds `device_check.cu` (`--fmad=false`, sm_89), + checks 15 functions on 2^32 inputs each with a single-threaded host reference + (`device_check.cu:146-168`): hours for the full set ⚠ (not timed); `--device expf` checks one. +- `WW_KOKKOS_LIBM`'s tables are generated from the build host's libmvec, so build and run happen + on the same workstation (`docs/TOOLCHAIN.md`). +- The CUDA builds in the main checkout are from 2026-10-16, before `WW_KOKKOS_LIBM`, and on the + `SHRD` switch: they rebuild from scratch. The toolchain stamp includes `CMAKE_PREFIX_PATH`, so + `build/ww3` also rebuilds inside `#cuda` and again on the next CPU run (`:480-487`). +- `run_dir`s are deleted at every start (`build/runs/ww3-gpu--cuda/`); `WW3GPU_RUNS` + moves them. +- Docker's CUDA image is another toolchain (CUDA 13.4, no OpenMP, counts check skipped): not + used for this run. + +### Phase A: preparation by agents (next session, no device run) + +Each unit ends with its gates and a commit; nothing in phase A types "yes" or launches a kernel. + +**A1. Freeze the commit.** A branch `goal/-gpu-first-run` from `main` after PR #191 merged. +The commit that phase B runs is its tip at the end of phase A, written into the run card (A7). +No other session works in that checkout during phase B: a separate worktree, `WW3GPU_RUNS` under +it. + +**A2. Make the tools able to judge this run.** Each change makes a comparison stricter or lets +it run the configuration asked; each is a change to a comparison or its invocation, so a person +reviews it before merge (AGENTS.md, "Never cheat a regtest tool"), and it merges before phase B. + +- A2a. `nccmp-tol`: a NaN or fill in the test where the reference has a finite value fails the + variable (today: passes with a note). Stricter, never looser. +- A2b. `nccmp-tol`: judge every `ww3.*.nc` the case writes, not the first only. +- A2c. `counts_resident.txt` on a device: the expected `not copied` count follows the number of + launches per step (or the line is checked on the host backend only, and the device run prints + its count for the record). Decide with the owner which; the CPU check stays as it is. +- A2d. `ww3-gpu-matrix-cuda` forwards `WW3GPU_KOKKOS` and `WW3GPU_LAB_SWITCH`, and a GPU entry's + verdict comes from `nccmp_tol`/`judged` with `test.comp` recorded (ADR-0014), selftest fixtures + for both. +- A2e. Ask, do not do: a 0/360 wrap for `dir`/`dp` in `compare.cpp`. A wrap changes what passes + near north, so it is the owner's decision; until then, `coast`'s direction fields are read by + hand when they fail near 0 or 360. +- A2f. `smoke-ww3-gpu`, `full-ww3-gpu`: accept a parts argument as `coast-ww3-gpu` does, so the + run card can use `just` and the preset is visible in the command. If not, the card uses + `scripts/ww3-gpu.sh` directly. + +**A3. Build everything the session will run, on the frozen commit, on the workstation.** +`JOBS=8 bash scripts/ww3-gpu.sh --ww3-gpu full --gpu --kokkos resident --build-only`, the same for +`smoke` and `coast` if their builds differ, and `scripts/libm_bits.sh --nvcc` (build only, if the +script separates it; else record that its build happens in the session). Then build `--ww3` in +the `#cuda` shell so the toolchain stamp matches (`--ww3 full --build-only` if supported ⚠). +Record nvcc, Kokkos, g++ host, gfortran, driver, and the `CMakeCache.txt` lines for +`WW_KOKKOS_LIBM`, `CMAKE_CUDA_ARCHITECTURES`, contraction flags. Read the warnings against +`docs/log/2026-10-18-cuda-build-only.md`'s list; a new warning in a wired part stops phase B. + +**A4. The same commands on the CPU first.** On the frozen commit: +`bash scripts/ww3-gpu.sh --ww3-gpu {smoke,full,coast} --cpu --kokkos resident`, each byte-identical +to the Fortran and to `--ww3` (NetCDF attribute aside); `WW_KOKKOS_VA_COUNT=1` and +`WW_KOKKOS_VA_POISON=1` on `full` and `coast`. These are phase B's reference numbers and prove +the commit is sound before the GPU is involved. + +**A5. Static readiness of the device kernels.** ptxas `--resource-usage` (registers, spills, +shared memory) of every kernel the preset launches, from A3's build; occupancy per kernel from +the 4090's limits; the expected launches per step on `full`. From them, the watchdog risk of +`W3SRCE` at 128 MiB and the batch size for the first run (32 MiB, as the handover proposed, +unless A5 says otherwise). Written into the run card; nothing tuned. + +**A6. Lane Q stays out of the first run.** The first run measures `main`'s `w3srce_team`, so +its number is the baseline lane Q must beat (plan 4). A6 records lane Q's state and its three +surviving mutants, nothing more. + +**A7. The run card.** Update `docs/GPU_SUPERVISED_RUN_CHECKLIST_202610.md` into one ordered card +with the exact commands of phase B (no `just` recipe whose default preset is not `resident`), the +frozen commit, the expected CPU numbers from A4, the per-field template for +`docs/data/ww3gpu_cuda_{smoke,full,coast}_.md`, the stop rule of each step, and the +`script -a` capture of the whole session. + +**A8. Phase A's PR.** Through `just pr`, the A2 changes flagged for human review in the body; +phase B starts after it merges. + +### Phase B: the owner's session (in order; the owner types every "yes") + +| Step | Command (from the run card) | Records | Stop rule | +|---|---|---|---| +| B0 | quiet machine: load below 1, no other session, `nvidia-smi -q` snapshot, `nvidia-smi --query-gpu=display_active --format=csv`, `script -a ` | machine block of the card | load above 1: wait | +| B1 | `scripts/libm_bits.sh --device expf` | TSV line, exit code | exit 2: CUDA error, stop and diagnose | +| B2 | `WW_KOKKOS_SRCE_BATCH_MIB=32 bash scripts/ww3-gpu.sh --ww3-gpu smoke --gpu --kokkos resident` | parts line all `=K`, `nccmp-tol` both references, every field's statistics; speed not read | any failure: stop, record | +| B3 | `WW_KOKKOS_SRCE_BATCH_MIB=32 bash scripts/ww3-gpu.sh --ww3-gpu full --gpu --kokkos resident --timers` | timers breakdown (Q3), `nccmp-tol` | `cudaErrorLaunchTimeout`: halve the batch once; again: stop | +| B4 | B3 without `--timers`, twice, with `WW3GPU_RUNS` apart; `cmp` the two Kokkos outputs | Q5 | not identical: record, skip B5's `--repeat` | +| B5 | `just perf-ww3-gpu full 3 resident` (with B3's batch variable if B3 needed it) | median of three (Q1); `results.py record` | `nccmp-tol` failure: no record | +| B6 | `WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1` on B4's command | NaN check, bytes moved (Resident spectrum rule) | a NaN in the output: a missed sync, stop | +| B7 | `coast` as B2, then without the batch variable once A2c is merged | every `OUTG` field, directions by hand near 0/360 | as B2 | +| B8 | `just ww3-gpu-matrix-cuda` on the entries U3 of PR #191 passed, if A2d merged | per-entry `nccmp_tol`, `test.comp`, `judged` | as the matrix reports | +| B9 | `scripts/libm_bits.sh --device` for all 15 functions, left running at the end | the full TSV (Q4) | none: a record | + +### Phase C: records and the decision + +- `docs/data/ww3gpu_cuda_{smoke,full,coast}_.md` (per-field statistics), + `bench/results/` from `results.py record` (`just results check`), a `docs/log/` entry with the + session's log and every stop rule that fired, `docs/GPU_LIBM_BITS_202610.md` with B1 and B9. +- `kokkos/PORT_STATUS.md` "On a GPU" and `port_score.py --write`; WFIP-0005 unit 6's done + criterion corrected to the resident preset (`just perf-ww3-gpu full 3 resident`); ADR-0014's + revisit after the first run, as it asks. +- The decision on plans 2 and 4, from B5's median against A4's CPU median, both on the same + commit: + +| GPU / CPU resident, `full` | What it says | Next | +|---|---|---| +| at least 5x | the device step is the faster design here | plan 4 (lane Q, launches, host remainder), then plan 2 for the GPU layout | +| 2x to 5x | the GPU wins but something bounds it | read B3's timers: `W3SRCE` dominant means lane Q first; launches or host means plan 4's launch work | +| below 2x | the GPU does not pay on this case yet | plan 2 on the CPU, and a larger case for the GPU before any further GPU work ⚠ | + +Whatever the ratio, the GPU number enters a table only with B5's `nccmp-tol` pass (ADR-0014) and +says "regtest pending" until WW3's regtests passed on its commit. + +## Risks the plan cannot remove + +- The 4090 is a desktop part with a display; an H100 has no watchdog and other limits, and + `cuda-h100-release` needs a HOPPER90 Kokkos picasso does not build. The ratio does not transfer + to an H100 ⚠. +- `full` is a 9 520-point regional case; GFS-Wave's global grid is larger, which favours the GPU, + and writes more output per step, which does not ⚠. +- g++ 14.4 for the CUDA build's host code: a host branch of the CUDA build may round differently + from the CPU build; A4 on the CPU build does not cover it ⚠. From d658fab64e4414e3cae3708d7d5c143a615b88e0 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 12:29:03 -0300 Subject: [PATCH 08/27] The resident VA: the boundary and track outputs read only their columns Goal abbb0456-6043, unit U8 of the production single-node plan. On the fork branch ww3-gpu/u8-out-readers (31a09522, 791e8c34 on 80644ba9; no gitlink change here), under WW_KOKKOS_IOPE=1 W3IOBC ('WRITE') reads back only the columns ISBPO and W3IOTR only the columns of its track points, from inside W3IOTR once the track file is read; the OASIS coupling's sync is dropped because it reads no VA; W3IOSF keeps its full read because it partitions every sea point. The shim keeps up to four column lists per grid so the point and boundary outputs do not reallocate at every call. PATCH.md section 23 and the docs/log entry carry the Fortran lines read, the static -O3 check of W3IOTR and the counters before and after; the whole-spectrum copies per run did not drop in the two entries, because another full sync follows in the same step. Tested: just kokkos-test openmp-release and serial-debug (11/11 each); WW3GPU_LAB_SWITCH=ON WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1 ww3_tp2.2 PR3_UQ resident test.comp 0/24, ww3_tr1 -i input_std SRCE,OUTG,IOPE,UPDV test.comp 0/23 via run_entry.cmake; bash scripts/ww3-gpu.sh --ww3-gpu smoke and full --cpu --kokkos resident byte-identical to the Fortran and --ww3 (D2H 37 and 1, unchanged); bash scripts/ww3-gpu.sh --self-test ok; scripts/vale.sh on the three Markdown files 0 errors. Co-Authored-By: Claude Opus 5.5 --- .../log/2026-10-18-resident-output-readers.md | 83 +++++++++++++++++++ docs/log/README.md | 1 + kokkos/src/fortran_iface/PATCH.md | 66 +++++++++++++++ kokkos/src/fortran_iface/parts.d/pnt.txt | 3 +- kokkos/src/fortran_iface/resident_c.hpp | 7 +- kokkos/src/fortran_iface/resident_shim.cpp | 67 +++++++++------ 6 files changed, 198 insertions(+), 29 deletions(-) create mode 100644 docs/log/2026-10-18-resident-output-readers.md diff --git a/docs/log/2026-10-18-resident-output-readers.md b/docs/log/2026-10-18-resident-output-readers.md new file mode 100644 index 00000000..ee09b09f --- /dev/null +++ b/docs/log/2026-10-18-resident-output-readers.md @@ -0,0 +1,83 @@ +# 2026-10-18: the boundary and track outputs read only their columns of the resident `VA` + +Goal `abbb0456-6043`, unit U8 of the production single-node plan +([`docs/compound/plans/2026-10-18-002-production-single-node-plan.md`](../compound/plans/2026-10-18-002-production-single-node-plan.md)). +Fork branch `ww3-gpu/u8-out-readers` of h0ffmann/WW3: `31a09522`, `791e8c34` on `80644ba9`. +The change is [`PATCH.md` section 23](../../kokkos/src/fortran_iface/PATCH.md#23-the-boundary-and-track-outputs-read-only-their-columns-ww3-gpuu8-out-readers). + +## What was tried + +The output block of `W3WAVE` copied the whole device `VA` to the host before each of `W3IOTR`, +`W3IOBC ('WRITE')`, `W3IOSF` and the OASIS coupling (`w3wavemd.F90:2934-2975` at `80644ba9`). +Read in the Fortran at the pin: + +- `W3IOBC ('WRITE')` reads the columns `ISBPO(1:NBO2(NFBPO))` (`w3iobcmd.F90:660-668`): now a + column read. +- `W3IOTR` reads `MAPFS(IY,IX)` for the points `MASK1` marks with `MAPSTA > 0` + (`w3iotrmd.F90:728-819`), known only after it reads the track file: now a column read inside + `W3IOTR`, before its section 4. +- `W3IOSF` partitions every sea point with `MAPSTA >= 0` (`w3iosfmd.F90:243-265`): the full read + stays. Partitioning on the device belongs to the `W3CPRT` unit. +- The OASIS coupling reads no `VA` (v, `grep` of `w3agcmmd.F90`, `w3ogcmmd.F90`, + `w3igcmmd.F90`, `w3oacpmd.F90`): its sync is removed; not compiled here ⚠. + +The new column reads are under `WW_KOKKOS_IOPE=1`, the opt-in flag of the point output's column +read. + +## Commands + +The superproject at `96ef800` plus this change, `WW3/` at the fork commit named, on an Intel Core +i9-14900 shared with other sessions, `OMP_NUM_THREADS=4`, never `--gpu`: + +```bash +# the matrix entry with track output, before (WW3 at 80644ba9) and after (791e8c34) +nix develop ./picasso/labs/pratico#ww3 --command cmake -S regtests -B build/m-after \ + -DWW3GPU_SELECTION=serial -DWW3GPU_OMP_THREADS=4 -DWW3GPU_DATA= \ + -DWW3GPU_KOKKOS=resident -DWW3GPU_SWITCH_WORDS='PR3;UQ;SHRD;!PDLIB;!MGP;!MGG' \ + -DWW3GPU_LAB_SWITCH=ON -DWW3GPU_STRICT=ON +WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1 \ + nix develop ./picasso/labs/pratico#ww3 --command ctest --test-dir build/m-after -R tp2.2 +# the entry with boundary output, through run_entry.cmake (see "Not resolved"), with a copy of +# build/m-after/ww3gpu_config.cmake whose WW3GPU_KOKKOS is SRCE,OUTG,IOPE,UPDV +WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1 nix develop ./picasso/labs/pratico#ww3 --command \ + cmake -DCONFIG=build/m-after/ww3gpu_config_tr1.cmake -DN=901 -DTEST=ww3_tr1 \ + '-DOPTS=-i input_std -w work_std' -P regtests/cmake/run_entry.cmake +# pair to pair with --ww3 +WW_KOKKOS_VA_COUNT=1 bash scripts/ww3-gpu.sh --ww3-gpu smoke --cpu --kokkos resident +WW_KOKKOS_VA_COUNT=1 bash scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos resident +``` + +## What it showed + +| Entry | test.comp after | `VA` before | `VA` after | +|---|---|---|---| +| `ww3_tp2.2 -s PR3_UQ`, resident, track every 4 h | 0/24 different, `track_o.ww3` and `ww3.202206_trck.nc` identical (v) | D2H 4 (60.07 MB), host reads 8, partial reads 3 | D2H 4 (60.07 MB), host reads 4, partial reads 7 (3 track copies, 5 columns, 4.4 kB) | +| `ww3_tr1 -i input_std`, `SRCE,OUTG,IOPE,UPDV`, boundary every 1 h | 0/23 different, `nest1.ww3` to `nest3.ww3` identical (v) | D2H 7 (0.57 MB), host reads 8, writes 6, partial reads 7 | D2H 7 (0.57 MB), host reads 1, writes 6, partial reads 14 (7 boundary copies of 8 columns, 70 kB) | +| `smoke`, `full`, resident | byte-identical to the Fortran and to `--ww3`, `nccmp-tol` PASS (v) | D2H 37, 1 (section 17) | D2H 37, 1 | + +With `WW_KOKKOS_VA_POISON=1` the stale host `VA` is NaN after every device write, so a column a +reader needs and the list missed would show in its output file; none did. The whole-spectrum +copies per run did not drop in either entry: the next full sync of the same step copies `VA` +anyway (in `ww3_tp2.2` the exit read of `W3WAVE` at the wind times, which are the track times; in +`ww3_tr1` the host write before the `PR1` Fortran propagation at every step). What changed is +what each output reader copies: the columns it reads, not the whole spectrum. + +## Found on the way + +- No KOKKOS switch without `PR3` compiled: `W3KOKKOS_MAP3` and `W3KOKKOS_MAPT` named + `W3ADATMD` variables declared only under `W3_PR3` (`w3adatmd.F90:520-525`, `:648-653`). Fixed in + `31a09522`; `ww3_tr1` (`SHRD RTD PR1`) builds with it (v). +- `W3KOKKOS_IOTR_READ` writes `VA` while `W3IOTR` holds it as `INTENT(IN)`; the `-O3` dump loads + nothing of it before the call (v, `PATCH.md` section 23), which is the compiler's code, not a + guarantee of the standard ⚠. + +## Not resolved + +- The matrix driver drops the empty options of a first-kept entry: `ww3gpu_select_entries` + (`regtests/cmake/ww3gpu_common.cmake:337-343`) appends `""` to an empty list, which adds nothing, + so `E_OPTS` is one shorter than `E_TESTS` and the configure stops ("list index: 3 out of range") + for the `SHRD;RTD;!PDLIB` selection, whose first entry is `ww3_tr1` with no options. Not fixed: + `regtests/` changes need a person's review ⚠. +- No runnable entry writes boundary output with `PR3`: `ww3_tp2.17` is `MPI`, and `ww3_tp2.6` and + `ww3_tp2.7` write it once, at the start, before the device holds newer bits ⚠. +- `W3IOSF` keeps its full read until partitioning runs on the device. diff --git a/docs/log/README.md b/docs/log/README.md index 39f78ba1..2e6ac2b4 100644 --- a/docs/log/README.md +++ b/docs/log/README.md @@ -68,3 +68,4 @@ Date, author, issue/PR/WFIP. | 2026-10-18 | [Where the Kokkos resident run of `full` spends its wall time on the CPU](2026-10-18-resident-profile.md) | #42, WFIP-0005 | | 2026-10-18 | [Five bit-neutral speed-ups of the resident preset: three built, two checked, one Fortran change tested](2026-10-18-resident-five-speedups.md) | #42, WFIP-0005 | | 2026-10-18 | [The CUDA build of `ww_kokkos` and of the fork for the resident preset, build only: 51 nvcc warnings, none in a wired part](2026-10-18-cuda-build-only.md) | plan 2026-10-18-002 U5, ADR-0014 | +| 2026-10-18 | [The boundary and track outputs read only their columns of the resident `VA`](2026-10-18-resident-output-readers.md) | #42, WFIP-0005 | diff --git a/kokkos/src/fortran_iface/PATCH.md b/kokkos/src/fortran_iface/PATCH.md index 33a002f0..e08ac926 100644 --- a/kokkos/src/fortran_iface/PATCH.md +++ b/kokkos/src/fortran_iface/PATCH.md @@ -1554,6 +1554,7 @@ section 12, not the whole `VA`. The ABI is `ww_kokkos` 0.8.0 (one entry point ad - **Not covered.** `W3IOTR` (track output) still reads the whole `VA`: the columns it reads come from the track positions it reads from its input file inside the routine (`w3iotrmd.F90:264`, `:819`), so listing them before the call means reproducing that reading; none of the cases writes tracks. + Section 23 covers it, and `W3IOBC ('WRITE')`, from inside `W3IOTR` once the track is read. `ww3_multi` (`wmwavemd.F90:1367` calls `W3IOPE` with no sync) is not supported by the resident `VA` at all (section 12). - **Memory contract**: phase 2. A device buffer of `NSPEC` floats per distinct column and a host one @@ -1833,3 +1834,68 @@ under its switch; without it the packet keeps 0 and the switch gate (`IS_LAB_SWI `W3SRCE` to the Fortran, as before. With the guard, `ww3_tp2.6` `ST4` and `ww3_ts2` `ST4_PR3_UQ` build, and their Kokkos runs stop at `WW_SRCE_INIT` code 5 because the preset asks for `SRCE` on a switch it does not translate (v, same matrix, entries 1 and 7). + +## 23. The boundary and track outputs read only their columns, `ww3-gpu/u8-out-readers` + +Two commits on `80644ba9` (section 22), pushed to h0ffmann/WW3 on 2026-10-18: `31a09522` (a +compile guard) and `791e8c34`. No ABI change: both use `ww_va_host_read_cols` of section 17, and +the fork still asks for `0.8`. They apply the resident-spectrum rule of `AGENTS.md` to the host +readers of `VA` in the output block of `W3WAVE` under `SHRD` (`w3wavemd.F90:2930-2990` at +`791e8c34`). The partial reads stay behind `WW_KOKKOS_IOPE=1`, opt-in, the flag of section 17. + +| Reader | What it reads of `VA` (at `80644ba9`) | Under `WW_KOKKOS_IOPE=1` now | +|---|---|---| +| `W3IOBC ('WRITE')`, boundary output | `VA(IS,ISEA)`, `ISEA = ISBPO(ISOUT)`, `ISOUT = 1..NBO2(NFBPO)`, in the `W3_SHRD` gather (`w3iobcmd.F90:660-668`); `ww3_grid` sets `ISBPO` to sea points only (`w3gridmd.F90:6005-6023`), read with the model definition | `W3KOKKOS_IOBC_READ` (`w3kokkos_pntmd.F90:90`) passes `ISBPO(1:NBO2(NFBPO))` in place of the full read (`w3wavemd.F90:2953-2958` at `791e8c34`) | +| `W3IOTR`, track output | `A(ITH,IK,JSEA)`, `JSEA = ISEA = MAPFS(IY,IX)`, for every point with `MASK1(IY,IX)` and `TSTSTR = 'SEA'`, which is `MAPSTA(IY,IX) > 0` (`w3iotrmd.F90:728-763`, `:774`, `:808-819`); its section 3 sets `MASK1` from the track file, inside the call (`:518-697`) | `W3WAVE` skips its read (`w3wavemd.F90:2936`); `W3IOTR` calls `W3KOKKOS_IOTR_READ` (`w3kokkos_pntmd.F90:107`) after section 3 and before section 4 (`w3iotrmd.F90:718-723` at `791e8c34`), with exactly those columns | +| `W3IOSF`, partitioned spectra | every `JSEA = 1..NSEAL` with `MAPSTA >= 0` (`w3iosfmd.F90:243-265`): every sea point that is not ice, dry or excluded | unchanged, the full read, which the rule allows for a reader of the whole field. A column read of the `MAPSTA >= 0` points would save only the ice and dry points; partitioning on the device belongs to the `W3CPRT` unit | +| The OASIS coupling (`W3_OASIS`, `J = 7`) | none: `SND_FIELDS_TO_ATMOS`, `_OCEAN` and `_ICE` read `W3ADATMD` fields and `ICEF`, and no `VA` (`w3agcmmd.F90:88-220`, `w3ogcmmd.F90:62-404`, `w3igcmmd.F90:76-167`, v by `grep`) | the sync is removed (`w3wavemd.F90:2978`); not compiled here, as there is no OASIS library ⚠ | + +- **Why inside `W3IOTR`.** The columns follow the track points that `W3IOTR` reads from + `track_i.ww3` in its section 3, with `BACKSPACE` and the `MASK2` of the previous call, so a list + made before the call would repeat that reading. The hunk is one `CALL` under + `W3_KOKKOS && W3_SHRD && !W3_MPI`; `W3KOKKOS_IOTR_READ` reads `MASK1` from `W3ODATMD`. It writes + the module `VA` while `W3IOTR` holds the same array as its `INTENT(IN)` dummy `A`. The `-O3` + code loads nothing of `A` before the call: in `gfortran -O3 -fdump-tree-optimized` of + `w3iotrmd.F90` with the entry's switch the call is at line 4477 of the dump and every use of + `a_1999(D)` is at lines 4847-4976 (v, 2026-10-18). The Fortran standard does not allow the actual + argument of a dummy to change through another name during the call, so this rests on the code + the compiler makes, not on the standard ⚠. +- **The resident state** stays `DEVICE` after each partial read, as in section 17: the next full + host read or write still copies the whole array, and the next kernel reads `d_va`. +- **The shim** (`resident_shim.cpp`) keeps up to four column lists per grid with their buffers, + the most recent first, in place of one: the point and the boundary outputs alternate at an output + time, and a single list would allocate the device buffers again at every call. +- **`31a09522`**: `W3KOKKOS_MAP3` and `W3KOKKOS_MAPT` named `W3ADATMD` variables that exist only + under `W3_PR3` (`w3adatmd.F90:520-525`, `:648-653`), so no KOKKOS switch without `PR3` compiled + (`ww3_tr1`, `SHRD RTD PR1`: "Symbol 'nmx0' referenced at (1) not found in module 'w3adatmd'", + v). The `USE` and the stores now sit under `W3KOKKOS_HAS_PR3`, as the calls did; without `PR3` + both parts stay off. +- **Without `KOKKOS`**, `gfortran -E -P` of `w3wavemd.F90` and `w3iotrmd.F90` with the entry's + switch is identical to the output from `WW3-noaa/`, blank lines dropped, apart from the include + path in one comment (v, 2026-10-18). `w3iotrmd.F90` is a fork file of the pnt unit (its `FILE` + line in `parts.d/pnt.txt`). + +The checkpoint on the CPU, 2026-10-18 (OpenMP backend, Kokkos 5.2.2, gfortran 16.2.0, Intel Core +i9-14900 shared with other sessions, `OMP_NUM_THREADS=4`; never `--gpu`). Each entry ran with +`WW3GPU_LAB_SWITCH=ON`, `WW3GPU_STRICT=ON`, `WW_KOKKOS_VA_POISON=1` and `WW_KOKKOS_VA_COUNT=1`, +before (`80644ba9`; for `ww3_tr1`, plus `31a09522`) and after (`791e8c34`). +[The log](../../../docs/log/2026-10-18-resident-output-readers.md) has the commands. + +| Entry, parts | Output it writes | test.comp, after | `VA` before | `VA` after | +|---|---|---|---|---| +| `ww3_tp2.2 -s PR3_UQ`, `resident` | track every 4 h (`track_o.ww3`, `ww3.202206_trck.nc`) | 0/24 files different (v) | D2H 4 (60.07 MB); host reads 8; partial reads 3, 2 of them copying 24 columns | D2H 4 (60.07 MB); host reads 4; partial reads 7, 5 copying 29 columns: the 3 track times after the first copy 5 columns, 4.4 kB | +| `ww3_tr1 -i input_std`, `SRCE,OUTG,IOPE,UPDV` | boundary every 1 h (`nest1.ww3` to `nest3.ww3`) | 0/23 files different (v) | D2H 7 (0.57 MB); host reads 8, writes 6; partial reads 7, 28 columns | D2H 7 (0.57 MB); host reads 1, writes 6; partial reads 14, 84 columns: 8 at each of the 7 boundary times, 70 kB in all | + +The host reads before these outputs are gone, and with the host copy poisoned the outputs still +have the Fortran's bits, so no column a reader needs was missed. The whole-spectrum copies per run +did not drop in these two entries. In `ww3_tp2.2` the track times are the wind times, where +`W3WAVE` returns and its exit read (`w3wavemd.F90:3160` at `791e8c34`) copies `VA` anyway; in +`ww3_tr1` (`PR1`) the host write before the Fortran propagation copies it at every step. What each +reader copied is now its columns, and the count per run drops only where no other full sync +follows in the same step. `bash scripts/ww3-gpu.sh --ww3-gpu smoke|full --cpu --kokkos resident`: +byte-identical to the Fortran and to `--ww3`, `nccmp-tol` PASS, D2H 37 (smoke) and 1 (full), as in +section 17 (v). The matrix driver drops the empty options of a first-kept entry +(`regtests/cmake/ww3gpu_common.cmake:337-343`: a `list(APPEND)` of an empty string onto an empty +list adds nothing), so `ww3_tr1` cannot be configured there; it ran through +`regtests/cmake/run_entry.cmake` directly, with a copy of the configured `ww3gpu_config.cmake` +whose parts are `SRCE,OUTG,IOPE,UPDV` ⚠. Nothing ran on a GPU ⚠. diff --git a/kokkos/src/fortran_iface/parts.d/pnt.txt b/kokkos/src/fortran_iface/parts.d/pnt.txt index 07e602f9..073723d7 100644 --- a/kokkos/src/fortran_iface/parts.d/pnt.txt +++ b/kokkos/src/fortran_iface/parts.d/pnt.txt @@ -1,3 +1,4 @@ -# The point output's spectra copied back alone from the resident VA, PATCH.md section 16 (resident_shim.cpp); opt-in, not under WW_KOKKOS_ALL +# The point, boundary and track outputs' spectra copied back alone from the resident VA, PATCH.md sections 17 and 23 (resident_shim.cpp); opt-in, not under WW_KOKKOS_ALL # Read by scripts/ww3-gpu.sh: PART (WW_KOKKOS_), COUNT , FILE ; PATCH.md section 9. PART IOPE +FILE model/src/w3iotrmd.F90 diff --git a/kokkos/src/fortran_iface/resident_c.hpp b/kokkos/src/fortran_iface/resident_c.hpp index e9a7d747..9c9c5114 100644 --- a/kokkos/src/fortran_iface/resident_c.hpp +++ b/kokkos/src/fortran_iface/resident_c.hpp @@ -33,7 +33,7 @@ // The last two, and the byte comparison of every model file in scripts/ww3-gpu.sh, came out of // the mutation audit of docs/log/2026-10-17-mutation-audit.md. // Under WW_KOKKOS_VA_RESIDENT=1 the partial read stays partial (a HOST array is pushed first, as -// every host call does), so a column W3IOPE reads and the list misses keeps stale bits there. +// every host call does), so a column W3IOPE, W3IOBC or W3IOTR reads and the list misses keeps stale bits there. // // SPDX-License-Identifier: MIT #pragma once @@ -48,8 +48,9 @@ int ww_va_host_read(int igrid, float* va, int nspec, int nsealm); /// Before a host write of VA: ww_va_host_read, then state HOST. int ww_va_host_write(int igrid, float* va, int nspec, int nsealm); -/// Before W3IOPE (point output) reads VA on the host, with WW_KOKKOS_IOPE=1 (the pnt unit, -/// PATCH.md section 16): when the device holds the newer bits (state DEVICE), copy back only the +/// Before W3IOPE (point output), W3IOBC ('WRITE', boundary output) or section 4 of W3IOTR +/// (track output) reads VA on the host, with WW_KOKKOS_IOPE=1 (the pnt unit, PATCH.md sections +/// 17 and 23): when the device holds the newer bits (state DEVICE), copy back only the /// columns cols[0..ncol) of VA, W3WAVE's JSEA in 0..nsealm (duplicates allowed), gathered on the /// device; otherwise (HOST, BOTH) nothing. The state is NOT changed: every other column of the /// host VA stays stale, so the next ww_va_host_read still copies the whole array. A column diff --git a/kokkos/src/fortran_iface/resident_shim.cpp b/kokkos/src/fortran_iface/resident_shim.cpp index 7e961c57..1c95fcf1 100644 --- a/kokkos/src/fortran_iface/resident_shim.cpp +++ b/kokkos/src/fortran_iface/resident_shim.cpp @@ -5,7 +5,7 @@ // // The Fortran owns VA; this file only copies it whole, host to device or back, when the // other side holds the newer bits, or (ww_va_host_read_cols, the pnt unit) copies back the few -// columns W3IOPE reads without changing which side is newer, or (ww_va_device_zero_cols, the updv +// columns W3IOPE, W3IOBC ('WRITE') or W3IOTR reads without changing which side is newer, or (ww_va_device_zero_cols, the updv // unit) repeats on d_va the zeroing of whole columns that W3UICE made on the host. A copy moves bits and computes nothing. Every call // comes from outside an OpenMP parallel region (the Fortran wrappers refuse one), so the // contexts and counters need no lock. Views die in Kokkos' finalize hook. @@ -27,26 +27,33 @@ #include #include #include +#include #include namespace ww::resident { namespace { +// One column list of the partial read and its buffers. Kept for the run: W3IOPE and W3IOBC ask +// for the same columns at every output time, so each list is allocated once. +struct ColSet { + std::vector cols; + Kokkos::View d_cols; + Kokkos::View d_buf; + Kokkos::View h_buf; +}; + struct Ctx { float* host = nullptr; int nspec = 0, nsealm = -1; DevVa d; Side side = Side::Host; - // The partial read: the sorted column list of the last call and its device copies, kept - // because W3IOPE asks for the same columns at every point-output time. - std::vector cols; + // The partial read: every sorted column list asked for, the most recent first. W3IOTR's list + // changes with the track; past kMaxColSets lists the oldest is dropped. + std::vector sets; // WW_KOKKOS_VA_CHECK=1: the host bits of VA when the two copies were last made equal, kept up to // date by the declared host writes of whole columns (the partial read, the zeroed columns). std::vector snap; bool snap_ok = false; - Kokkos::View d_cols; - Kokkos::View d_buf; - Kokkos::View h_buf; size_t count() const { return static_cast(nspec) * static_cast(nsealm + 1); } }; @@ -56,6 +63,10 @@ std::map& grids() { return *m; } +// The column lists a grid keeps buffers for: W3IOPE's and W3IOBC's, which repeat, and a few of +// W3IOTR's, which follow the track. +constexpr size_t kMaxColSets = 4; + struct Counts { long long h2d = 0, d2h = 0, host_reads = 0, host_writes = 0, device_calls = 0; long long part_calls = 0, part_d2h = 0, part_cols = 0; @@ -95,9 +106,7 @@ void release() { std::fprintf(stderr, "ww_kokkos: VA of grid %d was still newer on the device at finalize\n", igrid); c.d = DevVa(); - c.d_cols = {}; - c.d_buf = {}; - c.h_buf = {}; + c.sets.clear(); } grids().clear(); } @@ -263,23 +272,31 @@ int host_read_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const } if (want.empty()) return WW_KOKKOS_OK; const int n = static_cast(want.size()); - if (want != c->cols) { - c->cols = want; - c->d_cols = Kokkos::View( + const auto hit = std::find_if(c->sets.begin(), c->sets.end(), + [&](const ColSet& s) { return s.cols == want; }); + if (hit != c->sets.end()) { + std::rotate(c->sets.begin(), hit, hit + 1); + } else { + ColSet s; + s.cols = want; + s.d_cols = Kokkos::View( Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols"), n); - Kokkos::deep_copy(c->d_cols, Kokkos::View>( - c->cols.data(), n)); - c->d_buf = Kokkos::View( + Kokkos::deep_copy(s.d_cols, Kokkos::View>( + s.cols.data(), n)); + s.d_buf = Kokkos::View( Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols_buf"), nspec, n); - c->h_buf = Kokkos::View( + s.h_buf = Kokkos::View( Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols_host"), nspec, n); + if (c->sets.size() >= kMaxColSets) c->sets.pop_back(); + c->sets.insert(c->sets.begin(), std::move(s)); } + ColSet& cs = c->sets.front(); { ww::timers::Region tc("va.d2h.cols"); const DevVa d = c->d; - const auto dc = c->d_cols; - const auto buf = c->d_buf; + const auto dc = cs.d_cols; + const auto buf = cs.d_buf; const size_t ns = static_cast(nspec); Kokkos::parallel_for( "ww_va.gather_cols", Kokkos::RangePolicy<>(0, ns * static_cast(n)), @@ -287,17 +304,17 @@ int host_read_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const const size_t k = i / ns, s = i % ns; buf(s, k) = d(s, dc(k)); }); - Kokkos::deep_copy(c->h_buf, buf); + Kokkos::deep_copy(cs.h_buf, buf); Kokkos::fence("ww_va.d2h_cols"); } for (int k = 0; k < n; ++k) - std::memcpy(c->host + static_cast(c->cols[k]) * static_cast(nspec), - &c->h_buf(0, k), static_cast(nspec) * sizeof(float)); + std::memcpy(c->host + static_cast(cs.cols[k]) * static_cast(nspec), + &cs.h_buf(0, k), static_cast(nspec) * sizeof(float)); // The host columns just copied are part of what the host holds now. if (c->snap_ok) for (int k = 0; k < n; ++k) - std::memcpy(c->snap.data() + static_cast(c->cols[k]) * static_cast(nspec), - &c->h_buf(0, k), static_cast(nspec) * sizeof(float)); + std::memcpy(c->snap.data() + static_cast(cs.cols[k]) * static_cast(nspec), + &cs.h_buf(0, k), static_cast(nspec) * sizeof(float)); ++g_n.part_d2h; g_n.part_cols += n; g_n.part_bytes += static_cast(static_cast(n) * static_cast(nspec) * From 5fb30f1491c3882fa88493d5b62b9566c3f01f5d Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 12:30:27 -0300 Subject: [PATCH 09/27] docs/compound: plan 003 names the branch that builds the W3SDS4 and DIA changes Another session builds the W3SDS4 loop interchange, a DIA direction-halo layout and -fno-math-errno on goal/eb10363f-4821-noaa-progress-record. Plan 2 starts from that result, and both PRs need one resident regtest pass on the combined tree before a timing enters a table. Tested: nix shell nixpkgs#vale -c scripts/vale.sh on the plan, 0 errors Co-Authored-By: Claude Opus 5.5 --- .../2026-10-18-003-gpu-first-supervised-run-plan.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md index cdc7465e..4a6c47da 100644 --- a/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md +++ b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md @@ -102,6 +102,15 @@ every parity argument so far (`docs/log/2026-10-18-cuda-build-only.md`) ⚠. Not ranked: skipping `W3SDS4`'s zero-weight terms (12 of 23 products, profile log) changes the result for an infinite or NaN input, so it is the owner's decision and its own PR. +**Work in progress elsewhere (2026-10-18).** Branch `goal/eb10363f-4821-noaa-progress-record` +(another session, not pushed when this was written) builds the `W3SDS4` saturation-integral loop +interchange (`kokkos/src/ww_kokkos/st4_sds4.hpp`), a DIA direction-halo layout that turns the +index-table gathers into contiguous loads (`snl1_team.hpp`, `snl1_tables.{hpp,cpp}`), and +`-fno-math-errno` for the host build (`kokkos/src/ww_kokkos/CMakeLists.txt`). The halo layout +attacks the same gathers as plan 2 by another route; plan 2 starts from that branch's measured +result, not before it. Its timing and the GPU run of plan 1 need a resident regtest pass on the +tree that holds both PRs. + ### Plan 2 in short A team of eight points, one per AVX2 lane, each lane its own point's arithmetic in the From 1d9266c3500cd10c8d5e85e7c4a560334aab9095 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 12:35:39 -0300 Subject: [PATCH 10/27] docs/compound: plan 003 checks the CUDA build's host compiler before the GPU run The CUDA build compiles its host C++ with g++ 14.4.0, the CPU build and every parity argument with g++ 16.2.0; gfortran is 16.2.0 in both. Phase A3b runs the CPU backend under g++ 14.4.0 pair to pair, or compares the GIMPLE, so a GPU difference can be split between device and host. Tested: nix shell nixpkgs#vale -c scripts/vale.sh on the plan, 0 errors Co-Authored-By: Claude Opus 5.5 --- .../2026-10-18-003-gpu-first-supervised-run-plan.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md index 4a6c47da..98b9fa74 100644 --- a/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md +++ b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md @@ -240,6 +240,16 @@ Record nvcc, Kokkos, g++ host, gfortran, driver, and the `CMakeCache.txt` lines `WW_KOKKOS_LIBM`, `CMAKE_CUDA_ARCHITECTURES`, contraction flags. Read the warnings against `docs/log/2026-10-18-cuda-build-only.md`'s list; a new warning in a wired part stops phase B. +**A3b. The host compiler of the CUDA build.** gfortran is 16.2.0 in both builds, but +`nvcc_wrapper` compiles the host half of every `.cpp` with g++ 14.4.0, against 16.2.0 for the CPU +build and every static parity argument (`docs/log/2026-10-18-cuda-build-only.md:25-26`, +`:74-77`). Before phase B, one of two things: build the library on the OpenMP backend with g++ +14.4.0 and run `smoke` and `full` resident pair to pair with the Fortran (byte-identical, or a +defect to fix before the GPU run); or compare the `-O3` GIMPLE of the host-side shim paths under +both compilers. Record why the `#cuda` shell pins g++ 14.4 (presumably the newest host GCC nvcc +12.9 accepts ⚠, from NVIDIA's support matrix). Without it, a GPU difference cannot be split into +the device's share and the host compile's. + **A4. The same commands on the CPU first.** On the frozen commit: `bash scripts/ww3-gpu.sh --ww3-gpu {smoke,full,coast} --cpu --kokkos resident`, each byte-identical to the Fortran and to `--ww3` (NetCDF attribute aside); `WW_KOKKOS_VA_COUNT=1` and From 80a5c8a58ff7ee82d73c22e60b0fe12b02144f1d Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 12:41:26 -0300 Subject: [PATCH 11/27] kokkos: WW_KOKKOS_STRICT and WW_KOKKOS_PRESET, the resident list in presets.txt (goal abbb0456) Gaps 6 and 7 of docs/PRODUCTION_READINESS_202610.md, unit U2 + U4 of docs/compound/plans/2026-10-18-002-production-single-node-plan.md. part_flags.cpp reads WW_KOKKOS_STRICT and WW_KOKKOS_PRESET once, turns a preset's parts on as if each WW_KOKKOS_ were 1 (a set WW_KOKKOS_ still wins), and keeps the per-part count of calls left on the Fortran, printed in one line at exit. The resident part list moves from scripts/ww3-gpu.sh into kokkos/src/fortran_iface/presets.txt, which CMake compiles into the shim and the script and the regtest matrix read. The fork side is h0ffmann/WW3 branch ww3-gpu/u2-strict-preset (f0cbe70d strict mode, 237ca0de the preset, on 80644ba9); the WW3/ gitlink is not moved here, the integrator bumps it once. PATCH.md sections 24 and 25, GLOSSARY, kokkos/README.md. Needs human review before merge: the matrix's Kokkos run now sets WW_KOKKOS_STRICT=1 (ww3gpu_kokkos_env, regtests/cmake/ww3gpu_common.cmake) and every run unsets WW_KOKKOS_PRESET, which changes how a comparison is invoked: an entry whose Kokkos run would fall back to the Fortran now fails instead of comparing the Fortran with itself. The hot path: one new per-step function, W3KOKKOS_SRCE_STEP, outside the loop over sea points, adds a test of two LOGICALs per step; the fallback branch is one call and one increment; no getenv, string or lock per call or per point. Tested: just kokkos-test serial-debug (11/11 passed); just kokkos-test openmp-release (11/11 passed); just ww3-gpu-matrix-selftest (ok; 99 serial, 50 omp, 161 mpi, 310 all, 7 resident serial at 80644ba9, unchanged); bash -n and shellcheck scripts/ww3-gpu.sh (clean); bash scripts/ww3-gpu.sh --self-test (ok); vale on the four Markdown files (0 errors); bash scripts/ww3-gpu.sh --ww3-gpu smoke --cpu --kokkos resident before and after (every model file byte-identical to the Fortran's and to the run before this change); the same run with only WW_KOKKOS_PRESET=resident (parts line and all four model files identical); WW_KOKKOS_PRESET=bogus stops; WW_KOKKOS_ALL=1 with TAUWBUG=0 counts SRCE=2160 and with WW_KOKKOS_STRICT=1 stops naming the reason Co-Authored-By: Claude Opus 5.5 --- docs/GLOSSARY.md | 2 + docs/PRODUCTION_READINESS_202610.md | 21 +++++- kokkos/README.md | 11 ++- kokkos/src/fortran_iface/CMakeLists.txt | 20 ++++++ kokkos/src/fortran_iface/PATCH.md | 71 +++++++++++++++++++ kokkos/src/fortran_iface/part_flags.cpp | 86 ++++++++++++++++++++++++ kokkos/src/fortran_iface/presets.txt | 8 +++ kokkos/src/fortran_iface/ww_kokkos_c.hpp | 16 +++++ regtests/CMakeLists.txt | 3 +- regtests/cmake/run_entry.cmake | 4 +- regtests/cmake/selftest.cmake | 21 ++++-- regtests/cmake/ww3gpu_common.cmake | 48 +++++++++++-- scripts/ww3-gpu.sh | 21 +++++- 13 files changed, 313 insertions(+), 19 deletions(-) create mode 100644 kokkos/src/fortran_iface/presets.txt diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index e71fecb0..a66e5ebc 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -702,6 +702,8 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | `WW_KOKKOS_VA_RESIDENT` / `WW_KOKKOS_VA_COUNT` | diagnostics of the resident VA, ignoring `WW_KOKKOS_ALL`: `=1` makes every host sync call push a host-newer `VA` to the device and copy it back, so a host write without its sync call shows in the output; the count prints the traffic line and the copy counters at exit `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset. On the CPU both copies are `memcpy`, so the bits cannot change. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | | `WW_KOKKOS_VA_CHECK` / `WW_KOKKOS_VA_POISON` | diagnostics of the resident VA, read once, ignoring `WW_KOKKOS_ALL`, which compare or overwrite only the stale copy: `VA_CHECK=1` makes every sync call check what the state claims before it copies (unless `HOST`, the host `VA` still holds the bits of the last sync, so a host write without its `W3KOKKOS_VA_HOST_WRITE` stops the model; with `BOTH`, `d_va` holds them too, so a kernel that wrote `d_va` without marking it does); `VA_POISON=1` fills the host `VA` with a quiet NaN after every kernel write of `d_va`, so a host read without its sync, or a column the partial read misses, puts NaN in the output `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset; a correct run writes the same bits with either. Added by the mutation audit of 2026-10-17. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | | `WW_KOKKOS_OUTG_STAGED` | a diagnostic of the outg unit, read once: `=1` makes a host backend take the device's path of `ww_outg_va`, the inputs copied into the shim's own buffers, the fields computed into buffers, and only the fields a host reader needs copied back, so a field the copy-back selection misses keeps its old bits in WW3's array and shows in `out_grd.ww3` `(v)` [kokkos/src/fortran_iface/outg_shim.cpp](../kokkos/src/fortran_iface/outg_shim.cpp) | Default unset: on Serial and OpenMP the kernel writes WW3's arrays in place. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | +| `WW_KOKKOS_STRICT` | environment variable read once by `ww_kokkos_strict()`: `=1` makes every place where a part asked on Kokkos would run the Fortran stop the model with `EXTCDE` and a `NDSE` line naming the part and the reason: `KOKKOS_PART` under `WW_KOKKOS_ALL=1`, the two `W3SRCE` set-up fallbacks and the per-step `W3SRCE` gate (`W3KOKKOS_SRCE_STEP`). Unset, those places run the Fortran as before and the shim prints one line at exit, `ww_kokkos: calls left on the Fortran (...): SRCE= ...` `(v)` [kokkos/src/fortran_iface/part_flags.cpp](../kokkos/src/fortran_iface/part_flags.cpp) | Default unset. The regtest matrix's Kokkos run sets it, its Fortran runs unset it, so a pair cannot compare the Fortran with itself unseen. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 24 | +| `WW_KOKKOS_PRESET` / `presets.txt` | environment variable naming a line `PRESET ` of `kokkos/src/fortran_iface/presets.txt`: `ww_kokkos_flag` turns its parts on as if each `WW_KOKKOS_` were `1`; a set `WW_KOKKOS_` still wins, and `W3INIT` stops on a name the file lacks. `resident` is the only preset (`SRCE,XYP3,KTP3,OUTG,IOPE,UPDV`) `(v)` [kokkos/src/fortran_iface/presets.txt](../kokkos/src/fortran_iface/presets.txt) | The file is the one list: CMake compiles it into the shim, `scripts/ww3-gpu.sh --kokkos resident` and the matrix's `WW3GPU_KOKKOS=resident` read it. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 25 | | mutation audit / mutant (equivalent, surviving) | a deliberate, minimal breakage of a piece of the port, applied in a scratch tree and never committed, to ask whether the checks that call the piece correct would fail without it: a mutant is *detected* when a check fails, *surviving* when every check passes, and *equivalent* when it cannot change any bit the case computes (it is not a defect the case can show) `(v)` [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | First run 2026-10-17, on the CPU. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | | `WW_KOKKOS_ALL` / `ww_kokkos_flag` | the master run-time flag of every Kokkos part, and the one C function that resolves each `WW_KOKKOS_`: a set part flag wins (`1` on, anything else off), an unset one follows `WW_KOKKOS_ALL=1` (on) or stays off `(v)` [kokkos/src/fortran_iface/part_flags.cpp](../kokkos/src/fortran_iface/part_flags.cpp) | "100 %" means every wired part, not all of WW3; under the master a part whose translation does not cover the switch file stays Fortran. `W3INIT` logs one "Kokkos parts" line; `scripts/ww3-gpu.sh --kokkos all\|none\|` sets it. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | | W3KOKKOS_UNITSMD / W3KOKKOS_REGISTER / `parts.d` | the Fortran module the fork's CMake generates from every `model/src/w3kokkos_md.F90` (it `USE`s each unit module and calls each unit's set-up from `W3INIT`), the routine a unit's set-up calls to put a part on the "Kokkos parts" line, and the per-unit files `kokkos/src/fortran_iface/parts.d/.txt` that `scripts/ww3-gpu.sh` reads its parts from `(v)` h0ffmann/WW3 `ww3-gpu/kokkos-wire-base` | A wiring unit adds files and its own call-site hunks and edits no shared list. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | diff --git a/docs/PRODUCTION_READINESS_202610.md b/docs/PRODUCTION_READINESS_202610.md index c0be3219..056cd42b 100644 --- a/docs/PRODUCTION_READINESS_202610.md +++ b/docs/PRODUCTION_READINESS_202610.md @@ -42,8 +42,8 @@ regtest matrix yet"). 10 of the 310 rows of `kokkos/ports.tsv` are regtested and | 3 | MPI: four of the six resident parts are `SHRD` only, the preset stops on `DIST MPI`, the communicator is ignored, the device id is not per rank | Blocker | Major | A resident preset without `XYP3`, `OUTG`, `IOPE`, `UPDV`; one run on `switch_lab_mpi` | | 4 | Bit parity measured with gfortran and g++ only; the operational compiler and the GPU are not measured | Blocker | Major | Static argument for the Intel build's math calls; one smoke with it | | 5 | Build and packaging: a separate static library, C++20 and CMake 3.25; the install-only configure and the exact Kokkos check are done (U6) | Major | Major | A shared-library build and a recipe for the module systems of an operational centre | -| 6 | `WW_KOKKOS_ALL=1` and `W3KOKKOS_SRCE_ON` fall back to the Fortran without a word | Major | Minor | A strict flag that turns every fallback into a stop, and a per-run fallback count | -| 7 | Configuration is 61 environment variables, no namelist; the preset lives in a lab script | Major | Major | A namelist group read by `W3INIT`, the environment as override, the effective set printed once | +| 6 | `WW_KOKKOS_ALL=1` and `W3KOKKOS_SRCE_ON` fall back to the Fortran without a word | Major | Minor | Done on fork branch `ww3-gpu/u2-strict-preset`, not yet at the pin: `WW_KOKKOS_STRICT=1` stops, otherwise one fallback-count line at exit; the matrix's Kokkos runs set it | +| 7 | Configuration is 61 environment variables, no namelist; the preset lives in a lab script | Major | Major | Smallest form done on `ww3-gpu/u2-strict-preset`: `WW_KOKKOS_PRESET=resident` from one table, `presets.txt`; next, a namelist group read by `W3INIT` | | 8 | Observability: log lines on every rank, counters at exit to `stderr`, no provenance in the output files | Major | Minor | Gate the lines by `NAPLOG`; one NetCDF global attribute | | 9 | OpenMP inside MPI: the per-point parts are serialised by `!$OMP CRITICAL`; the thread budget per rank is not set by the shim | Major | Minor | Measure the hybrid entries of `ww3_ufs1.1` before claiming a speed for them | | 10 | Restart with the resident spectrum: covered by the sync calls, never regtested | Major | Minor | The restart-reproduction pair of `ww3_ufs1.1` (`work_c` against `work_d`) under the lab switch | @@ -266,6 +266,15 @@ asked for Kokkos and got the Fortran learns it only from one log line or a `*_CO - **(b) Minor.** The default is the Fortran anyway. - **Next step.** `WW_KOKKOS_STRICT=1`: `KOKKOS_PART` and the `SRCE` gate stop instead of falling back, and `W3KOKKOS_SRCE_ON` counts its Fortran calls for one line at the end of the run. +- **Done (2026-10-18), not yet at the pin.** Fork branch `ww3-gpu/u2-strict-preset` (`f0cbe70d`): + `KOKKOS_PART` under `WW_KOKKOS_ALL=1`, the two `W3SRCE` set-up fallbacks and a new per-step gate + `W3KOKKOS_SRCE_STEP` call `KOKKOS_FALLBACK`, which stops with `EXTCDE` and a `NDSE` line + naming the part and the reason under `WW_KOKKOS_STRICT=1`, and otherwise counts; the shim + prints `ww_kokkos: calls left on the Fortran (WW_KOKKOS_STRICT=1 stops instead): SRCE= ...` + at exit on every process, as the `*_COUNT` lines do. The flag is read once; the per-point + `W3KOKKOS_SRCE_ON` is unchanged. The regtest matrix's Kokkos run sets `WW_KOKKOS_STRICT=1` + (`regtests/cmake/ww3gpu_common.cmake`, `ww3gpu_kokkos_env`). `PATCH.md` section 24 has the + runs that show it. ## 7. Configuration surface @@ -283,6 +292,14 @@ script (v, `scripts/ww3-gpu.sh:190`), so a production job card would repeat it b ⚠ (convention, not a rule read today). - **Next step.** A `&KOKKOS_NML` group read in `W3INIT` that sets the same flags, the environment kept as override for tests, and the effective set written once to the log. +- **Done in its smallest form (2026-10-18), not yet at the pin.** `WW_KOKKOS_PRESET=resident`, + read by `W3INIT` through the shim (fork `dca49b00` on `ww3-gpu/u2-strict-preset`): the part + list is one line of `kokkos/src/fortran_iface/presets.txt`, which CMake compiles into + `part_flags.cpp` and `scripts/ww3-gpu.sh --kokkos resident` and the matrix's + `WW3GPU_KOKKOS=resident` read, so they cannot drift. A set `WW_KOKKOS_` still wins, an + unknown name stops the model, and the parts line stays the effective set, with the preset's + name on the line before it. A job card then carries one variable instead of six. The + namelist group remains the next step. `PATCH.md` section 25. ## 8. Observability diff --git a/kokkos/README.md b/kokkos/README.md index 7b8432e7..bb8f1ca4 100644 --- a/kokkos/README.md +++ b/kokkos/README.md @@ -411,11 +411,18 @@ this is the short form. | partial | `WW_KOKKOS_=1` for each part wanted | `--kokkos SNL1,SDS4` | the listed parts | | full | `WW_KOKKOS_ALL=1` | `--kokkos all` | every wired part whose translation covers the switch file | | GPU preset | `WW_KOKKOS_ALL=0 WW_KOKKOS_SRCE=1` | `--kokkos gpu` (the default of `--gpu`) | `W3SRCE` with the source terms it calls (ADR-0013) | -| resident preset | `WW_KOKKOS_ALL=0`, `WW_KOKKOS_=1` for `SRCE`, `XYP3`, `KTP3`, `OUTG`, `IOPE`, `UPDV` | `--kokkos resident` | the parts that keep the spectrum `VA` on the device (WFIP-0005 units 1 to 5, #173, #179); not a default: none of its parts is regtested yet ([`ports.tsv`](ports.tsv)) | +| resident preset | `WW_KOKKOS_PRESET=resident`, or `WW_KOKKOS_ALL=0` and `WW_KOKKOS_=1` for `SRCE`, `XYP3`, `KTP3`, `OUTG`, `IOPE`, `UPDV` (the line of [`presets.txt`](src/fortran_iface/presets.txt)) | `--kokkos resident` | the parts that keep the spectrum `VA` on the device (WFIP-0005 units 1 to 5, #173, #179); not a default: none of its parts is regtested yet ([`ports.tsv`](ports.tsv)) | An explicitly set `WW_KOKKOS_` wins over `WW_KOKKOS_ALL`; one C function, `ww_kokkos_flag` in `src/fortran_iface/part_flags.cpp`, resolves both for the Fortran and the -shims. `W3INIT` writes one line to `log.ww3` naming each part and where it ran. "Full" is 100 % +shims. `WW_KOKKOS_PRESET=` turns on the parts of one line of +[`src/fortran_iface/presets.txt`](src/fortran_iface/presets.txt) as if each were set to `1`; a set +`WW_KOKKOS_` still wins and an unknown name stops `W3INIT`. `W3INIT` writes one line to +`log.ww3` naming each part and where it ran, the effective set. Where a part asked on Kokkos would +run the Fortran (`WW_KOKKOS_ALL=1` on a switch file its translation does not cover, or a `W3SRCE` +step with a branch the kernel leaves to the Fortran), the model runs the Fortran and the shim +prints one line at exit with the count per part; `WW_KOKKOS_STRICT=1` stops the model there +instead, with a line naming the part and the reason (PATCH.md sections 24 and 25). "Full" is 100 % of the wired parts, not of WW3: their share of the profiled run time is in [`PORT_STATUS.md`](PORT_STATUS.md) (`python3 scripts/port_score.py --write`), and the I/O, set-up and the rest of the propagation stay Fortran. A part that is wired but whose `regtested` diff --git a/kokkos/src/fortran_iface/CMakeLists.txt b/kokkos/src/fortran_iface/CMakeLists.txt index 82f9badf..9b2a5549 100644 --- a/kokkos/src/fortran_iface/CMakeLists.txt +++ b/kokkos/src/fortran_iface/CMakeLists.txt @@ -23,6 +23,26 @@ target_sources(ww_kokkos PRIVATE timers.hpp timers.cpp) +# presets.txt, the one list of WW_KOKKOS_PRESET's presets (scripts/ww3-gpu.sh and the regtest +# matrix read the same file), as the initializer part_flags.cpp includes. +set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS + "${CMAKE_CURRENT_SOURCE_DIR}/presets.txt") +file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/presets.txt" _ww_preset_lines) +set(_ww_presets "") +foreach(_l IN LISTS _ww_preset_lines) + if(_l MATCHES "^[ \t]*(#|$)") + continue() + elseif(_l MATCHES "^PRESET ([a-z][a-z0-9_]*) ([A-Z][A-Z0-9_]*(,[A-Z][A-Z0-9_]*)*)$") + string(APPEND _ww_presets " {\"${CMAKE_MATCH_1}\", \"${CMAKE_MATCH_2}\"},\n") + else() + message(FATAL_ERROR "presets.txt: unknown line '${_l}'") + endif() +endforeach() +file(CONFIGURE OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/ww_kokkos_presets.inc" + CONTENT "// Generated from presets.txt by fortran_iface/CMakeLists.txt; do not edit.\n${_ww_presets}" + @ONLY) +target_include_directories(ww_kokkos PRIVATE "${CMAKE_CURRENT_BINARY_DIR}") + set(WW_W3KOKKOSMD "${CMAKE_CURRENT_SOURCE_DIR}/../../../WW3/model/src/w3kokkosmd.F90" CACHE FILEPATH "WW3's w3kokkosmd.F90 (the WW3/ submodule, branch ww3-gpu/kokkos-wire-base)") diff --git a/kokkos/src/fortran_iface/PATCH.md b/kokkos/src/fortran_iface/PATCH.md index e08ac926..951f15ed 100644 --- a/kokkos/src/fortran_iface/PATCH.md +++ b/kokkos/src/fortran_iface/PATCH.md @@ -1899,3 +1899,74 @@ section 17 (v). The matrix driver drops the empty options of a first-kept entry list adds nothing), so `ww3_tr1` cannot be configured there; it ran through `regtests/cmake/run_entry.cmake` directly, with a copy of the configured `ww3gpu_config.cmake` whose parts are `SRCE,OUTG,IOPE,UPDV` ⚠. Nothing ran on a GPU ⚠. + +## 24. `WW_KOKKOS_STRICT`: a fallback to the Fortran stops or is counted, `ww3-gpu/u2-strict-preset` + +Fork commit `f0cbe70d` on `ww3-gpu/u2-strict-preset`, branched from `80644ba9` and pushed to +h0ffmann/WW3 on 2026-10-18; not at the pin until the integrator merges it into +`ww3-gpu/kokkos-pnt`. Gap 6 of +[`PRODUCTION_READINESS_202610.md`](../../../docs/PRODUCTION_READINESS_202610.md). Three ABI +entries in [`part_flags.cpp`](part_flags.cpp) (`ww_kokkos_strict`, `ww_kokkos_fallback_slot`, +`ww_kokkos_fallback_add`); `w3kokkosmd.F90` +85 lines, `w3initmd.F90` +6 -1, `w3wavemd.F90` 2 +changed. + +Four places ran the Fortran for a part asked on Kokkos and said so only in the parts line: +`KOKKOS_PART` under `WW_KOKKOS_ALL=1` on a switch file a translation does not cover (`GATH`, +`SCAT`, `UWND`, `UTRN`, `MAP3`, `MAPT`; the opt-in parts are never on under the master), +`W3KOKKOS_SRCE_SETUP` on a switch file that is not the lab switch's source terms, `W3INIT` +without `ST4` and `NL1`, and the per-step `W3SRCE` gate (`IQTPE <= 0`, Liu's ice dispersion, +`TAUWBUG = 0` with `IT > 0`). Each now calls `KOKKOS_FALLBACK ( PART, REASON )` +(`w3kokkosmd.F90:1257-1269` at `dca49b00`): + +- **`WW_KOKKOS_STRICT=1`**: `*** WAVEWATCH III ERROR IN W3KOKKOSMD : WW_KOKKOS_STRICT=1 AND PART + WOULD RUN ON THE FORTRAN: ` on `NDSE`, then `EXTCDE(1)`, on every rank. +- **Unset (the default)**: the Fortran, as before, and the shim counts the call. At exit each + process prints one line to `stderr`, as the `*_COUNT` lines do: + `ww_kokkos: calls left on the Fortran (WW_KOKKOS_STRICT=1 stops instead): SRCE=0`. A set-up + fallback counts 1 per `W3INIT`; `SRCE` counts the time steps whose batch ran the Fortran. The + line appears only when some part could fall back. + +**The hot path.** The per-step gate is a new function, `W3KOKKOS_SRCE_STEP ( IT )` +(`w3kokkosmd.F90:1913-1918`), which `W3WAVE` calls once per step in place of +`W3KOKKOS_SRCE_ON` before the batch (`w3wavemd.F90:2407`), outside the `!$OMP PARALLEL` loop over +sea points. It returns `W3KOKKOS_SRCE_ON ( IT )` and adds one test of two `LOGICAL`s; only on +the fallback branch does it call `SRCE_FALLBACK`, which under the default is one call and one +increment of a plain counter (`ww_kokkos_fallback_add`; no lock, the caller is serial). The +counter's slot is made once in `W3KOKKOS_SRCE_SETUP`. `WW_KOKKOS_STRICT` is read once by the shim +(a function-local static) and cached in `KOKKOS_STRICT` by `W3KOKKOS_SETUP`. The per-point +`W3KOKKOS_SRCE_ON` of the Fortran loop (`w3wavemd.F90:2485`) is unchanged: that loop runs only +when the step's gate was false, so counting the step is complete without a counter in the loop. + +**The matrix.** `ww3gpu_kokkos_env` (`regtests/cmake/ww3gpu_common.cmake`) sets +`WW_KOKKOS_STRICT=1` in the Kokkos run of every pair and unsets it in the Fortran runs (`F`, +`G`), so a pair can no longer compare the Fortran with itself unseen: an entry whose Kokkos run +would fall back now fails at run time instead. This changes how a comparison is invoked and is +reviewed by a person before it merges (AGENTS.md, "Never cheat a regtest tool"). + +## 25. `WW_KOKKOS_PRESET`: the resident part list in one table, `ww3-gpu/u2-strict-preset` + +Fork commit `dca49b00` on `ww3-gpu/u2-strict-preset` (the child of `f0cbe70d`); not at the pin +until the integrator merges it. Gap 7 of +[`PRODUCTION_READINESS_202610.md`](../../../docs/PRODUCTION_READINESS_202610.md), smallest form. +One ABI entry, `ww_kokkos_preset_status`; `w3kokkosmd.F90` +24 -1 lines. + +[`presets.txt`](presets.txt) holds one line per preset, `PRESET `; today only +`PRESET resident SRCE,XYP3,KTP3,OUTG,IOPE,UPDV`, the list that was a line of +`scripts/ww3-gpu.sh`. Three readers, one file: + +- **The shim.** `CMakeLists.txt` here turns the file into `ww_kokkos_presets.inc` at configure + time, and `part_flags.cpp` includes it. `ww_kokkos_flag` reads `WW_KOKKOS_PRESET` once: a set + `WW_KOKKOS_` still wins, then a part of the preset is on as if set to `1` (so a part the + switch file does not cover stops the model, as under `--kokkos resident`), then the opt-in + list and `WW_KOKKOS_ALL` decide as before. +- **`scripts/ww3-gpu.sh`**: `kokkos_preset` reads the `PRESET` lines, so `--kokkos resident` + sets the same six `WW_KOKKOS_=1`. Both runs of a pair unset `WW_KOKKOS_PRESET`, and the + Fortran run unsets `WW_KOKKOS_STRICT`. +- **The matrix**: `ww3gpu_kokkos_preset` reads it for `WW3GPU_KOKKOS=`; `selftest.cmake` + checks a fixture table and that the real `resident` names only parts of `parts.d`. + +`W3KOKKOS_SETUP` (`w3kokkosmd.F90:1182-1188` at `dca49b00`) stops on a name the file lacks: +`*** WAVEWATCH III ERROR IN W3KOKKOSMD : WW_KOKKOS_PRESET= IS NOT A PRESET OF +kokkos/src/fortran_iface/presets.txt`. `W3KOKKOS_REPORT` writes ` Kokkos preset +(WW_KOKKOS_PRESET): resident` on the line before the parts line, which stays the effective set, +once per `W3INIT`. No namelist: the next step of gap 7. diff --git a/kokkos/src/fortran_iface/part_flags.cpp b/kokkos/src/fortran_iface/part_flags.cpp index f5c2d640..b26fe6fb 100644 --- a/kokkos/src/fortran_iface/part_flags.cpp +++ b/kokkos/src/fortran_iface/part_flags.cpp @@ -23,14 +23,69 @@ // while without it the shim reads the tables the Fortran built, the same bits by // construction. // +// WW_KOKKOS_PRESET= (PATCH.md section 25) turns on the parts of one line of presets.txt as if +// each WW_KOKKOS_ were "1", so a part the switch file does not cover stops the model as it +// does under --kokkos resident; a WW_KOKKOS_ that is set still wins, and the master decides +// the parts the preset does not name. WW_KOKKOS_STRICT=1 (section 24) is read here once as well, +// and the per-part count of calls left on the Fortran that it would have stopped is kept here and +// printed once at exit. Every variable is read once per process (a function-local static): no +// getenv on a per-call path. +// // SPDX-License-Identifier: MIT #include "fortran_iface/ww_kokkos_c.hpp" +#include #include #include namespace { +struct Preset { + const char* name; + const char* parts; // PART,PART,... without the WW_KOKKOS_ prefix +}; +constexpr Preset kPresets[] = { +#include "ww_kokkos_presets.inc" +}; +constexpr Preset kUnknown{"", ""}; + +// WW_KOKKOS_PRESET as read once: nullptr unset or empty, &kUnknown for a name presets.txt lacks. +const Preset* preset() { + static const Preset* const p = []() -> const Preset* { + const char* v = std::getenv("WW_KOKKOS_PRESET"); + if (v == nullptr || *v == '\0') return nullptr; + for (const Preset& q : kPresets) + if (std::strcmp(v, q.name) == 0) return &q; + return &kUnknown; + }(); + return p; +} + +// Whether `part` is one item of the comma list `list`. +bool in_list(const char* part, const char* list) { + const std::size_t n = std::strlen(part); + for (const char* s = list; *s != '\0';) { + const char* e = std::strchr(s, ','); + const std::size_t m = e ? static_cast(e - s) : std::strlen(s); + if (m == n && std::strncmp(s, part, n) == 0) return true; + if (e == nullptr) break; + s = e + 1; + } + return false; +} + +// The parts that can fall back to the Fortran, each with its count; filled at set-up, serially. +constexpr int kMaxSlots = 64; +char g_slot_name[kMaxSlots][25] = {}; +long long g_slot_count[kMaxSlots] = {}; +int g_nslots = 0; + +void print_fallbacks() { + std::fprintf(stderr, "ww_kokkos: calls left on the Fortran (WW_KOKKOS_STRICT=1 stops instead):"); + for (int i = 0; i < g_nslots; ++i) std::fprintf(stderr, " %s=%lld", g_slot_name[i], g_slot_count[i]); + std::fprintf(stderr, "\n"); +} + constexpr const char* kOptInOnly[] = {"WW_KOKKOS_XYP1", "WW_KOKKOS_KTP1", "WW_KOKKOS_XYP2", "WW_KOKKOS_KTP2", "WW_KOKKOS_XYPUG", "WW_KOKKOS_XYP3", "WW_KOKKOS_KTP3", "WW_KOKKOS_OUTG", "WW_KOKKOS_IOPE", @@ -40,8 +95,39 @@ constexpr const char* kOptInOnly[] = {"WW_KOKKOS_XYP1", "WW_KOKKOS_KTP1", "WW_KO extern "C" int ww_kokkos_flag(const char* name) { if (name == nullptr) return 0; if (const char* v = std::getenv(name)) return std::strcmp(v, "1") == 0 ? 1 : 0; + constexpr const char kPrefix[] = "WW_KOKKOS_"; + if (const Preset* p = preset(); p != nullptr && std::strncmp(name, kPrefix, sizeof kPrefix - 1) == 0 && + in_list(name + sizeof kPrefix - 1, p->parts)) + return 1; for (const char* p : kOptInOnly) if (std::strcmp(name, p) == 0) return 0; const char* all = std::getenv("WW_KOKKOS_ALL"); return (all != nullptr && std::strcmp(all, "1") == 0) ? 2 : 0; } + +extern "C" int ww_kokkos_preset_status() { + const Preset* p = preset(); + return p == nullptr ? 0 : (p == &kUnknown ? -1 : 1); +} + +extern "C" int ww_kokkos_strict() { + static const int strict = [] { + const char* v = std::getenv("WW_KOKKOS_STRICT"); + return (v != nullptr && std::strcmp(v, "1") == 0) ? 1 : 0; + }(); + return strict; +} + +extern "C" int ww_kokkos_fallback_slot(const char* part) { + if (part == nullptr || std::strlen(part) >= sizeof g_slot_name[0]) return -1; + for (int i = 0; i < g_nslots; ++i) + if (std::strcmp(g_slot_name[i], part) == 0) return i; + if (g_nslots == kMaxSlots) return -1; + if (g_nslots == 0) std::atexit(print_fallbacks); + std::strcpy(g_slot_name[g_nslots], part); + return g_nslots++; +} + +extern "C" void ww_kokkos_fallback_add(int slot) { + if (slot >= 0 && slot < g_nslots) ++g_slot_count[slot]; +} diff --git a/kokkos/src/fortran_iface/presets.txt b/kokkos/src/fortran_iface/presets.txt new file mode 100644 index 00000000..4a3985d9 --- /dev/null +++ b/kokkos/src/fortran_iface/presets.txt @@ -0,0 +1,8 @@ +# The run-time presets of WW_KOKKOS_PRESET= (part_flags.cpp, PATCH.md section 25), one line each: +# PRESET . The one list: CMake compiles it into part_flags.cpp, and +# scripts/ww3-gpu.sh --kokkos and the matrix's WW3GPU_KOKKOS= read it, so the three +# cannot drift. Every part is a PART of parts.d. +# +# resident: W3SRCE batched and the PR3 propagation, W3OUTG, the point output's read and W3UICE's +# write of VA on the resident spectrum (WFIP-0005; PATCH.md sections 11 and 15 to 18). +PRESET resident SRCE,XYP3,KTP3,OUTG,IOPE,UPDV diff --git a/kokkos/src/fortran_iface/ww_kokkos_c.hpp b/kokkos/src/fortran_iface/ww_kokkos_c.hpp index facba817..844c9f3e 100644 --- a/kokkos/src/fortran_iface/ww_kokkos_c.hpp +++ b/kokkos/src/fortran_iface/ww_kokkos_c.hpp @@ -874,6 +874,22 @@ int ww_st6_last_error(void); /// variable other than "1" is off whatever WW_KOKKOS_ALL says. int ww_kokkos_flag(const char* name); +/// WW_KOKKOS_PRESET (presets.txt): 0 unset or empty, 1 a preset of presets.txt, -1 any other +/// name, which W3KOKKOS_SETUP turns into a stop. Its parts are on in ww_kokkos_flag as if set to 1. +int ww_kokkos_preset_status(void); + +/// 1 when WW_KOKKOS_STRICT is "1": a part asked on Kokkos that would run the Fortran stops the +/// model instead (W3KOKKOSMD, KOKKOS_FALLBACK). Read once per process. +int ww_kokkos_strict(void); + +/// The counter of Fortran fallbacks of `part` (at most 24 characters), created on first use; -1 +/// when the name is too long or the table is full. Call at set-up, serially; the first call +/// registers one line at exit that prints every counter. +int ww_kokkos_fallback_slot(const char* part); + +/// One more call of the slot's part on the Fortran: one increment, no lock (callers are serial). +void ww_kokkos_fallback_add(int slot); + // --------------------------------------------------------------------------- // W3WAVE's per-step support routines (wave_support_shim.cpp): W3GATH, W3SCAT // (W3_SHRD), W3UWND (WNT1, RWND), W3UTRN (IC0), W3DZXY, W3MAP3, W3MAPT (PR3), and diff --git a/regtests/CMakeLists.txt b/regtests/CMakeLists.txt index fa4e9c43..6bc19e9e 100644 --- a/regtests/CMakeLists.txt +++ b/regtests/CMakeLists.txt @@ -23,7 +23,8 @@ # flag to 0. ADR-0011's ST4 checkpoint runs one at a time. # WW3GPU_KOKKOS empty (the default): the Kokkos run sets WW3GPU_ROUTINES as above. Otherwise # the flags of scripts/ww3-gpu.sh --kokkos, so a regtest and a timed run share -# them: resident (SRCE,XYP3,KTP3,OUTG,IOPE,UPDV), all (WW_KOKKOS_ALL=1), none, +# them: a preset of kokkos/src/fortran_iface/presets.txt (resident: +# SRCE,XYP3,KTP3,OUTG,IOPE,UPDV), all (WW_KOKKOS_ALL=1), none, # or a comma list of the parts of kokkos/src/fortran_iface/parts.d; the # selection is then WW3GPU_SWITCH_WORDS alone, and the Kokkos run must show # each part as =K in W3INIT's parts line diff --git a/regtests/cmake/run_entry.cmake b/regtests/cmake/run_entry.cmake index 07f769a3..36c0b856 100644 --- a/regtests/cmake/run_entry.cmake +++ b/regtests/cmake/run_entry.cmake @@ -14,7 +14,9 @@ # WW3GPU_KOKKOS the Kokkos run takes that preset's flags instead (ww3gpu_kokkos_env), kokkos_log # is W3INIT's parts line showing each part on Kokkos, and kokkos_spectra lists every # "W3 spectra|points computed" counter. Every run sets WW_KOKKOS_ALL and unsets the parts it -# does not turn on, so no flag is inherited from the shell. +# does not turn on, so no flag is inherited from the shell. The Kokkos run also sets +# WW_KOKKOS_STRICT=1: a part it asks for that would run the Fortran stops it, so a pair never +# compares the Fortran with itself unseen (kokkos/src/fortran_iface/PATCH.md section 24). cmake_minimum_required(VERSION 3.21) include("${CONFIG}") include("${CMAKE_CURRENT_LIST_DIR}/ww3gpu_common.cmake") diff --git a/regtests/cmake/selftest.cmake b/regtests/cmake/selftest.cmake index 5b241c48..99b55136 100644 --- a/regtests/cmake/selftest.cmake +++ b/regtests/cmake/selftest.cmake @@ -159,14 +159,27 @@ ww3gpu_kokkos_preset(f "SRCE,BOGUS" "${pp}") if(NOT f MATCHES "^ERROR:'BOGUS'") fail("preset with an unknown part: ${f}") endif() +# The presets come from presets.txt: a fixture table with another preset, and a name it lacks. +set(_real_presets "${WW3GPU_PRESETS_TXT}") +set(WW3GPU_PRESETS_TXT "${t}/presets.txt") +file(WRITE "${WW3GPU_PRESETS_TXT}" "# a table\nPRESET resident SRCE,XYP3\nPRESET lean SRCE\n") +ww3gpu_kokkos_preset(f resident "${pp}") +expect("${f}" "SRCE;XYP3" "preset resident read from presets.txt") +ww3gpu_kokkos_preset(f lean "${pp}") +expect("${f}" "SRCE" "a second preset of presets.txt") +ww3gpu_kokkos_preset(f bogus "${pp}") +if(NOT f MATCHES "^ERROR:'bogus' is not a preset") + fail("an unknown preset: ${f}") +endif() +set(WW3GPU_PRESETS_TXT "${_real_presets}") ww3gpu_kokkos_env(e K "SRCE" "SRCE;XYP3" "WW_KOKKOS_SRCE_COUNT") -expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_SRCE=1;WW_KOKKOS_XYP3=;WW_KOKKOS_SRCE_COUNT=1" "env, Kokkos run of a part list") +expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_STRICT=1;WW_KOKKOS_PRESET=;WW_KOKKOS_SRCE=1;WW_KOKKOS_XYP3=;WW_KOKKOS_SRCE_COUNT=1" "env, Kokkos run of a part list") ww3gpu_kokkos_env(e F "SRCE" "SRCE;XYP3" "WW_KOKKOS_SRCE_COUNT") -expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_SRCE=;WW_KOKKOS_XYP3=;WW_KOKKOS_SRCE_COUNT=" "env, Fortran run: ALL=0, nothing inherited") +expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_STRICT=;WW_KOKKOS_PRESET=;WW_KOKKOS_SRCE=;WW_KOKKOS_XYP3=;WW_KOKKOS_SRCE_COUNT=" "env, Fortran run: ALL=0, nothing inherited") ww3gpu_kokkos_env(e K "ALL" "SRCE" "") -expect("${e}" "WW_KOKKOS_ALL=1;WW_KOKKOS_SRCE=" "env, Kokkos run of all") +expect("${e}" "WW_KOKKOS_ALL=1;WW_KOKKOS_STRICT=1;WW_KOKKOS_PRESET=;WW_KOKKOS_SRCE=" "env, Kokkos run of all") ww3gpu_kokkos_env(e G "ALL" "SRCE" "") -expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_SRCE=" "env, the repeated Fortran run of all") +expect("${e}" "WW_KOKKOS_ALL=0;WW_KOKKOS_STRICT=;WW_KOKKOS_PRESET=;WW_KOKKOS_SRCE=" "env, the repeated Fortran run of all") set(pl " Kokkos parts (K Kokkos, F Fortran): SNL1=F SRCE=K GATH=F XYP3=K KTP3=K (3 of 5 on Kokkos)") ww3gpu_parts_line_check(m "${pl}" "SRCE;XYP3;KTP3") expect("${m}" "" "parts line, every part on Kokkos") diff --git a/regtests/cmake/ww3gpu_common.cmake b/regtests/cmake/ww3gpu_common.cmake index 54574422..fabc53e6 100644 --- a/regtests/cmake/ww3gpu_common.cmake +++ b/regtests/cmake/ww3gpu_common.cmake @@ -83,10 +83,32 @@ function(ww3gpu_parts_d parts counts dir) set(${counts} "${c}" PARENT_SCOPE) endfunction() +# The presets of WW_KOKKOS_PRESET (PATCH.md section 25), the file the shim compiles and +# scripts/ww3-gpu.sh reads; WW3GPU_PRESETS_TXT may point elsewhere (selftest.cmake). +if(NOT DEFINED WW3GPU_PRESETS_TXT) + set(WW3GPU_PRESETS_TXT "${CMAKE_CURRENT_LIST_DIR}/../../kokkos/src/fortran_iface/presets.txt") +endif() + +# ww3gpu_preset_parts( ): the PART,... list of PRESET in WW3GPU_PRESETS_TXT, or +# empty when the file has no such preset. +function(ww3gpu_preset_parts out name) + set(r "") + if(EXISTS "${WW3GPU_PRESETS_TXT}") + file(STRINGS "${WW3GPU_PRESETS_TXT}" lines REGEX "^PRESET ") + foreach(l IN LISTS lines) + if(l MATCHES "^PRESET ([a-z][a-z0-9_]*) ([A-Z][A-Z0-9_]*(,[A-Z][A-Z0-9_]*)*)$" AND CMAKE_MATCH_1 STREQUAL name) + set(r "${CMAKE_MATCH_2}") + break() + endif() + endforeach() + endif() + set(${out} "${r}" PARENT_SCOPE) +endfunction() + # ww3gpu_kokkos_preset( ): the flags the Kokkos run of WW3GPU_KOKKOS sets, as -# scripts/ww3-gpu.sh --kokkos does: "all" is ALL (WW_KOKKOS_ALL=1), "resident" the part list of -# that preset (kokkos_preset in ww3-gpu.sh), "none" nothing, and a comma list its parts, each of -# ; is ERROR: for anything else. +# scripts/ww3-gpu.sh --kokkos does: "all" is ALL (WW_KOKKOS_ALL=1), a preset of presets.txt +# (resident) its part list, "none" nothing, and a comma list its parts, each of ; is +# ERROR: for anything else. function(ww3gpu_kokkos_preset out spec parts) if(spec STREQUAL "all") set(${out} ALL PARENT_SCOPE) @@ -94,8 +116,13 @@ function(ww3gpu_kokkos_preset out spec parts) elseif(spec STREQUAL "none") set(${out} "" PARENT_SCOPE) return() - elseif(spec STREQUAL "resident") - set(spec "SRCE,XYP3,KTP3,OUTG,IOPE,UPDV") + elseif(spec MATCHES "^[a-z][a-z0-9_]*$") + ww3gpu_preset_parts(pl "${spec}") + if(pl STREQUAL "") + set(${out} "ERROR:'${spec}' is not a preset of ${WW3GPU_PRESETS_TXT}" PARENT_SCOPE) + return() + endif() + set(spec "${pl}") endif() string(REPLACE "," ";" l "${spec}") if(l STREQUAL "") @@ -114,7 +141,10 @@ endfunction() # ww3gpu_kokkos_env( ): the environment of one run, a list of # NAME=VALUE, NAME= meaning unset. Every WW_KOKKOS_ of is unset unless the Kokkos run # turns it on, WW_KOKKOS_ALL is 0 except in a Kokkos run whose has ALL, and the Kokkos run -# sets every counter of . The Fortran runs never inherit a flag from the shell. +# sets every counter of . The Kokkos run sets WW_KOKKOS_STRICT=1, so a part it asks for +# that would run the Fortran stops the run instead of comparing the Fortran with itself (PATCH.md +# section 24); the Fortran runs unset it. No run takes WW_KOKKOS_PRESET from the shell. The +# Fortran runs never inherit a flag from the shell. function(ww3gpu_kokkos_env out k flags parts counts) set(e "") if(k STREQUAL "K" AND "ALL" IN_LIST flags) @@ -122,6 +152,12 @@ function(ww3gpu_kokkos_env out k flags parts counts) else() list(APPEND e "WW_KOKKOS_ALL=0") endif() + if(k STREQUAL "K") + list(APPEND e "WW_KOKKOS_STRICT=1") + else() + list(APPEND e "WW_KOKKOS_STRICT=") + endif() + list(APPEND e "WW_KOKKOS_PRESET=") foreach(p IN LISTS parts) if(k STREQUAL "K" AND p IN_LIST flags) list(APPEND e "WW_KOKKOS_${p}=1") diff --git a/scripts/ww3-gpu.sh b/scripts/ww3-gpu.sh index 60509c13..efac30f6 100755 --- a/scripts/ww3-gpu.sh +++ b/scripts/ww3-gpu.sh @@ -188,12 +188,23 @@ cmp_files() { done < <({ model_files "$1"; model_files "$2"; } | sort -u) return "$bad" } +# The presets of WW_KOKKOS_PRESET, "PRESET " lines (PATCH.md section 25): the shim +# compiles the same file, so --kokkos resident and WW_KOKKOS_PRESET=resident cannot drift. +presets_txt="$root/kokkos/src/fortran_iface/presets.txt" kokkos_preset() { # : a preset's part list, anything else unchanged + local key name list case "$1" in - gpu) echo SRCE ;; - resident) echo SRCE,XYP3,KTP3,OUTG,IOPE,UPDV ;; - *) echo "$1" ;; + gpu) echo SRCE; return ;; + all|none|*,*) echo "$1"; return ;; esac + [ -f "$presets_txt" ] || { echo "ww3-gpu: no $presets_txt" >&2; exit 3; } + while read -r key name list; do + if [ "$key" = PRESET ] && [ "$name" = "$1" ]; then + [[ $list =~ ^[A-Z][A-Z0-9_]*(,[A-Z][A-Z0-9_]*)*$ ]] || { echo "ww3-gpu: $presets_txt: bad part list of $name" >&2; exit 3; } + echo "$list"; return + fi + done < "$presets_txt" + echo "$1" } if [ -n "$kokkos" ]; then [ "$mode" = ww3-gpu ] || [ "$mode" = self-test ] || { echo "ww3-gpu: --kokkos is for --ww3-gpu only" >&2; exit 64; } @@ -571,6 +582,10 @@ run_model() { # <0|1>: grid, strt, shel, ounf; prints ww3_shel's wall time local d="$1" k="$2" t0 t1 r local -a flags=() unset=() for r in "${all_parts[@]}"; do unset+=(-u "WW_KOKKOS_$r"); done + # No preset from the shell in either run (the Fortran run would turn its parts on); strict mode + # reaches the Kokkos run only (PATCH.md sections 24 and 25). + unset+=(-u WW_KOKKOS_PRESET) + [ "$k" = 1 ] || unset+=(-u WW_KOKKOS_STRICT) if [ -n "$kokkos" ]; then read -r -a flags <<<"$(kokkos_env "$k" "$kokkos")" [ "$k" = 0 ] || flags+=("${count_flags[@]}") From 84ea09d2b88ca4cf18ac00b486ba5367781f5255 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 12:44:01 -0300 Subject: [PATCH 12/27] regtests: an entry with no options keeps its item in _OPTS (goal abbb0456) ww3gpu_select_entries built _OPTS with list(APPEND), and a CMake list cannot hold an empty first item: when the first selected entry had no options (ww3_tr1's first line, "$rtst $ww3 ww3_tr1"), E_OPTS had one item fewer than E_TESTS, every later entry read the next one's options and configure stopped with "list index: 3 out of range (-3, 2)" (WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS="SHRD;RTD" just ww3-gpu-matrix-list serial, found by agent F). An entry with no options is now the placeholder WW3GPU_NO_OPTS, and ww3gpu_entry_opts reads it back as empty where regtests/CMakeLists.txt and stage.cmake take an entry's options. selftest.cmake gets a fixture whose first entry has no options, read back item by item. Needs human review before merge: this changes how entries are selected and which options each entry runs with. Tested: just ww3-gpu-matrix-selftest (ok; the new fixture gives 3 tests and 3 options, read back as [], [-s K1 -w work_K1], []); just ww3-gpu-matrix-list for serial, omp, mpi and all, default and WW3GPU_KOKKOS=resident: 99, 50, 161, 310 and 7, 0, 0, 7 tests, unchanged against the same commands before the fix; WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS="SHRD;RTD" serial: 4 tests, was the out-of-range error; the generated CTestTestfile gives the first ww3_tr1 entry OPTS= (empty) and the next ones their own options Co-Authored-By: Claude Opus 5.5 --- regtests/CMakeLists.txt | 2 +- regtests/cmake/selftest.cmake | 20 ++++++++++++++++++++ regtests/cmake/stage.cmake | 2 +- regtests/cmake/ww3gpu_common.cmake | 19 ++++++++++++++++++- 4 files changed, 40 insertions(+), 3 deletions(-) diff --git a/regtests/CMakeLists.txt b/regtests/CMakeLists.txt index 6bc19e9e..5cdd7a98 100644 --- a/regtests/CMakeLists.txt +++ b/regtests/CMakeLists.txt @@ -232,7 +232,7 @@ set_tests_properties(record:summary PROPERTIES FIXTURES_CLEANUP ww3-gpu LABELS " set(_n 0) foreach(test IN LISTS E_TESTS) - list(GET E_OPTS ${_n} opts) + ww3gpu_entry_opts(opts "${E_OPTS}" ${_n}) math(EXPR _n "${_n} + 1") ww3gpu_words(o "${opts}") ww3gpu_opt_of(sw -s ${o}) diff --git a/regtests/cmake/selftest.cmake b/regtests/cmake/selftest.cmake index 99b55136..cc85355a 100644 --- a/regtests/cmake/selftest.cmake +++ b/regtests/cmake/selftest.cmake @@ -77,6 +77,26 @@ expect("${M_OPTS}" "-s MP -w work_MP;-s MH -w work_MH -f -p $mpi -n $nr -t $nth; ww3gpu_select_entries(A "${t}/rt/bin/matrix.base" "${t}/rt" all) list(LENGTH A_TESTS na) expect("${na}" "12" "all selection, the three together") +# An entry with no options at all, first in the selection: its options must stay an item of +# _OPTS, or every later entry reads the options of the next one (and the last is out of range). +file(MAKE_DIRECTORY "${t}/rt0/c/input" "${t}/rt0/bin") +file(WRITE "${t}/rt0/c/input/switch" "F90 SHRD ST4 NL1\n") +file(WRITE "${t}/rt0/c/input/switch_K1" "F90 SHRD ST4 NL1\n") +file(WRITE "${t}/rt0/bin/matrix.base" [=[ + echo "$rtst $ww3 c" >> matrix.body + echo "$rtst -s K1 -w work_K1 $ww3 c" >> matrix.body + echo "$rtst $ww3 c" >> matrix.body +]=]) +ww3gpu_select_entries(Z "${t}/rt0/bin/matrix.base" "${t}/rt0" serial) +list(LENGTH Z_TESTS nzt) +list(LENGTH Z_OPTS nzo) +expect("${nzt}|${nzo}" "3|3" "an entry with no options first: one options item per test") +set(zo "") +foreach(i 0 1 2) + ww3gpu_entry_opts(o "${Z_OPTS}" ${i}) + list(APPEND zo "[${o}]") +endforeach() +expect("${zo}" "[];[-s K1 -w work_K1];[]" "an entry with no options first: the options, read back") ww3gpu_select_entries(T "${t}/rt/bin/matrix.base" "${t}/rt" serial ST4) expect("${T_TESTS}" "a;a;a;a;b;b;a;b" "serial selection by ST4, tests (NL3 and NL1X are in)") ww3gpu_physics(w SDS4 SNL1 SPR4 SIN4) diff --git a/regtests/cmake/stage.cmake b/regtests/cmake/stage.cmake index 2fdba263..234abdb8 100644 --- a/regtests/cmake/stage.cmake +++ b/regtests/cmake/stage.cmake @@ -25,7 +25,7 @@ if(WW3GPU_LAB_SWITCH) endif() set(i 0) foreach(test IN LISTS E_TESTS) - list(GET E_OPTS ${i} opts) + ww3gpu_entry_opts(opts "${E_OPTS}" ${i}) math(EXPR i "${i} + 1") ww3gpu_words(o "${opts}") ww3gpu_opt_of(inp -i ${o}) diff --git a/regtests/cmake/ww3gpu_common.cmake b/regtests/cmake/ww3gpu_common.cmake index fabc53e6..7f6e0bbd 100644 --- a/regtests/cmake/ww3gpu_common.cmake +++ b/regtests/cmake/ww3gpu_common.cmake @@ -296,11 +296,25 @@ endfunction() # ww3gpu_select_entries( [ # []]): every kept entry, in order, as two parallel lists _TESTS and _OPTS -# (the entry's options, $ww3 and the test dropped, space-joined). is a list of switch +# (the entry's options, $ww3 and the test dropped, space-joined; an entry with none is +# WW3GPU_NO_OPTS, since a CMake list cannot hold an empty first item, and ww3gpu_entry_opts reads +# it back as empty). is a list of switch # words, NL1 by default, ANY for no physics filter; every entry whose switch has one of them and # passes ww3gpu_switch_words on is in exactly one of serial (none of MPI, OMPG, OMPH), omp # (OMPG or OMPH, no MPI) and mpi (MPI); all is the three. A line whose switch file is missing is # dropped, and so is a switch with both ST4 and FLX5, which KOKKOS refuses. +set(WW3GPU_NO_OPTS "") + +# ww3gpu_entry_opts( ): the options of entry of a _OPTS list of +# ww3gpu_select_entries, empty for an entry that has none. +function(ww3gpu_entry_opts out optl i) + list(GET optl ${i} o) + if(o STREQUAL "${WW3GPU_NO_OPTS}") + set(o "") + endif() + set(${out} "${o}" PARENT_SCOPE) +endfunction() + function(ww3gpu_select_entries prefix base rt kind) set(phys "${ARGV4}") if(phys STREQUAL "") @@ -372,6 +386,9 @@ function(ww3gpu_select_entries prefix base rt kind) continue() endif() list(JOIN tok " " opts) + if(opts STREQUAL "") + set(opts "${WW3GPU_NO_OPTS}") + endif() list(APPEND tests "${test}") list(APPEND optl "${opts}") endforeach() From f7f5ff8ac003cfb2511a4085ce9249485019923b Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 12:46:40 -0300 Subject: [PATCH 13/27] PATCH.md 24 and 25: the runs that show strict mode and the preset (goal abbb0456) The table of what the two sections were checked on: the resident smoke before and after, the preset alone, an unknown preset, TAUWBUG = 0 counted and stopped under WW_KOKKOS_STRICT=1, and UWND left on the Fortran by a WNT2 switch, counted and stopped. The preset's fork commit is 237ca0de after a fix of its report line (the preset's name printed before the parts line, not in place of it). Tested: nix shell nixpkgs#vale -c scripts/vale.sh on the four Markdown files (0 errors, 0 warnings); the runs in the table, commands in PATCH.md section 24 Co-Authored-By: Claude Opus 5.5 --- docs/PRODUCTION_READINESS_202610.md | 2 +- kokkos/src/fortran_iface/PATCH.md | 26 ++++++++++++++++++++++---- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/docs/PRODUCTION_READINESS_202610.md b/docs/PRODUCTION_READINESS_202610.md index 056cd42b..6ffffa97 100644 --- a/docs/PRODUCTION_READINESS_202610.md +++ b/docs/PRODUCTION_READINESS_202610.md @@ -293,7 +293,7 @@ script (v, `scripts/ww3-gpu.sh:190`), so a production job card would repeat it b - **Next step.** A `&KOKKOS_NML` group read in `W3INIT` that sets the same flags, the environment kept as override for tests, and the effective set written once to the log. - **Done in its smallest form (2026-10-18), not yet at the pin.** `WW_KOKKOS_PRESET=resident`, - read by `W3INIT` through the shim (fork `dca49b00` on `ww3-gpu/u2-strict-preset`): the part + read by `W3INIT` through the shim (fork `237ca0de` on `ww3-gpu/u2-strict-preset`): the part list is one line of `kokkos/src/fortran_iface/presets.txt`, which CMake compiles into `part_flags.cpp` and `scripts/ww3-gpu.sh --kokkos resident` and the matrix's `WW3GPU_KOKKOS=resident` read, so they cannot drift. A set `WW_KOKKOS_` still wins, an diff --git a/kokkos/src/fortran_iface/PATCH.md b/kokkos/src/fortran_iface/PATCH.md index 951f15ed..88d01a51 100644 --- a/kokkos/src/fortran_iface/PATCH.md +++ b/kokkos/src/fortran_iface/PATCH.md @@ -1916,7 +1916,7 @@ Four places ran the Fortran for a part asked on Kokkos and said so only in the p `W3KOKKOS_SRCE_SETUP` on a switch file that is not the lab switch's source terms, `W3INIT` without `ST4` and `NL1`, and the per-step `W3SRCE` gate (`IQTPE <= 0`, Liu's ice dispersion, `TAUWBUG = 0` with `IT > 0`). Each now calls `KOKKOS_FALLBACK ( PART, REASON )` -(`w3kokkosmd.F90:1257-1269` at `dca49b00`): +(`w3kokkosmd.F90:1257-1269` at `237ca0de`): - **`WW_KOKKOS_STRICT=1`**: `*** WAVEWATCH III ERROR IN W3KOKKOSMD : WW_KOKKOS_STRICT=1 AND PART WOULD RUN ON THE FORTRAN: ` on `NDSE`, then `EXTCDE(1)`, on every rank. @@ -1927,7 +1927,7 @@ without `ST4` and `NL1`, and the per-step `W3SRCE` gate (`IQTPE <= 0`, Liu's ice line appears only when some part could fall back. **The hot path.** The per-step gate is a new function, `W3KOKKOS_SRCE_STEP ( IT )` -(`w3kokkosmd.F90:1913-1918`), which `W3WAVE` calls once per step in place of +(`w3kokkosmd.F90:1912-1917`), which `W3WAVE` calls once per step in place of `W3KOKKOS_SRCE_ON` before the batch (`w3wavemd.F90:2407`), outside the `!$OMP PARALLEL` loop over sea points. It returns `W3KOKKOS_SRCE_ON ( IT )` and adds one test of two `LOGICAL`s; only on the fallback branch does it call `SRCE_FALLBACK`, which under the default is one call and one @@ -1943,9 +1943,23 @@ when the step's gate was false, so counting the step is complete without a count would fall back now fails at run time instead. This changes how a comparison is invoked and is reviewed by a person before it merges (AGENTS.md, "Never cheat a regtest tool"). +**What it was checked on** (2026-10-18, i9-14900, `OMP_NUM_THREADS=8`, the machine shared with +three other jobs, so every wall time is one run and ⚠). The runs re-ran `ww3_grid`, `ww3_strt`, +`ww3_shel` and `ww3_ounf` in a copy of the smoke case's Kokkos directory with every `WW_KOKKOS_*` +unset and only the variables named: + +| Run | Result | +|---|---| +| `bash scripts/ww3-gpu.sh --ww3-gpu smoke --cpu --kokkos resident`, before (`80644ba9`) and after | every model file byte-identical to the Fortran's, and the after run's four files identical (`cmp`) to the before run's (v); `ww3_shel` with Kokkos 3.96 s before, 3.65 s and 10.93 s after (load average 17 during the second) ⚠ | +| the same smoke, `WW_KOKKOS_ALL=1` | `SRCE=0` in the line at exit (v) | +| smoke with `TAUWBUG = 0` in `&SIN4`, `WW_KOKKOS_ALL=1` | `ww_kokkos: calls left on the Fortran (WW_KOKKOS_STRICT=1 stops instead): SRCE=2160` (v) | +| the same, `WW_KOKKOS_STRICT=1` | stops: `... WW_KOKKOS_STRICT=1 AND PART SRCE WOULD RUN ON THE FORTRAN: TAUWBUG = 0 (SINTAILPAR(4) <= 0.5) WITH IT > 0, WHICH NO TRANSLATION CAN REPRODUCE` (v) | +| `switch_lab_ompg` with `WNT2` for `WNT1` (a scratch copy, not committed), `--kokkos all` | `UWND=1 SRCE=0` at exit; the pair byte-identical (v) | +| the same build, `WW_KOKKOS_ALL=1 WW_KOKKOS_STRICT=1` | stops: `... PART UWND WOULD RUN ON THE FORTRAN: WW_KOKKOS_ALL=1 ASKS FOR IT, BUT ITS KOKKOS TRANSLATION DOES NOT COVER THIS SWITCH FILE` (v) | + ## 25. `WW_KOKKOS_PRESET`: the resident part list in one table, `ww3-gpu/u2-strict-preset` -Fork commit `dca49b00` on `ww3-gpu/u2-strict-preset` (the child of `f0cbe70d`); not at the pin +Fork commit `237ca0de` on `ww3-gpu/u2-strict-preset` (the child of `f0cbe70d`); not at the pin until the integrator merges it. Gap 7 of [`PRODUCTION_READINESS_202610.md`](../../../docs/PRODUCTION_READINESS_202610.md), smallest form. One ABI entry, `ww_kokkos_preset_status`; `w3kokkosmd.F90` +24 -1 lines. @@ -1965,8 +1979,12 @@ One ABI entry, `ww_kokkos_preset_status`; `w3kokkosmd.F90` +24 -1 lines. - **The matrix**: `ww3gpu_kokkos_preset` reads it for `WW3GPU_KOKKOS=`; `selftest.cmake` checks a fixture table and that the real `resident` names only parts of `parts.d`. -`W3KOKKOS_SETUP` (`w3kokkosmd.F90:1182-1188` at `dca49b00`) stops on a name the file lacks: +`W3KOKKOS_SETUP` (`w3kokkosmd.F90:1182-1188` at `237ca0de`) stops on a name the file lacks: `*** WAVEWATCH III ERROR IN W3KOKKOSMD : WW_KOKKOS_PRESET= IS NOT A PRESET OF kokkos/src/fortran_iface/presets.txt`. `W3KOKKOS_REPORT` writes ` Kokkos preset (WW_KOKKOS_PRESET): resident` on the line before the parts line, which stays the effective set, once per `W3INIT`. No namelist: the next step of gap 7. + +Checked the same way as section 24 (2026-10-18): `WW_KOKKOS_PRESET=resident` alone gives the +parts line of `--kokkos resident` character for character and the same four model files (`cmp`, +v); `WW_KOKKOS_PRESET=bogus` stops with the line above (v). From c2a0acfcff7a88657859d66dbef9a83493c57edf Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 13:03:50 -0300 Subject: [PATCH 14/27] docs/compound: plan to pin WW3 to NOAA's production/GFS.v17 The owner asked for a stable WW3 pin. NOAA-EMC/WW3's only release tag is 6.07.1 (2019); NOAA ships through production branches. The owner chose production/GFS.v17 (7a23cd77) in its own PR after #191: every file of the KOKKOS patch and two ported modules changed there, ST4 and the DIA did not. Tested: nix shell nixpkgs#vale -c scripts/vale.sh on the plan, 0 errors; facts from git ls-remote, gh release list and git diff --stat in WW3-noaa/ Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-10-18-004-pin-gfs-v17-plan.md | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md diff --git a/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md b/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md new file mode 100644 index 00000000..b2fd77b7 --- /dev/null +++ b/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md @@ -0,0 +1,78 @@ +--- +title: Pin WW3 to NOAA's production/GFS.v17 branch - Plan +type: feat +date: 2026-10-18 +artifact_contract: ce-unified-plan/v1 +product_contract_source: owner, session of 2026-10-18 (goal abbb0456-6043): "pin WW3 in a latest stable version", GFS.v17 in its own PR after #191 +execution: code +--- + +# Pin WW3 to NOAA's `production/GFS.v17` - Plan + +## Goal Capsule + +- **Objective:** `WW3-noaa/` and the fork's KOKKOS patch sit on the WW3 that NOAA's next + operational GFS runs, so every parity result, timing and regtest of ww3-gpu is about the code + GFS-Wave will use. +- **Authority:** AGENTS.md ("WW3 source is read-only", "Bit for bit", "Translate, don't + improve"); the owner chose `production/GFS.v17` on 2026-10-18, in its own PR after #191. +- **Stop conditions:** a translated routine whose Fortran changed between the two bases and whose + translation no longer matches it; a regtest difference that a Fortran/Fortran run does not + reproduce. Either stops the pin move; it is never solved by keeping old Fortran. +- **Execution profile:** large; its own branch from `main` after PR #191 merges. + +## Why GFS.v17 and not a release tag + +Facts read 2026-10-18 (v, `git ls-remote --tags` and `gh release list -R NOAA-EMC/WW3`, +`git rev-list` in `WW3-noaa/`): + +| Candidate | Commit, date | Against the pin `03adda91` (`develop`, 2026-10-05) | +|---|---|---| +| `develop` | `03adda91`, 2026-10-05 | the pin itself | +| release `6.07.1`, the only tag | 2019-04-19 | 404 commits older; before the CMake build the patch uses | +| `production/GFS.v16.3`, operational GFS-Wave today | `41e7c9ca`, 2025-02-21 | split 2020-03-04; 372 of the pin's commits missing, 18 of its own | +| `production/GFS.v17` | `7a23cd77`, 2026-05-14 | split 2025-11-26 (`9d0cb947`); 143 of its own, 31 of the pin's missing | + +NOAA ships WW3 through `production/*` branches, not release tags. That `GFS.v17` is the next +operational GFS, and its implementation date, are not read in a NOAA notice ⚠ (the `noaa-notices` +skill is the place to check). + +## What moves + +`git diff --stat 03adda91 origin/production/GFS.v17 -- model/src`: 64 files, 11 661 insertions, +2 878 deletions (v). Every file of the KOKKOS patch changed upstream: `CMakeLists.txt`, +`cmake/switches.json`, `w3initmd.F90`, `w3srcemd.F90`, `w3wavemd.F90` (v). So did the Fortran of +two ported routines' modules, `w3pro3md.F90` (`W3XYP3`, `W3KTP3`) and `w3iobcmd.F90` (the +boundary output of U8) (v). `w3src4md.F90` (ST4) and `w3snl1md.F90` (the DIA), 66 % of the +resident wall time, did not change (v, absent from the name list). `wminiomd.F90` and +`w3iotrmd.F90` did not change either. + +## Units + +1. **Read GFS.v17's switch files and record them** (plan 3 of + `2026-10-18-003-gpu-first-supervised-run-plan.md`): `model/bin/switch_NCEP_st4` and the switch + the GFS.v17 workflow builds with ⚠ (where that file lives is not read). Run the `W3SRCE` gate + of gap 1 on it: if `SEED` or `FLD2` is there, the Kokkos `W3SRCE` does not run for GFS-Wave. +2. **Diff the Fortran of every translated routine** between `03adda91` and `7a23cd77`, from + `kokkos/ports.tsv`'s file:line columns. Each routine whose lines changed is listed with the + hunk; its translation is updated in its own commit (translate, don't improve), with the static + parity argument of ADR-0016 redone. +3. **Rebase the fork.** `WW3-noaa/` to `7a23cd77`; a new fork branch from it carrying the KOKKOS + patch (`ww3-gpu/kokkos-pnt` after #191's pin bump), each hunk re-placed, `PATCH.md`'s line + numbers updated. `scripts/ww3-gpu.sh`'s fork-versus-NOAA check must pass unchanged. +4. **Rerun the evidence on the new base:** `smoke`, `full`, `coast` resident pair to pair; the + resident matrix with `WW3GPU_LAB_SWITCH=ON` (the entries of #191's U3); a `ww3_multi` entry; + `WW_KOKKOS_VA_POISON=1` and `WW_KOKKOS_VA_COUNT=1`. Then `port_score.py --write`, and every + page that cites `model/src/.F90:` at `80644ba9` or `03adda91` updated to the new + pin (AGENTS.md says the pin's commit at the time of writing). +5. **Docs:** AGENTS.md, CONTRIBUTING.md and `docs/TOOLCHAIN.md` name the new pin and why it is + a production branch; `README.md` and `README.pt-BR.md` together. + +## Risks + +- `GFS.v17` lacks 31 commits of `develop`; any of them that the KOKKOS patch or a translation + relies on must be found in unit 2 or 3 ⚠ (not read). +- A production branch moves with NOAA's fixes; the pin is a commit, and moving it again is a + deliberate change, as for every submodule here. +- The other session's NOAA study and its `W3SDS4` patch are written against `03adda91`; the + patch is checked again on the new base with `scripts/ww3_fortran_ab.sh`. From ff7fe364d99223ebf92de4b20a7df8df59e5255c Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 13:04:20 -0300 Subject: [PATCH 15/27] docs/compound: plan 004's front matter parses c487900 committed a product_contract_source value with an unquoted ': ', which YAML refuses; vale's failure was hidden by a pipe. Quoted. Tested: nix shell nixpkgs#vale -c scripts/vale.sh on the plan, rc 0, 0 errors Co-Authored-By: Claude Opus 5.5 --- docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md b/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md index b2fd77b7..99602044 100644 --- a/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md +++ b/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md @@ -3,7 +3,7 @@ title: Pin WW3 to NOAA's production/GFS.v17 branch - Plan type: feat date: 2026-10-18 artifact_contract: ce-unified-plan/v1 -product_contract_source: owner, session of 2026-10-18 (goal abbb0456-6043): "pin WW3 in a latest stable version", GFS.v17 in its own PR after #191 +product_contract_source: "owner, session of 2026-10-18 (goal abbb0456-6043), asked for a stable WW3 pin; GFS.v17 in its own PR after #191" execution: code --- From 7a13a2b24a96b354e23b9c9481d718bf38c5f668 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 12:42:26 -0300 Subject: [PATCH 16/27] ww3_multi and the resident VA: column syncs at the exchanges, no read at W3WAVE's return Goal abbb0456-6043, units U1 and U7 of the production single-node plan (gaps 2 and 13 of docs/PRODUCTION_READINESS_202610.md). With the resident VA, ww3_multi's WMIOHG and WMIOEG wrote a grid's VA on the host between W3WAVE calls with no sync; the matrix with the lab switch showed it: WW_KOKKOS_VA_CHECK=1 stops mww3_test_02 to _04, and without it they differ from the Fortran in 5, 10 and 4 files (ww3_self 0). The owner refused a whole-VA copy at W3WAVE's exit, so the fork branch ww3-gpu/u1-multi-va (f330dc53, five commits on 80644ba9; no gitlink change here) syncs only the columns each exchange reads or writes and drops the read at W3WAVE's return. Here: ww_va_host_wrote_cols (ww_kokkos 0.10.0), a column scatter sharing the column lists of the partial read (now up to 8 per grid); PATCH.md section 24 with every host use of VA outside W3WAVE; the fork's wminiomd.F90 and wmwavemd.F90 as FILE lines of parts.d/resident.txt; scripts/ww3-gpu.sh prints the "still newer on the device at finalize" line instead of failing on it, since no read at the return is now the design (a gate change for a person to review); the docs/log entry with before and after; ports rows and reports for WMIOHG and WMIOEG; the glossary rows; gaps 2 and 13 updated. Tested: WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,!PDLIB,!MGP,!MGG' just ww3-gpu-matrix 2 serial -R '^0(33|34|38|43):' and WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,MGP,!PDLIB' just ww3-gpu-matrix 2 omp -R '^00(2|8):', with the fork at f330dc53's tree, plain and with WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_CHECK=1: test.comp 0 files different on all six; scripts/ww3-gpu.sh --ww3-gpu full|smoke --cpu --kokkos resident, plain and poisoned: byte-identical to the Fortran and to --ww3, whole-VA D2H on full 1 -> 0; just kokkos-test serial-debug and openmp-release 11/11; bash scripts/ww3-gpu.sh --self-test ok; shellcheck scripts/ww3-gpu.sh; python3 scripts/port_score.py --write then --check current; scripts/vale.sh on the changed Markdown: 0 errors Co-Authored-By: Claude Opus 5.5 --- docs/GLOSSARY.md | 4 +- docs/PRODUCTION_READINESS_202610.md | 30 ++++- .../ww3gpu_matrix_multi_va_2026-10-18.tsv | 124 +++++++++++++++++ docs/log/2026-10-18-ww3-multi-resident-va.md | 118 ++++++++++++++++ docs/log/README.md | 1 + kokkos/CMakeLists.txt | 6 +- kokkos/PORT_RANKING.md | 2 + kokkos/PORT_STATUS.md | 6 +- kokkos/PORT_TABLE.tsv | 2 + kokkos/port_report/WMIOEG_KOKKOS.md | 35 +++++ kokkos/port_report/WMIOHG_KOKKOS.md | 47 +++++++ kokkos/ports.tsv | 2 + kokkos/src/fortran_iface/PATCH.md | 101 ++++++++++++++ kokkos/src/fortran_iface/parts.d/resident.txt | 4 +- kokkos/src/fortran_iface/resident_c.hpp | 14 +- kokkos/src/fortran_iface/resident_shim.cpp | 127 ++++++++++++++---- scripts/ww3-gpu.sh | 8 +- site/data/port_score.tsv | 10 +- 18 files changed, 596 insertions(+), 45 deletions(-) create mode 100644 docs/data/ww3gpu_matrix_multi_va_2026-10-18.tsv create mode 100644 docs/log/2026-10-18-ww3-multi-resident-va.md create mode 100644 kokkos/port_report/WMIOEG_KOKKOS.md create mode 100644 kokkos/port_report/WMIOHG_KOKKOS.md diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index a66e5ebc..c0144de7 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -484,7 +484,8 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | AINIT / AINIT2 / INFLAGS1 | "Flag for array initialization" (`AINIT(2)`) and "Array consolidating the above six" input flags `(v)` `w3adatmd.F90:318`, `w3idatmd.F90:99` | The conditions under which `W3SETA` re-points its alias groups. | [kokkos/port_report/W3SETA_KOKKOS.md](../kokkos/port_report/W3SETA_KOKKOS.md) | | NAPROC / UNDEF | "Number of processors for computation" / "Value for undefined parameters in gridded output fields" `(v)` `w3odatmd.F90:102`, `:82-83` | `W3XDMA` sizes the full-grid X arrays as `NSEALM*NAPROC` and fills most with `UNDEF`. | [kokkos/port_report/W3XDMA_KOKKOS.md](../kokkos/port_report/W3XDMA_KOKKOS.md) | | OUTFLAGS / NXXX / X* arrays | `W3XDMA`'s `LOGICAL OUTFLAGS(NOGRP,NGRPP)` argument, its local full-grid size `NSEALM*NAPROC`, and the 93 full-grid output copies of `WADAT` (`XHS` ... `XUSERO`) `(v)` `w3adatmd.F90:1555`, `:1560`, `:1596`, `:414-497` | Allocated by `W3XDMA`, gathered into by `DO_OUTPUT_EXCHANGES`, named by `W3XETA`'s aliases; `XUSERO`'s shape follows `OUTFLAGS(10,2)` alone, a WW3 defect kept as written. | [kokkos/port_report/W3XDMA_KOKKOS.md](../kokkos/port_report/W3XDMA_KOKKOS.md) | -| WMWAVE / WMIOHG / WMIOEG | "Run multi-grid version of WAVEWATCH III" / "Gather internal high-to-low data for a given model" / "Gather internal same-rank data for a given model" `(v)` `model/src/wmwavemd.F90:85`, `wminiomd.F90:1714`, `:2814` | `ww3_multi`'s driver and two of its exchanges between grids; the exchanges write a grid's `VA` on the host between its `W3WAVE` calls, which the resident spectrum's sync calls do not see. | [docs/PRODUCTION_READINESS_202610.md](PRODUCTION_READINESS_202610.md) | +| WMWAVE / WMIOHG / WMIOEG | "Run multi-grid version of WAVEWATCH III" / "Gather internal high-to-low data for a given model" / "Gather internal same-rank data for a given model" `(v)` `model/src/wmwavemd.F90:85`, `wminiomd.F90:1714`, `:2814` | `ww3_multi`'s driver and two of its exchanges between grids; the exchanges write a grid's `VA` on the host between its `W3WAVE` calls; with the resident `VA` they store the columns they wrote into the device copy (`kokkos/src/fortran_iface/PATCH.md`, section 26). | [docs/PRODUCTION_READINESS_202610.md](PRODUCTION_READINESS_202610.md) | +| WMIOBS / WMIOHS / WMIOES / W3MPIP | "Stage internal boundary data in the data structure BPSTGE" / "Stage internal high-to-low data in the data structure HGSTGE" / "Stage internal same-rank data in the data structure EQSTGE" / "Prepare MPI persistent communication needed for WAVEWATCH I/O routines" `(v)` `WW3-noaa/model/src/wminiomd.F90:121`, `:1402`, `:2516`, `w3initmd.F90:5412` at `03adda91` | The sending side of `ww3_multi`'s exchanges, which read a grid's `VA` on the host between its `W3WAVE` calls, and the set-up of the MPI requests that send point-output spectra from `VA`; with the resident `VA` each reads its columns back first (`kokkos/src/fortran_iface/PATCH.md`, section 26). | [docs/log/2026-10-18-ww3-multi-resident-va.md](log/2026-10-18-ww3-multi-resident-va.md) | | W3FLDG / W3FLDO | "Update input fields in the WAVEWATCH III generic shell from a WAVEWATCH III shell data file or write from preprocessor" / "Open and prepare WAVEWATCH III field files as used by the generic shell and the field preprocessor" `(v)` `w3fldsmd.F90:964-965`, `:116-117` | `W3FLDG` reads or writes one time record and the field records of `wind.ww3`, `level.ww3`, `current.ww3` and the others, gfortran unformatted big-endian; `W3FLDO` opens the unit and writes or checks its 40-byte header record (file id, field id, grid dimensions, filler, tide flag). | [kokkos/port_report/W3FLDG_KOKKOS.md](../kokkos/port_report/W3FLDG_KOKKOS.md), [kokkos/port_report/W3FLDO_KOKKOS.md](../kokkos/port_report/W3FLDO_KOKKOS.md) | ## HPC and software @@ -2147,6 +2148,7 @@ Every term above, alphabetically, with the section it lives in. - wm*md.F90 (WMESMFMD: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - wm*md.F90 (WMESMFMD, WMSCRPMD, …): [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - WMSCRPMD: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) +- WMIOBS / WMIOHS / WMIOES / W3MPIP: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - WMWAVE / WMIOHG / WMIOEG: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - WW3GPU_LAB_SWITCH: [This repository's own names](#this-repositorys-own-names) - WW_KOKKOS_STRICT / WW_BUILD_TESTS (proposed): [This repository's own names](#this-repositorys-own-names) diff --git a/docs/PRODUCTION_READINESS_202610.md b/docs/PRODUCTION_READINESS_202610.md index 6ffffa97..66e6dc88 100644 --- a/docs/PRODUCTION_READINESS_202610.md +++ b/docs/PRODUCTION_READINESS_202610.md @@ -38,7 +38,7 @@ regtest matrix yet"). 10 of the 310 rows of `kokkos/ports.tsv` are regtested and | # | Gap | (a) | (b) | Smallest next step | |---|---|---|---|---| | 1 | No regtest switch has the source-term set the Kokkos `W3SRCE` translates; NOAA's UFS regtests miss it too | Blocker | Blocker | Read GFS-Wave's operational switch file at its tag; run the `WW3GPU_LAB_SWITCH` matrix on the MPI and `ww3_multi` entries | -| 2 | `ww3_multi` writes `VA` on the host between `W3WAVE` calls with no sync | Blocker | Blocker | Mark the host newer when `W3WAVE` returns; run `mww3_test_0x` with `WW_KOKKOS_VA_CHECK=1` | +| 2 | `ww3_multi` writes `VA` on the host between `W3WAVE` calls with no sync; fixed under `SHRD` on 2026-10-18 by column syncs at the exchanges (section 2) | Minor ⚠ | Minor ⚠ | Move the pin to the fork branch; `mww3_test_08` under MPI when the owner runs MPI | | 3 | MPI: four of the six resident parts are `SHRD` only, the preset stops on `DIST MPI`, the communicator is ignored, the device id is not per rank | Blocker | Major | A resident preset without `XYP3`, `OUTG`, `IOPE`, `UPDV`; one run on `switch_lab_mpi` | | 4 | Bit parity measured with gfortran and g++ only; the operational compiler and the GPU are not measured | Blocker | Major | Static argument for the Intel build's math calls; one smoke with it | | 5 | Build and packaging: a separate static library, C++20 and CMake 3.25; the install-only configure and the exact Kokkos check are done (U6) | Major | Major | A shared-library build and a recipe for the module systems of an operational centre | @@ -49,7 +49,7 @@ regtest matrix yet"). 10 of the 310 rows of `kokkos/ports.tsv` are regtested and | 10 | Restart with the resident spectrum: covered by the sync calls, never regtested | Major | Minor | The restart-reproduction pair of `ww3_ufs1.1` (`work_c` against `work_d`) under the lab switch | | 11 | `PDLIB` and unstructured grids: translated, not wired, gated out | Minor ⚠ | Major | None for GFS-Wave v16; a wiring unit for `PDLIB` when the owner asks | | 12 | The fork patch: 4 082 lines of new Fortran, 819 changed lines in three WW3 files, 58 218 lines of C++ in another repository, most of it LGPL-3.0-or-later, some MIT, 76 files with no licence line | Minor | Blocker | ADR-0018 (Proposed) names the three places `ww_kokkos` could live; the owner chooses, after a legal reading ⚠ | -| 13 | A stale contract remark: "one spectral grid per process" | Minor | Minor | Edit the remark in the fork | +| 13 | A stale contract remark: "one spectral grid per process"; edited on the fork branch `ww3-gpu/u1-multi-va` (section 13) | Minor | Minor | Move the pin | The sections below give the evidence of each gap. The ranked top five closes the page. @@ -131,6 +131,24 @@ per grid too (v, `snl1_shim.cpp:83-89`), so several grids in one process are sup costs one host-to-device copy of `VA` at the next call; run the `mww3_test_01` to `_03` entries with `WW_KOKKOS_VA_CHECK=1` under the lab switch. +**Update, 2026-10-18 (plan 002, U1).** Reproduced and fixed under `SHRD` +([log](log/2026-10-18-ww3-multi-resident-va.md), `PATCH.md` section 26). With the lab switch +and the resident preset, `WW_KOKKOS_VA_CHECK=1` stopped `mww3_test_02`, `_03` and `_04` +(`grdset_a`), and without it they differed from the Fortran in 5, 10 and 4 files, with +`ww3_self` 0 (v). The step above was not taken: on a GPU it adds a whole copy of `VA` per grid +per call, which the owner refused. Instead each `ww3_multi` routine that reads or writes `VA` +between `W3WAVE` calls (`WMIOBS`, `WMIOHS`, `WMIOES`, `WMIOHG`, `WMIOEG`, and `WMWAVE`'s +unified point output) syncs only the columns of its own loop, and `W3WAVE` no longer reads +`VA` back at its return, since `ww3_shel` reads none after it. This adds `wminiomd.F90` and +`wmwavemd.F90` to the fork's diff. After, the six `mww3_test_01` to `_05` entries gave +`test.comp` 0 files different, also with the host copy poisoned, and no entry's whole-array +copies went up: `_04` went from 864 to 0 device-to-host copies, and `ww3_shel` full from 1 to 0 +(v). Left open: `mww3_test_08` (MPI), whose six differing files the same defect explains by +reading ⚠, not run (ADR-0017); the fork branch waits for the pin bump. + +- **(a), (b) now Minor** under `SHRD` ⚠: shown on the four `PR3 UQ` `grdset_a`-type entries and + one moving grid, with the lab switch's source terms, not on GFS-Wave's grids. + ## 3. MPI *Scope, 2026-10-18:* out of scope by decision. The port is single node, and an MPI build keeps @@ -397,6 +415,10 @@ grid's tables". The shim keeps one context per grid index in a map (v, `snl1_shi `W3INIT` calls `WW_SNL1_INIT` with `IMOD` (v, `w3initmd.F90:764`). The remark describes phase 1 and misleads a reviewer of (b). Remark b of the same block, on concurrent calls, is still true. +**Update, 2026-10-18 (plan 002, U7).** Remark a now says one context per grid, keyed by the grid +index, with `W3INIT` passing `IMOD`; remark b is kept (v, fork commit `f330dc53` on +`ww3-gpu/u1-multi-va`, `w3kokkosmd.F90:99-104`). Closed once the pin moves. + ## Ranked top five Ranked by what blocks both targets first, then by how much of GFS-Wave's run it decides. @@ -408,7 +430,9 @@ Ranked by what blocks both targets first, then by how much of GFS-Wave's run it 2. **`ww3_multi` writes `VA` behind the resident copy.** `wminiomd.F90:2264`, `:3261-3330` write it between `W3WAVE` calls; `W3WAVE` returns with "both equal" (`w3wavemd.F90:3153`, `resident_shim.cpp:221-240`). GFS-Wave is a `ww3_multi` mosaic. Next: a host write at - `W3WAVE`'s return, then `mww3_test_0x` with `WW_KOKKOS_VA_CHECK=1`. + `W3WAVE`'s return, then `mww3_test_0x` with `WW_KOKKOS_VA_CHECK=1`. Update, 2026-10-18: + fixed under `SHRD` with column syncs at the exchanges and no read at the return, the + `mww3_test_01` to `_05` entries byte-identical (section 2); `mww3_test_08` (MPI) not run. 3. **The resident preset cannot run under `DIST MPI`.** `XYP3`, `OUTG`, `IOPE`, `UPDV` are `SHRD` only and the preset sets them explicitly, which stops the model (`w3kokkos_prop3md.F90:127-129`, `w3kokkosmd.F90:1197-1201`); the batched `W3SRCE` never ran diff --git a/docs/data/ww3gpu_matrix_multi_va_2026-10-18.tsv b/docs/data/ww3gpu_matrix_multi_va_2026-10-18.tsv new file mode 100644 index 00000000..4a3395de --- /dev/null +++ b/docs/data/ww3gpu_matrix_multi_va_2026-10-18.tsv @@ -0,0 +1,124 @@ +# The regtest matrix summaries of docs/log/2026-10-18-ww3-multi-resident-va.md, as written by +# record:summary, one block per run. The VA diagnostics were set in the environment and the +# summary does not record them: each block's first line names them. Before is WW3 80644ba9; +# after is WW3 8f6ba104, whose tree equals ww3-gpu/u1-multi-va f330dc53 (git diff --stat empty). +# +# === before_res_check: WW_KOKKOS_VA_CHECK=1 WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T14:53:39Z +# ww3-gpu commit: 4f87fec (dirty) +# WW3 commit: 80644ba9 (origin/ww3-gpu/kokkos-pnt) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 serial -R ^0(33|34|38|43): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +33 mww3_test_01 PR3_UQ yes yes yes 0/22 PASS hs 28 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ +34 mww3_test_02 PR3_UQ yes fortran-only yes n/a n/a - 41 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +38 mww3_test_03 PR3_UQ yes fortran-only yes n/a n/a - 46 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +43 mww3_test_04 PR3_UQ yes fortran-only yes n/a n/a - 74 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +# +# === before_res_nocheck: WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:01:19Z +# ww3-gpu commit: 4f87fec (dirty) +# WW3 commit: 80644ba9 (origin/ww3-gpu/kokkos-pnt) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 serial -R ^0(33|34|38|43): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +33 mww3_test_01 PR3_UQ yes yes yes 0/22 PASS hs 31 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ +34 mww3_test_02 PR3_UQ yes yes yes 5/23 FAIL dp,fp,hs 61 SRCE:0,SDS4:0,SIN4:0,SPR4:0 0/23 -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +38 mww3_test_03 PR3_UQ yes yes yes 10/34 FAIL dp,fp,hs 72 SRCE:0,SDS4:0,SIN4:0,SPR4:0 0/34 -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +43 mww3_test_04 PR3_UQ yes yes yes 4/25 FAIL dir,dp,fp,hs 150 SRCE:0,SDS4:0,SIN4:0,SPR4:0 0/25 -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +# +# === before_sub_check: WW_KOKKOS_VA_CHECK=1 WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:07:44Z +# ww3-gpu commit: 4f87fec (dirty) +# WW3 commit: 80644ba9 (origin/ww3-gpu/kokkos-pnt) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,MGP,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 omp -R ^00(2|8): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words PR3 UQ SHRD MGP !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV = SRCE,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +2 mww3_test_05 ST1_PR3_UQ_OMP yes fortran-only yes n/a n/a - 184 SRCE:10147,SDS4:0,SIN4:0,SPR4:0 - -s ST1_PR3_UQ_OMP -w work_ST1_PR3_UQ_OMP -m grdset_a -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes fortran-only yes n/a n/a - 181 SRCE:10147,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np +# +# === before_sub_nocheck: WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:11:49Z +# ww3-gpu commit: 4f87fec (dirty) +# WW3 commit: 80644ba9 (origin/ww3-gpu/kokkos-pnt) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,MGP,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 omp -R ^00(2|8): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words PR3 UQ SHRD MGP !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV = SRCE,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +2 mww3_test_05 ST1_PR3_UQ_OMP yes yes yes 0/27 PASS hs 210 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST1_PR3_UQ_OMP -w work_ST1_PR3_UQ_OMP -m grdset_a -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes yes yes 0/27 PASS hs 217 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np +# +# === after_res_poison: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_CHECK=1 WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:47:56Z +# ww3-gpu commit: 1ebd62f +# WW3 commit: 8f6ba104 () +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 serial -R ^0(33|34|38|43): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +33 mww3_test_01 PR3_UQ yes yes yes 0/22 PASS hs 31 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ +34 mww3_test_02 PR3_UQ yes yes yes 0/23 PASS dp,fp,hs 48 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +38 mww3_test_03 PR3_UQ yes yes yes 0/34 PASS dp,fp,hs 57 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +43 mww3_test_04 PR3_UQ yes yes yes 0/25 PASS dir,dp,fp,hs 121 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +# +# === after_res_nocheck: WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:54:32Z +# ww3-gpu commit: 1ebd62f +# WW3 commit: 8f6ba104 () +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 serial -R ^0(33|34|38|43): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +33 mww3_test_01 PR3_UQ yes yes yes 0/22 PASS hs 34 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ +34 mww3_test_02 PR3_UQ yes yes yes 0/23 PASS dp,fp,hs 43 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +38 mww3_test_03 PR3_UQ yes yes yes 0/34 PASS dp,fp,hs 56 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +43 mww3_test_04 PR3_UQ yes yes yes 0/25 PASS dir,dp,fp,hs 120 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +# +# === after_sub_poison: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_CHECK=1 WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T15:51:48Z +# ww3-gpu commit: 1ebd62f +# WW3 commit: 8f6ba104 () +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,MGP,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 omp -R ^00(2|8): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words PR3 UQ SHRD MGP !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV = SRCE,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +2 mww3_test_05 ST1_PR3_UQ_OMP yes yes yes 0/27 PASS hs 200 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST1_PR3_UQ_OMP -w work_ST1_PR3_UQ_OMP -m grdset_a -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes yes yes 0/27 PASS hs 199 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np +# +# === after_sub_nocheck: WW_KOKKOS_VA_COUNT=1 +# date: 2026-10-18T16:00:52Z +# ww3-gpu commit: 1ebd62f +# WW3 commit: 8f6ba104 () +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,MGP,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 omp -R ^00(2|8): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words PR3 UQ SHRD MGP !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV = SRCE,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +2 mww3_test_05 ST1_PR3_UQ_OMP yes yes yes 0/27 PASS hs 345 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST1_PR3_UQ_OMP -w work_ST1_PR3_UQ_OMP -m grdset_a -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes yes yes 0/27 PASS hs 347 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np diff --git a/docs/log/2026-10-18-ww3-multi-resident-va.md b/docs/log/2026-10-18-ww3-multi-resident-va.md new file mode 100644 index 00000000..aa96e367 --- /dev/null +++ b/docs/log/2026-10-18-ww3-multi-resident-va.md @@ -0,0 +1,118 @@ +# `ww3_multi` and the resident `VA`: the defect reproduced, column syncs at the exchanges, no read at `W3WAVE`'s return + +2026-10-18, an agent for M. Hoffmann (goal `abbb0456-6043`); unit U1 + U7 of +[the production single-node plan](../compound/plans/2026-10-18-002-production-single-node-plan.md), +gaps 2 and 13 of [`PRODUCTION_READINESS_202610.md`](../PRODUCTION_READINESS_202610.md), +[WFIP-0005](../WFIPs/WFIP-0005-device-resident-step.md). The fork change is +[`PATCH.md` section 26](../../kokkos/src/fortran_iface/PATCH.md#26-ww3_multi-and-the-resident-va-column-syncs-no-read-at-w3waves-return-ww3-gpuu1-multi-va). + +## Question + +Does `ww3_multi` change a grid's `VA` on the host between two `W3WAVE` calls behind the resident +device copy, as gap 2 read from the code? If so, can the syncs sit where `ww3_multi` reads and +writes `VA`, moving only those columns, with the whole-array read at `W3WAVE`'s return removed, +and still give the Fortran's bytes? + +## What was done + +On the owner's Intel Core i9-14900 (32 hardware threads, shared with three other sessions; at +most 8 threads used: 2 matrix jobs of 4), in the pinned `ww3` shell (gfortran and g++ 16.2.0), +Kokkos OpenMP backend, never `--gpu`. + +1. Before, `WW3/` at `80644ba9`: the `ww3_multi` entries that are `SHRD` with `PR3 UQ`, with the + lab switch's source terms, the resident preset and the VA diagnostics in the environment: + + ```bash + WW_KOKKOS_VA_CHECK=1 WW_KOKKOS_VA_COUNT=1 WW3GPU_OMP_THREADS=4 WW3GPU_LAB_SWITCH=ON \ + WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,!PDLIB,!MGP,!MGG' \ + just ww3-gpu-matrix 2 serial -R '^0(33|34|38|43|61):' + ``` + + then the same without `WW_KOKKOS_VA_CHECK`, and the moving-grid entries of `mww3_test_05`, + where `XYP3` and `KTP3` refuse (`w3kokkos_prop3md.F90:123-133`): `WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV + WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,MGP,!PDLIB' just ww3-gpu-matrix 2 omp -R '^00(2|8):'`, with + and without the check. The task's subset `SRCE,KTP3,OUTG,IOPE,UPDV` stops in `W3INIT` there, + `KTP3` refusing the moving grid (v, `WW_KOKKOS_KTP3=1, BUT ITS KOKKOS TRANSLATION DOES NOT COVER + THIS SWITCH FILE`), so `KTP3` was dropped from it. The before runs used `kokkos/` with the new + entry point already added; `80644ba9` does not call it. +2. The change, on the fork branch `ww3-gpu/u1-multi-va` (`3bbd1557` to `f330dc53`), and the shim's + `ww_va_host_wrote_cols` here (`PATCH.md` section 26). A first plan (B1) marked the host newer at + `W3WAVE`'s return; the owner rejected it the same day, since on a GPU it adds a whole copy of + `VA` per grid per call, and asked that the read at the return go too. +3. After, `WW3/` at `8f6ba104`, whose tree equals `f330dc53` (`diff --stat` empty): the same + entries with `WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_CHECK=1` and plain, all with + `WW_KOKKOS_VA_COUNT=1` (the script `run_set.sh` of the session, which only sets these + variables around the two commands above). +4. `ww3_shel`, before and after: `WW_KOKKOS_VA_COUNT=1 OMP_NUM_THREADS=8 JOBS=8 bash + scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos resident`, and after also `smoke`, and both with + `WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_CHECK=1`. +5. Without `KOKKOS`: `gfortran -E -P` of `w3wavemd.F90`, `wminiomd.F90`, `wmwavemd.F90` of the + fork and of `WW3-noaa/` with every word of `switches/switch_lab_shrd`, then of + `mww3_test_08`'s `switch_ST4_PR3_UQ_MPI`, blank lines dropped. With `KOKKOS` and that MPI + switch: `cmake -S -B build/mpi_check/b -DSWITCH= -DWW_KOKKOS=ON + -DCMAKE_PREFIX_PATH=kokkos/build/openmp-release/install` and `cmake --build build/mpi_check/b + -j 8 --target ww3_multi`. +6. `just kokkos-test serial-debug`; `bash scripts/ww3-gpu.sh --self-test`; `shellcheck + scripts/ww3-gpu.sh`. + +The summaries are in +[`ww3gpu_matrix_multi_va_2026-10-18.tsv`](../data/ww3gpu_matrix_multi_va_2026-10-18.tsv). + +## Result + +- (v) Gap 2 is real. Before, with `WW_KOKKOS_VA_CHECK=1`, `mww3_test_02`, `_03` and `_04` + (`grdset_a`) stop: "2 columns of the host VA of grid 1 changed since the last sync, and no + W3KOKKOS_VA_HOST_WRITE declared it" at a kernel's sync (02), 110 columns of grid 1 at a host read + (03), 240 columns of grid 2 at a kernel's sync (04). Without the check they run and differ: + `test.comp` 5/23, 10/34 and 4/25 files different, `ww3_self` 0 each (a Fortran/Fortran run + gives the same bits), `nccmp-tol` FAIL; in 02 `out_grd.coarse`, `out_pnt.coarse`, the + spectra, track and field NetCDF. `mww3_test_01` (one grid) passes, 0/22. +- (v) On the moving grid of `mww3_test_05` (`SRCE,OUTG,IOPE,UPDV`), the check stops both entries + (138 columns of grid 2, at a host write), and without it both pass, 0/27: the host write before + the Fortran propagation declares the host newer before any kernel reads `d_va`, so the defect is + hidden there, as gap 2 said it might be. +- (v) After: all six entries give `test.comp` 0 files different, plain, with the check, and with + the host copy poisoned, every part shown as `=K` in the parts line. +- (v) Whole-`VA` copies device to host per run, before → after (`WW_KOKKOS_VA_COUNT=1`, plain runs): + `mww3_test_01` 24 → 12, `_02` 384 → 8, `_03` 216 → 216, `_04` 864 → 0, `_05` 156 → 156; none + went up. In their place the exchanges moved columns: in `_04`, 384 372 columns read + (737.99 MB) and 71 388 stored (137.07 MB) over the run, against 3 002.99 MB of whole copies + before. The 216 of `_03` are the `W3OUTG` fallback inside `W3WAVE` (0 Kokkos `W3OUTG` calls in + that run), and the 156 of `_05` the Fortran propagation's host write: both inside `W3WAVE`. +- (v) `ww3_shel` full, after, plain and poisoned: every model file byte-identical to the Fortran's, + every file but the NetCDF to `--ww3`'s, `nccmp-tol` PASS; whole-`VA` D2H 1 → 0 (21.21 MB less + per run), H2D 1 → 1 (the bind). Smoke: the same verdicts, D2H 37 → 37 (the output readers of + `PATCH.md` section 23). The run ends with the device copy newer; `scripts/ww3-gpu.sh` failed on + that line, as mutant A2 of the mutation audit had made it, and now prints it. +- ⚠ Wall time of full, one run each on a shared machine: 40.86 s before, 43.99 s and 32.35 s after + (the second run poisoned). The extra column copies of `ww3_multi` were not timed; no conclusion. +- (v) Without `KOKKOS`, the three files preprocess to `WW3-noaa`'s text with both switches (the + include path in one comment aside). With `KOKKOS` and the MPI switch, `ww3_multi` builds; it + was not run (ADR-0017). `just kokkos-test serial-debug`: 11/11; the self-test and shellcheck + pass. +- (v) `mww3_test_07` was not compared: with the lab switch its `ww3_grid` stops in the Fortran + run too, reading its `SDS4` namelist. + +**`mww3_test_08`** (`ww3_multi`, `MPI`, `SRCE,KTP3`), read and not run (ADR-0017). Its +`ST4_PR3_UQ_MPI` entry uses `input/ww3_multi_grdset_a.inp`: grids `outer` rank 1 and `inner` +rank 2 in one group, so the inner grid feeds the outer one (`WMIOHS` reads the inner `VA`, +`WMIOHG` writes the outer `VA`) and the outer grid sends boundary spectra to the inner one +(`WMIOBS`). Under `DIST MPI` the batched `W3SRCE` and `W3KTP3` run on the device copy and +`XYP3` does not (v, `docs/log/2026-10-18-resident-mpi-lab-switch.md`). At `80644ba9`: `W3WAVE` +of the outer grid returns with "both equal", `WMIOHG` writes its overlap columns on the host, and +in the next step `W3KTP3` on the device reads `d_va` without them; the host write before the +Fortran `W3XYP3` (`w3wavemd.F90:2037`) then copies `d_va` back over the host and the nesting +update is lost. The inner grid's boundary, taken from those outer columns by `WMIOBS`, differs. +This is the defect fixed here, and it matches the six differing files there, all of the inner +grid ⚠: why the outer grid's own outputs did not differ (its overlap points are masked by +`FLGHG1`/`FLGHG2`) was not traced, and the entry was not run with the fix. + +## What it changes + +- Gap 2's blocker for `ww3_multi` under `SHRD` is closed on these entries; the fork branch + `ww3-gpu/u1-multi-va` waits for the integrator's pin bump. `ww_kokkos` is 0.10.0. +- `mww3_test_08` is the regtest that would show the same fix under MPI, when the owner runs MPI. +- Left inside `W3WAVE`, for the output-reader units: the `W3OUTG` fallback's whole read per step + (`mww3_test_03`), and the Fortran propagation's host write per step when `XYP3` refuses. +- The change to `scripts/ww3-gpu.sh` turns a gate's failure into a printed line; a person reviews + it before it merges (`AGENTS.md`, "Never cheat a regtest tool"). diff --git a/docs/log/README.md b/docs/log/README.md index 2e6ac2b4..5df6b9fe 100644 --- a/docs/log/README.md +++ b/docs/log/README.md @@ -69,3 +69,4 @@ Date, author, issue/PR/WFIP. | 2026-10-18 | [Five bit-neutral speed-ups of the resident preset: three built, two checked, one Fortran change tested](2026-10-18-resident-five-speedups.md) | #42, WFIP-0005 | | 2026-10-18 | [The CUDA build of `ww_kokkos` and of the fork for the resident preset, build only: 51 nvcc warnings, none in a wired part](2026-10-18-cuda-build-only.md) | plan 2026-10-18-002 U5, ADR-0014 | | 2026-10-18 | [The boundary and track outputs read only their columns of the resident `VA`](2026-10-18-resident-output-readers.md) | #42, WFIP-0005 | +| 2026-10-18 | [`ww3_multi` and the resident `VA`: the defect reproduced, column syncs at the exchanges, no read at `W3WAVE`'s return](2026-10-18-ww3-multi-resident-va.md) | plan 2026-10-18-002 U1 + U7, WFIP-0005 | diff --git a/kokkos/CMakeLists.txt b/kokkos/CMakeLists.txt index c1906757..0917357f 100644 --- a/kokkos/CMakeLists.txt +++ b/kokkos/CMakeLists.txt @@ -18,7 +18,7 @@ if(WW_KOKKOS_BUILD_TESTS) else() set(ww_languages CXX) endif() -project(ww3_gpu_kokkos VERSION 0.9.0 LANGUAGES ${ww_languages}) +project(ww3_gpu_kokkos VERSION 0.10.0 LANGUAGES ${ww_languages}) set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED ON) @@ -100,8 +100,8 @@ install(EXPORT ww_kokkosTargets NAMESPACE ww_kokkos:: # architecture are written into the config, which refuses any other (cmake/ww_kokkosConfig.cmake.in). configure_file(cmake/ww_kokkosConfig.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/ww_kokkosConfig.cmake @ONLY) # Entry points are only added, never removed or changed, so an install satisfies any -# request of the same major version up to its own (0.9.0 accepts find_package(ww_kokkos -# 0.3), 0.4, 0.5, 0.6, 0.7 and 0.8). Removing or changing an entry point needs a major bump (PATCH.md, "The fork today"). +# request of the same major version up to its own (0.10.0 accepts find_package(ww_kokkos +# 0.3), 0.4, 0.5, 0.6, 0.7, 0.8 and 0.9). Removing or changing an entry point needs a major bump (PATCH.md, "The fork today"). write_basic_package_version_file(${CMAKE_CURRENT_BINARY_DIR}/ww_kokkosConfigVersion.cmake COMPATIBILITY SameMajorVersion) install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ww_kokkosConfig.cmake diff --git a/kokkos/PORT_RANKING.md b/kokkos/PORT_RANKING.md index 7a42367d..b5207a61 100644 --- a/kokkos/PORT_RANKING.md +++ b/kokkos/PORT_RANKING.md @@ -384,3 +384,5 @@ The record is the `regtested` column of `kokkos/ports.tsv`. | `w3pro2md:w3xyp2` | ported | `prop_xyp2.hpp`, `prop_params2.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist | | `w3pro2md:w3ktp2` | ported | `prop_ktp2.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist | | `w3profsmd:w3xypug` | ported | `prop_xypug.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once the four scheme hooks, a shim and the W3WAVE wiring exist | +| `wminiomd:wmiohg` | ported | `resident_shim.cpp` | yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), WW3GPU_LAB_SWITCH=ON matrix entries mww3_test_01 to _04 (resident) and mww3_test_05 (SRCE,OUTG,IOPE,UPDV), test.comp 0 files different, plain and with WW_KOKKOS_VA_POISON=1 (docs/log/2026-10-18-ww3-multi-resident-va.md) | +| `wminiomd:wmioeg` | ported | `resident_shim.cpp` | yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), the entries of WMIOHG, test.comp 0 files different; whether they reach 2.b was not checked | diff --git a/kokkos/PORT_STATUS.md b/kokkos/PORT_STATUS.md index 66fd9add..e073c1fe 100644 --- a/kokkos/PORT_STATUS.md +++ b/kokkos/PORT_STATUS.md @@ -45,11 +45,11 @@ Profile: full (bench_small, 120 x 80, 9 520 sea points, 12 h), Release -pg, swit | level | run time | lines | routines | files | what it counts (WW3@03adda91) | |---|---|---|---|---|---| -| translated | **99.3 %** | 67699 of 263375 (25.70 %) | 310 of 1023 (30.30 %) | 63 of 147 | a Kokkos translation exists in kokkos/src | +| translated | **99.3 %** | 67711 of 263375 (25.71 %) | 312 of 1023 (30.50 %) | 64 of 147 | a Kokkos translation exists in kokkos/src | | ported | **99.3 %** | 17022 of 263375 (6.46 %) | 60 of 1023 (5.87 %) | 23 of 147 | WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag | -| regtested | **48.6 %** | 2677 of 263375 (1.02 %) | 10 of 1023 (0.98 %) | 5 of 147 | WW3's regtests ran with it on Kokkos and agreed with the Fortran | +| regtested | **48.6 %** | 2689 of 263375 (1.02 %) | 12 of 1023 (1.17 %) | 6 of 147 | WW3's regtests ran with it on Kokkos and agreed with the Fortran | -The port itself: 58497 lines in 347 files, the C++ under kokkos/src (every .cpp and .hpp line). +The port itself: 58327 lines in 347 files, the C++ under kokkos/src (every .cpp and .hpp line). On a GPU: none yet. The CUDA build links; the RTX 4090 run, `just smoke-ww3-gpu gpu`, is the owner's, supervised. diff --git a/kokkos/PORT_TABLE.tsv b/kokkos/PORT_TABLE.tsv index f06dee59..d96daf12 100644 --- a/kokkos/PORT_TABLE.tsv +++ b/kokkos/PORT_TABLE.tsv @@ -408,3 +408,5 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3wavset:trig_compute_lh_stress model/src/w3wavset.F90:448-584 — — translated — yes: kokkos/src/ww_kokkos/wavset_trig_compute_lh_stress.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/TRIG_COMPUTE_LH_STRESS_KOKKOS.md — w3wavset:trig_wave_setup_apply_fct model/src/w3wavset.F90:1020-1107 — — translated — yes: kokkos/src/ww_kokkos/wavset_trig_wave_setup_apply_fct.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/TRIG_WAVE_SETUP_APPLY_FCT_KOKKOS.md — w3wavset:trig_wave_setup_apply_precond model/src/w3wavset.F90:892-1005 — — translated — yes: kokkos/src/ww_kokkos/wavset_trig_wave_setup_apply_precond.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/TRIG_WAVE_SETUP_APPLY_PRECOND_KOKKOS.md +— wminiomd:wmioeg model/src/wminiomd.F90:3258-3262 — — regtested — yes: kokkos/src/fortran_iface/resident_shim.cpp no: called on the fork branch ww3-gpu/u1-multi-va (wminiomd.F90:3341-3363, :3457-3463 at f330dc53); the WW3/ pin moves with the integrator passed yes not recorded — none: ADR-0010 kokkos/port_report/WMIOEG_KOKKOS.md +— wminiomd:wmiohg model/src/wminiomd.F90:2260-2266 — — regtested — yes: kokkos/src/fortran_iface/resident_shim.cpp no: called on the fork branch ww3-gpu/u1-multi-va (wminiomd.F90:2318-2327 at f330dc53); the WW3/ pin moves with the integrator passed yes not recorded — none: ADR-0010 kokkos/port_report/WMIOHG_KOKKOS.md diff --git a/kokkos/port_report/WMIOEG_KOKKOS.md b/kokkos/port_report/WMIOEG_KOKKOS.md new file mode 100644 index 00000000..f6456693 --- /dev/null +++ b/kokkos/port_report/WMIOEG_KOKKOS.md @@ -0,0 +1,35 @@ +# WMIOEG_KOKKOS + +**Status:** ported, the stores of `VA` only, wired and regtested on the CPU (2026-10-18) on the +fork branch h0ffmann/WW3 `ww3-gpu/u1-multi-va` (`f330dc53`), under the resident `VA`; not at the +`WW3/` pin until the integrator moves it ⚠. + +`WMIOEG` updates grid `IMOD`'s `VA` from the grids of equal rank: it scales the native columns +(2.a) and adds the averaged, converted spectra of every other grid (2.b, or copies them on an SMC +grid). With the resident `VA` it is the same column path as +[`WMIOHG_KOKKOS.md`](WMIOHG_KOKKOS.md), with a read before, since 2.a and 2.b read the columns +they write. + +## What was ported + +| | | +|---|---| +| WW3 routine | `WMIOEG`, the stores of 2.a, `model/src/wminiomd.F90:3258-3262` at NOAA-EMC/WW3 `03adda91` (`VA(:,JSEA) = WGHT * VA(:,JSEA)`), and of 2.b (`:3278`, `:3326`, `:3330`) | +| Kokkos code | `ww_va_host_read_cols` (gather) and `ww_va_host_wrote_cols` (scatter) in [`../src/fortran_iface/resident_shim.cpp`](../src/fortran_iface/resident_shim.cpp) | +| Fortran side | `W3KOKKOS_VA_READ_COLS` before 2.a and `W3KOKKOS_VA_WRITE_COLS` after 2.b, with the union of `EQSTGE(IMOD,IMOD)%JSEA` and `EQSTGE(IMOD,J)%JSEA` ([`PATCH.md`](../src/fortran_iface/PATCH.md), section 26) | + +## Arithmetic + +None on the device; the Fortran statements run unchanged on the host columns, which the gather +made current first. Bits move, as in `WMIOHG_KOKKOS.md`. + +## Benchmarks + +None of its own: the regtest entries of `WMIOHG_KOKKOS.md` count both routines' columns +together, untimed ⚠. + +## How it was regtested + +The entries of [`WMIOHG_KOKKOS.md`](WMIOHG_KOKKOS.md). Whether those inputs reach `WMIOEG`'s 2.b +(grids of equal rank with overlap) was not checked ⚠: `WMIOEG` runs on every grid and its 2.a +loop runs when `EQSTGE(IMOD,IMOD)%NREC > 0`. diff --git a/kokkos/port_report/WMIOHG_KOKKOS.md b/kokkos/port_report/WMIOHG_KOKKOS.md new file mode 100644 index 00000000..5e7cde1f --- /dev/null +++ b/kokkos/port_report/WMIOHG_KOKKOS.md @@ -0,0 +1,47 @@ +# WMIOHG_KOKKOS + +**Status:** ported, the store of `VA` only, wired and regtested on the CPU (2026-10-18) on the +fork branch h0ffmann/WW3 `ww3-gpu/u1-multi-va` (`f330dc53`), under the resident `VA`; not at the +`WW3/` pin until the integrator moves it ⚠. + +`WMIOHG` gathers the spectra a higher-ranked grid sends to grid `IMOD` (two-way nesting), +averages and converts them in Fortran, and stores them into `VA(:,JSEA)` of grid `IMOD`. With the +resident `VA` the device copy must get the same columns. What runs on the device is the store's +data movement, nothing of the averaging or the conversion. + +## What was ported + +| | | +|---|---| +| WW3 routine | `WMIOHG`, the store loop 2.e, `model/src/wminiomd.F90:2260-2266` at NOAA-EMC/WW3 `03adda91`: `VA(I,JSEA) = SPEC2(I,IS) / SIG2(I) * CG(1+(I-1)/NTH,ISEA)`, `JSEA = HGSTGE(IMOD,J)%LJSEA(IS)`, every `I = 1..NSPEC` | +| Kokkos code | `ww_va_host_wrote_cols` in [`../src/fortran_iface/resident_shim.cpp`](../src/fortran_iface/resident_shim.cpp) (`host_wrote_cols`, kernel `ww_va.scatter_cols`), declared in [`../src/fortran_iface/resident_c.hpp`](../src/fortran_iface/resident_c.hpp) | +| Fortran side | `W3KOKKOS_VA_WRITE_COLS` of the fork's `w3kokkos_residentmd.F90`, called after the loop with `LJSEA(1:NTOT)` ([`PATCH.md`](../src/fortran_iface/PATCH.md), section 26) | +| Plan | [the production single-node plan](../../docs/compound/plans/2026-10-18-002-production-single-node-plan.md), unit U1 | + +## Arithmetic + +None on the device. The Fortran statement runs unchanged on the host array. The shim then copies +the host columns `LJSEA(1:NTOT)` into a buffer (`memcpy`), the buffer to the device +(`deep_copy`), and stores each element into `d_va(s, col)` (`ww_va.scatter_cols`). A copy of a +`float` moves its bits, so each device column holds the host column's bits (v, by reading the +code: no arithmetic in the kernel). The list is the JSEA of the Fortran loop itself, so no column +is missed or added; `WW_KOKKOS_VA_CHECK=1` would stop a run where another column changed, and +`WW_KOKKOS_VA_POISON=1` would put NaN into the output if a host reader missed a column. + +## Benchmarks + +| Case | Machine | Command | Result | +|---|---|---|---| +| `mww3_test_02`, `_03`, `_04` (`grdset_a`), resident preset, lab switch | i9-14900, OpenMP, 4 threads | `docs/log/2026-10-18-ww3-multi-resident-va.md`, step 3 | columns stored over the run: 18 432 (44.24 MB), 49 680 (95.39 MB), 71 388 (137.07 MB), with `WMIOEG`'s; not timed ⚠ | + +No timing: the matrix runs are not timed runs, and the machine was shared. + +## How it was regtested + +WW3's own regtests through the matrix driver (`WW3GPU_LAB_SWITCH=ON`, `WW3GPU_STRICT=ON`), the +Fortran and the Kokkos run of one executable compared by `test.comp`: `mww3_test_01` to `_04` +(`PR3_UQ`, resident preset) and `mww3_test_05` (`ST1`/`ST4_PR3_UQ_OMP`, `SRCE,OUTG,IOPE,UPDV`), +0 files different, plain and with the host copy poisoned (v, 2026-10-18; summaries in +[`ww3gpu_matrix_multi_va_2026-10-18.tsv`](../../docs/data/ww3gpu_matrix_multi_va_2026-10-18.tsv)). +Before the change `_02`, `_03`, `_04` differed in 5, 10 and 4 files. Under MPI it builds and was +not run (ADR-0017). diff --git a/kokkos/ports.tsv b/kokkos/ports.tsv index 8d1d1789..bd48ea02 100644 --- a/kokkos/ports.tsv +++ b/kokkos/ports.tsv @@ -333,3 +333,5 @@ w3pro1md:w3ktp1 model/src/w3pro1md.F90:900-1230 331 ported translated 2026-10-16 w3pro2md:w3xyp2 model/src/w3pro2md.F90:508-1264 757 ported translated 2026-10-16 under switch_lab_shrd with PR2 for PR3; static parity by -O3 GIMPLE and assembly comparison (literals, MIN/MAX and the CGN tree, NINT as lroundf, W3QCK3 calls); a fresh reviewer found no code defect and fixed the report's pin; compiled on serial-debug, never run no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist kokkos/src/ww_kokkos/prop_xyp2.hpp,kokkos/src/ww_kokkos/prop_params2.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_XYP2=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) w3pro2md:w3ktp2 model/src/w3pro2md.F90:1306-1710 405 ported translated 2026-10-16 under switch_lab_shrd with PR2 for PR3; static parity by -O3 GIMPLE and assembly comparison (FDDMAX four-lane grouping, MIN/MAX operand order, W3QCK1/W3QCK2 arguments); a fresh reviewer found no arithmetic defect and fixed two citations; NaN-payload order (port report); compiled on serial-debug, never run no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist kokkos/src/ww_kokkos/prop_ktp2.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_KTP2=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) w3profsmd:w3xypug model/src/w3profsmd.F90:69-272 204 ported translated 2026-10-16 under the switch of ww3_tp2.6 (SHRD PR3, no PDLIB); static parity by -O3 GIMPLE and object comparison (final MAX as maxss with a +0 source, no FMA); the schemes W3XYPFSN2, W3XYPFSPSI2, W3XYPFSFCT2, W3XYPFSNIMP and DSEC21 are host hooks, SETDEPTH called ported; a fresh reviewer found no arithmetic defect and fixed one citation; compiled on serial-debug, never run no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once the four scheme hooks, a shim and the W3WAVE wiring exist kokkos/src/ww_kokkos/prop_xypug.hpp no: header-only host function; the four schemes are hooks; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_XYPUG=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) +wminiomd:wmiohg model/src/wminiomd.F90:2260-2266 7 ported the store loop 2.e repeated on the resident VA as a column scatter (ww_va_host_wrote_cols, kernel ww_va.scatter_cols), 2026-10-18; no arithmetic on the device, the Fortran statement runs unchanged on the host (PATCH.md section 26, port report) yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), WW3GPU_LAB_SWITCH=ON matrix entries mww3_test_01 to _04 (resident) and mww3_test_05 (SRCE,OUTG,IOPE,UPDV), test.comp 0 files different, plain and with WW_KOKKOS_VA_POISON=1 (docs/log/2026-10-18-ww3-multi-resident-va.md) kokkos/src/fortran_iface/resident_shim.cpp no: called on the fork branch ww3-gpu/u1-multi-va (wminiomd.F90:2318-2327 at f330dc53); the WW3/ pin moves with the integrator +wminiomd:wmioeg model/src/wminiomd.F90:3258-3262 5 ported the stores of 2.a (and of 2.b, :3278, :3326, :3330) repeated on the resident VA: column gather before, column scatter after (ww_va_host_read_cols, ww_va_host_wrote_cols), 2026-10-18; no arithmetic on the device (PATCH.md section 26, port report) yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), the entries of WMIOHG, test.comp 0 files different; whether they reach 2.b was not checked kokkos/src/fortran_iface/resident_shim.cpp no: called on the fork branch ww3-gpu/u1-multi-va (wminiomd.F90:3341-3363, :3457-3463 at f330dc53); the WW3/ pin moves with the integrator diff --git a/kokkos/src/fortran_iface/PATCH.md b/kokkos/src/fortran_iface/PATCH.md index 88d01a51..fc79b67e 100644 --- a/kokkos/src/fortran_iface/PATCH.md +++ b/kokkos/src/fortran_iface/PATCH.md @@ -7,6 +7,7 @@ Read with `git -C WW3 fetch origin && git -C WW3 branch -r` and `git rev-list -- | h0ffmann/WW3 branch | Head | Against the `WW3/` pin | What it wires | |---|---|---|---| +| `ww3-gpu/u1-multi-va` | `f330dc53` | five commits on `80644ba9`, pushed 2026-10-18 (v, `ls-remote`) | `ww3_multi`'s exchanges sync only the columns of `VA` they read or write (`W3KOKKOS_VA_READ_COLS`, `W3KOKKOS_VA_WRITE_COLS`, `ww_va_host_wrote_cols`), no read at `W3WAVE`'s return, the ABI `0.10` ([section 26](#26-ww3_multi-and-the-resident-va-column-syncs-no-read-at-w3waves-return-ww3-gpuu1-multi-va)) | | `ww3-gpu/kokkos-outg` | `e0cc8ee9` | the `WW3/` pin of h0ffmann/ww3-gpu `goal/42-gpu-step-units45`; one commit on `ww3-gpu/kokkos-prop3dev`, pushed 2026-10-17 (v) | `W3OUTG`'s integrals on the device copy of `VA` (`ww_outg_va`, `W3KOKKOS_OUTGMD`), the host read of `VA` at an output time split per reader, the ABI `0.7` ([section 15](#15-w3outg-on-the-resident-spectrum-ww3-gpukokkos-outg)) | | `ww3-gpu/kokkos-prop3dev` | `90631fad` | the parent of `e0cc8ee9`, the `WW3/` pin of h0ffmann/ww3-gpu `goal/42-srce-prop3dev`; one commit on `ww3-gpu/kokkos-resident`, pushed 2026-10-17 (v) | `W3XYP3` and `W3KTP3` batched on the device copy of `VA` (`ww_xyp3_batch_va`, `ww_ktp3_batch_va`), their host syncs only on the Fortran path, the ABI `0.6` ([section 13](#13-the-propagation-on-the-resident-spectrum-ww3-gpukokkos-prop3dev)) | | `ww3-gpu/kokkos-resident` | `513954a6` | the parent of `90631fad`, the `WW3/` pin of h0ffmann/ww3-gpu `goal/42-srce-resident`; two commits on `ww3-gpu/kokkos-prop3`, pushed 2026-10-17 (v) | the resident spectrum: `VA` synced at W3WAVE's host statements (`W3KOKKOS_RESIDENTMD`) and `W3SRCE` batched on the device copy of `VA` (`ww_srce_batch_va`), the ABI `0.5` ([section 12](#12-the-resident-spectrum-on-ww3-gpukokkos-resident)) | @@ -1988,3 +1989,103 @@ once per `W3INIT`. No namelist: the next step of gap 7. Checked the same way as section 24 (2026-10-18): `WW_KOKKOS_PRESET=resident` alone gives the parts line of `--kokkos resident` character for character and the same four model files (`cmp`, v); `WW_KOKKOS_PRESET=bogus` stops with the line above (v). +## 26. `ww3_multi` and the resident `VA`: column syncs, no read at `W3WAVE`'s return, `ww3-gpu/u1-multi-va` + +Five commits on `80644ba9` (section 22), pushed to h0ffmann/WW3 on 2026-10-18: `3bbd1557` +(`W3KOKKOS_RESIDENTMD` and the ABI), `bb5c516e` (`wminiomd.F90`), `f4df761a` (`wmwavemd.F90`), +`5ef94512` (`w3wavemd.F90`) and `f330dc53` (the remark of gap 13 of +[`PRODUCTION_READINESS_202610.md`](../../../docs/PRODUCTION_READINESS_202610.md)). Unit U1 + U7 +of [the production single-node plan](../../../docs/compound/plans/2026-10-18-002-production-single-node-plan.md), +under the resident-spectrum rule of `AGENTS.md`. The ABI is `ww_kokkos` 0.10.0 (one entry point +added, `ww_va_host_wrote_cols`); the fork asks for `0.10`. + +**The defect.** `ww3_multi` reads and writes the `VA` of a grid on the host between two `W3WAVE` +calls of that grid. Before this change `W3WAVE` ended with a whole-array read +(`w3wavemd.F90:3151-3153` at `80644ba9`), which left both copies "equal"; the writes below then +changed the host copy with no sync, and the next kernel on the device copy (`W3XYP3`, `W3KTP3` or +the batched `W3SRCE`) read it without the nesting update. `WW_KOKKOS_VA_CHECK=1` stops such a run; +without it the outputs differ (the table below). + +**Every host use of `VA` outside `W3WAVE`** in the drivers at `80644ba9`, read with +`grep -n "VA" ww3_shel.F90 wm*.F90` and every routine those files call that takes `VA` (v, +2026-10-18): + +| Site (line at `80644ba9`) | Use | Under `W3_KOKKOS` now | +|---|---|---| +| `WMIOBS`, `wminiomd.F90:374` | reads `VA(:,JSEA)` at the boundary points it sends, `JSEA` from `NBI2S(IOFF+I,2)` (`SHRD`) or `INIT_GET_JSEA_ISPROC` (`DIST`, local points only) | `W3KOKKOS_VA_READ_COLS` with that list, per receiving grid (`wminiomd.F90:365-384` at `f330dc53`) | +| `WMIOHS`, `:1636`, `:1640` | reads `VA(:,JSEA)`, `JSEA = HGSTGE(J,IMOD)%ISEND(I,1)`, `I = 1..NR` | `W3KOKKOS_VA_READ_COLS` with `ISEND(1:NR,1)` (`:1651-1659`) | +| `WMIOHG`, `:2264` | writes every element of `VA(:,JSEA)`, `JSEA = HGSTGE(IMOD,J)%LJSEA(IS)`, `IS = 1..NTOT`, reads none | `W3KOKKOS_VA_WRITE_COLS` with `LJSEA(1:NTOT)` after the loop (`:2318-2327`) | +| `WMIOES`, `:2726`, `:2732`, `:2736` | reads `VA(:,JSEA)`, `JSEA = EQSTGE(J,IMOD)%SJS(I)`, `I = 1..NR` | `W3KOKKOS_VA_READ_COLS` with `SJS(1:NR)` (`:2772-2780`) | +| `WMIOEG`, `:3261`, `:3278`, `:3326`, `:3330` | reads and writes `VA(:,JSEA)`: `JSEA = EQSTGE(IMOD,IMOD)%JSEA(I)` (2.a) and `EQSTGE(IMOD,J)%JSEA(I)` for every other grid with `NREC > 0` (2.b, SMC included) | the union of those lists, read before 2.a and stored after 2.b (`:3341-3363`, `:3457-3463`) | +| `WMWAVE`, `wmwavemd.F90:1367` | `W3IOPE ( VA )`, the unified point output | as before `W3WAVE`'s own `W3IOPE`: `W3KOKKOS_IOPE_READ` under `WW_KOKKOS_IOPE=1`, otherwise `W3KOKKOS_VA_HOST_READ`; under `MPI` a host read before the `W3MPIP` requests start (`wmwavemd.F90:1365-1370`, `:1379-1387`) | +| `ww3_shel.F90` | none: `VA` is named only in commented-out lines (`:1950-2003`) | nothing | +| `WMINIT` (`wminitmd.F90:2993-3006`) | calls `WMIOBS` before any kernel ran | covered by the `WMIOBS` row | + +`W3IOBC`, `W3IORS`, `W3IOSF`, `W3IOTR` and `W3OUTG` read `VA` only when `W3WAVE` calls them, +and the sites of sections 12, 15, 17 and 23 sync them. `wmupdtmd.F90:1407-1844` names a local +scalar `VA`, not the spectrum. `WMIOBG` writes `BBPI0`/`BBPIN`, not `VA`. `PDLIB_exchange2Dreal_zero +( VA )` after `WMIOHG` and `WMIOEG` runs only under `PDLIB`, where no resident part is allowed +(sections 13, 15, 17), so the state stays `HOST` there. + +- **`W3KOKKOS_VA_READ_COLS ( N, COLS )`**, `W3KOKKOS_VA_WRITE_COLS ( N, COLS )` + (`w3kokkos_residentmd.F90:86-132` at `f330dc53`): `COLS` are `JSEA`, `0..NSEALM`, of the + current grid (`IGRID`, `VA` of `W3SETG`/`W3SETW`); `N < 1` does nothing; both stop the model + inside an OpenMP parallel region, as every sync. The read is `ww_va_host_read_cols` (section 17). +- **`ww_va_host_wrote_cols`** (`resident_shim.cpp`, declared in `resident_c.hpp`): with the state + `DEVICE` or `BOTH`, the listed host columns are packed into the list's host buffer, copied once + to the device buffer and scattered into `d_va` by one `parallel_for` (`ww_va.scatter_cols`); with + `HOST`, nothing. The state is not changed. It shares the column lists of the partial read + (section 23), now up to eight per grid (`kMaxColSets`): `ww3_multi` uses one list per + neighbouring grid and direction at every exchange. Bits move and nothing is computed. Under + `WW_KOKKOS_VA_CHECK=1` the listed columns are taken as written and every other column must still + hold the bits of the last sync; under `WW_KOKKOS_VA_RESIDENT=1` `d_va` is then copied back + whole, so a column written and not listed shows in the output. +- **`W3WAVE`'s return** no longer reads `VA` back (section 12's `:3083` row, `:3151-3153` at + `80644ba9`). A run of `ww3_shel` may end with the device copy newer; `resident_shim.cpp` + still says so at finalize, and `scripts/ww3-gpu.sh` now prints that line instead of failing on + it (the check of mutant A2 in section 21 guarded the read this change removes). The check that + a host reader missed its sync is `WW_KOKKOS_VA_POISON=1`. +- **Why `wminiomd.F90` and `wmwavemd.F90` join the fork's diff.** The plan's first choice (B1) + marked the host newer at `W3WAVE`'s return, inside the files the fork already changed; the owner + rejected it on 2026-10-18, because on a GPU it costs a whole host-to-device copy of `VA` per + grid per call. The syncs belong where `ww3_multi` reads and writes, and they move only the + columns exchanged. Both files are `FILE` lines of `parts.d/resident.txt`, so + `scripts/ww3-gpu.sh --ww3` still accepts the fork. +- **Without `KOKKOS`**, `gfortran -E -P` of `w3wavemd.F90`, `wminiomd.F90` and `wmwavemd.F90` with + `switches/switch_lab_shrd` and with `mww3_test_08`'s `switch_ST4_PR3_UQ_MPI` is identical to + `WW3-noaa/`'s, blank lines and the include path in one comment aside (v, 2026-10-18). With + `KOKKOS` and that MPI switch, `ww3_multi` builds (v, `cmake -S -DSWITCH= + -DWW_KOKKOS=ON`, `cmake --build ... --target ww3_multi`); it was not run (ADR-0017). + +The checkpoint on the CPU, 2026-10-18 (OpenMP backend, gfortran 16.2.0, Intel Core i9-14900 +shared with other sessions, 2 matrix jobs of 4 threads; never `--gpu`), with +`WW3GPU_LAB_SWITCH=ON` and `WW3GPU_STRICT=ON`, before (`80644ba9`) and after (`f330dc53`'s tree). +[The log](../../../docs/log/2026-10-18-ww3-multi-resident-va.md) has the commands; the summaries +are [`ww3gpu_matrix_multi_va_2026-10-18.tsv`](../../../docs/data/ww3gpu_matrix_multi_va_2026-10-18.tsv). + +| Entry, parts | Before, `VA_CHECK=1` | Before, plain: test.comp | After, poisoned and plain: test.comp | Whole `VA` D2H, before → after | Columns after (read; stored) | +|---|---|---|---|---|---| +| 33 `mww3_test_01` `PR3_UQ`, `resident` | runs | 0/22 | 0/22 | 24 (24.13 MB) → 12 (12.07 MB) | 1 440 (3.46 MB); 0 | +| 34 `mww3_test_02` `PR3_UQ` `grdset_a`, `resident` | stops: 2 columns of grid 1 changed | 5/23, `ww3_self` 0/23 | 0/23 | 384 (853.40 MB) → 8 (32.29 MB) | 28 032 (67.28 MB); 18 432 (44.24 MB) | +| 38 `mww3_test_03` `PR3_UQ` `grdset_a`, `resident` | stops: 110 columns of grid 1 | 10/34, `ww3_self` 0/34 | 0/34 | 216 (441.68 MB) → 216 (441.68 MB) | 100 224 (192.43 MB); 49 680 (95.39 MB) | +| 43 `mww3_test_04` `PR3_UQ` `grdset_a`, `resident` | stops: 240 columns of grid 2 | 4/25, `ww3_self` 0/25 | 0/25 | 864 (3 002.99 MB) → 0 | 384 372 (737.99 MB); 71 388 (137.07 MB) | +| 2, 8 `mww3_test_05` `ST1`/`ST4_PR3_UQ_OMP` (moving grid), `SRCE,OUTG,IOPE,UPDV` | stops: 138 columns of grid 2 | 0/27 | 0/27 | 156 (790.79 MB) → 156 (790.79 MB) | 66 120 (158.69 MB); 6 312 (15.15 MB) | + +- Every after run is byte-identical with the host copy poisoned, so no host reader of `VA` outside + `W3WAVE` on these entries reads a column it did not sync (v). +- The whole-array copies went down or stayed; none went up (v, `WW_KOKKOS_VA_COUNT=1`). In 38 the + 216 that stay are one per step of the `W3OUTG` fallback inside `W3WAVE` (`W3OUTG Kokkos calls: 0` + in that run), and in `mww3_test_05` the 156 each way are the host write of the Fortran + propagation (`XYP3` and `KTP3` refuse a moving grid): the output block and the Fortran + propagation, not the exchanges. +- `mww3_test_05` passes before without the diagnostic because, with `XYP3` and `KTP3` off, a + host write of the Fortran propagation (`w3wavemd.F90:1885` or `:2037`) comes before any kernel + in each step and declares the host newer, as gap 2 predicted. +- `ww3_shel`: `bash scripts/ww3-gpu.sh --ww3-gpu full|smoke --cpu --kokkos resident`, plain and + with `WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_CHECK=1`: every model file byte-identical to the + Fortran's and every file but the NetCDF to `--ww3`'s, `nccmp-tol` PASS; whole-`VA` D2H on full + 1 → 0 (one copy of 21.21 MB per run), on smoke 37 → 37 (the output readers of section 23) (v). + Wall time of full, one run each on the shared machine ⚠: 40.86 s before, 43.99 s and 32.35 s + after; no difference can be read from these. +- `mww3_test_07` (`default`) was not compared: with the lab switch its `ww3_grid` stops in the + Fortran run too (`ERROR IN READING SDS4`, the entry's namelist against `ST4`'s). diff --git a/kokkos/src/fortran_iface/parts.d/resident.txt b/kokkos/src/fortran_iface/parts.d/resident.txt index 33c6669a..c7cb1011 100644 --- a/kokkos/src/fortran_iface/parts.d/resident.txt +++ b/kokkos/src/fortran_iface/parts.d/resident.txt @@ -1,3 +1,5 @@ -# The resident spectrum VA, WFIP-0005 units 1 and 2 (resident_shim.cpp): no part of its own +# The resident spectrum VA, WFIP-0005 units 1 and 2 (resident_shim.cpp): no part of its own; its column syncs in ww3_multi (PATCH.md section 26) # Read by scripts/ww3-gpu.sh: PART (WW_KOKKOS_), COUNT , FILE ; PATCH.md section 9. COUNT WW_KOKKOS_VA_COUNT +FILE model/src/wminiomd.F90 +FILE model/src/wmwavemd.F90 diff --git a/kokkos/src/fortran_iface/resident_c.hpp b/kokkos/src/fortran_iface/resident_c.hpp index 9c9c5114..a3a4eb6b 100644 --- a/kokkos/src/fortran_iface/resident_c.hpp +++ b/kokkos/src/fortran_iface/resident_c.hpp @@ -1,7 +1,7 @@ // kokkos/src/fortran_iface/resident_c.hpp // The C ABI of the resident spectrum (WFIP-0005 units 1 and 2, the partial read of the pnt -// unit, PATCH.md section 16, and the zeroed columns of the updv unit, section 17): declarations -// only. +// unit, PATCH.md section 16, the zeroed columns of the updv unit, section 17, and the partial +// write of ww3_multi's exchanges, section 26): declarations only. // resident_shim.cpp implements the ww_va_* calls, srce_shim.cpp ww_srce_batch_va; // model/src/w3kokkos_residentmd.F90 and w3kokkosmd.F90 on the fork // (h0ffmann/WW3 ww3-gpu/kokkos-resident) mirror them. The rules of ww_kokkos_c.hpp hold: @@ -57,6 +57,16 @@ int ww_va_host_write(int igrid, float* va, int nspec, int nsealm); /// outside 0..nsealm is BAD_SHAPE. va, nspec, nsealm as for ww_va_host_read. int ww_va_host_read_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const int* cols); +/// After host statements that wrote the columns cols[0..ncol) of VA and no other (ww3_multi's +/// WMIOHG and WMIOEG between W3WAVE calls, PATCH.md section 26; a column they also read needs +/// ww_va_host_read_cols first): with state DEVICE or BOTH those host columns are copied into +/// d_va, packed into one buffer (only their bytes cross); with HOST nothing. The state is NOT +/// changed. Duplicates allowed; a column outside 0..nsealm is BAD_SHAPE. Under +/// WW_KOKKOS_VA_CHECK=1 the listed columns are taken as written and every other column must +/// still hold the bits of the last sync. Under WW_KOKKOS_VA_RESIDENT=1 d_va is then copied back +/// whole, so a column written and not listed shows in the output. +int ww_va_host_wrote_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const int* cols); + /// Before a host statement whose only writes of VA set whole columns to 0, which /// ww_va_device_zero_cols then repeats on d_va (W3UICE under WW_KOKKOS_UPDV=1, the updv unit, /// PATCH.md section 17): no copy and no change of state. Under WW_KOKKOS_VA_RESIDENT=1 it is a diff --git a/kokkos/src/fortran_iface/resident_shim.cpp b/kokkos/src/fortran_iface/resident_shim.cpp index 1c95fcf1..19b42720 100644 --- a/kokkos/src/fortran_iface/resident_shim.cpp +++ b/kokkos/src/fortran_iface/resident_shim.cpp @@ -63,16 +63,19 @@ std::map& grids() { return *m; } -// The column lists a grid keeps buffers for: W3IOPE's and W3IOBC's, which repeat, and a few of -// W3IOTR's, which follow the track. -constexpr size_t kMaxColSets = 4; +// The column lists a grid keeps buffers for: W3IOPE's and W3IOBC's, which repeat, a few of +// W3IOTR's, which follow the track, and ww3_multi's exchanges (PATCH.md section 26), one list per +// neighbouring grid and direction, read and written at each exchange. +constexpr size_t kMaxColSets = 8; struct Counts { long long h2d = 0, d2h = 0, host_reads = 0, host_writes = 0, device_calls = 0; long long part_calls = 0, part_d2h = 0, part_cols = 0; long long zero_calls = 0, zero_launches = 0, zero_cols = 0, zero_checks = 0; + long long wrote_calls = 0, wrote_h2d = 0, wrote_cols = 0; long long va_checks = 0, poisons = 0; double h2d_bytes = 0.0, d2h_bytes = 0.0, spectrum_bytes = 0.0, part_bytes = 0.0; + double wrote_bytes = 0.0; }; Counts g_n; bool g_hooks = false; @@ -247,6 +250,33 @@ int host_sync(int igrid, float* va, int nspec, int nsealm, bool write) { return WW_KOKKOS_OK; } +/// The buffers of the sorted column list `want` (n > 0 columns), moved to the front of the +/// grid's lists; allocated on its first use, dropping the oldest list past kMaxColSets. The +/// partial reads and the partial writes share the lists. +ColSet& col_set(Ctx& c, const std::vector& want) { + const int n = static_cast(want.size()); + const auto hit = + std::find_if(c.sets.begin(), c.sets.end(), [&](const ColSet& s) { return s.cols == want; }); + if (hit != c.sets.end()) { + std::rotate(c.sets.begin(), hit, hit + 1); + } else { + ColSet s; + s.cols = want; + s.d_cols = Kokkos::View( + Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols"), n); + Kokkos::deep_copy(s.d_cols, Kokkos::View>( + s.cols.data(), n)); + s.d_buf = Kokkos::View( + Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols_buf"), c.nspec, n); + s.h_buf = Kokkos::View( + Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols_host"), c.nspec, n); + if (c.sets.size() >= kMaxColSets) c.sets.pop_back(); + c.sets.insert(c.sets.begin(), std::move(s)); + } + return c.sets.front(); +} + /// The partial read: d_va(:, cols) gathered into d_buf on the device, one copy of d_buf back, /// and its columns stored into the Fortran array. Bits move and nothing is computed, so each /// host column copied holds what ww_va_host_read would have put there. The state is left as it @@ -272,26 +302,7 @@ int host_read_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const } if (want.empty()) return WW_KOKKOS_OK; const int n = static_cast(want.size()); - const auto hit = std::find_if(c->sets.begin(), c->sets.end(), - [&](const ColSet& s) { return s.cols == want; }); - if (hit != c->sets.end()) { - std::rotate(c->sets.begin(), hit, hit + 1); - } else { - ColSet s; - s.cols = want; - s.d_cols = Kokkos::View( - Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols"), n); - Kokkos::deep_copy(s.d_cols, Kokkos::View>( - s.cols.data(), n)); - s.d_buf = Kokkos::View( - Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols_buf"), nspec, n); - s.h_buf = Kokkos::View( - Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_va.cols_host"), nspec, n); - if (c->sets.size() >= kMaxColSets) c->sets.pop_back(); - c->sets.insert(c->sets.begin(), std::move(s)); - } - ColSet& cs = c->sets.front(); + ColSet& cs = col_set(*c, want); { ww::timers::Region tc("va.d2h.cols"); const DevVa d = c->d; @@ -322,6 +333,66 @@ int host_read_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const return WW_KOKKOS_OK; } +/// The partial write: after host statements that wrote the columns cols of VA and no other +/// (ww3_multi's nesting and equal-rank updates, PATCH.md section 26), those host columns packed +/// into the list's h_buf, one copy of it to d_buf, and a scatter into d_va. Bits move and nothing is +/// computed, so each device column holds the host column's bits. The state is left as it was: +/// with DEVICE the other device columns are still the newer ones, with BOTH both copies are equal +/// again, and with HOST d_va is stale anyway and nothing moves. +int host_wrote_cols(int igrid, float* va, int nspec, int nsealm, int ncol, const int* cols) { + ww::timers::Region t("va.host.wcols"); + Ctx* c = bind(igrid, va, nspec, nsealm); + if (c == nullptr) return WW_KOKKOS_ERR_BAD_SHAPE; + if (ncol < 0 || (ncol > 0 && cols == nullptr)) return WW_KOKKOS_ERR_BAD_SHAPE; + std::vector want(cols, cols + ncol); + std::sort(want.begin(), want.end()); + want.erase(std::unique(want.begin(), want.end()), want.end()); + if (!want.empty() && (want.front() < 0 || want.back() > nsealm)) { + say("ww_va_host_wrote_cols: a column outside 0..NSEALM"); + return WW_KOKKOS_ERR_BAD_SHAPE; + } + const size_t ns = static_cast(nspec); + // The listed columns are declared written: the snapshot takes their new host bits first, so + // the check below still catches a column the host wrote and the list misses. + if (c->snap_ok) + for (const int j : want) + std::memcpy(c->snap.data() + static_cast(j) * ns, + c->host + static_cast(j) * ns, ns * sizeof(float)); + if (const int rc = check(*c, igrid, "a partial host write"); rc != WW_KOKKOS_OK) return rc; + if (c->side == Side::Host || want.empty()) { + if (resident_diag() && c->side == Side::Host) { + if (!Kokkos::is_initialized()) return WW_KOKKOS_ERR_NOT_INITIALISED; + h2d(*c); + } + return WW_KOKKOS_OK; + } + if (!Kokkos::is_initialized()) return WW_KOKKOS_ERR_NOT_INITIALISED; + const int n = static_cast(want.size()); + ColSet& cs = col_set(*c, want); + for (int k = 0; k < n; ++k) + std::memcpy(&cs.h_buf(0, k), c->host + static_cast(cs.cols[k]) * ns, + ns * sizeof(float)); + { + ww::timers::Region tc("va.h2d.cols"); + Kokkos::deep_copy(cs.d_buf, cs.h_buf); + const DevVa d = c->d; + const auto dc = cs.d_cols; + const auto buf = cs.d_buf; + Kokkos::parallel_for( + "ww_va.scatter_cols", Kokkos::RangePolicy<>(0, ns * static_cast(n)), + KOKKOS_LAMBDA(const size_t i) { + const size_t k = i / ns, s = i % ns; + d(s, dc(k)) = buf(s, k); + }); + Kokkos::fence("ww_va.h2d_cols"); + } + ++g_n.wrote_h2d; + g_n.wrote_cols += n; + g_n.wrote_bytes += static_cast(static_cast(n) * ns * sizeof(float)); + if (resident_diag()) d2h(*c); // a column the host wrote and the list missed is overwritten + return WW_KOKKOS_OK; +} + /// The device side of a host statement that set whole columns of VA to 0 and wrote nothing else: /// the same statement, VA(:,JSEA) = 0., on d_va. A store of +0.0f, so the device column holds /// the bits of the host column. HOST: the host write is the newer one and d_va is stale anyway. @@ -452,6 +523,12 @@ int ww_va_host_read_cols(int igrid, float* va, int nspec, int nsealm, int ncol, return guarded([&]() { return host_read_cols(igrid, va, nspec, nsealm, ncol, cols); }); } +int ww_va_host_wrote_cols(int igrid, float* va, int nspec, int nsealm, int ncol, + const int* cols) { + ++g_n.wrote_calls; + return guarded([&]() { return host_wrote_cols(igrid, va, nspec, nsealm, ncol, cols); }); +} + int ww_va_host_update(int igrid, float* va, int nspec, int nsealm) { ++g_n.zero_calls; return guarded([&]() { @@ -504,6 +581,10 @@ void ww_va_report(void) { std::fprintf(stderr, "ww_kokkos: VA partial host reads: %lld\n", g_n.part_calls); std::fprintf(stderr, "ww_kokkos: VA partial device-to-host copies: %lld\n", g_n.part_d2h); std::fprintf(stderr, "ww_kokkos: VA partial columns copied: %lld\n", g_n.part_cols); + std::fprintf(stderr, "ww_kokkos: VA partial host writes: %lld\n", g_n.wrote_calls); + std::fprintf(stderr, "ww_kokkos: VA partial host-to-device copies: %lld\n", g_n.wrote_h2d); + std::fprintf(stderr, "ww_kokkos: VA partial columns stored: %lld (%.4f MB)\n", g_n.wrote_cols, + g_n.wrote_bytes / 1.0e6); std::fprintf(stderr, "ww_kokkos: VA host updates repeated on the device: %lld\n", g_n.zero_calls); std::fprintf(stderr, "ww_kokkos: VA device zeroing launches: %lld\n", g_n.zero_launches); std::fprintf(stderr, "ww_kokkos: VA device columns zeroed: %lld\n", g_n.zero_cols); diff --git a/scripts/ww3-gpu.sh b/scripts/ww3-gpu.sh index efac30f6..b05da369 100755 --- a/scripts/ww3-gpu.sh +++ b/scripts/ww3-gpu.sh @@ -647,11 +647,11 @@ if [ -n "$timers" ]; then if (name ~ /\.call$/) in_kokkos += s } END { printf ">> %-22s %21s %10.3f s %6.1f %%\n", "outside Kokkos", "", w - in_kokkos, (w > 0 ? 100 * (w - in_kokkos) / w : 0) }' fi -# The resident VA must be current on the host when W3WAVE returns: resident_shim.cpp says so at -# finalize when it is not (a missing W3KOKKOS_VA_HOST_READ at the return, which no output shows). +# W3WAVE no longer reads VA back at its return (PATCH.md section 26): each host reader outside it +# syncs what it reads, and ww3_shel reads no VA after its last W3WAVE, so a run may end with the +# device copy newer. A host reader that misses its sync shows under WW_KOKKOS_VA_POISON=1. if grep -q "still newer on the device at finalize" "$d1/ww3_shel.out"; then - grep -h "still newer on the device at finalize" "$d1/ww3_shel.out" >&2 - echo "ww3-gpu: the Kokkos run ended with the device's VA newer than WW3's" >&2; exit 1 + echo ">> $(grep -h "still newer on the device at finalize" "$d1/ww3_shel.out" | head -1) (expected: no read at W3WAVE's return)" fi if [ -n "$kokkos" ]; then parts_line="$(grep -h "Kokkos parts" "$d1/log.ww3" | head -1 || true)" diff --git a/site/data/port_score.tsv b/site/data/port_score.tsv index 70f7a5bf..2705b4ce 100644 --- a/site/data/port_score.tsv +++ b/site/data/port_score.tsv @@ -1,10 +1,10 @@ # The global numbers of the Kokkos port, written by scripts/port_score.py --write; do not edit. # WW3@03adda91. The website reads this file (site/content). # name share count what -of the run time, translated 99.3 % 310 of 1 023 routines a Kokkos translation exists in kokkos/src; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) +of the run time, translated 99.3 % 312 of 1 023 routines a Kokkos translation exists in kokkos/src; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) of the run time, ported 99.3 % 60 of 1 023 routines WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) -of the run time, regtested 48.6 % 10 of 1 023 routines WW3's regtests ran with it on Kokkos and agreed with the Fortran; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) -of the lines of code, translated 25.70 % 67 699 of 263 375 Fortran lines of the translated routines, of every line in WW3's model/src, in 63 of 147 files +of the run time, regtested 48.6 % 12 of 1 023 routines WW3's regtests ran with it on Kokkos and agreed with the Fortran; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) +of the lines of code, translated 25.71 % 67 711 of 263 375 Fortran lines of the translated routines, of every line in WW3's model/src, in 64 of 147 files of the lines of code, ported 6.46 % 17 022 of 263 375 Fortran lines of the ported routines, of every line in WW3's model/src, in 23 of 147 files -of the lines of code, regtested 1.02 % 2 677 of 263 375 Fortran lines of the regtested routines, of every line in WW3's model/src, in 5 of 147 files -lines of Kokkos C++ 58 497 in 347 files under kokkos/src: the port itself +of the lines of code, regtested 1.02 % 2 689 of 263 375 Fortran lines of the regtested routines, of every line in WW3's model/src, in 6 of 147 files +lines of Kokkos C++ 58 327 in 347 files under kokkos/src: the port itself From d39b9e6ff360fff4293a21506dffb1848fdba202 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 13:22:52 -0300 Subject: [PATCH 17/27] kokkos: port score after U1, U2 and U8 together The C++ line count of the port changed once U1's shim changes joined U2's and U8's on one branch; regenerated, not hand-merged. Tested: python3 scripts/port_score.py --write, then --check, current Co-Authored-By: Claude Opus 5.5 --- kokkos/PORT_STATUS.md | 2 +- site/data/port_score.tsv | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/kokkos/PORT_STATUS.md b/kokkos/PORT_STATUS.md index e073c1fe..a7edbc8b 100644 --- a/kokkos/PORT_STATUS.md +++ b/kokkos/PORT_STATUS.md @@ -49,7 +49,7 @@ Profile: full (bench_small, 120 x 80, 9 520 sea points, 12 h), Release -pg, swit | ported | **99.3 %** | 17022 of 263375 (6.46 %) | 60 of 1023 (5.87 %) | 23 of 147 | WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag | | regtested | **48.6 %** | 2689 of 263375 (1.02 %) | 12 of 1023 (1.17 %) | 6 of 147 | WW3's regtests ran with it on Kokkos and agreed with the Fortran | -The port itself: 58327 lines in 347 files, the C++ under kokkos/src (every .cpp and .hpp line). +The port itself: 58429 lines in 347 files, the C++ under kokkos/src (every .cpp and .hpp line). On a GPU: none yet. The CUDA build links; the RTX 4090 run, `just smoke-ww3-gpu gpu`, is the owner's, supervised. diff --git a/site/data/port_score.tsv b/site/data/port_score.tsv index 2705b4ce..fee25f47 100644 --- a/site/data/port_score.tsv +++ b/site/data/port_score.tsv @@ -7,4 +7,4 @@ of the run time, regtested 48.6 % 12 of 1 023 routines WW3's regtests ran with i of the lines of code, translated 25.71 % 67 711 of 263 375 Fortran lines of the translated routines, of every line in WW3's model/src, in 64 of 147 files of the lines of code, ported 6.46 % 17 022 of 263 375 Fortran lines of the ported routines, of every line in WW3's model/src, in 23 of 147 files of the lines of code, regtested 1.02 % 2 689 of 263 375 Fortran lines of the regtested routines, of every line in WW3's model/src, in 6 of 147 files -lines of Kokkos C++ 58 327 in 347 files under kokkos/src: the port itself +lines of Kokkos C++ 58 429 in 347 files under kokkos/src: the port itself From ea0de76c7a58fe95432dcc72f8d99fb3f935196b Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 13:36:42 -0300 Subject: [PATCH 18/27] docs/compound: plan 003 links PR #193 and its reported DIA and W3SDS4 numbers Tested: nix shell nixpkgs#vale -c scripts/vale.sh on the plan, rc 0 Co-Authored-By: Claude Opus 5.5 --- .../2026-10-18-003-gpu-first-supervised-run-plan.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md index 98b9fa74..60f6c2d6 100644 --- a/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md +++ b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md @@ -102,12 +102,15 @@ every parity argument so far (`docs/log/2026-10-18-cuda-build-only.md`) ⚠. Not ranked: skipping `W3SDS4`'s zero-weight terms (12 of 23 products, profile log) changes the result for an infinite or NaN input, so it is the owner's decision and its own PR. -**Work in progress elsewhere (2026-10-18).** Branch `goal/eb10363f-4821-noaa-progress-record` -(another session, not pushed when this was written) builds the `W3SDS4` saturation-integral loop +**Work in progress elsewhere (2026-10-18).** PR #193 (branch +`goal/eb10363f-4821-noaa-progress-record`, another session) builds the `W3SDS4` saturation-integral loop interchange (`kokkos/src/ww_kokkos/st4_sds4.hpp`), a DIA direction-halo layout that turns the index-table gathers into contiguous loads (`snl1_team.hpp`, `snl1_tables.{hpp,cpp}`), and `-fno-math-errno` for the host build (`kokkos/src/ww_kokkos/CMakeLists.txt`). The halo layout -attacks the same gathers as plan 2 by another route; plan 2 starts from that branch's measured +attacks the same gathers as plan 2 by another route. That session reports the DIA kernel alone +(`ww_bench_snl1`, medians of seven) 1.93x faster at one thread and 1.19x at 32 with the halo, and +the `W3SDS4` interchange about 7 % faster unless `SATWEIGHTS` and `SATINDICES` are transposed; +end to end not measured ⚠ (PR #193's own record is the source). Plan 2 starts from that result, not before it. Its timing and the GPU run of plan 1 need a resident regtest pass on the tree that holds both PRs. From 32554077b9e0ea692f104a1f0cf4d534f37d2a93 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 14:01:57 -0300 Subject: [PATCH 19/27] The resident VA: W3CPRT on the device copy, partition fields without VA Goal abbb0456-6043, unit U9 of docs/compound/plans/2026-10-18-002-production-single-node-plan.md. At every output time with partitioning on, W3WAVE copied the whole VA back from the device for W3CPRT and W3IOSF, and W3KOKKOS_OUTG fell back to the Fortran W3OUTG for any partition field. W3CPRT is now translated (iosf_w3cprt.hpp: per point the Fortran's serial code and the existing W3PART translation, called unchanged; the DTPRT/ICPRT table placed by an integer prefix sum) and run on the resident VA by cprt_shim.cpp (ww_cprt_va, ww_cprt_fetch); W3OUTG section 6 is its own entry point, ww_outg_part, moved unchanged out of ww::iogo::w3outg. Only the table and the partition fields W3IOGO writes come back. WW_KOKKOS_CPRT is opt-in. Fork: h0ffmann/WW3 ww3-gpu/u9-cprt 4f126d85 (gitlink not moved). PATCH.md section 27, port report W3CPRT_KOKKOS.md, ports.tsv row w3iosfmd:w3cprt, the W3PART family's regtested and wired cells, GLOSSARY, docs/log/2026-10-18-cprt-resident.md. Tested: just kokkos-test serial-debug (11/11, no warning) and openmp-release (11/11); static parity by -O3 GIMPLE of w3iosfmd.F90 and cprt_shim.cpp (port report); scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV[,CPRT] on two bench_small partition variants (switch_lab_shrd, switch_lab_ompg), WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1: every model file and partition.ww3 byte-identical to the Fortran and to --ww3, VA D2H 13 -> 1 in 12 h; smoke byte-identical; WW3 ww3_tpt1.1 PTM1-5 through regtests/cmake/run_entry.cmake (PR3 UQ for PR0): test.comp 0/22 each, nccmp-tol PASS; python3 scripts/port_score.py --write then --check; vale 0 errors on the changed Markdown Co-Authored-By: Claude Opus 5.5 --- docs/GLOSSARY.md | 4 + docs/log/2026-10-18-cprt-resident.md | 93 +++++ docs/log/README.md | 1 + kokkos/PORT_RANKING.md | 19 +- kokkos/PORT_STATUS.md | 4 +- kokkos/PORT_TABLE.tsv | 162 ++++----- kokkos/port_report/W3CPRT_KOKKOS.md | 82 +++++ kokkos/port_report/W3OUTG_KOKKOS.md | 2 +- kokkos/ports.tsv | 19 +- kokkos/src/fortran_iface/PATCH.md | 73 ++++ kokkos/src/fortran_iface/cprt_c.hpp | 69 ++++ kokkos/src/fortran_iface/cprt_shim.cpp | 424 ++++++++++++++++++++++ kokkos/src/fortran_iface/outg_c.hpp | 43 +++ kokkos/src/fortran_iface/outg_shim.cpp | 141 ++++++- kokkos/src/fortran_iface/part_flags.cpp | 6 +- kokkos/src/fortran_iface/parts.d/cprt.txt | 4 + kokkos/src/ww_kokkos/CMakeLists.txt | 1 + kokkos/src/ww_kokkos/iogo_w3outg.hpp | 43 ++- kokkos/src/ww_kokkos/iosf_w3cprt.hpp | 121 ++++++ site/data/port_score.tsv | 6 +- 20 files changed, 1190 insertions(+), 127 deletions(-) create mode 100644 docs/log/2026-10-18-cprt-resident.md create mode 100644 kokkos/port_report/W3CPRT_KOKKOS.md create mode 100644 kokkos/src/fortran_iface/cprt_c.hpp create mode 100644 kokkos/src/fortran_iface/cprt_shim.cpp create mode 100644 kokkos/src/fortran_iface/parts.d/cprt.txt create mode 100644 kokkos/src/ww_kokkos/iosf_w3cprt.hpp diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index c0144de7..fcb2b01e 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -432,6 +432,7 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | W3UTAU / W3URHO / W3UIC1 | "Interpolate atmosphere momentum fields to the given time", "Interpolate air density field to the given time", "Update ice thickness in the wave model" `(v)` `w3updtmd.F90:835`, `:2626`, `:1555` | Forcing updates in `W3WAVE` not translated yet; hooks of the Kokkos `W3WAVE`. | [kokkos/port_report/W3WAVE_KOKKOS.md](../kokkos/port_report/W3WAVE_KOKKOS.md) | | W3UBPT | "Update spectra at the active boundary points" `(v)` `w3updtmd.F90:1329` | Hooks of the Kokkos `W3WAVE`. | [kokkos/port_report/W3WAVE_KOKKOS.md](../kokkos/port_report/W3WAVE_KOKKOS.md) | | W3IOPE / W3IOSF / W3CPRT | "Extract point output data and store in output COMMONs", "Write partitioned spectral data to file", "Partitioning of spectra into fields for all grid points that are locally stored" `(v)` `w3iopomd.F90:873`, `w3iosfmd.F90:360`, `:109` | The output and boundary I/O `W3WAVE` calls; hooks of the Kokkos `W3WAVE`. | [kokkos/port_report/W3WAVE_KOKKOS.md](../kokkos/port_report/W3WAVE_KOKKOS.md) | +| ICPRT / DTPRT / FINSIZ | `W3ODATMD`'s partition table: `ICPRT(JSEA,1)` the number of partitions plus one of point `JSEA` (0 when skipped), `ICPRT(JSEA,2)` its first column of `DTPRT(DIMP,FINSIZ)`, which holds `XP(1:15,0:NP)` of every point one after the other; built by `W3CPRT`, read by `W3OUTG` section 6 and `W3IOSF` `(v)` `w3iosfmd.F90:221-326` | A few values per point and partition, against `NTH*NK` of the spectrum. | [kokkos/port_report/W3CPRT_KOKKOS.md](../kokkos/port_report/W3CPRT_KOKKOS.md) | | MAPFS | the sea-point index of a grid point, `MAPFS(IY,IX)`; "Some array start at 0 because MAPFS(IY,IX)=0 for missing points" `(v)` `w3gdatmd.F90:1723`, `:1735` | `0` on land, so `W3IOPE` reads element 0 of the `(0:NSEA)` arrays for a land corner, with weight 0. | [kokkos/port_report/W3IOPE_KOKKOS.md](../kokkos/port_report/W3IOPE_KOKKOS.md) | | LPDLIB | "Logical for using the PDLIB library" `(v)` `constants.F90:102` | A run-time flag, `.FALSE.` unless WW3 was built with `PDLIB` (`w3initmd.F90:642-645`); `W3IOPE` interpolates `ICEFO` with a different sum under it. | [kokkos/port_report/W3IOPE_KOKKOS.md](../kokkos/port_report/W3IOPE_KOKKOS.md) | | SPPNT | "Point output buffer" `(v)` `w3adatmd.F90:306`, allocated `SPPNT(NTH,NK,4)` `:1451` | The four corner spectra `W3IOPE` copies from `A` (as `SP`) before interpolating them. | [kokkos/port_report/W3IOPE_KOKKOS.md](../kokkos/port_report/W3IOPE_KOKKOS.md) | @@ -714,6 +715,7 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | `WW_KOKKOS_OUTG` / `WW_KOKKOS_OUTG_COUNT` / `ww_outg_va` / W3KOKKOS_OUTG | the run-time flag, counter and C entry point of the outg wiring unit (`W3OUTG`'s integrals on the resident VA, WFIP-0005 unit 5), and its Fortran wrapper on the fork; opt-in only: `WW_KOKKOS_ALL=1` does not turn it on; the wrapper calls the Fortran `W3OUTG` when partition, `CALC_U3STOKES`, `SKEWNESS` or `CALC_WBT` fields are requested; the count prints calls, sea points and the fields and bytes copied back at exit `(v)` [kokkos/src/fortran_iface/outg_shim.cpp](../kokkos/src/fortran_iface/outg_shim.cpp) | Wired on h0ffmann/WW3 `ww3-gpu/kokkos-outg`, 2026-10-17; run byte-identical on the CPU, not regtested. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | | `WW_KOKKOS_IOPE` / `ww_va_host_read_cols` / W3KOKKOS_IOPE_READ | the run-time flag, C entry point and Fortran wrapper of the pnt wiring unit (WFIP-0005): before `W3IOPE` (point output) the host copy of the resident `VA` is refreshed only in the columns `W3IOPE` reads, the sea points `MAPFS` gives at the four interpolation corners `IPTINT` of each output point, gathered on the device and copied back; the resident state is not changed, so the other columns stay stale until the next full host read; opt-in only: `WW_KOKKOS_ALL=1` does not turn it on; counted under `WW_KOKKOS_VA_COUNT` (`VA partial host reads`, `... device-to-host copies`, `... columns copied`) `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Wired on h0ffmann/WW3 `ww3-gpu/kokkos-pnt`, 2026-10-17; run byte-identical on the CPU, not regtested. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | | `WW_KOKKOS_UPDV` / `ww_va_device_zero_cols` / W3KOKKOS_UICE | the run-time flag, C entry point and Fortran wrapper of the updv wiring unit (WFIP-0005): at an ice update the Fortran `W3UICE` runs on the host `VA`, the columns it set to 0 are found from the ice bit of `MAPST2` before and after, and the same columns are set to 0 in the resident `VA` on the device, so no `VA` crosses; with `ww_va_host_update` before it; under `WW_KOKKOS_VA_RESIDENT=1` the two copies are compared after each update and a difference stops the model; opt-in only: `WW_KOKKOS_ALL=1` does not turn it on; `W3ULEV` is not covered `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Wired on h0ffmann/WW3 `ww3-gpu/kokkos-pnt`, 2026-10-17; run byte-identical on the CPU, not regtested. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | +| `WW_KOKKOS_CPRT` / `ww_cprt_va` / `ww_cprt_fetch` / `ww_outg_part` / W3KOKKOS_CPRT | the run-time flag, C entry points and Fortran wrapper of the cprt wiring unit: at an output time with partitioning, `W3CPRT` (the watershed partitioning of every local sea point, `W3PART` per point) runs on the resident `VA` as a Kokkos launch over the points, each point the Fortran's serial code, and only the table `ICPRT` and the used columns of `DTPRT` come back to the host; `ww_outg_part` fills `W3OUTG`'s partition fields (section 6) from that table without `VA`, so a gridded output with partition fields no longer falls back to the Fortran `W3OUTG`; a NaN depth or a spectrum with no peak falls back to the Fortran `W3CPRT` after a host read; opt-in only: `WW_KOKKOS_ALL=1` does not turn it on; `WW_KOKKOS_CPRT_COUNT=1` prints its counters, `WW_KOKKOS_CPRT_CHUNK` the points per launch `(v)` [kokkos/src/fortran_iface/cprt_shim.cpp](../kokkos/src/fortran_iface/cprt_shim.cpp) | Wired on h0ffmann/WW3 `ww3-gpu/u9-cprt`, 2026-10-18; run byte-identical on the CPU. | [kokkos/port_report/W3CPRT_KOKKOS.md](../kokkos/port_report/W3CPRT_KOKKOS.md) | | `scripts/libm_bits.sh` / `kokkos/tools/libm_bits/` / `ww::libm` | the checker and the host-and-device copies of the libm functions the `W3SRCE` unit calls per sea point (glibc 2.44's `expf`, `logf`, `powf`, `sincosf`, libmvec's `_ZGVbN4v_expf` and `_ZGVbN4vv_powf`, CORE-MATH's `tanhf`, `log10f`, `atan2f`, `coshf`, `sinhf`, and double `exp`, `tanh`), in namespace `ww::libm` `(v)` [scripts/libm_bits.sh](../scripts/libm_bits.sh) header, #177, #178 | `probe` and `fingerprint` name this CPU's glibc build, `all` runs every host check, `--nvcc` compiles the device check, `--device` runs it under the owner only (exit 7 without a typed "yes"). On the host every copy equals glibc on every input checked; on a device it is ⚠ until `--device` runs. | [docs/GPU_LIBM_BITS_202610.md](GPU_LIBM_BITS_202610.md), [docs/WFIPs/WFIP-0006-device-libm-bits.md](WFIPs/WFIP-0006-device-libm-bits.md) | | `libm_device.hpp` / `ww::dm` / `WW_KOKKOS_LIBM` | the header through which every libm call of the kernels `WW_KOKKOS_ALL` runs goes, its namespace, and the CMake option (CUDA builds only, `ON` by default) that points its device branch at `ww::libm`; the host branch is the Kokkos call the kernel made before, so the CPU build calls glibc as it did `(v)` [kokkos/src/ww_kokkos/libm_device.hpp](../kokkos/src/ww_kokkos/libm_device.hpp), [kokkos/src/ww_kokkos/CMakeLists.txt](../kokkos/src/ww_kokkos/CMakeLists.txt) | WFIP-0006 units 1 to 4 (#178). Overloads for the other argument type are deleted, so a float/double promotion does not compile. `-DWW_KOKKOS_LIBM=OFF` keeps libdevice and ADR-0014's verdict by tolerance. | [docs/WFIPs/WFIP-0006-device-libm-bits.md](WFIPs/WFIP-0006-device-libm-bits.md), [AGENTS.md](../AGENTS.md) | | `gen_tables` / `ww_libm_tables.hpp` / `WW_LIBM_HOST_FMA` | a host program the CUDA build compiles with the host g++ (nvcc's front end fails on it) and runs at build time; it writes a header with libmvec's `powf` tables read from the loaded `libmvec.so.1`, this host's `rcpps` values, and `WW_LIBM_HOST_FMA`, glibc's fma or sse2 build decided from glibc's own `exp` `(v)` [kokkos/src/ww_kokkos/CMakeLists.txt](../kokkos/src/ww_kokkos/CMakeLists.txt), [kokkos/tools/libm_bits/gen_tables.cpp](../kokkos/tools/libm_bits/gen_tables.cpp) | It follows `GLIBC_TUNABLES`; the run host must be the build host, or have the same glibc build and `rcpps`, which nothing checks at run time ⚠. | [docs/WFIPs/WFIP-0006-device-libm-bits.md](WFIPs/WFIP-0006-device-libm-bits.md) | @@ -1182,6 +1184,7 @@ Every term above, alphabetically, with the section it lives in. - IC3TABLE_CHENG: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - ICL DAP: [HPC and software](#hpc-and-software) - ICLO / ICLO_NONE / ICLO_GRDI / ICLO_GRDJ / ICLO_TRDL / ICLO_TRPL: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) +- ICPRT / DTPRT / FINSIZ: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - IFILE4: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - IJKCel3: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) - IJKUFc: [WW3 modules, routines and regtests](#ww3-modules-routines-and-regtests) @@ -2236,6 +2239,7 @@ Every term above, alphabetically, with the section it lives in. - `ww_kokkos_c.hpp`: [This repository's own names](#this-repositorys-own-names) - `ww_kokkos_c.hpp` / `snl1_shim.cpp` / `w3kokkosmd.F90`: [This repository's own names](#this-repositorys-own-names) - `WW_KOKKOS_BUILD_TESTS` / `WW3GPU_KOKKOS_PREFIX`: [This repository's own names](#this-repositorys-own-names) +- `WW_KOKKOS_CPRT` / `ww_cprt_va` / `ww_cprt_fetch` / `ww_outg_part` / W3KOKKOS_CPRT: [This repository's own names](#this-repositorys-own-names) - `WW_KOKKOS_DEVICE_ID`: [This repository's own names](#this-repositorys-own-names) - `WW_KOKKOS_TIMERS` / `--timers` / `ww::timers::Region`: [This repository's own names](#this-repositorys-own-names) - `WW_KOKKOS_ERR_RUNTIME`: [This repository's own names](#this-repositorys-own-names) diff --git a/docs/log/2026-10-18-cprt-resident.md b/docs/log/2026-10-18-cprt-resident.md new file mode 100644 index 00000000..404a4a5f --- /dev/null +++ b/docs/log/2026-10-18-cprt-resident.md @@ -0,0 +1,93 @@ +# W3CPRT on the resident VA: partitioning without copying the spectrum back + +2026-10-18, agent G of the production single-node goal (unit U9 of +[`docs/compound/plans/`](../compound/plans/), branch `goal/abbb0456-6043-u9-cprt`; fork branch +`ww3-gpu/u9-cprt` of h0ffmann/WW3, on `80644ba9`). + +## Question + +At every output time with partitioning on (gridded partition fields, or the partition file), +`W3WAVE` read the whole `VA` back from the device for `W3CPRT` (`w3wavemd.F90:2688-2693` at +`80644ba9`), and `W3KOKKOS_OUTG` fell back to the Fortran `W3OUTG` when a partition field was +requested. Can `W3CPRT` run on the resident `VA`, with only its table coming back, bit for bit? + +## What was done + +- `W3PART` and everything it calls were translated on 2026-10-16 and never wired + ([`W3PART_KOKKOS.md`](../../kokkos/port_report/W3PART_KOKKOS.md)). `W3CPRT` itself is 236 lines: + per sea point a conversion of `N` to `E = N 2 pi sigma / c_g`, one `W3PART` call, and the + storage of `XP(1:15,0:NP)` one point after the other. Points are independent, so a launch over + points with each point's serial code is the translation; the table is placed by an integer + prefix sum. Kernel [`iosf_w3cprt.hpp`](../../kokkos/src/ww_kokkos/iosf_w3cprt.hpp), shim + [`cprt_shim.cpp`](../../kokkos/src/fortran_iface/cprt_shim.cpp), fork unit + `w3kokkos_cprtmd.F90`, details in + [`PATCH.md` section 27](../../kokkos/src/fortran_iface/PATCH.md). +- The first run with `CPRT` on still showed 13 device-to-host copies of `VA` in 12 hours: the + host read before `W3IOSF` (`w3wavemd.F90:2963-2966`). `W3IOSF` reads no `VA` (its `USE` lists, + `w3iosfmd.F90:429-453`, v), so that read is skipped under `KOKKOS_CPRT`. +- `W3OUTG`'s section 6 (the partition fields from the table) is now its own entry point, + `ww_outg_part`, so `W3KOKKOS_OUTG` no longer falls back for `FLPART`. +- Static parity: the `-O3` GIMPLE of `w3iosfmd.F90` and of the shim + ([`W3CPRT_KOKKOS.md`](../../kokkos/port_report/W3CPRT_KOKKOS.md)). +- CPU only, never `--gpu`. i9-14900, shared with other agents' jobs, `OMP_NUM_THREADS=4`, + `JOBS=4`, gfortran/g++ 16.2.0, Kokkos 5.2.2 OpenMP backend. + +Runs. `bench_small` (`kokkos/tests/fixtures/bench_small`: 120 x 80, 9 520 sea points, +`NK = NTH = 24`, 12 hours) writes no output, so two throwaway variants of its `ww3_shel.nml` (not +committed, restored after each run) turned partitioning on, both with hourly field output and the +hourly partition file (`DATE%PARTITION`, `TYPE%PARTITION` over the whole grid, unformatted): + +- `p6`: fields `HS T02 DIR` (which `W3OUTG` on Kokkos computes) and the partition file; +- `pf`: the same plus all 17 partition fields (`PHS PTP PLP PDIR PSPR PWS PDP PQP PPE PGW PSW + PTM10 PT01 PT02 PEP TWS PNR`). + +Each ran as `WW3GPU_SWITCH=switch_lab_ WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1 JOBS=4 +OMP_NUM_THREADS=4 bash scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos `, "before" with +`SRCE,XYP3,KTP3,OUTG,IOPE,UPDV` and "after" with `CPRT` added, and `partition.ww3` was compared +with `cmp` by hand (the script does not list it). WW3's regtest `ww3_tpt1.1` (one multi-modal +spectrum on 18 sea points, one grid per partitioning method `PTM = 1..5`, not in `matrix.base`) +ran through the matrix driver's `regtests/cmake/run_entry.cmake` with the configuration of +`WW3GPU_KOKKOS=CPRT WW3GPU_SWITCH_WORDS=SHRD just ww3-gpu-matrix-list serial`, options `-N -g +PTM`, `WW_KOKKOS_VA_POISON=1`. + +## Result + +| Run | `VA` device-to-host copies, 12 h, 13 output times | Files | +|---|---|---| +| `p6`, `switch_lab_shrd`, before | 13 (275.71 MB) | every model file and `partition.ww3` byte-identical to the Fortran of the same executable and to `--ww3` (v) | +| `p6`, `switch_lab_shrd`, after | 1 (21.21 MB, `W3WAVE`'s read at its return, which U1 removes on its own branch) | the same (v) | +| `pf`, `switch_lab_shrd`, before (with `ww_outg_part`) | 13 | the same (v) | +| `pf`, `switch_lab_shrd`, after | 1 | the same (v) | +| `pf`, `switch_lab_ompg`, after | 1 | the same (v) | + +- (v) In every run `VA host copies poisoned` was 97: the host `VA` was filled with NaN after each + kernel write, and no output changed, so no host reader read a stale `VA`. +- (v) After, per output time, what a device would move instead of the 21.21 MB of `VA` + (`WW_KOKKOS_CPRT_COUNT=1` and `WW_KOKKOS_OUTG_COUNT=1`, projected on the host backend, totals + divided by 13 calls): the table back, 1 135 640 B (`ICPRT` 76 168 B and the used `DTPRT` + columns); the inputs in, 2 212 324 B (`CG` and `WN` are 1.98 MB of it); with the 17 partition + fields requested, those fields back, 3 387 072 B. 0 fallbacks to the Fortran. +- (v) `ww3_tpt1.1`, `PTM = 1` to `5`: `test.comp` found 0 of 22 files different and `nccmp-tol` + passed on each; each Kokkos run made 2 `W3CPRT` Kokkos calls on 18 sea points, 0 fallbacks. + The staged switch had `PR3 UQ` in place of the test's `PR0` ⚠: the fork's `KOKKOS` build does + not compile with `PR0` (`w3kokkosmd.F90:3183` at `80644ba9` uses `W3ADATMD`'s PR3 arrays), so + neither `ww3_tpt1.1` as written nor `ww3_tnc1` (the one serial `matrix.base` entry with partition + fields, `PR0`) can be built with `KOKKOS`; both runs of each pair used the one executable. Two of + the five entries first failed to link (`DSO missing from command line`, `libgfortran`) in a + build directory left by an interrupted run; removing `build_903` and `build_905` fixed it ⚠. +- (v) A WW3 defect found on the way: the formatted partition file (`TYPE%PARTITION%FORMAT = T`) + stops `ww3_shel` with a gfortran format error in the Fortran itself. Format 942 is + `(I3,3F8.2,2F9.2,F7.2)` (`w3iosfmd.F90:739`) and `:706` writes `IP` and 15 values of + `DTPRT(:,IP+IOFF)`; the format reverts to `I3` for a real (backtrace under `gdb`, Fortran run of + the same executable). The runs above use `FORMAT = F`. Not changed: WW3 is read-only. +- ⚠ Nothing ran on a GPU, and the CUDA build of the new code was not compiled. The wall times the + script prints are one run each on a shared machine and are not quoted. + +## What it changes + +`W3CPRT` and `W3OUTG`'s partition fields no longer copy the spectrum back: with `CPRT` on, an +output time with partitioning moves only the table and the fields written. Open: `CPRT` is not +in the `resident` preset (section 25 of PATCH.md), the fallbacks are not reported to +`WW_KOKKOS_STRICT` (section 24; this fork branch predates it), the regtest matrix has no +`KOKKOS`-buildable entry with partitioning, and the formatted partition file is broken in WW3 +at the pin. diff --git a/docs/log/README.md b/docs/log/README.md index 5df6b9fe..970d8e9e 100644 --- a/docs/log/README.md +++ b/docs/log/README.md @@ -70,3 +70,4 @@ Date, author, issue/PR/WFIP. | 2026-10-18 | [The CUDA build of `ww_kokkos` and of the fork for the resident preset, build only: 51 nvcc warnings, none in a wired part](2026-10-18-cuda-build-only.md) | plan 2026-10-18-002 U5, ADR-0014 | | 2026-10-18 | [The boundary and track outputs read only their columns of the resident `VA`](2026-10-18-resident-output-readers.md) | #42, WFIP-0005 | | 2026-10-18 | [`ww3_multi` and the resident `VA`: the defect reproduced, column syncs at the exchanges, no read at `W3WAVE`'s return](2026-10-18-ww3-multi-resident-va.md) | plan 2026-10-18-002 U1 + U7, WFIP-0005 | +| 2026-10-18 | [`W3CPRT` on the resident `VA`: partitioning without copying the spectrum back](2026-10-18-cprt-resident.md) | plan 2026-10-18-002 U9, WFIP-0005 | diff --git a/kokkos/PORT_RANKING.md b/kokkos/PORT_RANKING.md index b5207a61..f313cd1e 100644 --- a/kokkos/PORT_RANKING.md +++ b/kokkos/PORT_RANKING.md @@ -92,7 +92,7 @@ The record is the `regtested` column of `kokkos/ports.tsv`. | `w3uqckmd:w3qck3` | ported | `prop_qck3.hpp` | no: no regtest matrix yet; the CPU replay ran it inside W3XYP3: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS; the transparency branches with obstructions were not checked for coverage | | `w3uqckmd:w3qck1` | ported | `prop_qck1.hpp` | no: no regtest matrix yet; the CPU replay ran it inside W3KTP3 (refraction on): scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS | | `w3pro3md:w3ktp3` | ported | `prop_ktp3.hpp`, `prop3_shim.cpp`, `prop3_c.hpp` | no: no regtest matrix yet; the CPU replay ran it: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS, 441 792 point calls in 48 calls, refraction only: FLCK and FLCUR are false in bench_small, so the wavenumber shift (W3QCK2) never ran; on the resident VA (ww_ktp3_batch_va, fork 90631fad, WFIP-0005 unit 3b, PATCH.md section 13) the same replay byte-identical on 2026-10-17 on switch_lab_ompg and switch_lab_shrd, with and without WW_KOKKOS_VA_RESIDENT=1, VA traffic H2D 1 / D2H 13 per full run | -| `w3dispmd:wavnu1` | ported | `prop_wavnu1.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist | +| `w3dispmd:wavnu1` | ported | `prop_wavnu1.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md); only PTMEAN's call, not W3INIT's or W3ULEV's | | `w3uqckmd:w3qck2` | ported | `prop_qck2.hpp` | no: never run: W3KTP3 calls it only when FLCK is true, and the full and smoke cases have no current-induced wavenumber shift | | `w3wavemd:w3gath` | ported | `wave_gath_scat.hpp`, `wave_support_shim.cpp` | yes: scripts/ww3-gpu.sh --ww3-gpu full (bench_small) --cpu on 2026-10-16 (OpenMP backend, gfortran/g++ 16.2.0), with WW_KOKKOS_ALL=1 and with the W3WAVE support parts alone: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, the binary files byte-identical to --ww3, 13 824 calls (kokkos/src/fortran_iface/PATCH.md, section 8) | | `w3wavemd:w3scat` | ported | `wave_gath_scat.hpp`, `wave_support_shim.cpp` | yes: scripts/ww3-gpu.sh --ww3-gpu full (bench_small) --cpu on 2026-10-16 (OpenMP backend, gfortran/g++ 16.2.0), with WW_KOKKOS_ALL=1 and with the W3WAVE support parts alone: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, the binary files byte-identical to --ww3, 13 824 calls (kokkos/src/fortran_iface/PATCH.md, section 8) | @@ -126,12 +126,13 @@ The record is the `regtested` column of `kokkos/ports.tsv`. | `pdlib_w3profsmd:apply_boundary_condition` | ported | `pdlib_boundary.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim with the MPI hooks | | `pdlib_w3profsmd:pdlib_w3xypfsfct2` | ported | `pdlib_fsfct2.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); no regtest selects EXPFSFCT | | `pdlib_w3profsmd:pdlib_w3xypug` | ported | `pdlib_w3xypug.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3WAVE hunk | -| `w3partmd:ptnghb` | ported | `part_ptnghb.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk | -| `w3partmd:pt_fld` | ported | `part_fld.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk | -| `w3partmd:ptsort` | ported | `part_ptsort.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk | -| `w3partmd:ptmean` | ported | `part_ptmean.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs W3PART and a shim | +| `w3partmd:ptnghb` | ported | `part_ptnghb.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3partmd:pt_fld` | ported | `part_fld.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3partmd:ptsort` | ported | `part_ptsort.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3partmd:ptmean` | ported | `part_ptmean.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | | `w3snl3md:insnl3` | ported | `snl3_tables.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3IOGR hunk | -| `w3partmd:w3part` | ported | `part_w3part.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3IOSF hunk | +| `w3partmd:w3part` | ported | `part_w3part.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3iosfmd:w3cprt` | ported | `iosf_w3cprt.hpp`, `cprt_shim.cpp` | no: regtest matrix pending; the CPU replay ran it on 2026-10-18: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT on two partition variants of bench_small (switch_lab_shrd, and switch_lab_ompg for the fields variant), WW_KOKKOS_VA_POISON=1: every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 (docs/log/2026-10-18-cprt-resident.md) | | `w3snl3md:w3snl3` | ported | `snl3_gmd.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3SRCE hunk | | `w3src0md:w3spr0` | ported | `srce_st0_spr0.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests | | `w3src1md:w3spr1` | ported | `srce_st1_spr1.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests | @@ -259,9 +260,9 @@ The record is the `regtested` column of `kokkos/ports.tsv`. | `w3uostmd:uost_sourceterm_setgrid` | ported | `uost_uost_sourceterm_setgrid.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | | `w3uostmd:uost_srctrmcompute` | ported | `uost_uost_srctrmcompute.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | | `w3wavset:fd_wave_setup_apply_fct` | ported | `wavset_fd_wave_setup_apply_fct.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | -| `w3partmd:fifo_add` | ported | `part_fld.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | -| `w3partmd:fifo_first` | ported | `part_fld.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | -| `w3partmd:fifo_empty` | ported | `part_fld.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | +| `w3partmd:fifo_add` | ported | `part_fld.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3partmd:fifo_first` | ported | `part_fld.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | +| `w3partmd:fifo_empty` | ported | `part_fld.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | | `w3wavset:differentiate_xydir` | ported | `wavset_differentiate_xydir.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | | `w3wavset:fd_wave_setup_scalar_prod` | ported | `wavset_fd_wave_setup_scalar_prod.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | | `w3wavset:trig_compute_diff` | ported | `wavset_trig_compute_diff.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk | diff --git a/kokkos/PORT_STATUS.md b/kokkos/PORT_STATUS.md index a7edbc8b..12eff9c8 100644 --- a/kokkos/PORT_STATUS.md +++ b/kokkos/PORT_STATUS.md @@ -45,11 +45,11 @@ Profile: full (bench_small, 120 x 80, 9 520 sea points, 12 h), Release -pg, swit | level | run time | lines | routines | files | what it counts (WW3@03adda91) | |---|---|---|---|---|---| -| translated | **99.3 %** | 67711 of 263375 (25.71 %) | 312 of 1023 (30.50 %) | 64 of 147 | a Kokkos translation exists in kokkos/src | +| translated | **99.3 %** | 67947 of 263375 (25.80 %) | 313 of 1023 (30.60 %) | 65 of 147 | a Kokkos translation exists in kokkos/src | | ported | **99.3 %** | 17022 of 263375 (6.46 %) | 60 of 1023 (5.87 %) | 23 of 147 | WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag | | regtested | **48.6 %** | 2689 of 263375 (1.02 %) | 12 of 1023 (1.17 %) | 6 of 147 | WW3's regtests ran with it on Kokkos and agreed with the Fortran | -The port itself: 58429 lines in 347 files, the C++ under kokkos/src (every .cpp and .hpp line). +The port itself: 59232 lines in 350 files, the C++ under kokkos/src (every .cpp and .hpp line). On a GPU: none yet. The CUDA build links; the RTX 4090 run, `just smoke-ww3-gpu gpu`, is the owner's, supervised. diff --git a/kokkos/PORT_TABLE.tsv b/kokkos/PORT_TABLE.tsv index d96daf12..26798ec0 100644 --- a/kokkos/PORT_TABLE.tsv +++ b/kokkos/PORT_TABLE.tsv @@ -26,78 +26,77 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi 18 w3initmd:w3mpip model/src/w3initmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — 19 w3iogomd:w3readflgrd model/src/w3iogomd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — 20 w3iopomd:nf90_err_check model/src/w3iopomd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -21 w3iosfmd:w3cprt model/src/w3iosfmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -22 w3nmlshelmd:read_domain_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -23 w3nmlshelmd:read_homogeneous_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -24 w3nmlshelmd:read_input_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -25 w3nmlshelmd:read_output_date_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -26 w3nmlshelmd:read_output_path_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -27 w3nmlshelmd:read_output_type_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -28 w3nmlshelmd:report_domain_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -29 w3nmlshelmd:report_homogeneous_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -30 w3nmlshelmd:report_input_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -31 w3nmlshelmd:report_output_date_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -32 w3nmlshelmd:report_output_path_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -33 w3nmlshelmd:report_output_type_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -34 w3nmlshelmd:w3nmlshel model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -35 w3odatmd:w3dmo2 model/src/w3odatmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -36 w3odatmd:w3dmo5 model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -37 w3odatmd:w3nout model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -38 w3odatmd:w3seto model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -39 w3parall:synchronize_ipgl_etc_array model/src/w3parall.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -40 w3servmd:itrace model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -41 w3servmd:nextln model/src/w3servmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -42 w3servmd:print_memcheck model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -43 w3servmd:str_to_upper model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -44 w3servmd:strsplit model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -45 w3servmd:wwdate model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -46 w3servmd:wwtime model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -47 w3timemd:d2j model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -48 w3timemd:dsec21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -49 w3timemd:stme21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -50 w3timemd:t2d model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -51 w3timemd:tdiff model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -52 w3timemd:tick21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -53 w3triamd:coordmax model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -54 w3triamd:fix_periodcity model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -55 w3triamd:is_in_ungrid model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -56 w3triamd:nvectri model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -57 w3triamd:spatial_grid model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -58 w3triamd:ug_gradients model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -59 w3triamd:w3nestug model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -60 w3updtmd:w3ubpt model/src/w3updtmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -61 w3updtmd:w3ucur model/src/w3updtmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -62 w3wdatmd:w3dimw model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -63 w3wdatmd:w3ndat model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -64 w3wdatmd:w3setw model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — -65 yowelementpool:belongto model/src/yowelementpool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -66 yowexchangemodule:creatempitypes model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -67 yowexchangemodule:finalizeexchangemodule model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -68 yowexchangemodule:initnbrdomains model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -69 yowexchangemodule:pdlib_exchange1dreal model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -70 yowexchangemodule:pdlib_exchange2dreal_zero model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -71 yowexchangemodule:setdimsize model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -72 yowfunction:computelistnp_listnpa_listiplg model/src/yowfunction.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -73 yowfunction:computelistnp_listnpa_listiplg_kernel model/src/yowfunction.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -74 yownodepool:connnodes model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -75 yownodepool:insertconnnode model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -76 yownodepool:nodes model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -77 yowpdlibmain:assignmesh model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -78 yowpdlibmain:computeia_ja_posi_nnz model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -79 yowpdlibmain:computetria_ien_si_ccon model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -80 yowpdlibmain:exchangeghostids model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -81 yowpdlibmain:findconndomains model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -82 yowpdlibmain:findconnnodes model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -83 yowpdlibmain:findghostnodes model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -84 yowpdlibmain:initfromgriddim model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -85 yowpdlibmain:initmpi model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -86 yowpdlibmain:postpartition model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -87 yowpdlibmain:postpartition2 model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -88 yowpdlibmain:prepartition model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -89 yowpdlibmain:real_mpi_barrier_pdlib model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -90 yowpdlibmain:runparmetis model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -91 yowrankmodule:exchangeiplg model/src/yowrankmodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — -92 yowrankmodule:initrankmodule model/src/yowrankmodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +21 w3nmlshelmd:read_domain_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +22 w3nmlshelmd:read_homogeneous_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +23 w3nmlshelmd:read_input_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +24 w3nmlshelmd:read_output_date_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +25 w3nmlshelmd:read_output_path_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +26 w3nmlshelmd:read_output_type_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +27 w3nmlshelmd:report_domain_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +28 w3nmlshelmd:report_homogeneous_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +29 w3nmlshelmd:report_input_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +30 w3nmlshelmd:report_output_date_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +31 w3nmlshelmd:report_output_path_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +32 w3nmlshelmd:report_output_type_nml model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +33 w3nmlshelmd:w3nmlshel model/src/w3nmlshelmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +34 w3odatmd:w3dmo2 model/src/w3odatmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +35 w3odatmd:w3dmo5 model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +36 w3odatmd:w3nout model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +37 w3odatmd:w3seto model/src/w3odatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +38 w3parall:synchronize_ipgl_etc_array model/src/w3parall.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +39 w3servmd:itrace model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +40 w3servmd:nextln model/src/w3servmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +41 w3servmd:print_memcheck model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +42 w3servmd:str_to_upper model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +43 w3servmd:strsplit model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +44 w3servmd:wwdate model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +45 w3servmd:wwtime model/src/w3servmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +46 w3timemd:d2j model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +47 w3timemd:dsec21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +48 w3timemd:stme21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +49 w3timemd:t2d model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +50 w3timemd:tdiff model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +51 w3timemd:tick21 model/src/w3timemd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +52 w3triamd:coordmax model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +53 w3triamd:fix_periodcity model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +54 w3triamd:is_in_ungrid model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +55 w3triamd:nvectri model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +56 w3triamd:spatial_grid model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +57 w3triamd:ug_gradients model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +58 w3triamd:w3nestug model/src/w3triamd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +59 w3updtmd:w3ubpt model/src/w3updtmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +60 w3updtmd:w3ucur model/src/w3updtmd.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +61 w3wdatmd:w3dimw model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +62 w3wdatmd:w3ndat model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +63 w3wdatmd:w3setw model/src/w3wdatmd.F90 0.00 full todo — no no — no — — none: ADR-0010 — +64 yowelementpool:belongto model/src/yowelementpool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +65 yowexchangemodule:creatempitypes model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +66 yowexchangemodule:finalizeexchangemodule model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +67 yowexchangemodule:initnbrdomains model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +68 yowexchangemodule:pdlib_exchange1dreal model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +69 yowexchangemodule:pdlib_exchange2dreal_zero model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +70 yowexchangemodule:setdimsize model/src/yowexchangemodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +71 yowfunction:computelistnp_listnpa_listiplg model/src/yowfunction.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +72 yowfunction:computelistnp_listnpa_listiplg_kernel model/src/yowfunction.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +73 yownodepool:connnodes model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +74 yownodepool:insertconnnode model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +75 yownodepool:nodes model/src/yownodepool.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +76 yowpdlibmain:assignmesh model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +77 yowpdlibmain:computeia_ja_posi_nnz model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +78 yowpdlibmain:computetria_ien_si_ccon model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +79 yowpdlibmain:exchangeghostids model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +80 yowpdlibmain:findconndomains model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +81 yowpdlibmain:findconnnodes model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +82 yowpdlibmain:findghostnodes model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +83 yowpdlibmain:initfromgriddim model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +84 yowpdlibmain:initmpi model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +85 yowpdlibmain:postpartition model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +86 yowpdlibmain:postpartition2 model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +87 yowpdlibmain:prepartition model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +88 yowpdlibmain:real_mpi_barrier_pdlib model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +89 yowpdlibmain:runparmetis model/src/yowpdlibmain.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +90 yowrankmodule:exchangeiplg model/src/yowrankmodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — +91 yowrankmodule:initrankmodule model/src/yowrankmodule.F90 0.00 pdlib todo — no no — no — — none: ADR-0010 — — w3snl1md:w3snl1 model/src/w3snl1md.F90:115-473 48.35 full regtested — yes: kokkos/src/ww_kokkos/snl1_dia.cpp, kokkos/src/fortran_iface/snl1_shim.cpp yes: WW3@c9a6a0c6 (h0ffmann/WW3 ww3-gpu/kokkos-st4, which carries the hoff/kokkos-snl1 hunks) calls it under WW_KOKKOS_SNL1=1 passed yes byte-identical on all 100 (test.comp); byte-identical on all 25 (test.comp) nccmp-tol 71 PASS, 29 n/a, 0 FAIL; nccmp-tol 25 PASS none: ADR-0010 kokkos/port_report/W3SNL1_KOKKOS.md — pdlib_w3profsmd:pdlib_jacobi_gauss_seidel_block model/src/w3profsmd_pdlib.F90:5463-6406 37.54 pdlib translated — yes: kokkos/src/ww_kokkos/pdlib_jgs_block.hpp no: header-only; no shim and no W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/PDLIB_JACOBI_GAUSS_SEIDEL_BLOCK_KOKKOS.md — w3src4md:w3sds4 model/src/w3src4md.F90:2043-2659 23.23 full wired — yes: kokkos/src/ww_kokkos/st4_sds4.hpp,kokkos/src/ww_kokkos/st4_libmvec.hpp,kokkos/src/fortran_iface/st4_shim.cpp yes: WW3@c9a6a0c6 (h0ffmann/WW3 ww3-gpu/kokkos-st4) calls it under WW_KOKKOS_SDS4/SIN4/SPR4=1 pending no — — none: ADR-0010 kokkos/port_report/W3SDS4_KOKKOS.md @@ -119,15 +118,15 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3wavemd:w3wave model/src/w3wavemd.F90:229-2997 0.15 pdlib translated — yes: kokkos/src/ww_kokkos/wave_w3wave.hpp,kokkos/src/ww_kokkos/wave_w3wave_state.hpp,kokkos/src/ww_kokkos/wave_w3wave_hooks.hpp no: header-only host driver; no shim, no hook table filled, no W3WAVE hunk on the fork pending no — — none: ADR-0010 kokkos/port_report/W3WAVE_KOKKOS.md — w3sln1md:w3sln1 model/src/w3sln1md.F90:56-211 0.13 full wired — yes: kokkos/src/ww_kokkos/srce_ln1.hpp yes: inside W3SRCE/WW_KOKKOS_SRCE at 357f575c pending no — — none: ADR-0010 kokkos/port_report/W3SLN1_KOKKOS.md — w3wavemd:w3gath model/src/w3wavemd.F90:3024-3329 0.11 full regtested — yes: kokkos/src/ww_kokkos/wave_gath_scat.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_GATH=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/W3GATH_KOKKOS.md -— w3partmd:pt_fld model/src/w3partmd.F90:839-1156 0.05 pdlib translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no W3PART hunk yet pending no — — none: ADR-0010 kokkos/port_report/PT_FLD_KOKKOS.md +— w3partmd:pt_fld model/src/w3partmd.F90:839-1156 0.05 pdlib translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/PT_FLD_KOKKOS.md — w3iogomd:w3outg model/src/w3iogomd.F90:1201-2380 0.04 pdlib wired — yes: kokkos/src/ww_kokkos/iogo_w3outg.hpp,kokkos/src/fortran_iface/outg_shim.cpp,kokkos/src/fortran_iface/outg_c.hpp yes: WW_KOKKOS_OUTG (opt-in, not under WW_KOKKOS_ALL), in place of CALL W3OUTG at w3wavemd.F90 section 4.b (W3KOKKOS_OUTG in w3kokkos_outgmd.F90, ww_outg_va in kokkos/src/fortran_iface/outg_shim.cpp) at e0cc8ee9, on the resident VA; the Fortran W3OUTG when partitions, CALC_U3STOKES, SKEWNESS or CALC_WBT fields are requested pending no — — none: ADR-0010 kokkos/port_report/W3OUTG_KOKKOS.md -— w3partmd:ptmean model/src/w3partmd.F90:1175-1538 0.02 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptmean.hpp no: header-only; no shim and no W3PART/W3IOSF hunk yet pending no — — none: ADR-0010 kokkos/port_report/PTMEAN_KOKKOS.md -— w3partmd:ptsort model/src/w3partmd.F90:511-613 0.02 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptsort.hpp no: header-only; no shim and no W3PART hunk yet pending no — — none: ADR-0010 kokkos/port_report/PTSORT_KOKKOS.md +— w3partmd:ptmean model/src/w3partmd.F90:1175-1538 0.02 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptmean.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/PTMEAN_KOKKOS.md +— w3partmd:ptsort model/src/w3partmd.F90:511-613 0.02 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptsort.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/PTSORT_KOKKOS.md — pdlib_w3profsmd:apply_boundary_condition_va model/src/w3profsmd_pdlib.F90:5037-5148 0.01 pdlib translated — yes: kokkos/src/ww_kokkos/pdlib_boundary_va.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/APPLY_BOUNDARY_CONDITION_VA_KOKKOS.md — w3adatmd:w3dima model/src/w3adatmd.F90:847-1509 0.01 full translated — yes: kokkos/src/ww_kokkos/adat_w3dima.hpp,kokkos/src/ww_kokkos/adat_wadat.hpp no: host-side allocation routine; no shim and no W3INIT/W3IOGO hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3DIMA_KOKKOS.md — w3gdatmd:w3dims model/src/w3gdatmd.F90:1929-2154 0.01 full translated — yes: kokkos/src/ww_kokkos/gdat_w3dims.hpp,kokkos/src/ww_kokkos/gdat_sgrd.hpp no: host-only header; no shim and no W3IOGR hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3DIMS_KOKKOS.md — w3parall:init_get_isea model/src/w3parall.F90:1396-1504 0.01 pdlib translated — yes: kokkos/src/ww_kokkos/parall_init_get_isea.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/INIT_GET_ISEA_KOKKOS.md -— w3partmd:w3part model/src/w3partmd.F90:138-497 0.01 pdlib translated — yes: kokkos/src/ww_kokkos/part_w3part.hpp no: header-only; no shim and no W3IOSF hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3PART_KOKKOS.md +— w3partmd:w3part model/src/w3partmd.F90:138-497 0.01 pdlib translated — yes: kokkos/src/ww_kokkos/part_w3part.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/W3PART_KOKKOS.md — w3src4md:calc_ustar model/src/w3src4md.F90:1849-2011 0.01 pdlib wired — yes: kokkos/src/ww_kokkos/st4_calc_ustar.hpp yes: WW3@c9a6a0c6 (h0ffmann/WW3 ww3-gpu/kokkos-st4) calls it under WW_KOKKOS_SDS4/SIN4/SPR4=1 pending no — — none: ADR-0010 kokkos/port_report/CALC_USTAR_KOKKOS.md — w3updtmd:w3ulev model/src/w3updtmd.F90:2066-2614 0.01 pdlib translated — yes: kokkos/src/ww_kokkos/updt_w3ulev.hpp no: header-only launcher; no shim and no W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3ULEV_KOKKOS.md — w3updtmd:w3uwnd model/src/w3updtmd.F90:488-822 0.01 full regtested — yes: kokkos/src/ww_kokkos/updt_uwnd.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_UWND=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/W3UWND_KOKKOS.md @@ -149,7 +148,7 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3adatmd:w3xdma model/src/w3adatmd.F90:1522-2409 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/adat_w3xdma.hpp,kokkos/src/ww_kokkos/adat_w3xdma_wadat.hpp no: host-side allocation routine; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3XDMA_KOKKOS.md — w3adatmd:w3xeta model/src/w3adatmd.F90:3112-3300 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/adat_w3xeta.hpp no: host-side pointer selection; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3XETA_KOKKOS.md — w3dispmd:distab model/src/w3dispmd.F90:552-683 0.00 full regtested — yes: kokkos/src/ww_kokkos/disp_distab.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_DISTAB=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/DISTAB_KOKKOS.md -— w3dispmd:wavnu1 model/src/w3dispmd.F90:85-202 0.00 full translated — yes: kokkos/src/ww_kokkos/prop_wavnu1.hpp no: header-only device function; no shim and no W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/WAVNU1_KOKKOS.md +— w3dispmd:wavnu1 model/src/w3dispmd.F90:85-202 0.00 full translated — yes: kokkos/src/ww_kokkos/prop_wavnu1.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/WAVNU1_KOKKOS.md — w3dispmd:wavnu2 model/src/w3dispmd.F90:204-346 0.00 full regtested — yes: kokkos/src/ww_kokkos/disp_distab.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it inside DISTAB under WW_KOKKOS_DISTAB=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/WAVNU2_KOKKOS.md — w3dispmd:wavnu3 model/src/w3dispmd.F90:348-454 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/disp_wavnu3.hpp no: header-only device function; no shim yet pending no — — none: ADR-0010 kokkos/port_report/WAVNU3_KOKKOS.md — w3fldsmd:w3fldg model/src/w3fldsmd.F90:935-1474 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/flds_w3fldg.hpp no: host-staged field I/O routine; no shim and no ww3_shel hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3FLDG_KOKKOS.md @@ -169,10 +168,11 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3iopomd:w3iopo model/src/w3iopomd.F90:2131-2626 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/iopo_w3iopo.hpp,kokkos/src/ww_kokkos/iopo_state.hpp no: header-only host driver; no shim and no W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3IOPO_KOKKOS.md — w3iopomd:w3iopp model/src/w3iopomd.F90:242-859 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/iopo_w3iopp.hpp,kokkos/src/ww_kokkos/iopo_state.hpp no: header-only host driver; no shim and no W3INIT hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3IOPP_KOKKOS.md — w3iorsmd:w3iors model/src/w3iorsmd.F90:113-1600 0.00 full translated — yes: kokkos/src/ww_kokkos/iors_w3iors.hpp,kokkos/src/ww_kokkos/iors_stream.hpp no: header-only host driver; no shim and no W3INIT or W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3IORS_KOKKOS.md +— w3iosfmd:w3cprt model/src/w3iosfmd.F90:109-344 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/iosf_w3cprt.hpp, kokkos/src/fortran_iface/cprt_shim.cpp no: wired on the fork branch ww3-gpu/u9-cprt (W3KOKKOS_CPRT under WW_KOKKOS_CPRT=1), not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/W3CPRT_KOKKOS.md — w3parall:get_jsea_ibelong model/src/w3parall.F90:1265-1384 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/parall_get_jsea_ibelong.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/GET_JSEA_IBELONG_KOKKOS.md — w3parall:init_get_jsea_isproc model/src/w3parall.F90:1152-1252 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/parall_init_get_jsea_isproc.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/INIT_GET_JSEA_ISPROC_KOKKOS.md — w3parall:set_up_nseal_nsealm model/src/w3parall.F90:1030-1139 0.00 full translated — yes: kokkos/src/ww_kokkos/parall_set_up_nseal_nsealm.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/SET_UP_NSEAL_NSEALM_KOKKOS.md -— w3partmd:ptnghb model/src/w3partmd.F90:630-821 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptnghb.hpp no: header-only; no shim and no W3PART hunk yet pending no — — none: ADR-0010 kokkos/port_report/PTNGHB_KOKKOS.md +— w3partmd:ptnghb model/src/w3partmd.F90:630-821 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/part_ptnghb.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/PTNGHB_KOKKOS.md — w3pro3md:w3map3 model/src/w3pro3md.F90:139-518 0.00 full regtested — yes: kokkos/src/ww_kokkos/prop_map3.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_MAP3=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/W3MAP3_KOKKOS.md — w3pro3md:w3mapt model/src/w3pro3md.F90:527-625 0.00 full regtested — yes: kokkos/src/ww_kokkos/prop_mapt.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_MAPT=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp passed yes not recorded — none: ADR-0010 kokkos/port_report/W3MAPT_KOKKOS.md — w3sbt4md:insbt4 model/src/w3sbt4md.F90:145-231 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/srce_bt4.hpp no: header-only; no shim and no W3SRCE hunk yet pending no — — none: ADR-0010 kokkos/port_report/INSBT4_KOKKOS.md @@ -213,9 +213,9 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3iogomd:vplus_d model/src/w3iogomd.F90:4820-4860 — — translated — yes: kokkos/src/ww_kokkos/iogo_vplus_d.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/VPLUS_D_KOKKOS.md — w3parall:prop_freq_shift model/src/w3parall.F90:604-739 — — translated — yes: kokkos/src/ww_kokkos/prop_freq_shift.hpp no: header-only device function; no shim yet pending no — — none: ADR-0010 kokkos/port_report/PROP_FREQ_SHIFT_KOKKOS.md — w3parall:prop_refraction_pr1 model/src/w3parall.F90:285-431 — — translated — yes: kokkos/src/ww_kokkos/parall_prop_refraction_pr1.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/PROP_REFRACTION_PR1_KOKKOS.md -— w3partmd:fifo_add model/src/w3partmd.F90:1103-1113 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/FIFO_ADD_KOKKOS.md -— w3partmd:fifo_empty model/src/w3partmd.F90:1122-1133 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/FIFO_EMPTY_KOKKOS.md -— w3partmd:fifo_first model/src/w3partmd.F90:1142-1152 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/FIFO_FIRST_KOKKOS.md +— w3partmd:fifo_add model/src/w3partmd.F90:1103-1113 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/FIFO_ADD_KOKKOS.md +— w3partmd:fifo_empty model/src/w3partmd.F90:1122-1133 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/FIFO_EMPTY_KOKKOS.md +— w3partmd:fifo_first model/src/w3partmd.F90:1142-1152 — — translated — yes: kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/FIFO_FIRST_KOKKOS.md — w3pro1md:w3ktp1 model/src/w3pro1md.F90:900-1230 — — translated — yes: kokkos/src/ww_kokkos/prop_ktp1.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_KTP1=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) pending no — — none: ADR-0010 kokkos/port_report/W3KTP1_KOKKOS.md — w3pro1md:w3xyp1 model/src/w3pro1md.F90:301-873 — — translated — yes: kokkos/src/ww_kokkos/prop_xyp1.hpp,kokkos/src/ww_kokkos/prop_params1.hpp,kokkos/src/ww_kokkos/prop_params.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_XYP1=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) pending no — — none: ADR-0010 kokkos/port_report/W3XYP1_KOKKOS.md — w3pro2md:w3ktp2 model/src/w3pro2md.F90:1306-1710 — — translated — yes: kokkos/src/ww_kokkos/prop_ktp2.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_KTP2=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) pending no — — none: ADR-0010 kokkos/port_report/W3KTP2_KOKKOS.md diff --git a/kokkos/port_report/W3CPRT_KOKKOS.md b/kokkos/port_report/W3CPRT_KOKKOS.md new file mode 100644 index 00000000..859581a4 --- /dev/null +++ b/kokkos/port_report/W3CPRT_KOKKOS.md @@ -0,0 +1,82 @@ +# W3CPRT_KOKKOS + +**Status:** ported, phase 2 (on the resident spectrum), wired on the fork branch +`ww3-gpu/u9-cprt` (PATCH.md section 27, 2026-10-18): `W3KOKKOS_CPRT` replaces `W3WAVE`'s host read +of `VA` and `CALL W3CPRT ( IMOD )` under `WW_KOKKOS_CPRT=1` (opt-in, not under `WW_KOKKOS_ALL`). +Not at the `WW3/` pin until the integrator merges the branch. Run byte-identical on the CPU +(below); a GPU never ran it ⚠. + +`W3CPRT` partitions the spectrum of every local sea point at an output time with partitioning on +(`FLPART`: gridded partition fields or the partition file, `w3wavemd.F90:2680-2682` at `80644ba9`). +Per point it turns the action spectrum `N(k,theta)` of `VA` into the energy spectrum +`E(f,theta) = N * 2 pi sigma / c_g` (`w3iosfmd.F90:262-267`), calls the watershed partitioning +`W3PART` (`w3partmd.F90:138-497`) and stores the `NP+1` columns `XP(1:15,0:NP)` it returns, the +whole spectrum first and then each partition, in `DTPRT`, one point after the other, with +`ICPRT(JSEA,1) = NP+1` and `ICPRT(JSEA,2)` the point's first column (`w3iosfmd.F90:284-326`). +`W3OUTG` section 6 and `W3IOSF` read only that table. On the Fortran path the whole `VA` came back +from the device at every such output time for it; here only the table does. + +## What was ported + +| | | +|---|---| +| WW3 routine | `W3CPRT`, `model/src/w3iosfmd.F90:109-344` at h0ffmann/WW3 `80644ba9` (the `WW3/` pin), identical at NOAA-EMC/WW3 `03adda91` (v, `git log 03adda91..80644ba9 -- model/src/w3iosfmd.F90 model/src/w3partmd.F90 model/src/w3dispmd.F90` empty, 2026-10-18) | +| Kokkos code | [`../src/ww_kokkos/iosf_w3cprt.hpp`](../src/ww_kokkos/iosf_w3cprt.hpp): `w3cprt_point` (sections 2-4, `:245-279`) and `w3cprt_store` (section 5, `:284-303`); the launches are in the shim | +| Callees | `W3PART` and what it calls, translated before and called unchanged: [`part_w3part.hpp`](../src/ww_kokkos/part_w3part.hpp) ([`W3PART_KOKKOS.md`](W3PART_KOKKOS.md)), `ptsort`, `ptnghb` (the `NEIGH` table, built on the host once per `NK`, `NTH` and copied in), `pt_fld`, `ptmean`, `wavnu1` | +| Not ported, by switch | `W3_T` (test output, `:313-317`), `W3_S` (`STRACE`). `W3_MPI`: `INIT_GET_ISEA` is taken as `ISEA = JSEA` (`W3_SHRD`, `w3parall.F90:1236`); `W3KOKKOS_CPRT_SETUP` refuses the flag under `MPI` and `T` | +| `PTMETH` | 1 to 5, all of `W3PART`'s branches; `DIMXP` is 2 for 4 and 5 and `((NK+1)/2)*((NTH-1)/2)` otherwise (`:213-219`) | +| Left to the Fortran | A sea point with a NaN depth (`:270-274`: two lines and `STOP`) and a spectrum `PTMEAN` cannot take (`IFPMAX = 0`, where the Fortran reads out of bounds, [`PTMEAN_KOKKOS.md`](PTMEAN_KOKKOS.md)): `W3KOKKOS_CPRT` then reads `VA` to the host and calls the Fortran `W3CPRT` whole | +| Structural changes | Points run in parallel, each one the Fortran's serial code for that point. The Fortran's `TMP` array grows by `TSFAC*NSEAL` columns and is copied into `DTPRT(:,1:FINSIZ)` (`:288-326`); every column `1..FINSIZ` is written once by `:303`, so `DTPRT` is the concatenation of the points' `XP(:,0:NP)` in `JSEA` order, which an integer prefix sum over `NP+1` places directly. `PTMEAN`'s format 1000 (`XP ARRAY TOO SMALL`) is counted per call on the device and written by `W3KOKKOS_CPRT` the same number of times, with the same text and value `DIMXP+1` | +| C shim | [`../src/fortran_iface/cprt_shim.cpp`](../src/fortran_iface/cprt_shim.cpp), `ww_cprt_va` and `ww_cprt_fetch` of [`cprt_c.hpp`](../src/fortran_iface/cprt_c.hpp) (ABI `ww_kokkos` 0.10); the Fortran side is `W3KOKKOS_CPRT` of `model/src/w3kokkos_cprtmd.F90` on `ww3-gpu/u9-cprt` | +| Partition fields | `ww_outg_part` of [`outg_c.hpp`](../src/fortran_iface/outg_c.hpp) runs `W3OUTG` section 6 (`w3iogomd.F90:2242-2340`, `ww::iogo::w3outg_part`, the code that was inside `w3outg`, moved unchanged) after `ww_outg_va`, so a gridded output with partition fields no longer makes `W3KOKKOS_OUTG` fall back to the Fortran `W3OUTG` and its host read of `VA` | + +## Arithmetic + +The reference is the `-O3` GIMPLE (`-fdump-tree-optimized-lineno`) of `w3iosfmd.F90` compiled with +the flags and defines of WW3's CMake Release build under `switch_lab_shrd` (gfortran 16.2.0, the +`pratico#ww3` shell), read beside the GIMPLE of `cprt_shim.cpp` with the `openmp-release` flags +(`-O3 -march=x86-64-v3 -std=c++20 -ffp-contract=off -fopenmp`, g++ 16.2.0), 2026-10-18. + +- (v) `UABS = U10*ASF` is one product in both (`_198 = _187 * _197`, `uabs_99 = _95 * _98`). +- (v) `UDIR = U10D*RADE`: both multiply by the same folded constant, `5.72957763671875e+1`. +- (v) `FACT = TPI*SIG(IK)/CG(IK,ISEA)`: both compute `SIG*6.283185482025146484375e+0` and divide by + `CG` (`fact_569 = _232 / _245`, `fact_125 = _115 / _124`); the product is commuted, which changes + no bit. +- (v) `E2(IK,ITH) = VA(ITH+(IK-1)*NTH,JSEA)*FACT`, one product per bin, commuted in the C++. +- (v) No FMA: contraction is off in the C++ (`-ffp-contract=off`), and WW3's Fortran flags have + no `-march` (`-O3 -g -fno-second-underscore ...`, `flags.make` of the build), so the x86-64 + baseline has no FMA instruction to contract into; the products and the quotient above stay + separate statements in both GIMPLE dumps. +- (v) The libm functions the shim's object imports are `cosf`, `logf`, `atan2f`, `sqrtf`, `fmodf` + and `roundf` (`nm -u`), those of `W3PART`'s translation; their parity with `w3partmd.o` + (`cosf`, `logf`, `atan2f`, `lroundf`, inline `sqrtss` and `fprem`) is argued in + [`W3PART_KOKKOS.md`](W3PART_KOKKOS.md) and [`PTMEAN_KOKKOS.md`](PTMEAN_KOKKOS.md). +- (v) `ICPRT` and the `DTPRT` columns are integers and copies: no arithmetic. +- ⚠ On a device `cosf`, `logf`, `atan2f`, `fmodf` and `roundf` are CUDA's, not glibc's (as for + `W3PART`). The CUDA build of the shim was not compiled. + +## Benchmarks + +None. No timing without parity (`AGENTS.md`). The wall times the runs below print are one run +each on a machine shared with other jobs ⚠ and are not quoted. + +## How it was regtested + +`VA` traffic, the checks and the commands are in +[`PATCH.md` section 27](../src/fortran_iface/PATCH.md) and +[`docs/log/2026-10-18-cprt-resident.md`](../../docs/log/2026-10-18-cprt-resident.md). In short, on +the CPU (OpenMP backend, gfortran/g++ 16.2.0, i9-14900 shared, never `--gpu`): + +- `scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT` on two + throwaway variants of `bench_small` with partitioning on, `WW_KOKKOS_VA_POISON=1`: every model + file and `partition.ww3` byte-identical to the Fortran of the same executable and to `--ww3` + (v); see the log for the five runs. +- `VA` device-to-host copies in those 12-hour runs (13 output times): 13 before, 1 after, the read + at `W3WAVE`'s return (v). +- WW3's regtest `ww3_tpt1.1`, the five partitioning methods `PTM = 1..5`, through the matrix + driver's `regtests/cmake/run_entry.cmake` with `WW3GPU_KOKKOS=CPRT`: `test.comp` found 0 of 22 + files different and `nccmp-tol` passed on each; 2 Kokkos `W3CPRT` calls on 18 sea points per + run, no fallback (v). The staged switch had `PR3 UQ` for the test's `PR0`, since the fork's + `KOKKOS` build needs `PR3` ⚠; `ww3_tpt1.1` is not in `matrix.base`, and the one serial + `matrix.base` entry with partition fields, `ww3_tnc1`, is `PR0` and does not build with + `KOKKOS`, so the matrix proper has not run this port ⚠. diff --git a/kokkos/port_report/W3OUTG_KOKKOS.md b/kokkos/port_report/W3OUTG_KOKKOS.md index faf070b0..890ad12c 100644 --- a/kokkos/port_report/W3OUTG_KOKKOS.md +++ b/kokkos/port_report/W3OUTG_KOKKOS.md @@ -35,7 +35,7 @@ before every gridded output. | Not ported, by switch | `W3_IG1` `HSIG` (`:1556-1558`), `W3_O9` negative Hs (`:1995-2003`), `W3_O8` filter (`:2058-2068`), `W3_T` prints (`:2219-2237`, `:2368-2375`), `W3_S` STRACE. Neither lab switch sets them; `W3KOKKOS_OUTG_SETUP` refuses `WW_KOKKOS_OUTG=1` under any of them and under `MPI`. The `W3_OMPG` directives are covered: `W3OutgScalars::ompg` selects the object's two choices that differ (below) | | Not computed | Locals that reach no output: `NKH` (`:1512-1513`), `FXPMC` (`:1388`), `ETMSSL`, `ETMSSCL` (`:1577-1589`), `ETUSCX`, `ETUSCY` (`:1617-1618`), `UBS` (`:1590`, `:1975`), `SCREST` (`:1809-1811`). The object keeps the arrays as stores that nothing reads (`NKH` with its `logf`) and folds the scalar `FXPMC` away; none of them has an effect a Release build can observe (no `errno` in Fortran, no trapping: `-ffpe-trap` is in WW3's Debug flags only) | | Structural changes | The Fortran runs each section as a loop over `JSEA`; here one thread runs sections 2-4 for one point, which reads only that point's data. The local `EBD(NK,NSEAL)` is a caller work view; `SKEWNESS`'s `F2`, `SIGHF`, `DFIMHF`, `FAK` are work views with one column per point. `FLOLOC` is computed on the host and passed as booleans | -| C shim | [`../src/fortran_iface/outg_shim.cpp`](../src/fortran_iface/outg_shim.cpp), `ww_outg_va` of [`outg_c.hpp`](../src/fortran_iface/outg_c.hpp) (ABI `ww_kokkos` 0.7): `A` is the resident `VA` (`ww::resident::device_va`), the fields are W3ADATMD's arrays in place on Serial and OpenMP, device buffers on a device with only the fields a host reader needs copied back; the Fortran side is `W3KOKKOS_OUTG` of `model/src/w3kokkos_outgmd.F90` on `ww3-gpu/kokkos-outg` ([`PATCH.md` section 15](../src/fortran_iface/PATCH.md#15-w3outg-on-the-resident-spectrum-ww3-gpukokkos-outg)). The Fortran `W3OUTG` runs instead when `FLPART` (section 6, which reads `W3CPRT`'s table), `CALC_U3STOKES` (`FLOLOC(6,8)`, `(6,12)`), `SKEWNESS` (`FLOLOC(8,7:9)`, translated, not wired) or `CALC_WBT` (`FLOLOC(2,17)`) is requested, or a requested 3-D field is not allocated | +| C shim | [`../src/fortran_iface/outg_shim.cpp`](../src/fortran_iface/outg_shim.cpp), `ww_outg_va` of [`outg_c.hpp`](../src/fortran_iface/outg_c.hpp) (ABI `ww_kokkos` 0.7): `A` is the resident `VA` (`ww::resident::device_va`), the fields are W3ADATMD's arrays in place on Serial and OpenMP, device buffers on a device with only the fields a host reader needs copied back; the Fortran side is `W3KOKKOS_OUTG` of `model/src/w3kokkos_outgmd.F90` on `ww3-gpu/kokkos-outg` ([`PATCH.md` section 15](../src/fortran_iface/PATCH.md#15-w3outg-on-the-resident-spectrum-ww3-gpukokkos-outg)). Since 2026-10-18 section 6 (`FLPART`, which reads `W3CPRT`'s table and no `A`) runs through `ww_outg_part` after `ww_outg_va` ([`PATCH.md` section 27](../src/fortran_iface/PATCH.md), [`W3CPRT_KOKKOS.md`](W3CPRT_KOKKOS.md)). The Fortran `W3OUTG` runs instead when `CALC_U3STOKES` (`FLOLOC(6,8)`, `(6,12)`), `SKEWNESS` (`FLOLOC(8,7:9)`, translated, not wired) or `CALC_WBT` (`FLOLOC(2,17)`) is requested, or a requested 3-D field is not allocated | | Plan | [`W3WAVE_SUPPORT_KOKKOS_PLAN_202610.md`](../../docs/W3WAVE_SUPPORT_KOKKOS_PLAN_202610.md), issue #42 | ## Arithmetic diff --git a/kokkos/ports.tsv b/kokkos/ports.tsv index bd48ea02..34706985 100644 --- a/kokkos/ports.tsv +++ b/kokkos/ports.tsv @@ -41,7 +41,7 @@ w3pro3md:w3xyp3 model/src/w3pro3md.F90:641-1477 837 ported translated 2026-10-16 w3uqckmd:w3qck3 model/src/w3uqckmd.F90:925-1336 412 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run; wired 2026-10-17 (kokkos/src/fortran_iface/PATCH.md section 11) no: no regtest matrix yet; the CPU replay ran it inside W3XYP3: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS; the transparency branches with obstructions were not checked for coverage kokkos/src/ww_kokkos/prop_qck3.hpp yes: inside the Kokkos W3XYP3 (its two sweeps), under WW_KOKKOS_XYP3 at w3wavemd.F90:2022-2023 at fcdfabef w3uqckmd:w3qck1 model/src/w3uqckmd.F90:117-480 364 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run; wired 2026-10-17 (kokkos/src/fortran_iface/PATCH.md section 11) no: no regtest matrix yet; the CPU replay ran it inside W3KTP3 (refraction on): scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS kokkos/src/ww_kokkos/prop_qck1.hpp yes: inside the Kokkos W3KTP3 (the direction pass), under WW_KOKKOS_KTP3 at w3wavemd.F90:1864-1865 and 2224-2225 at fcdfabef w3pro3md:w3ktp3 model/src/w3pro3md.F90:1519-1961 443 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run; wired 2026-10-17 (kokkos/src/fortran_iface/PATCH.md section 11) no: no regtest matrix yet; the CPU replay ran it: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos XYP3,KTP3,SRCE on 2026-10-17 (OpenMP backend, gfortran/g++ 16.2.0), switch_lab_ompg and switch_lab_shrd, and WW_KOKKOS_PROP3_BATCH_MIB=64: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, out_grd.ww3 byte-identical to --ww3, nccmp-tol PASS, 441 792 point calls in 48 calls, refraction only: FLCK and FLCUR are false in bench_small, so the wavenumber shift (W3QCK2) never ran; on the resident VA (ww_ktp3_batch_va, fork 90631fad, WFIP-0005 unit 3b, PATCH.md section 13) the same replay byte-identical on 2026-10-17 on switch_lab_ompg and switch_lab_shrd, with and without WW_KOKKOS_VA_RESIDENT=1, VA traffic H2D 1 / D2H 13 per full run kokkos/src/ww_kokkos/prop_ktp3.hpp,kokkos/src/fortran_iface/prop3_shim.cpp,kokkos/src/fortran_iface/prop3_c.hpp yes: WW_KOKKOS_KTP3 (opt-in, not under WW_KOKKOS_ALL), every sea point of an intra-spectral part in one call at w3wavemd.F90:1864-1865 and 2224-2225 (W3KOKKOS_KTP3 in w3kokkos_prop3md.F90, ww_ktp3_batch in kokkos/src/fortran_iface/prop3_shim.cpp) at fcdfabef; on the resident VA (ww_ktp3_batch_va) at 90631fad -w3dispmd:wavnu1 model/src/w3dispmd.F90:85-202 118 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist kokkos/src/ww_kokkos/prop_wavnu1.hpp no: header-only device function; no shim and no W3WAVE hunk yet +w3dispmd:wavnu1 model/src/w3dispmd.F90:85-202 118 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md); only PTMEAN's call, not W3INIT's or W3ULEV's kokkos/src/ww_kokkos/prop_wavnu1.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it w3uqckmd:w3qck2 model/src/w3uqckmd.F90:517-891 375 ported translated 2026-10-16; static parity by -O3 GIMPLE and assembly comparison, a fresh reviewer (port report); compiled on serial-debug, openmp-release and cuda-release, never run; wired 2026-10-17 (kokkos/src/fortran_iface/PATCH.md section 11) no: never run: W3KTP3 calls it only when FLCK is true, and the full and smoke cases have no current-induced wavenumber shift kokkos/src/ww_kokkos/prop_qck2.hpp yes: inside the Kokkos W3KTP3 (the wavenumber pass), under WW_KOKKOS_KTP3 at w3wavemd.F90:1864-1865 and 2224-2225 at fcdfabef w3wavemd:w3gath model/src/w3wavemd.F90:3024-3329 306 ported translated 2026-10-16; static parity by -O3 GIMPLE and objdump comparison, a fresh reviewer (port report); instantiated by wave_support.cpp in the serial-debug, openmp-release and cuda-release builds (CUDA build only), never run yes: scripts/ww3-gpu.sh --ww3-gpu full (bench_small) --cpu on 2026-10-16 (OpenMP backend, gfortran/g++ 16.2.0), with WW_KOKKOS_ALL=1 and with the W3WAVE support parts alone: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, the binary files byte-identical to --ww3, 13 824 calls (kokkos/src/fortran_iface/PATCH.md, section 8) kokkos/src/ww_kokkos/wave_gath_scat.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_GATH=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp w3wavemd:w3scat model/src/w3wavemd.F90:3345-3644 300 ported translated 2026-10-16; static parity by -O3 GIMPLE and objdump comparison, a fresh reviewer (port report); instantiated by wave_support.cpp in the serial-debug, openmp-release and cuda-release builds (CUDA build only), never run yes: scripts/ww3-gpu.sh --ww3-gpu full (bench_small) --cpu on 2026-10-16 (OpenMP backend, gfortran/g++ 16.2.0), with WW_KOKKOS_ALL=1 and with the W3WAVE support parts alone: out_grd.ww3 and the NetCDF field byte-identical (cmp) to the Fortran of the same executable, the binary files byte-identical to --ww3, 13 824 calls (kokkos/src/fortran_iface/PATCH.md, section 8) kokkos/src/ww_kokkos/wave_gath_scat.hpp,kokkos/src/fortran_iface/wave_support_shim.cpp yes: WW3@357f575c (h0ffmann/WW3 ww3-gpu/kokkos-wave-support) calls it under WW_KOKKOS_SCAT=1 or WW_KOKKOS_ALL=1, through kokkos/src/fortran_iface/wave_support_shim.cpp @@ -75,12 +75,13 @@ pdlib_w3profsmd:pdlib_explicit_block model/src/w3profsmd_pdlib.F90:6408-6685 278 pdlib_w3profsmd:apply_boundary_condition model/src/w3profsmd_pdlib.F90:5150-5336 187 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB; a fresh review found no defect (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim with the MPI hooks kokkos/src/ww_kokkos/pdlib_boundary.hpp no: header-only; no shim yet pdlib_w3profsmd:pdlib_w3xypfsfct2 model/src/w3profsmd_pdlib.F90:1504-1877 374 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison with and without REF1; a fresh review found it matches wherever the Fortran is defined; WW3 runs the WII loop over NE, so the Fortran overflows the heap unless NE == npa, and the C++ bounds it to min(NE,npa), a departure awaiting the owner ⚠ (port report); no regtest reaches it (EXPFSFCT = F everywhere); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); no regtest selects EXPFSFCT kokkos/src/ww_kokkos/pdlib_fsfct2.hpp no: header-only; no shim yet pdlib_w3profsmd:pdlib_w3xypug model/src/w3profsmd_pdlib.F90:626-866 241 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB and REF1; a fresh review found no defect (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3WAVE hunk kokkos/src/ww_kokkos/pdlib_w3xypug.hpp no: header-only; no shim and no W3WAVE hunk yet -w3partmd:ptnghb model/src/w3partmd.F90:630-821 192 ported translated 2026-10-16; static parity by -O3 GIMPLE comparison under switch_NCEP_st4 (integer only, object identical to the CMake build); a fresh review found the translation exact in slot order, values, zero fill and rebuild rule and corrected a comment on the neighbour count, no code defect (port report); host only, no CUDA build needed no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk kokkos/src/ww_kokkos/part_ptnghb.hpp no: header-only; no shim and no W3PART hunk yet -w3partmd:pt_fld model/src/w3partmd.F90:839-1156 318 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4 (MAXVAL seed and maxss order, ABS and compare, FIFO ring of NSPEC kept), no WW3 defect found; a fresh review found no defect (FIFO ring, MAXVAL NaN and empty cases on host and in the CUDA SASS, the five sweeps, the NEIGH layout shared with part_ptnghb.hpp) (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no W3PART hunk yet -w3partmd:ptsort model/src/w3partmd.F90:511-613 103 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4: integer counting sort, the four loops in the Fortran's order and bounds, stable in I; a fresh review found no defect in the translation and fixed two report wording errors (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3PART hunk kokkos/src/ww_kokkos/part_ptsort.hpp no: header-only; no shim and no W3PART hunk yet -w3partmd:ptmean model/src/w3partmd.F90:1175-1538 364 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4, every sum in the Fortran's order, the nine MAX/MIN sites in the object's operand order, SIGN folds, constants and the SIGP**(-5) power tree; a fresh review found no defect and corrected comments; WW3's out-of-bounds read when IFPMAX stays 0 (a NaN spectrum only) returns kPtmeanNoPeak so the shim calls the Fortran there (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs W3PART and a shim kokkos/src/ww_kokkos/part_ptmean.hpp no: header-only; no shim and no W3PART/W3IOSF hunk yet +w3partmd:ptnghb model/src/w3partmd.F90:630-821 192 ported translated 2026-10-16; static parity by -O3 GIMPLE comparison under switch_NCEP_st4 (integer only, object identical to the CMake build); a fresh review found the translation exact in slot order, values, zero fill and rebuild rule and corrected a comment on the neighbour count, no code defect (port report); host only, no CUDA build needed no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_ptnghb.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3partmd:pt_fld model/src/w3partmd.F90:839-1156 318 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4 (MAXVAL seed and maxss order, ABS and compare, FIFO ring of NSPEC kept), no WW3 defect found; a fresh review found no defect (FIFO ring, MAXVAL NaN and empty cases on host and in the CUDA SASS, the five sweeps, the NEIGH layout shared with part_ptnghb.hpp) (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3partmd:ptsort model/src/w3partmd.F90:511-613 103 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4: integer counting sort, the four loops in the Fortran's order and bounds, stable in I; a fresh review found no defect in the translation and fixed two report wording errors (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_ptsort.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3partmd:ptmean model/src/w3partmd.F90:1175-1538 364 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4, every sum in the Fortran's order, the nine MAX/MIN sites in the object's operand order, SIGN folds, constants and the SIGP**(-5) power tree; a fresh review found no defect and corrected comments; WW3's out-of-bounds read when IFPMAX stays 0 (a NaN spectrum only) returns kPtmeanNoPeak so the shim calls the Fortran there (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_ptmean.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it w3snl3md:insnl3 model/src/w3snl3md.F90:788-1649 862 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_glwu (NOGRB) and switch_UoM_nl3; the four LOG(AUXFR) of :1046 and :1273 follow the object's libmvec/scalar split; NKD is 1300 under ST4 by use association and 275 without, kept as a flag; a fresh review found no defect and corrected a call count in the report (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3IOGR hunk kokkos/src/ww_kokkos/snl3_tables.hpp no: header-only; no shim and no W3IOGR hunk yet -w3partmd:w3part model/src/w3partmd.F90:138-497 360 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4 (UPAR maxss order, MINVAL/MAXVAL/MAXLOC NaN seeds, NINT as lroundf with the IHMAX clamp, the wind-sea sort and combination in the Fortran's order); calls ptsort, pt_fld, ptmean and the PtnghbTable, none re-translated; a fresh review found no defect (NINT range-guarded and equal to lroundf for every float, the CUDA FFMA are libdevice and IEEE division only) and fixed the shim contract on kPtmeanNoPeak (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3IOSF hunk kokkos/src/ww_kokkos/part_w3part.hpp no: header-only; no shim and no W3IOSF hunk yet +w3partmd:w3part model/src/w3partmd.F90:138-497 360 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4 (UPAR maxss order, MINVAL/MAXVAL/MAXLOC NaN seeds, NINT as lroundf with the IHMAX clamp, the wind-sea sort and combination in the Fortran's order); calls ptsort, pt_fld, ptmean and the PtnghbTable, none re-translated; a fresh review found no defect (NINT range-guarded and equal to lroundf for every float, the CUDA FFMA are libdevice and IEEE division only) and fixed the shim contract on kPtmeanNoPeak (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_w3part.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3iosfmd:w3cprt model/src/w3iosfmd.F90:109-344 236 ported translated and wired 2026-10-18 on the resident VA (kokkos/src/fortran_iface/PATCH.md section 27); static parity by -O3 GIMPLE of w3iosfmd.F90 and of the shim under switch_lab_shrd: UABS, UDIR with the same folded RADE, FACT = (SIG*TPI)/CG, E2 = VA*FACT, no FMA, W3PART called unchanged, ICPRT and DTPRT integer placement (port report) no: regtest matrix pending; the CPU replay ran it on 2026-10-18: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT on two partition variants of bench_small (switch_lab_shrd, and switch_lab_ompg for the fields variant), WW_KOKKOS_VA_POISON=1: every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 (docs/log/2026-10-18-cprt-resident.md) kokkos/src/ww_kokkos/iosf_w3cprt.hpp, kokkos/src/fortran_iface/cprt_shim.cpp no: wired on the fork branch ww3-gpu/u9-cprt (W3KOKKOS_CPRT under WW_KOKKOS_CPRT=1), not at the WW3/ pin until the integrator merges it w3snl3md:w3snl3 model/src/w3snl3md.F90:181-777 597 ported translated 2026-10-16 (non-PDLIB WAVNU1 variant, with EXPAND and EXPND2; no NL3 switch has PDLIB); static parity by -O3 GIMPLE and object comparison under switch_NCEP_glwu (NCEP2 -> NOGRB, SCRIP/SCRIPNC dropped) and switch_UoM_nl3, including the libmvec powf grouping of the SCALE loop; a fresh review found no defect and corrected two statements in the report (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3SRCE hunk kokkos/src/ww_kokkos/snl3_gmd.hpp no: header-only; no shim and no W3SRCE hunk yet w3src0md:w3spr0 model/src/w3src0md.F90:81-231 151 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd with ST0 FLX2; a fresh review found no arithmetic defect and corrected where the report placed the reversed NaN-payload operands; compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests kokkos/src/ww_kokkos/srce_st0_spr0.hpp yes: WW_KOKKOS_SPR0, w3srcemd.F90:1086, :2201 at 357f575c w3src1md:w3spr1 model/src/w3src1md.F90:87-240 154 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd with ST1 FLX1; a fresh review found no defect; compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests kokkos/src/ww_kokkos/srce_st1_spr1.hpp yes: WW_KOKKOS_SPR1, w3srcemd.F90:1098, :2212 at 357f575c @@ -208,9 +209,9 @@ w3uostmd:uost_sourceterm_compute model/src/w3uostmd.F90:1056-1128 73 ported tran w3uostmd:uost_sourceterm_setgrid model/src/w3uostmd.F90:606-667 62 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_mixa (lab + TIDE UOST FLD2 IG1 IC3 IS2 BT8); a fresh review found the vector path taken at NTH = HUGE(0) where the object stays scalar (and an i+4 overflow), fixed and read again in the object no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/uost_uost_sourceterm_setgrid.hpp no: header-only; no shim and no caller hunk yet w3uostmd:uost_srctrmcompute model/src/w3uostmd.F90:298-343 46 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd+TIDE UOST FLD2 IG1 IC3 IS2 BT8 (mixa); a fresh review found no defect (forwarding call only; object and vtable slot read again) no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/uost_uost_srctrmcompute.hpp no: header-only; no shim and no caller hunk yet w3wavset:fd_wave_setup_apply_fct model/src/w3wavset.F90:1945-2026 82 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB+SETUP; a fresh review found no defect no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/wavset_fd_wave_setup_apply_fct.hpp no: header-only; no shim and no caller hunk yet -w3partmd:fifo_add model/src/w3partmd.F90:1103-1113 11 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd (inlined at all six PT_FLD sites; store, +1, signed > NSPEC wrap to 1 as cmovg in both objects); already the member Fifo::add of the PT_FLD translation, no code added; a fresh review found no defect (report wording on the NSPEC load corrected) no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet -w3partmd:fifo_first model/src/w3partmd.F90:1142-1152 11 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd; a fresh review found no defect no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet -w3partmd:fifo_empty model/src/w3partmd.F90:1122-1133 12 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd (inlined at three sites in PT_FLD as one integer IQ_START/IQ_END compare, same in the C++; already the member Fifo::empty of part_fld.hpp, translated with PT_FLD, not duplicated); a fresh review found no defect no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/part_fld.hpp no: header-only; no shim and no caller hunk yet +w3partmd:fifo_add model/src/w3partmd.F90:1103-1113 11 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd (inlined at all six PT_FLD sites; store, +1, signed > NSPEC wrap to 1 as cmovg in both objects); already the member Fifo::add of the PT_FLD translation, no code added; a fresh review found no defect (report wording on the NSPEC load corrected) no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3partmd:fifo_first model/src/w3partmd.F90:1142-1152 11 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd; a fresh review found no defect no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it +w3partmd:fifo_empty model/src/w3partmd.F90:1122-1133 12 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd (inlined at three sites in PT_FLD as one integer IQ_START/IQ_END compare, same in the C++; already the member Fifo::empty of part_fld.hpp, translated with PT_FLD, not duplicated); a fresh review found no defect no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_fld.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it w3wavset:differentiate_xydir model/src/w3wavset.F90:363-435 73 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB+SETUP (ww3_tp2.6, DIST MPI PDLIB SCOTCH); a fresh review found no arithmetic defect (both objects one tail jmp to the callee); fixed two off-by-one line citations no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/wavset_differentiate_xydir.hpp no: header-only; no shim and no caller hunk yet w3wavset:fd_wave_setup_scalar_prod model/src/w3wavset.F90:2655-2729 75 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB+SETUP; a fresh review found no defect (only a NaN-payload operand order of the multiply, which the Fortran object itself varies by path) no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/wavset_fd_wave_setup_scalar_prod.hpp no: header-only; no shim and no caller hunk yet w3wavset:trig_compute_diff model/src/w3wavset.F90:598-685 88 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_PDLIB (ww3_tp2.6) + SETUP; a fresh review found no defect in the header; the report named the caller at :1712 wrongly, fixed no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and its caller's hunk kokkos/src/ww_kokkos/wavset_trig_compute_diff.hpp no: header-only; no shim and no caller hunk yet diff --git a/kokkos/src/fortran_iface/PATCH.md b/kokkos/src/fortran_iface/PATCH.md index fc79b67e..f139a95d 100644 --- a/kokkos/src/fortran_iface/PATCH.md +++ b/kokkos/src/fortran_iface/PATCH.md @@ -2089,3 +2089,76 @@ are [`ww3gpu_matrix_multi_va_2026-10-18.tsv`](../../../docs/data/ww3gpu_matrix_m after; no difference can be read from these. - `mww3_test_07` (`default`) was not compared: with the lab switch its `ww3_grid` stops in the Fortran run too (`ERROR IN READING SDS4`, the entry's namelist against `ST4`'s). + +## 27. `W3CPRT` on the resident `VA`, and `W3OUTG`'s partition fields without it, `ww3-gpu/u9-cprt` + +One commit on `80644ba9` (section 22), `4f126d85`, pushed to h0ffmann/WW3 on 2026-10-18; not at the +pin until the integrator merges it. ABI `ww_kokkos` 0.10 (three entry points added: `ww_cprt_va`, +`ww_cprt_fetch` of [`cprt_c.hpp`](cprt_c.hpp), `ww_outg_part` of [`outg_c.hpp`](outg_c.hpp)). +Unit U9 of the production plan: at every output time with partitioning on, `W3WAVE` read the whole +`VA` back from the device for `W3CPRT` (`w3wavemd.F90:2688-2693` at `80644ba9`), again before +`W3IOSF` (`:2963-2966`), and `W3KOKKOS_OUTG` fell back to the Fortran `W3OUTG`, with its own host +read, whenever a partition field was asked for (`FLPART`). GFS-Wave writes partitions. + +- **The fork.** A unit file, `model/src/w3kokkos_cprtmd.F90` (`W3KOKKOS_CPRTMD`): the flag + `KOKKOS_CPRT` from `WW_KOKKOS_CPRT` (opt-in, `kOptInOnly` of [`part_flags.cpp`](part_flags.cpp)), + refused outside `SHRD` and under `MPI` or `T`; `W3KOKKOS_CPRT ( IMOD )` packs what `W3CPRT` + reads, calls `ww_cprt_va`, allocates `ICPRT` and `DTPRT` as `W3CPRT` does (`w3iosfmd.F90:221-229`, + `:320-326`), writes `PTMEAN`'s format 1000 the number of times the kernel counted, and calls + `ww_cprt_fetch`. A sea point with a NaN depth, or a spectrum `PTMEAN` cannot take, makes it read + `VA` to the host and call the Fortran `W3CPRT` whole instead (the first stops the model there, + `:270-274`; the second reads out of bounds in the Fortran, + [`PTMEAN_KOKKOS.md`](../../port_report/PTMEAN_KOKKOS.md)). In `w3wavemd.F90`, two hunks: at + `:2688` the host read and `CALL W3CPRT` become `IF ( LOCAL .AND. FLPART .AND. KOKKOS_CPRT ) CALL + W3KOKKOS_CPRT ( IMOD ) ELSE` the old lines; before `W3IOSF` the host read becomes + `IF ( .NOT. KOKKOS_CPRT )`. `W3IOSF` reads `ICPRT`, `DTPRT`, `DW`, `U10`, `U10D`, `CX`, `CY`, + `ASF`, `TIME` and the grid, and no `VA` (its `USE` lists, `w3iosfmd.F90:429-453`, v), so it needs + no host copy once `W3CPRT` is on Kokkos. In `w3kokkos_outgmd.F90`, `FLPART` no longer forces the + Fortran `W3OUTG`: after `ww_outg_va` (run with `FLPART` false, as before), `W3KOKKOS_OUTG` calls + `ww_outg_part` for section 6, which reads the table and no `VA` (`w3iogomd.F90:2242-2340`) and + whose fields no other section reads. That holds whichever of `W3CPRT` or `W3KOKKOS_CPRT` built + the table. `model/src/CMakeLists.txt` asks for `ww_kokkos 0.10`. +- **The kernel** ([`../ww_kokkos/iosf_w3cprt.hpp`](../ww_kokkos/iosf_w3cprt.hpp), the shim + [`cprt_shim.cpp`](cprt_shim.cpp)). The sea points go in chunks (`WW_KOKKOS_CPRT_CHUNK`, by default + four per thread on a host backend, 1024 on a device), one work slot per point. Per chunk: a launch + runs `W3CPRT` sections 2-4 and `W3PART` (unchanged, + [`W3PART_KOKKOS.md`](../../port_report/W3PART_KOKKOS.md)) per point in the Fortran's serial + order; an integer reduction counts format-1000 messages and no-peak points; an integer prefix sum + over `NP+1` gives each point its first `DTPRT` column and `ICPRT(JSEA+1,2)`; a launch copies + `XP(:,0:NP)` there. Integer sums are exact in any order, so the table is in the Fortran's `JSEA` + order by construction. The static parity argument is in + [`W3CPRT_KOKKOS.md`](../../port_report/W3CPRT_KOKKOS.md). +- **Memory contract**: phase 2. `VA` is read in `d_va` (`ww::resident::device_va`). On a device the + inputs (`MAPSF`, `MAPSTA`, `U10`, `U10D`, `ASF`, `DW`, `CG`, `WN`, the frequency and direction + tables, `EWN1`, `ECG1`) are copied in per call, `CG` and `WN` being most of it; `ICPRT` and the + `FINSIZ` used columns of `DTPRT` come back. The work slots, 288 kB per point for `NK = NTH = 24` + and `IHMAX = 50` (`w3cprt_nreal` and `w3cprt_nint`) (`PTMEAN`'s sums are sized for `NSPEC+1` partitions), live for the run. `ww_outg_part` + copies `ICPRT` and `DTPRT` in on a device and only the partition fields `W3IOGO` writes back + (`FLOGRD(4,k)` and `FLOGR2(4,k)`, `w3iogomd.F90:3283-3367`). No Unified Memory. +- **Counters**: `WW_KOKKOS_CPRT_COUNT=1` (calls, sea points, fallbacks, bytes in and back); + `WW_KOKKOS_OUTG_COUNT=1` adds the partition-field calls and bytes. **Timers**: `cprt.call`, + `cprt.h2d`, `cprt.kernel`, `cprt.d2h`, `outg.part`. +- **Not covered.** `W3_MPI` (the `DIST` runs keep the single host read of section 11); the + `WW_KOKKOS_STRICT` count of section 24 (the two fallbacks above are not reported to it: this + branch is on `80644ba9`, without `KOKKOS_FALLBACK`); the `resident` preset of section 25 does not + list `CPRT`, so the runs below name the parts. + +The checkpoint on the CPU, 2026-10-18 (OpenMP backend, Kokkos 5.2.2, gfortran/g++ 16.2.0, i9-14900 +shared with other agents' jobs, `OMP_NUM_THREADS=4`; never `--gpu`); the commands, the two +throwaway `bench_small` variants with partitioning on and the numbers are in +[`docs/log/2026-10-18-cprt-resident.md`](../../../docs/log/2026-10-18-cprt-resident.md): + +- `bash scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV[,CPRT]` with + `WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1`, five runs on `switch_lab_shrd` and + `switch_lab_ompg`: every model file and `partition.ww3` byte-identical to the Fortran of the + same executable and to `--ww3` (v). `VA` device-to-host copies in 12 hours with 13 output times: + 13 before, 1 after (the read at `W3WAVE`'s return, which section 26 removes), with only the + partition file and with all 17 partition fields (v). Per output time a device would move back + 1.14 MB of table and, with every partition field requested, 3.39 MB of fields, against the + 21.21 MB of `VA`, and in 2.21 MB of inputs (projected on the host backend). +- WW3's `ww3_tpt1.1`, the five partitioning methods (`-N -g PTM1` .. `PTM5`), through + `regtests/cmake/run_entry.cmake` with `WW3GPU_KOKKOS=CPRT`: `test.comp` 0 of 22 files different + and `nccmp-tol` PASS on each, 2 Kokkos calls on 18 sea points each (v), with `PR3 UQ` in the + staged switch for its `PR0` ⚠, because the fork's `KOKKOS` build needs `PR3`. No `matrix.base` + entry with partitioning builds with `KOKKOS` (`ww3_tnc1` is `PR0`), so the matrix proper did not + run this part ⚠. diff --git a/kokkos/src/fortran_iface/cprt_c.hpp b/kokkos/src/fortran_iface/cprt_c.hpp new file mode 100644 index 00000000..1cccbd57 --- /dev/null +++ b/kokkos/src/fortran_iface/cprt_c.hpp @@ -0,0 +1,69 @@ +// kokkos/src/fortran_iface/cprt_c.hpp +// The C ABI of the cprt wiring unit: W3CPRT's spectral partitioning on the resident VA (PATCH.md +// section 27). Declarations only; cprt_shim.cpp implements them, and model/src/w3kokkos_cprtmd.F90 +// on the fork (h0ffmann/WW3 ww3-gpu/u9-cprt) mirrors them. The rules of ww_kokkos_c.hpp hold: +// Fortran order, caller-owned arrays, float is WW3's REAL, nothing throws. +// +// SPDX-License-Identifier: MIT +#pragma once + +extern "C" { + +/// The packed integers of ww_cprt_va, in this order (1-based in the Fortran). +enum WwCprtIpar { + WW_CPRT_NK = 0, + WW_CPRT_NTH, + WW_CPRT_NSEAL, + WW_CPRT_NSEA, ///< rows of MAPSF, U10, U10D; columns 0:NSEA of CG, WN and DW + WW_CPRT_NX, ///< MAPSTA(NY,NX) + WW_CPRT_NY, + WW_CPRT_NASF, ///< SIZE(ASF), at least NSEA + WW_CPRT_PTMETH, ///< PTMETH, IHMAX, FLCOMB (1 or 0) of W3ODATMD for the current output grid + WW_CPRT_IHMAX, + WW_CPRT_FLCOMB, + WW_CPRT_DIMP, ///< DIMP of W3ODATMD, 15 + WW_CPRT_NAR1D, ///< NAR1D and N1MAX of W3DISPMD + WW_CPRT_N1MAX, + WW_CPRT_NIPAR +}; + +/// The packed reals of ww_cprt_va, in this order. +enum WwCprtRpar { + WW_CPRT_DTH = 0, ///< DTH, XFR, FACHFE, FTE of W3GDATMD + WW_CPRT_XFR, + WW_CPRT_FACHFE, + WW_CPRT_FTE, + WW_CPRT_HSPMIN, ///< HSPMIN, WSMULT, WSCUT, PTFCUT of W3ODATMD + WW_CPRT_WSMULT, + WW_CPRT_WSCUT, + WW_CPRT_PTFCUT, + WW_CPRT_UNDEF, ///< UNDEF of CONSTANTS (a variable, constants.F90:85) + WW_CPRT_DSIE, ///< DSIE of W3DISPMD + WW_CPRT_NRPAR +}; + +/// Sections 1-5 of W3CPRT (w3iosfmd.F90:109-344) on d_va of grid `igrid` +/// (ww::resident::device_va: VA crosses only when the host holds the newer bits). The partition +/// table is kept in the shim's buffers until ww_cprt_fetch. Out: *finsiz, FINSIZ = ICPRT(NSEAL+1,2)-1 +/// (:312); *nxpfull, the number of PTMEAN calls that met a full XP array, each a format 1000 line +/// the caller writes on NDSE when IAPROC == NAPERR (w3partmd.F90:1416); *fallback, 1 when a point +/// met PTMEAN's IFPMAX = 0 (a NaN spectrum, part_ptmean.hpp), where the Fortran reads out of +/// bounds: the caller then discards this result, reads VA to the host and calls the Fortran +/// W3CPRT. The caller has stopped already on a NaN depth (:270-274). va is VA(1,0), nspec = +/// SIZE(VA,1), nsealm = UBOUND(VA,2); mapsf (NSEA,3), mapsta (NY,NX), u10, u10d (NSEA), asf +/// (NASF), dw (0:NSEA), cg, wn (0:NK+1,0:NSEA), sig, dsip (0:NK+1), dsii (NK), th, ecos, esin (NTH), +/// ewn1, ecg1 (0:NAR1D). Returns a WwKokkosStatus. +int ww_cprt_va(int igrid, const int* ipar, int nipar, const float* rpar, int nrpar, float* va, + int nspec, int nsealm, const int* mapsf, const int* mapsta, const float* u10, + const float* u10d, const float* asf, const float* dw, const float* cg, + const float* wn, const float* sig, const float* dsip, const float* dsii, + const float* th, const float* ecos, const float* esin, const float* ewn1, + const float* ecg1, int* finsiz, int* nxpfull, int* fallback); + +/// The table of the last ww_cprt_va of grid `igrid` into WW3's arrays: icprt is +/// ICPRT(NSEALM+1,2), every entry written; dtprt DTPRT(DIMP,MAX(1,FINSIZ)), columns 1..finsiz +/// written (the caller zeroes DTPRT when finsiz is 0, :325). Returns a WwKokkosStatus, +/// WW_KOKKOS_ERR_BAD_SHAPE when igrid, nsealm or finsiz differ from that call's. +int ww_cprt_fetch(int igrid, int nsealm, int* icprt, int finsiz, float* dtprt); + +} // extern "C" diff --git a/kokkos/src/fortran_iface/cprt_shim.cpp b/kokkos/src/fortran_iface/cprt_shim.cpp new file mode 100644 index 00000000..1d6ea05d --- /dev/null +++ b/kokkos/src/fortran_iface/cprt_shim.cpp @@ -0,0 +1,424 @@ +// kokkos/src/fortran_iface/cprt_shim.cpp +// The bind(C) shim of the cprt wiring unit (PATCH.md section 27): W3CPRT's spectral partitioning +// on the resident VA. Marshalling and the order of the launches only; the arithmetic is the +// translated headers iosf_w3cprt.hpp and part_w3part.hpp, called unchanged. +// +// The caller is W3WAVE (w3wavemd.F90) through W3KOKKOS_CPRT of w3kokkos_cprtmd.F90 on the fork, +// at an output time with FLPART, outside any OpenMP region, in place of CALL W3CPRT ( IMOD ) and +// of the host read of VA before it. +// +// Order. The local sea points go in chunks of `chunk` points (WW_KOKKOS_CPRT_CHUNK, read once; +// by default four per thread of the execution space on a host backend, 1024 on a device), each +// point with its own work slot. Per chunk: one launch runs sections 2-4 (W3PART) for every point; +// a reduction counts XP-full messages and the points PTMEAN cannot take; an integer prefix sum +// gives each point its first DTPRT column and ICPRT(JSEA+1,2); a last launch copies XP(:,0:NP) to +// those columns. Integer sums are exact in any order, so DTPRT and ICPRT are the Fortran's JSEA +// order by construction, and each point's floating-point arithmetic is W3PART's serial code. +// +// Memory contract (phase 2). VA is read in d_va (ww::resident::device_va), so it crosses only +// when the host holds the newer bits, and W3WAVE makes no host read of VA for this call. On a +// backend whose execution space reads host memory (Serial, OpenMP) the inputs are WW3's own +// arrays, read in place. On a device the inputs (MAPSF, MAPSTA, U10, U10D, ASF, DW, CG, WN, the +// frequency and direction tables, DISTAB's EWN1 and ECG1) are copied in per call, which is per +// output time. ICPRT (NSEALM+1 by 2 ints) and the FINSIZ columns of DTPRT (15 Reals each) are +// built in the shim's buffers and copied back by ww_cprt_fetch; nothing else comes back. The work +// slots are device buffers of `chunk` slots that live for the run. No Unified Memory. +// +// WW_KOKKOS_CPRT_COUNT=1 prints, at exit, the calls, the sea points, the fallbacks to the Fortran +// and the bytes copied in and back (on a host backend the copies a device would make, labelled as +// such: none is made). WW_KOKKOS_TIMERS=1 times cprt.call, cprt.h2d, cprt.kernel and cprt.d2h. +// +// SPDX-License-Identifier: MIT + +#include "fortran_iface/cprt_c.hpp" +#include "fortran_iface/timers.hpp" +#include "fortran_iface/ww_kokkos_c.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ww_kokkos/iosf_w3cprt.hpp" +#include "ww_kokkos/part_ptnghb.hpp" +#include "ww_kokkos/part_w3part.hpp" +#include "ww_kokkos/real.hpp" +#include "ww_kokkos/resident_state.hpp" + +#if defined(_OPENMP) +#include +#endif + +namespace ww::cprtshim { +namespace { + +using ww::Real; +using Exec = Kokkos::DefaultExecutionSpace; +using Mem = Exec::memory_space; +static_assert(std::is_same_v, "the C ABI passes REAL as float"); +constexpr bool kHostReadable = Kokkos::SpaceAccessibility::accessible; + +template +using H1 = Kokkos::View>; +template +using H2 = Kokkos::View>; +template +using D1 = Kokkos::View; +template +using D2 = Kokkos::View; +template +using U1 = Kokkos::View>; +template +using U2 = Kokkos::View>; + +std::atomic g_calls{0}, g_points{0}, g_fallbacks{0}, g_h2d_bytes{0}, g_d2h_bytes{0}; + +void print_counts() { + const char* proj = kHostReadable ? " (projected for a device, none made)" : ""; + std::fprintf(stderr, "ww_kokkos: W3CPRT Kokkos calls: %lld\n", g_calls.load()); + std::fprintf(stderr, "ww_kokkos: W3CPRT sea points: %lld\n", g_points.load()); + std::fprintf(stderr, "ww_kokkos: W3CPRT fallbacks to the Fortran (no peak): %lld\n", g_fallbacks.load()); + std::fprintf(stderr, "ww_kokkos: W3CPRT input bytes copied H2D%s: %lld\n", proj, g_h2d_bytes.load()); + std::fprintf(stderr, "ww_kokkos: W3CPRT table bytes copied back D2H%s: %lld\n", proj, g_d2h_bytes.load()); +} + +/// WW_KOKKOS_CPRT_COUNT=1 registers the exit print when the library loads, so a run in which +/// the entry point was never called still says 0. +[[maybe_unused]] const bool g_count_on = [] { + const char* v = std::getenv("WW_KOKKOS_CPRT_COUNT"); + const bool c = v != nullptr && std::strcmp(v, "1") == 0; + if (c) std::atexit(print_counts); + return c; +}(); + +int fail(int code, const char* what) { + std::fprintf(stderr, "ww_kokkos: cprt: %s (code %d)\n", what, code); + return code; +} + +/// Points per chunk: WW_KOKKOS_CPRT_CHUNK when a positive integer, else the default above. +int chunk_size() { + static const int n = [] { + if (const char* v = std::getenv("WW_KOKKOS_CPRT_CHUNK")) { + char* end = nullptr; + const long c = std::strtol(v, &end, 10); + if (end != v && *end == '\0' && c > 0 && c <= (1L << 20)) return static_cast(c); + std::fprintf(stderr, "ww_kokkos: cprt: WW_KOKKOS_CPRT_CHUNK='%s' ignored\n", v); + } + return kHostReadable ? 4 * Exec().concurrency() : 1024; + }(); + return n; +} + +/// The shim's buffers, kept between calls. +struct Bufs { + // Inputs on a device (unused on a host backend). + D2 mapsf, mapsta; + D1 u10, u10d, asf, dw, sig, dsip, dsii, th, ecos, esin, ewn1, ecg1; + D2 cg, wn; + D1 neigh; + ww::part::PtnghbTable table; + // Work slots and per-point results of a chunk. + D1 wr; + D1 wi, np, nxp, rc, first; + // The table, until ww_cprt_fetch. + D2 icprt; + D2 dtprt; + int igrid = -1, nsealm = -1, finsiz = -1; +}; +Bufs& bufs() { + static auto* b = new Bufs(); // never destroyed, as every shim's state + static bool hooked = false; + if (!hooked && Kokkos::is_initialized()) { + hooked = true; + Kokkos::push_finalize_hook([] { *b = Bufs(); }); + } + return *b; +} + +template +void ensure(V& v, const char* name, size_t n0, size_t n1 = 0) { + if constexpr (V::rank == 1) { + if (v.extent(0) != n0) v = V(Kokkos::view_alloc(Kokkos::WithoutInitializing, std::string(name)), n0); + } else { + if (v.extent(0) != n0 || v.extent(1) != n1) + v = V(Kokkos::view_alloc(Kokkos::WithoutInitializing, std::string(name)), n0, n1); + } +} + +/// A read-only input: WW3's array itself on a host backend; else its device buffer, refreshed. +template +U1 in1(D1& buf, const char* name, const T* p, size_t n) { + if (kHostReadable) return U1(p, n); + ensure(buf, name, n); + Kokkos::deep_copy(buf, H1(p, n)); + g_h2d_bytes.fetch_add(static_cast(n * sizeof(T)), std::memory_order_relaxed); + return U1(buf.data(), n); +} +template +U2 in2(D2& buf, const char* name, const T* p, size_t n0, size_t n1) { + if (kHostReadable) return U2(p, n0, n1); + ensure(buf, name, n0, n1); + Kokkos::deep_copy(buf, H2(p, n0, n1)); + g_h2d_bytes.fetch_add(static_cast(n0 * n1 * sizeof(T)), std::memory_order_relaxed); + return U2(buf.data(), n0, n1); +} +/// What a device would copy in, counted on a host backend too. +void count_in(size_t bytes) { + if (kHostReadable) g_h2d_bytes.fetch_add(static_cast(bytes), std::memory_order_relaxed); +} + +bool nested_in_openmp() { +#if defined(_OPENMP) + return omp_get_level() > 0; +#else + return false; +#endif +} + +int run(int igrid, const int* ipar, const float* rpar, float* va, int nspec, int nsealm, const int* mapsf, + const int* mapsta, const float* u10, const float* u10d, const float* asf, const float* dw, + const float* cg, const float* wn, const float* sig, const float* dsip, const float* dsii, + const float* th, const float* ecos, const float* esin, const float* ewn1, const float* ecg1, + int* finsiz_out, int* nxpfull_out, int* fallback_out) { + const int nk = ipar[WW_CPRT_NK], nth = ipar[WW_CPRT_NTH], nseal = ipar[WW_CPRT_NSEAL]; + const int nsea = ipar[WW_CPRT_NSEA], nx = ipar[WW_CPRT_NX], ny = ipar[WW_CPRT_NY]; + const int nasf = ipar[WW_CPRT_NASF], ptmeth = ipar[WW_CPRT_PTMETH], ihmax = ipar[WW_CPRT_IHMAX]; + const int nar1d = ipar[WW_CPRT_NAR1D], n1max = ipar[WW_CPRT_N1MAX]; + // Under W3_SHRD NSEAL = NSEA = NSEALM; JSEA = ISEA is what the kernel assumes. + if (nk < 1 || nth < 1 || nspec != nk * nth || nseal < 0 || nseal != nsea || nsealm < nseal || nx < 1 || + ny < 1 || nasf < nsea || ihmax < 1 || ipar[WW_CPRT_DIMP] != ww::part::kDimp || nar1d < 2 || n1max < 0 || + n1max > nar1d - 1) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_va: bad sizes"); + const int dimxp = ww::iosf::w3cprt_dimxp(nk, nth, ptmeth); + if (dimxp < 1) return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_va: DIMXP < 1"); + + ww::timers::Region tcall("cprt.call"); + Bufs& b = bufs(); + b.finsiz = -1; + ww::resident::DevVa dva; + const int st = ww::resident::device_va(igrid, va, nspec, nsealm, dva); + if (st != WW_KOKKOS_OK) return fail(st, "the resident VA refused this call"); + if (dva.stride(1) != static_cast(nspec)) return fail(WW_KOKKOS_ERR_BAD_SHAPE, "the resident VA is padded"); + + ww::iosf::W3cprtArgs a; + ww::part::PtmeanGrid g{}; + { + ww::timers::Region t("cprt.h2d"); + const size_t nk2 = static_cast(nk) + 2, ns1 = static_cast(nsea) + 1; + const size_t ns = static_cast(nsea), nt = static_cast(nth); + a.va = U2(dva.data(), static_cast(nspec), static_cast(nsealm) + 1); + a.mapsf = in2(b.mapsf, "ww_cprt.mapsf", mapsf, ns, 3); + a.mapsta = in2(b.mapsta, "ww_cprt.mapsta", mapsta, static_cast(ny), static_cast(nx)); + a.u10 = in1(b.u10, "ww_cprt.u10", u10, ns); + a.u10d = in1(b.u10d, "ww_cprt.u10d", u10d, ns); + a.asf = in1(b.asf, "ww_cprt.asf", asf, ns); + a.dw = in1(b.dw, "ww_cprt.dw", dw, ns1); + a.cg = in2(b.cg, "ww_cprt.cg", cg, nk2, ns1); + a.wn = in2(b.wn, "ww_cprt.wn", wn, nk2, ns1); + count_in((ns * 3 + static_cast(nx) * static_cast(ny)) * sizeof(int) + + (3 * ns + ns1 + 2 * nk2 * ns1) * sizeof(Real)); + const auto s_sig = in1(b.sig, "ww_cprt.sig", sig, nk2); + const auto s_dsip = in1(b.dsip, "ww_cprt.dsip", dsip, nk2); + const auto s_dsii = in1(b.dsii, "ww_cprt.dsii", dsii, static_cast(nk)); + const auto s_th = in1(b.th, "ww_cprt.th", th, nt); + const auto s_ecos = in1(b.ecos, "ww_cprt.ecos", ecos, nt); + const auto s_esin = in1(b.esin, "ww_cprt.esin", esin, nt); + const size_t na = static_cast(nar1d) + 1; + const auto s_ewn1 = in1(b.ewn1, "ww_cprt.ewn1", ewn1, na); + const auto s_ecg1 = in1(b.ecg1, "ww_cprt.ecg1", ecg1, na); + count_in((2 * nk2 + static_cast(nk) + 3 * nt + 2 * na) * sizeof(Real)); + // PTNGHB's table (w3partmd.F90:630-821): a function of NK and NTH, rebuilt when they change. + if (b.table.update(nk, nth) || b.neigh.extent(0) != b.table.neigh.size()) { + ensure(b.neigh, "ww_cprt.neigh", b.table.neigh.size()); + Kokkos::deep_copy(b.neigh, H1(b.table.neigh.data(), b.table.neigh.size())); + } + g.nk = nk; + g.nth = nth; + g.dth = rpar[WW_CPRT_DTH]; + g.xfr = rpar[WW_CPRT_XFR]; + g.fachfe = rpar[WW_CPRT_FACHFE]; + g.fte = rpar[WW_CPRT_FTE]; + g.sig = s_sig.data(); + g.dsip = s_dsip.data(); + g.dsii = s_dsii.data(); + g.th = s_th.data(); + g.ecos = s_ecos.data(); + g.esin = s_esin.data(); + g.disp.ewn1 = s_ewn1.data(); + g.disp.ecg1 = s_ecg1.data(); + g.disp.dsie = rpar[WW_CPRT_DSIE]; + g.disp.n1max = n1max; + } + const ww::part::PtmeanOpts o{rpar[WW_CPRT_HSPMIN], rpar[WW_CPRT_WSMULT], ptmeth, rpar[WW_CPRT_UNDEF]}; + const ww::part::W3partOpts po{ihmax, rpar[WW_CPRT_WSCUT], rpar[WW_CPRT_PTFCUT], ipar[WW_CPRT_FLCOMB] != 0}; + + const int chunk = nseal < chunk_size() ? (nseal > 0 ? nseal : 1) : chunk_size(); + const size_t nreal = static_cast(ww::iosf::w3cprt_nreal(nk, nth, dimxp)); + const size_t nint = static_cast(ww::iosf::w3cprt_nint(nk, nth, dimxp, ihmax)); + const size_t nxp = static_cast(ww::part::kDimp) * (static_cast(dimxp) + 1); + const size_t nspec_s = static_cast(nspec); + if (b.wr.extent(0) < nreal * chunk) b.wr = D1(Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_cprt.wr"), nreal * chunk); + if (b.wi.extent(0) < nint * chunk) b.wi = D1(Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_cprt.wi"), nint * chunk); + for (D1* v : {&b.np, &b.nxp, &b.rc, &b.first}) + if (v->extent(0) < static_cast(chunk)) *v = D1(Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_cprt.pt"), chunk); + // ICPRT = 0, ICPRT(1,2) = 1 (:230-231). + b.icprt = D2("ww_cprt.icprt", static_cast(nsealm) + 1, 2); + Kokkos::deep_copy(Kokkos::subview(b.icprt, 0, 1), 1); + if (b.dtprt.extent(0) != static_cast(ww::part::kDimp) || b.dtprt.extent(1) == 0) + b.dtprt = D2(Kokkos::view_alloc(Kokkos::WithoutInitializing, "ww_cprt.dtprt"), ww::part::kDimp, + static_cast(nseal > 0 ? nseal : 1)); + + const auto wr = b.wr; + const auto wi = b.wi; + const auto vnp = b.np; + const auto vnxp = b.nxp; + const auto vrc = b.rc; + const auto vfirst = b.first; + const auto icprt = b.icprt; + const int* neigh = b.neigh.data(); + int base = 0, nxpfull = 0; + { + ww::timers::Region t("cprt.kernel"); + for (int c0 = 0; c0 < nseal; c0 += chunk) { + const int n = (nseal - c0 < chunk) ? nseal - c0 : chunk; + // Sections 2-4, one point per index. + Kokkos::parallel_for( + "ww::iosf::w3cprt_point", Kokkos::RangePolicy(0, n), KOKKOS_LAMBDA(const int i) { + Real* r = wr.data() + nreal * static_cast(i); + int* ib = wi.data() + nint * static_cast(i); + Real* e2 = r; + Real* xp = e2 + nspec_s; + const ww::part::W3partScratch w = ww::part::w3part_carve(xp + nxp, ib, nk, nth, dimxp, ihmax); + int np = 0, nfull = 0; + vrc(i) = ww::iosf::w3cprt_point(a, g, o, po, neigh, dimxp, c0 + i + 1, e2, xp, w, np, nfull); + vnp(i) = np; + vnxp(i) = nfull; + }); + int nopeak = 0, nfull = 0; + Kokkos::parallel_reduce( + "ww::iosf::w3cprt_status", Kokkos::RangePolicy(0, n), + KOKKOS_LAMBDA(const int i, int& sx, int& sp) { + sx += vnxp(i); + if (vrc(i) == ww::part::kPtmeanNoPeak) sp += 1; + }, + nfull, nopeak); + if (nopeak > 0) { + *finsiz_out = 0; + *nxpfull_out = 0; + *fallback_out = 1; + g_fallbacks.fetch_add(1, std::memory_order_relaxed); + return WW_KOKKOS_OK; + } + nxpfull += nfull; + // ICPRT(JSEA,2) = 1 + the columns of the points before JSEA (:251, :286). + int total = 0; + const int b0 = base; + Kokkos::parallel_scan( + "ww::iosf::w3cprt_scan", Kokkos::RangePolicy(0, n), + KOKKOS_LAMBDA(const int i, int& acc, const bool final) { + const int np = vnp(i); + const int cols = np >= 0 ? np + 1 : 0; + if (final) { + vfirst(i) = b0 + 1 + acc; + icprt(c0 + i + 1, 1) = b0 + 1 + acc + cols; // ICPRT(JSEA+1,2) + } + acc += cols; + }, + total); + if (static_cast(base) + static_cast(total) > b.dtprt.extent(1)) { + size_t cap = 2 * b.dtprt.extent(1); + if (cap < static_cast(base) + static_cast(total)) + cap = static_cast(base) + static_cast(total); + Kokkos::resize(b.dtprt, static_cast(ww::part::kDimp), cap); + } + const auto dtprt = b.dtprt; + Kokkos::parallel_for( + "ww::iosf::w3cprt_store", Kokkos::RangePolicy(0, n), KOKKOS_LAMBDA(const int i) { + const Real* xp = wr.data() + nreal * static_cast(i) + nspec_s; + ww::iosf::w3cprt_store(c0 + i + 1, vnp(i), vfirst(i), xp, icprt, dtprt); + }); + base += total; + } + Exec().fence("ww_cprt_va"); + } + b.igrid = igrid; + b.nsealm = nsealm; + b.finsiz = base; // FINSIZ = ICPRT(NSEAL+1,2) - 1 (:312) + *finsiz_out = base; + *nxpfull_out = nxpfull; + *fallback_out = 0; + g_calls.fetch_add(1, std::memory_order_relaxed); + g_points.fetch_add(nseal, std::memory_order_relaxed); + return WW_KOKKOS_OK; +} + +} // namespace +} // namespace ww::cprtshim + +extern "C" int ww_cprt_va(int igrid, const int* ipar, int nipar, const float* rpar, int nrpar, float* va, + int nspec, int nsealm, const int* mapsf, const int* mapsta, const float* u10, + const float* u10d, const float* asf, const float* dw, const float* cg, + const float* wn, const float* sig, const float* dsip, const float* dsii, + const float* th, const float* ecos, const float* esin, const float* ewn1, + const float* ecg1, int* finsiz, int* nxpfull, int* fallback) { + using namespace ww::cprtshim; + if (ipar == nullptr || rpar == nullptr || nipar != WW_CPRT_NIPAR || nrpar != WW_CPRT_NRPAR) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_va: bad packet"); + if (va == nullptr || mapsf == nullptr || mapsta == nullptr || u10 == nullptr || u10d == nullptr || + asf == nullptr || dw == nullptr || cg == nullptr || wn == nullptr || sig == nullptr || dsip == nullptr || + dsii == nullptr || th == nullptr || ecos == nullptr || esin == nullptr || ewn1 == nullptr || + ecg1 == nullptr || finsiz == nullptr || nxpfull == nullptr || fallback == nullptr) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_va: a null pointer"); + try { + if (!Kokkos::is_initialized()) return fail(WW_KOKKOS_ERR_NOT_INITIALISED, "before ww_kokkos_init"); + if (nested_in_openmp()) return fail(WW_KOKKOS_ERR_UNSUPPORTED, "inside an OpenMP region"); + // W3CPRT reads VA and never writes it: d_va keeps the state device_va left (BOTH or DEVICE). + return run(igrid, ipar, rpar, va, nspec, nsealm, mapsf, mapsta, u10, u10d, asf, dw, cg, wn, sig, dsip, + dsii, th, ecos, esin, ewn1, ecg1, finsiz, nxpfull, fallback); + } catch (const std::bad_alloc& e) { + return fail(WW_KOKKOS_ERR_RUNTIME, e.what()); + } catch (const std::exception& e) { + return fail(WW_KOKKOS_ERR_KERNEL, e.what()); + } catch (...) { + return fail(WW_KOKKOS_ERR_KERNEL, "unknown exception"); + } +} + +extern "C" int ww_cprt_fetch(int igrid, int nsealm, int* icprt, int finsiz, float* dtprt) { + using namespace ww::cprtshim; + if (icprt == nullptr || dtprt == nullptr) return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_fetch: a null pointer"); + try { + if (!Kokkos::is_initialized()) return fail(WW_KOKKOS_ERR_NOT_INITIALISED, "before ww_kokkos_init"); + Bufs& b = bufs(); + if (b.finsiz < 0 || igrid != b.igrid || nsealm != b.nsealm || finsiz != b.finsiz) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_cprt_fetch: no table of this grid and size"); + ww::timers::Region t("cprt.d2h"); + const size_t nrow = static_cast(nsealm) + 1; + Kokkos::deep_copy(H2(icprt, nrow, 2), b.icprt); + long long bytes = static_cast(nrow * 2 * sizeof(int)); + if (finsiz > 0) { + const size_t nc = static_cast(finsiz); + Kokkos::deep_copy(H2(dtprt, static_cast(ww::part::kDimp), nc), + Kokkos::subview(b.dtprt, Kokkos::ALL, std::make_pair(size_t{0}, nc))); + bytes += static_cast(nc * ww::part::kDimp * sizeof(Real)); + } + Exec().fence("ww_cprt_fetch"); + g_d2h_bytes.fetch_add(bytes, std::memory_order_relaxed); + b.finsiz = -1; + return WW_KOKKOS_OK; + } catch (const std::bad_alloc& e) { + return fail(WW_KOKKOS_ERR_RUNTIME, e.what()); + } catch (const std::exception& e) { + return fail(WW_KOKKOS_ERR_KERNEL, e.what()); + } catch (...) { + return fail(WW_KOKKOS_ERR_KERNEL, "unknown exception"); + } +} diff --git a/kokkos/src/fortran_iface/outg_c.hpp b/kokkos/src/fortran_iface/outg_c.hpp index 298f83d6..a4975dc9 100644 --- a/kokkos/src/fortran_iface/outg_c.hpp +++ b/kokkos/src/fortran_iface/outg_c.hpp @@ -81,4 +81,47 @@ int ww_outg_va(int igrid, const int* ipar, int nipar, const float* rpar, int nrp const float* ec2, const float* es2, const float* esc, const int* mapsf, const int* mapsta, float* const* fields, int nfields); +/// The packed integers of ww_outg_part, in this order. +enum WwOutgPartIpar { + WW_OUTGP_NSEAL = 0, + WW_OUTGP_NSEA, ///< rows of MAPSF + WW_OUTGP_NX, ///< MAPSTA(NY,NX) + WW_OUTGP_NY, + WW_OUTGP_NROW, ///< SIZE(PWST) = NSEALM: the rows of every partition field + WW_OUTGP_NOSWLL, ///< NOSWLL: the partition fields are (NROW, 0:NOSWLL) + WW_OUTGP_NICP, ///< SIZE(ICPRT,1) = NSEALM+1 + WW_OUTGP_NDTP, ///< SIZE(DTPRT,2), the columns of DTPRT(DIMP,:) + WW_OUTGP_DIMP, ///< DIMP, 15 + WW_OUTGP_NOGRP, ///< FLOGRD(NOGRP,NGRPP) + WW_OUTGP_NGRPP, + WW_OUTGP_FLOUTG, ///< 1 or 0 + WW_OUTGP_FLOUTG2, ///< 1 or 0 + WW_OUTGP_NIPAR +}; + +/// The packed reals of ww_outg_part. +enum WwOutgPartRpar { WW_OUTGP_UNDEF = 0, WW_OUTGP_WSCUT, WW_OUTGP_NRPAR }; + +/// The partition fields of ww_outg_part, the order of `fields` (W3ADATMD arrays): the first +/// fifteen are (NROW, 0:NOSWLL), PWST and PNR (NROW). FLOGRD(4,k) names field k+1. +enum WwOutgPartField { + WW_OUTGP_PHS = 0, WW_OUTGP_PTP, WW_OUTGP_PLP, WW_OUTGP_PDIR, WW_OUTGP_PSI, WW_OUTGP_PWS, + WW_OUTGP_PTHP0, WW_OUTGP_PQP, WW_OUTGP_PPE, WW_OUTGP_PGW, WW_OUTGP_PSW, WW_OUTGP_PTM1, + WW_OUTGP_PT1, WW_OUTGP_PT2, WW_OUTGP_PEP, + WW_OUTGP_N2D, + WW_OUTGP_PWST = WW_OUTGP_N2D, WW_OUTGP_PNR, + WW_OUTGP_NFIELD +}; + +/// Section 6 of W3OUTG (w3iogomd.F90:2242-2340), the partition fields from W3CPRT's table, alone +/// (ww::iogo::w3outg_part). It reads no VA. W3KOKKOS_OUTG calls it after ww_outg_va when FLPART +/// is the only reason the Fortran W3OUTG would run (PATCH.md section 27). On a backend that reads +/// host memory every field is written in place; on a device ICPRT and DTPRT are copied in and only +/// the fields W3IOGO ('WRITE') writes (FLOGRD(4,k) under FLOUTG, FLOGR2(4,k) under FLOUTG2, +/// w3iogomd.F90:3283-3367) come back. mapsf (NSEA,3), mapsta (NY,NX), icprt (NICP,2), dtprt +/// (DIMP,NDTP), flogrd, flogr2 LOGICAL(4) (NOGRP,NGRPP). Returns a WwKokkosStatus. +int ww_outg_part(const int* ipar, int nipar, const float* rpar, int nrpar, const int* flogrd, + const int* flogr2, const int* mapsf, const int* mapsta, const int* icprt, + const float* dtprt, float* const* fields, int nfields); + } // extern "C" diff --git a/kokkos/src/fortran_iface/outg_shim.cpp b/kokkos/src/fortran_iface/outg_shim.cpp index 524c8d7f..f4d94b8a 100644 --- a/kokkos/src/fortran_iface/outg_shim.cpp +++ b/kokkos/src/fortran_iface/outg_shim.cpp @@ -5,9 +5,10 @@ // The caller is W3WAVE (w3wavemd.F90) through W3KOKKOS_OUTG of w3kokkos_outgmd.F90 on the fork, // at a field-output time, outside any OpenMP region, in place of CALL W3OUTG (VA, FLPFLD, FLOUTG, // FLOUTG2). W3KOKKOS_OUTG calls the Fortran W3OUTG instead when a field this unit does not compute -// is requested: the partitions of section 6 (FLPFLD; they read W3CPRT's table, built on the host -// from VA), CALC_U3STOKES (FLOLOC(6,8), (6,12)), SKEWNESS (FLOLOC(8,7..9)) and CALC_WBT -// (FLOLOC(2,17)). So ww::iogo::w3outg runs here with FLPART false and no hook requested. +// is requested: CALC_U3STOKES (FLOLOC(6,8), (6,12)), SKEWNESS (FLOLOC(8,7..9)) and CALC_WBT +// (FLOLOC(2,17)). So ww::iogo::w3outg runs here with FLPART false and no hook requested; the +// partition fields of section 6 (FLPFLD), which read W3CPRT's table and no VA, are ww_outg_part's, +// called after it (PATCH.md section 27). // // Memory contract (phase 2). A reads d_va of the grid (ww::resident::device_va), so VA crosses // only when the host holds the newer bits, and W3WAVE makes no host read of VA for this call. On a @@ -376,9 +377,143 @@ int run(int igrid, const int* ipar, const float* rpar, const int* flogrd, const return WW_KOKKOS_OK; } +std::atomic g_part_calls{0}, g_part_back{0}, g_part_back_bytes{0}; + +void print_part_counts() { + const char* proj = kHostReadable ? " (projected for a device, none made)" : ""; + std::fprintf(stderr, "ww_kokkos: W3OUTG partition-field Kokkos calls: %lld\n", g_part_calls.load()); + std::fprintf(stderr, "ww_kokkos: W3OUTG partition fields copied back D2H%s: %lld\n", proj, g_part_back.load()); + std::fprintf(stderr, "ww_kokkos: W3OUTG partition bytes copied back D2H%s: %lld\n", proj, + g_part_back_bytes.load()); +} +[[maybe_unused]] const bool g_part_count_on = [] { + const char* v = std::getenv("WW_KOKKOS_OUTG_COUNT"); + const bool c = v != nullptr && std::strcmp(v, "1") == 0; + if (c) std::atexit(print_part_counts); + return c; +}(); + +/// The device buffers of section 6, kept between calls (unused on a host backend). +struct PartBufs { + D2 mapsf, mapsta, icprt; + D2 dtprt; + D2 f2[WW_OUTGP_N2D]; + D1 f1[WW_OUTGP_NFIELD - WW_OUTGP_N2D]; +}; +PartBufs& part_bufs() { + static auto* b = new PartBufs(); // never destroyed, as every shim's state + static bool hooked = false; + if (!hooked && Kokkos::is_initialized()) { + hooked = true; + Kokkos::push_finalize_hook([] { *b = PartBufs(); }); + } + return *b; +} + +int run_part(const int* ipar, const float* rpar, const int* flogrd, const int* flogr2, const int* mapsf, + const int* mapsta, const int* icprt, const float* dtprt, float* const* fields) { + const int nseal = ipar[WW_OUTGP_NSEAL], nsea = ipar[WW_OUTGP_NSEA], nx = ipar[WW_OUTGP_NX]; + const int ny = ipar[WW_OUTGP_NY], nrow = ipar[WW_OUTGP_NROW], noswll = ipar[WW_OUTGP_NOSWLL]; + const int nicp = ipar[WW_OUTGP_NICP], ndtp = ipar[WW_OUTGP_NDTP], nogrp = ipar[WW_OUTGP_NOGRP]; + const int ngrpp = ipar[WW_OUTGP_NGRPP]; + if (nseal < 0 || nsea < nseal || nx < 1 || ny < 1 || nrow < nseal || noswll < 0 || nicp < nseal + 1 || + ndtp < 1 || ipar[WW_OUTGP_DIMP] != 15 || nogrp < 4 || ngrpp < 17) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_outg_part: bad sizes"); + for (int k = 0; k < WW_OUTGP_NFIELD; ++k) + if (fields[k] == nullptr) return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_outg_part: a null field"); + const bool floutg = ipar[WW_OUTGP_FLOUTG] != 0, floutg2 = ipar[WW_OUTGP_FLOUTG2] != 0; + const auto want = [&](int j) { + const int k = 3 + (j - 1) * nogrp; // FLOGRD(4,j) + return (floutg && flogrd[k] != 0) || (floutg2 && flogr2[k] != 0); + }; + + ww::timers::Region tcall("outg.part"); + PartBufs& b = part_bufs(); + ww::iogo::W3OutgScalars s{}; + s.nseal = nseal; + s.noswll = noswll; + s.undef = rpar[WW_OUTGP_UNDEF]; + s.wscut = rpar[WW_OUTGP_WSCUT]; + Args g; + const size_t nr = static_cast(nrow), nc = static_cast(noswll) + 1; + const auto pin2 = [&](D2& buf, const char* name, const int* p, size_t n0, size_t n1) { + if (in_place()) return U2(p, n0, n1); + ensure(buf, name, n0, n1); + Kokkos::deep_copy(buf, H2(p, n0, n1)); + return U2(buf.data(), n0, n1); + }; + g.mapsf = pin2(b.mapsf, "ww_outg.part.mapsf", mapsf, static_cast(nsea), 3); + g.mapsta = pin2(b.mapsta, "ww_outg.part.mapsta", mapsta, static_cast(ny), static_cast(nx)); + g.icprt = pin2(b.icprt, "ww_outg.part.icprt", icprt, static_cast(nicp), 2); + if (in_place()) { + g.dtprt = U2(dtprt, 15, static_cast(ndtp)); + } else { + ensure(b.dtprt, "ww_outg.part.dtprt", 15, static_cast(ndtp)); + Kokkos::deep_copy(b.dtprt, H2(dtprt, 15, static_cast(ndtp))); + g.dtprt = U2(b.dtprt.data(), 15, static_cast(ndtp)); + } + Args::V2* o2[WW_OUTGP_N2D] = {&g.phs, &g.ptp, &g.plp, &g.pdir, &g.psi, &g.pws, &g.pthp0, &g.pqp, + &g.ppe, &g.pgw, &g.psw, &g.ptm1, &g.pt1, &g.pt2, &g.pep}; + for (int k = 0; k < WW_OUTGP_N2D; ++k) { + if (in_place()) { + *o2[k] = U2(fields[k], nr, nc); + } else { + ensure(b.f2[k], "ww_outg.part.field2", nr, nc); + *o2[k] = U2(b.f2[k].data(), nr, nc); + } + } + Args::V1* o1[WW_OUTGP_NFIELD - WW_OUTGP_N2D] = {&g.pwst, &g.pnr}; + for (int k = 0; k < WW_OUTGP_NFIELD - WW_OUTGP_N2D; ++k) { + if (in_place()) { + *o1[k] = U1(fields[WW_OUTGP_N2D + k], nr); + } else { + ensure(b.f1[k], "ww_outg.part.field1", nr); + *o1[k] = U1(b.f1[k].data(), nr); + } + } + ww::iogo::w3outg_part(Exec(), s, g); + Exec().fence("ww_outg_part"); + for (int k = 0; k < WW_OUTGP_NFIELD; ++k) { + if (!want(k + 1)) continue; + const size_t n = k < WW_OUTGP_N2D ? nr * nc : nr; + if (!in_place()) { + if (k < WW_OUTGP_N2D) + Kokkos::deep_copy(H2(fields[k], nr, nc), b.f2[k]); + else + Kokkos::deep_copy(H1(fields[k], nr), b.f1[k - WW_OUTGP_N2D]); + } + g_part_back.fetch_add(1, std::memory_order_relaxed); + g_part_back_bytes.fetch_add(static_cast(n * sizeof(float)), std::memory_order_relaxed); + } + if (!in_place()) Exec().fence("ww_outg_part d2h"); + g_part_calls.fetch_add(1, std::memory_order_relaxed); + return WW_KOKKOS_OK; +} + } // namespace } // namespace ww::outgshim +extern "C" int ww_outg_part(const int* ipar, int nipar, const float* rpar, int nrpar, const int* flogrd, + const int* flogr2, const int* mapsf, const int* mapsta, const int* icprt, + const float* dtprt, float* const* fields, int nfields) { + using namespace ww::outgshim; + if (ipar == nullptr || rpar == nullptr || nipar != WW_OUTGP_NIPAR || nrpar != WW_OUTGP_NRPAR || + nfields != WW_OUTGP_NFIELD || fields == nullptr || flogrd == nullptr || flogr2 == nullptr || + mapsf == nullptr || mapsta == nullptr || icprt == nullptr || dtprt == nullptr) + return fail(WW_KOKKOS_ERR_BAD_SHAPE, "ww_outg_part: bad packet"); + try { + if (!Kokkos::is_initialized()) return fail(WW_KOKKOS_ERR_NOT_INITIALISED, "before ww_kokkos_init"); + if (nested_in_openmp()) return fail(WW_KOKKOS_ERR_UNSUPPORTED, "inside an OpenMP region"); + return run_part(ipar, rpar, flogrd, flogr2, mapsf, mapsta, icprt, dtprt, fields); + } catch (const std::bad_alloc& e) { + return fail(WW_KOKKOS_ERR_RUNTIME, e.what()); + } catch (const std::exception& e) { + return fail(WW_KOKKOS_ERR_KERNEL, e.what()); + } catch (...) { + return fail(WW_KOKKOS_ERR_KERNEL, "unknown exception"); + } +} + extern "C" int ww_outg_va(int igrid, const int* ipar, int nipar, const float* rpar, int nrpar, const int* flogrd, const int* flogr2, const int* flogrr, float* va, int nspec, int nsealm, const float* cg, const float* wn, const float* dw, const float* sig, diff --git a/kokkos/src/fortran_iface/part_flags.cpp b/kokkos/src/fortran_iface/part_flags.cpp index b26fe6fb..d17c8d04 100644 --- a/kokkos/src/fortran_iface/part_flags.cpp +++ b/kokkos/src/fortran_iface/part_flags.cpp @@ -15,7 +15,8 @@ // Fortran unless their own flag is "1", so WW_KOKKOS_ALL=1 cannot route them to Kokkos. The PR3 // propagation of the prop3 unit (W3XYP3, W3KTP3, PATCH.md section 11), W3OUTG of the outg unit // (PATCH.md section 15), the partial read of VA before W3IOPE of the pnt unit (section 16) and -// W3UICE's zeroing of VA repeated on the device by the updv unit (section 17) are opt-in the same +// W3UICE's zeroing of VA repeated on the device by the updv unit (section 17) and W3CPRT of the +// cprt unit (section 27) are opt-in the same // way until their rows are regtested. ST4_TABLES is opt-in as well (the owner, 2026-10-17): the // C++ rebuild of the ST4 tables gives the Fortran's values ("0 values differ", the // ST4_TABLES row of PATCH.md) but cost about 8 s serial per run (coast, SRCE 18.3-18.6 s against 26.6-26.7 s with ST4_TABLES, @@ -89,7 +90,8 @@ void print_fallbacks() { constexpr const char* kOptInOnly[] = {"WW_KOKKOS_XYP1", "WW_KOKKOS_KTP1", "WW_KOKKOS_XYP2", "WW_KOKKOS_KTP2", "WW_KOKKOS_XYPUG", "WW_KOKKOS_XYP3", "WW_KOKKOS_KTP3", "WW_KOKKOS_OUTG", "WW_KOKKOS_IOPE", - "WW_KOKKOS_UPDV", "WW_KOKKOS_ST4_TABLES"}; + "WW_KOKKOS_UPDV", "WW_KOKKOS_CPRT", + "WW_KOKKOS_ST4_TABLES"}; } // namespace extern "C" int ww_kokkos_flag(const char* name) { diff --git a/kokkos/src/fortran_iface/parts.d/cprt.txt b/kokkos/src/fortran_iface/parts.d/cprt.txt new file mode 100644 index 00000000..47b15ccb --- /dev/null +++ b/kokkos/src/fortran_iface/parts.d/cprt.txt @@ -0,0 +1,4 @@ +# W3CPRT's spectral partitioning on the resident VA, PATCH.md section 27 (cprt_shim.cpp); opt-in, not under WW_KOKKOS_ALL +# Read by scripts/ww3-gpu.sh: PART (WW_KOKKOS_), COUNT , FILE ; PATCH.md section 9. +PART CPRT +COUNT WW_KOKKOS_CPRT_COUNT diff --git a/kokkos/src/ww_kokkos/CMakeLists.txt b/kokkos/src/ww_kokkos/CMakeLists.txt index c3a4fd4c..35d6d9f7 100644 --- a/kokkos/src/ww_kokkos/CMakeLists.txt +++ b/kokkos/src/ww_kokkos/CMakeLists.txt @@ -73,6 +73,7 @@ target_sources(ww_kokkos PRIVATE part_fld.hpp part_ptmean.hpp part_w3part.hpp + iosf_w3cprt.hpp pdlib_params.hpp pdlib_fsn2.hpp pdlib_fspsi2.hpp diff --git a/kokkos/src/ww_kokkos/iogo_w3outg.hpp b/kokkos/src/ww_kokkos/iogo_w3outg.hpp index d6ecc50d..ed268bac 100644 --- a/kokkos/src/ww_kokkos/iogo_w3outg.hpp +++ b/kokkos/src/ww_kokkos/iogo_w3outg.hpp @@ -660,6 +660,31 @@ KOKKOS_INLINE_FUNCTION void w3outg_part_point(const W3OutgArgs& g, co } } +/// Section 6 (:2242-2340): 6.a the UNDEF fill of the partition fields over their NSEALM rows, +/// 6.b the copy from ICPRT and DTPRT per local sea point. It reads no A: of g it uses mapsf, +/// mapsta, icprt, dtprt and the partition fields only, and s.nseal, s.noswll, s.undef, s.wscut. +/// No other section reads what it writes, so it may run apart from the rest of W3OUTG. +template +void w3outg_part(const ExecSpace& exec, const W3OutgScalars& s, const W3OutgArgs& g) { + const Real undef = s.undef; + const int nsealm = static_cast(g.pwst.extent(0)); + const int ncol = s.noswll + 1; + Kokkos::parallel_for( + "ww::iogo::w3outg_part_init", Kokkos::RangePolicy(exec, 0, nsealm), KOKKOS_LAMBDA(const int j) { + for (int c = 0; c < ncol; ++c) { + g.phs(j, c) = undef; g.ptp(j, c) = undef; g.plp(j, c) = undef; g.pdir(j, c) = undef; + g.psi(j, c) = undef; g.pws(j, c) = undef; g.pthp0(j, c) = undef; g.pqp(j, c) = undef; + g.ppe(j, c) = undef; g.pgw(j, c) = undef; g.psw(j, c) = undef; g.ptm1(j, c) = undef; + g.pt1(j, c) = undef; g.pt2(j, c) = undef; g.pep(j, c) = undef; + } + g.pwst(j) = undef; + g.pnr(j) = undef; + }); + Kokkos::parallel_for( + "ww::iogo::w3outg_part", Kokkos::RangePolicy(exec, 1, s.nseal + 1), + KOKKOS_LAMBDA(const int jsea) { w3outg_part_point(g, s, jsea); }); +} + /// W3OUTG(A, FLPART, FLOUTG, FLOUTG2), :1201-2380, as switch_lab_shrd (s.ompg false) or /// switch_lab_ompg (s.ompg true) compiles it. template @@ -700,23 +725,7 @@ void w3outg(const ExecSpace& exec, const W3OutgScalars& s, const W3OutgFlags& f, KOKKOS_LAMBDA(const int jsea) { w3outg_point(g, s, l, jsea); }); // 6. (:2242-2340) - if (f.flpart) { - const int ncol = s.noswll + 1; - Kokkos::parallel_for( - "ww::iogo::w3outg_part_init", Kokkos::RangePolicy(exec, 0, nsealm), KOKKOS_LAMBDA(const int j) { - for (int c = 0; c < ncol; ++c) { - g.phs(j, c) = undef; g.ptp(j, c) = undef; g.plp(j, c) = undef; g.pdir(j, c) = undef; - g.psi(j, c) = undef; g.pws(j, c) = undef; g.pthp0(j, c) = undef; g.pqp(j, c) = undef; - g.ppe(j, c) = undef; g.pgw(j, c) = undef; g.psw(j, c) = undef; g.ptm1(j, c) = undef; - g.pt1(j, c) = undef; g.pt2(j, c) = undef; g.pep(j, c) = undef; - } - g.pwst(j) = undef; - g.pnr(j) = undef; - }); - Kokkos::parallel_for( - "ww::iogo::w3outg_part", Kokkos::RangePolicy(exec, 1, s.nseal + 1), - KOKKOS_LAMBDA(const int jsea) { w3outg_part_point(g, s, jsea); }); - } + if (f.flpart) w3outg_part(exec, s, g); // :2347-2362 const bool u3s1 = floloc(6, 8); diff --git a/kokkos/src/ww_kokkos/iosf_w3cprt.hpp b/kokkos/src/ww_kokkos/iosf_w3cprt.hpp new file mode 100644 index 00000000..756b54ef --- /dev/null +++ b/kokkos/src/ww_kokkos/iosf_w3cprt.hpp @@ -0,0 +1,121 @@ +// kokkos/src/ww_kokkos/iosf_w3cprt.hpp +// W3CPRT: the spectral partitioning of every local sea point at an output time, ported. Per point +// it converts the action spectrum N(k,theta) of VA to the energy spectrum E(f,theta) = N*TPI*SIG/CG, +// calls W3PART (part_w3part.hpp, unchanged) and stores NP+1 columns XP(1:15,0:NP) in DTPRT one point +// after the other, with ICPRT(JSEA,1) = NP+1 and ICPRT(JSEA,2) the first column of point JSEA. +// Phase 2: it reads the resident VA (ww::resident::device_va), so an output time with partitioning +// copies the partition table back to the host instead of the spectrum. +// +// WW3 heritage: SUBROUTINE W3CPRT, WW3-noaa/model/src/w3iosfmd.F90:109-344 at NOAA-EMC/WW3 03adda91 +// (identical in the h0ffmann/WW3 fork at the WW3/ pin 80644ba9), author H. L. Tolman, with C. Bunney's +// DIMXP of 25-Jul-2018. Only what W3_SHRD compiles is ported: INIT_GET_ISEA gives ISEA = JSEA +// (w3parall.F90:1236, NAPROC = 1). Not ported: W3_T (test output), W3_S (STRACE). The NaN depth stop +// (:270-274) is the caller's (W3KOKKOS_CPRT runs it before the call, which changes no output: the +// Fortran stops before it writes anything a later reader sees). +// +// Two parts, both in the Fortran's order: +// w3cprt_point sections 2-4 (:245-279) for one JSEA: the MAPSTA test, UABS, UDIR, DEPTH, E2 +// and the W3PART call. Each point reads only its own column, so points run in +// parallel and each one is the Fortran's serial code. +// w3cprt_store section 5 (:284-303) for one JSEA after an integer prefix sum: the counters +// and the copy of XP(:,0:NP) to its columns. The Fortran's TMP array grows by +// TSFAC*NSEAL columns and its new columns are zeroed (:288-299), but every column +// 1..FINSIZ is written once by :303 before DTPRT = TMP(:,1:FINSIZ) (:320-326), so +// DTPRT is the concatenation of the XP(:,0:NP) of the points in JSEA order, which +// is what the prefix sum places. Integer sums are exact in any order. +// +// Arithmetic, from the -O3 GIMPLE of w3iosfmd.F90 (gfortran 16.2.0, WW3's Release flags, the +// lab switch; port report kokkos/port_report/W3CPRT_KOKKOS.md): +// - UABS = U10*ASF, UDIR = U10D*RADE, one mulss each. +// - FACT = (TPI*SIG(IK))/CG(IK,ISEA): the product with the folded REAL TPI first, then divss. +// - E2(IK,ITH) = VA(ITH+(IK-1)*NTH,JSEA)*FACT, one mulss. No FMA (contraction off). +// +// SPDX-License-Identifier: LGPL-3.0-or-later +#pragma once + +#include + +#include "part_w3part.hpp" +#include "real.hpp" +#include "st4_params.hpp" + +namespace ww::iosf { + +using ww::Real; + +/// W3PART's result for a point the MAPSTA test of :253 skips (its ICPRT(JSEA,1) stays 0). +inline constexpr int kCprtSkipped = -2; + +/// DIMXP of :213-219. +KOKKOS_INLINE_FUNCTION int w3cprt_dimxp(int nk, int nth, int ptmeth) { + if (ptmeth == 4 || ptmeth == 5) return 2; + return ((nk + 1) / 2) * ((nth - 1) / 2); +} + +/// The Reals and ints of one point's work: E2(NK,NTH), XP(DIMP,0:DIMXP) and W3PART's scratch. +KOKKOS_INLINE_FUNCTION int w3cprt_nreal(int nk, int nth, int dimxp) { + return nk * nth + ww::part::kDimp * (dimxp + 1) + ww::part::w3part_nreal(nk, nth, dimxp); +} +KOKKOS_INLINE_FUNCTION int w3cprt_nint(int nk, int nth, int dimxp, int ihmax) { + return ww::part::w3part_nint(nk, nth, dimxp, ihmax); +} + +/// The arrays of one call, index 0 at the Fortran's lower bound, LayoutLeft: +/// va VA(NSPEC,0:NSEALM) va(isp-1, jsea) +/// mapsf MAPSF(NSEA,3) mapsf(isea-1, 0) = IX; mapsta MAPSTA(NY,NX) +/// u10, u10d, asf (1:NSEA) u10(isea-1) +/// dw DW(0:NSEA) dw(isea); cg, wn CG, WN(0:NK+1,0:NSEA), cg(ik, isea) +template +struct W3cprtArgs { + template + using V1 = Kokkos::View>; + template + using V2 = Kokkos::View>; + V2 va; + V2 mapsf, mapsta; + V1 u10, u10d, asf, dw; + V2 cg, wn; +}; + +/// Sections 2-4 (:245-279) for JSEA = jsea (1-based). Returns W3PART's status (kPtmeanOk, +/// kPtmeanXpFull, kPtmeanNoPeak) and sets np (kCprtSkipped when MAPSTA < 0), xp as W3PART's +/// XP(J,N) = xp[(J-1) + 15*N], nxpfull W3PART's count. e2 holds NK*NTH Reals; w is carved. +template +KOKKOS_INLINE_FUNCTION int w3cprt_point(const W3cprtArgs& a, const ww::part::PtmeanGrid& g, + const ww::part::PtmeanOpts& o, const ww::part::W3partOpts& po, + const int* neigh, int dimxp, int jsea, Real* e2, Real* xp, + const ww::part::W3partScratch& w, int& np, int& nxpfull) { + const int nk = g.nk, nth = g.nth; + const int isea = jsea; // INIT_GET_ISEA under W3_SHRD + const int ix = a.mapsf(isea - 1, 0); + const int iy = a.mapsf(isea - 1, 1); + nxpfull = 0; + if (a.mapsta(iy - 1, ix - 1) < 0) { // :253 + np = kCprtSkipped; + return ww::part::kPtmeanOk; + } + const Real uabs = a.u10(isea - 1) * a.asf(isea - 1); // :258-260 + const Real udir = a.u10d(isea - 1) * ww::st4::kRade; + const Real depth = a.dw(isea); + for (int ik = 1; ik <= nk; ++ik) { // :262-267 + const Real fact = ww::st4::kTpi * g.sig[ik] / a.cg(ik, isea); + for (int ith = 1; ith <= nth; ++ith) + e2[(ik - 1) + (ith - 1) * nk] = a.va(ith + (ik - 1) * nth - 1, jsea) * fact; + } + return ww::part::w3part(g, o, po, neigh, e2, uabs, udir, depth, &a.wn(1, isea), np, xp, dimxp, w, + nxpfull); // :278-279 +} + +/// Section 5 (:284-303) for one point: c is NP+1 when NP >= 0 and 0 otherwise, first its +/// ICPRT(JSEA,2) (1-based column of DTPRT). icprt is ICPRT(NSEALM+1,2), dtprt DTPRT(DIMP,:). +template +KOKKOS_INLINE_FUNCTION void w3cprt_store(int jsea, int np, int first, const Real* xp, const V2i& icprt, + const V2r& dtprt) { + // ICPRT(JSEA+1,2) = ICPRT(JSEA,2) [+ NP + 1] (:251, :286) is the caller's prefix sum. + if (np < 0) return; + icprt(jsea - 1, 0) = np + 1; + for (int n = 0; n <= np; ++n) + for (int j = 0; j < ww::part::kDimp; ++j) dtprt(j, first - 1 + n) = xp[j + ww::part::kDimp * n]; +} + +} // namespace ww::iosf diff --git a/site/data/port_score.tsv b/site/data/port_score.tsv index fee25f47..759ca9e1 100644 --- a/site/data/port_score.tsv +++ b/site/data/port_score.tsv @@ -1,10 +1,10 @@ # The global numbers of the Kokkos port, written by scripts/port_score.py --write; do not edit. # WW3@03adda91. The website reads this file (site/content). # name share count what -of the run time, translated 99.3 % 312 of 1 023 routines a Kokkos translation exists in kokkos/src; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) +of the run time, translated 99.3 % 313 of 1 023 routines a Kokkos translation exists in kokkos/src; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) of the run time, ported 99.3 % 60 of 1 023 routines WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) of the run time, regtested 48.6 % 12 of 1 023 routines WW3's regtests ran with it on Kokkos and agreed with the Fortran; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) -of the lines of code, translated 25.71 % 67 711 of 263 375 Fortran lines of the translated routines, of every line in WW3's model/src, in 64 of 147 files +of the lines of code, translated 25.80 % 67 947 of 263 375 Fortran lines of the translated routines, of every line in WW3's model/src, in 65 of 147 files of the lines of code, ported 6.46 % 17 022 of 263 375 Fortran lines of the ported routines, of every line in WW3's model/src, in 23 of 147 files of the lines of code, regtested 1.02 % 2 689 of 263 375 Fortran lines of the regtested routines, of every line in WW3's model/src, in 6 of 147 files -lines of Kokkos C++ 58 429 in 347 files under kokkos/src: the port itself +lines of Kokkos C++ 59 232 in 350 files under kokkos/src: the port itself From 4094dd049131d5ed0f3e5c0c45c51ff463d5463f Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 14:14:58 -0300 Subject: [PATCH 20/27] The resident preset adds CPRT, W3CPRT on the resident VA presets.txt's resident line becomes SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT, so --kokkos resident, WW_KOKKOS_PRESET=resident and the matrix's WW3GPU_KOKKOS=resident partition on the device copy of VA (PATCH.md section 27) instead of reading it whole at each output time with partitioning on. CPRT refuses no switch the preset did not refuse already: W3KOKKOS_CPRT_SETUP accepts SHRD without MPI or T, and W3KOKKOS_OUTG_SETUP, already in the preset, accepts SHRD without MPI, IG1, O8, O9 or T (WW3@711afb80 w3kokkos_cprtmd.F90:70-76, w3kokkos_outgmd.F90:74-80), so the selection words stay. The consumers follow: the self-test of scripts/ww3-gpu.sh, the fixture parts.d and the expected lists of regtests/cmake/selftest.cmake, and the docs that restate the list (README.md, README.pt-BR.md, kokkos/README.md, docs/GLOSSARY.md, the regtests/CMakeLists.txt comment). Goal abbb0456-6043. Tested: just ww3-gpu-matrix-selftest (ok, 7 serial / 0 omp / 0 mpi entries of the resident preset at WW3@711afb80); bash scripts/ww3-gpu.sh --self-test (0); shellcheck scripts/ww3-gpu.sh (0); just kokkos-test serial-debug and openmp-release (11/11 each); python3 scripts/port_score.py --write then --check (current); scripts/vale.sh on README.md kokkos/README.md docs/GLOSSARY.md (0 errors); matrix entry 38 ww3_tnc1 with WW3GPU_KOKKOS=resident, WW3GPU_LAB_SWITCH=ON, WW3GPU_STRICT=ON, WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1: CPRT=K, test.comp 0/30, failed only its parts-line check for XYP3,KTP3 on a PR0 switch Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- README.pt-BR.md | 2 +- docs/GLOSSARY.md | 4 ++-- kokkos/README.md | 2 +- kokkos/src/fortran_iface/presets.txt | 7 ++++--- regtests/CMakeLists.txt | 2 +- regtests/cmake/selftest.cmake | 6 +++--- scripts/ww3-gpu.sh | 7 ++++--- 8 files changed, 17 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index a87c85e9..688f0065 100644 --- a/README.md +++ b/README.md @@ -149,7 +149,7 @@ with `--kokkos`: | `none` | none | the Fortran only, in the Kokkos build | | `all` | all the parts of the build (`WW_KOKKOS_ALL=1`), but not the opt-in parts below | the widest test of the translation | | `gpu` | `SRCE` | the default of `--gpu`: all of `W3SRCE` in each call ([ADR-0013](docs/ADRs/ADR-0013-operational-parts-default-to-kokkos.md)) | -| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV` | the full time step on Kokkos, with the spectrum kept on the Kokkos side ([next section](#what---kokkos-resident-does)) | +| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT` | the full time step on Kokkos, with the spectrum kept on the Kokkos side ([next section](#what---kokkos-resident-does)) | | a list, for example `SNL1,SDS4` | the parts in the list | a test of one part at a time | ### what `--kokkos resident` does diff --git a/README.pt-BR.md b/README.pt-BR.md index d3cde672..89a8d871 100644 --- a/README.pt-BR.md +++ b/README.pt-BR.md @@ -150,7 +150,7 @@ duas rodadas. Escolha as partes com `--kokkos`: | `none` | nenhuma | só o Fortran, no build com Kokkos | | `all` | todas as partes do build (`WW_KOKKOS_ALL=1`), menos as opcionais abaixo | o teste mais amplo da tradução | | `gpu` | `SRCE` | o padrão do `--gpu`: o `W3SRCE` inteiro em cada chamada ([ADR-0013](docs/ADRs/ADR-0013-operational-parts-default-to-kokkos.md)) | -| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV` | o passo de tempo inteiro em Kokkos, com o espectro guardado do lado do Kokkos ([próxima seção](#o-que-o---kokkos-resident-faz)) | +| `resident` | `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT` | o passo de tempo inteiro em Kokkos, com o espectro guardado do lado do Kokkos ([próxima seção](#o-que-o---kokkos-resident-faz)) | | uma lista, por exemplo `SNL1,SDS4` | as partes da lista | o teste de uma parte por vez | ### o que o `--kokkos resident` faz diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md index fcb2b01e..096ca35c 100644 --- a/docs/GLOSSARY.md +++ b/docs/GLOSSARY.md @@ -705,7 +705,7 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | `WW_KOKKOS_VA_CHECK` / `WW_KOKKOS_VA_POISON` | diagnostics of the resident VA, read once, ignoring `WW_KOKKOS_ALL`, which compare or overwrite only the stale copy: `VA_CHECK=1` makes every sync call check what the state claims before it copies (unless `HOST`, the host `VA` still holds the bits of the last sync, so a host write without its `W3KOKKOS_VA_HOST_WRITE` stops the model; with `BOTH`, `d_va` holds them too, so a kernel that wrote `d_va` without marking it does); `VA_POISON=1` fills the host `VA` with a quiet NaN after every kernel write of `d_va`, so a host read without its sync, or a column the partial read misses, puts NaN in the output `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset; a correct run writes the same bits with either. Added by the mutation audit of 2026-10-17. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | | `WW_KOKKOS_OUTG_STAGED` | a diagnostic of the outg unit, read once: `=1` makes a host backend take the device's path of `ww_outg_va`, the inputs copied into the shim's own buffers, the fields computed into buffers, and only the fields a host reader needs copied back, so a field the copy-back selection misses keeps its old bits in WW3's array and shows in `out_grd.ww3` `(v)` [kokkos/src/fortran_iface/outg_shim.cpp](../kokkos/src/fortran_iface/outg_shim.cpp) | Default unset: on Serial and OpenMP the kernel writes WW3's arrays in place. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | | `WW_KOKKOS_STRICT` | environment variable read once by `ww_kokkos_strict()`: `=1` makes every place where a part asked on Kokkos would run the Fortran stop the model with `EXTCDE` and a `NDSE` line naming the part and the reason: `KOKKOS_PART` under `WW_KOKKOS_ALL=1`, the two `W3SRCE` set-up fallbacks and the per-step `W3SRCE` gate (`W3KOKKOS_SRCE_STEP`). Unset, those places run the Fortran as before and the shim prints one line at exit, `ww_kokkos: calls left on the Fortran (...): SRCE= ...` `(v)` [kokkos/src/fortran_iface/part_flags.cpp](../kokkos/src/fortran_iface/part_flags.cpp) | Default unset. The regtest matrix's Kokkos run sets it, its Fortran runs unset it, so a pair cannot compare the Fortran with itself unseen. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 24 | -| `WW_KOKKOS_PRESET` / `presets.txt` | environment variable naming a line `PRESET ` of `kokkos/src/fortran_iface/presets.txt`: `ww_kokkos_flag` turns its parts on as if each `WW_KOKKOS_` were `1`; a set `WW_KOKKOS_` still wins, and `W3INIT` stops on a name the file lacks. `resident` is the only preset (`SRCE,XYP3,KTP3,OUTG,IOPE,UPDV`) `(v)` [kokkos/src/fortran_iface/presets.txt](../kokkos/src/fortran_iface/presets.txt) | The file is the one list: CMake compiles it into the shim, `scripts/ww3-gpu.sh --kokkos resident` and the matrix's `WW3GPU_KOKKOS=resident` read it. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 25 | +| `WW_KOKKOS_PRESET` / `presets.txt` | environment variable naming a line `PRESET ` of `kokkos/src/fortran_iface/presets.txt`: `ww_kokkos_flag` turns its parts on as if each `WW_KOKKOS_` were `1`; a set `WW_KOKKOS_` still wins, and `W3INIT` stops on a name the file lacks. `resident` is the only preset (`SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT`) `(v)` [kokkos/src/fortran_iface/presets.txt](../kokkos/src/fortran_iface/presets.txt) | The file is the one list: CMake compiles it into the shim, `scripts/ww3-gpu.sh --kokkos resident` and the matrix's `WW3GPU_KOKKOS=resident` read it. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 25 | | mutation audit / mutant (equivalent, surviving) | a deliberate, minimal breakage of a piece of the port, applied in a scratch tree and never committed, to ask whether the checks that call the piece correct would fail without it: a mutant is *detected* when a check fails, *surviving* when every check passes, and *equivalent* when it cannot change any bit the case computes (it is not a defect the case can show) `(v)` [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | First run 2026-10-17, on the CPU. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) | | `WW_KOKKOS_ALL` / `ww_kokkos_flag` | the master run-time flag of every Kokkos part, and the one C function that resolves each `WW_KOKKOS_`: a set part flag wins (`1` on, anything else off), an unset one follows `WW_KOKKOS_ALL=1` (on) or stays off `(v)` [kokkos/src/fortran_iface/part_flags.cpp](../kokkos/src/fortran_iface/part_flags.cpp) | "100 %" means every wired part, not all of WW3; under the master a part whose translation does not cover the switch file stays Fortran. `W3INIT` logs one "Kokkos parts" line; `scripts/ww3-gpu.sh --kokkos all\|none\|` sets it. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | | W3KOKKOS_UNITSMD / W3KOKKOS_REGISTER / `parts.d` | the Fortran module the fork's CMake generates from every `model/src/w3kokkos_md.F90` (it `USE`s each unit module and calls each unit's set-up from `W3INIT`), the routine a unit's set-up calls to put a part on the "Kokkos parts" line, and the per-unit files `kokkos/src/fortran_iface/parts.d/.txt` that `scripts/ww3-gpu.sh` reads its parts from `(v)` h0ffmann/WW3 `ww3-gpu/kokkos-wire-base` | A wiring unit adds files and its own call-site hunks and edits no shared list. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | @@ -720,7 +720,7 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md). | `libm_device.hpp` / `ww::dm` / `WW_KOKKOS_LIBM` | the header through which every libm call of the kernels `WW_KOKKOS_ALL` runs goes, its namespace, and the CMake option (CUDA builds only, `ON` by default) that points its device branch at `ww::libm`; the host branch is the Kokkos call the kernel made before, so the CPU build calls glibc as it did `(v)` [kokkos/src/ww_kokkos/libm_device.hpp](../kokkos/src/ww_kokkos/libm_device.hpp), [kokkos/src/ww_kokkos/CMakeLists.txt](../kokkos/src/ww_kokkos/CMakeLists.txt) | WFIP-0006 units 1 to 4 (#178). Overloads for the other argument type are deleted, so a float/double promotion does not compile. `-DWW_KOKKOS_LIBM=OFF` keeps libdevice and ADR-0014's verdict by tolerance. | [docs/WFIPs/WFIP-0006-device-libm-bits.md](WFIPs/WFIP-0006-device-libm-bits.md), [AGENTS.md](../AGENTS.md) | | `gen_tables` / `ww_libm_tables.hpp` / `WW_LIBM_HOST_FMA` | a host program the CUDA build compiles with the host g++ (nvcc's front end fails on it) and runs at build time; it writes a header with libmvec's `powf` tables read from the loaded `libmvec.so.1`, this host's `rcpps` values, and `WW_LIBM_HOST_FMA`, glibc's fma or sse2 build decided from glibc's own `exp` `(v)` [kokkos/src/ww_kokkos/CMakeLists.txt](../kokkos/src/ww_kokkos/CMakeLists.txt), [kokkos/tools/libm_bits/gen_tables.cpp](../kokkos/tools/libm_bits/gen_tables.cpp) | It follows `GLIBC_TUNABLES`; the run host must be the build host, or have the same glibc build and `rcpps`, which nothing checks at run time ⚠. | [docs/WFIPs/WFIP-0006-device-libm-bits.md](WFIPs/WFIP-0006-device-libm-bits.md) | | `WW_KOKKOS_TIMERS` / `--timers` / `ww::timers::Region` | a diagnostic environment variable, read once and ignoring `WW_KOKKOS_ALL`: `=1` makes every shim time its regions, named `.` (`.call` the entry point, `.h2d` the copies to the device, `.kernel` the launch, `.d2h` the copies back; `st4.tables.call` the ST4 table rebuild), fencing the default execution space at each start and stop, and print one line per region at exit `(v)` [kokkos/src/fortran_iface/timers.hpp](../kokkos/src/fortran_iface/timers.hpp) | `scripts/ww3-gpu.sh --ww3-gpu --timers` sets it and prints each region's share of `ww3_shel` and the time outside Kokkos; refused with `--repeat`, because a fenced run is a breakdown, not a timing. Off, a region is one branch (#175). | [scripts/ww3-gpu.sh](../scripts/ww3-gpu.sh), [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) | -| `--kokkos gpu` / `--kokkos resident` / `just perf-ww3-gpu` | the two part-list presets of `scripts/ww3-gpu.sh`: `gpu` is `SRCE` alone (ADR-0013), `resident` is `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV`, the parts that keep `VA` on the device (WFIP-0005 units 1 to 5); and the recipe that times a case with `--gpu --repeat` `(v)` [scripts/ww3-gpu.sh](../scripts/ww3-gpu.sh) `kokkos_preset`, [justfile](../justfile) | `resident` is not a default: none of its parts is regtested yet (`kokkos/ports.tsv`). With it the full case copies `VA` to the device once and back once, on the CPU (#179); nothing has run on a GPU. `just perf-ww3-gpu` asks the supervising human for "yes". | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), [bench/README.md](../bench/README.md) | +| `--kokkos gpu` / `--kokkos resident` / `just perf-ww3-gpu` | the two part-list presets of `scripts/ww3-gpu.sh`: `gpu` is `SRCE` alone (ADR-0013), `resident` is `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT`, the parts that keep `VA` on the device (WFIP-0005 units 1 to 5, and `W3CPRT`, PATCH.md section 27); and the recipe that times a case with `--gpu --repeat` `(v)` [scripts/ww3-gpu.sh](../scripts/ww3-gpu.sh) `kokkos_preset`, [justfile](../justfile) | `resident` is not a default: none of its parts is regtested yet (`kokkos/ports.tsv`). With it the full case copies `VA` to the device once and back once, on the CPU (#179); nothing has run on a GPU. `just perf-ww3-gpu` asks the supervising human for "yes". | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), [bench/README.md](../bench/README.md) | | `ww_srce_init` / `ww_srce` / `ww_srce_batch` / `ww_srce_enabled` / `ww_srce_last_error` | the C entry points of the W3SRCE port; `ww_srce_batch` takes every active sea point of a time step in one call (`W3KOKKOS_SRCE_BATCH` on the fork) `(v)` [kokkos/src/fortran_iface/ww_kokkos_c.hpp](../kokkos/src/fortran_iface/ww_kokkos_c.hpp) | `ww_srce_init` needs the DIA and ST4 contexts of the same grid | | GIMPLE | GCC's three-address intermediate representation; `-fdump-tree-optimized` writes it after the optimisations `(v)` [ADR-0011](ADRs/ADR-0011-port-in-units-regtest-at-checkpoints.md) | The static-parity argument compares the Fortran's and the C++'s, statement by statement | | `maxss` / `minss` / `maxps` | the x86 SSE scalar and packed float max/min instructions; `maxss a,b` keeps `b` only when `b > a`, so a NaN or a `+0`/`-0` tie returns `a` `(v)` [kokkos/port_report/W3SRCE_KOKKOS.md](../kokkos/port_report/W3SRCE_KOKKOS.md) | The operand order a Fortran `MAX`/`MIN` gets in the object, which the port copies; GCC may swap it after GIMPLE | diff --git a/kokkos/README.md b/kokkos/README.md index bb8f1ca4..4bb7aa66 100644 --- a/kokkos/README.md +++ b/kokkos/README.md @@ -411,7 +411,7 @@ this is the short form. | partial | `WW_KOKKOS_=1` for each part wanted | `--kokkos SNL1,SDS4` | the listed parts | | full | `WW_KOKKOS_ALL=1` | `--kokkos all` | every wired part whose translation covers the switch file | | GPU preset | `WW_KOKKOS_ALL=0 WW_KOKKOS_SRCE=1` | `--kokkos gpu` (the default of `--gpu`) | `W3SRCE` with the source terms it calls (ADR-0013) | -| resident preset | `WW_KOKKOS_PRESET=resident`, or `WW_KOKKOS_ALL=0` and `WW_KOKKOS_=1` for `SRCE`, `XYP3`, `KTP3`, `OUTG`, `IOPE`, `UPDV` (the line of [`presets.txt`](src/fortran_iface/presets.txt)) | `--kokkos resident` | the parts that keep the spectrum `VA` on the device (WFIP-0005 units 1 to 5, #173, #179); not a default: none of its parts is regtested yet ([`ports.tsv`](ports.tsv)) | +| resident preset | `WW_KOKKOS_PRESET=resident`, or `WW_KOKKOS_ALL=0` and `WW_KOKKOS_=1` for `SRCE`, `XYP3`, `KTP3`, `OUTG`, `IOPE`, `UPDV`, `CPRT` (the line of [`presets.txt`](src/fortran_iface/presets.txt)) | `--kokkos resident` | the parts that keep the spectrum `VA` on the device (WFIP-0005 units 1 to 5, #173, #179); not a default: none of its parts is regtested yet ([`ports.tsv`](ports.tsv)) | An explicitly set `WW_KOKKOS_` wins over `WW_KOKKOS_ALL`; one C function, `ww_kokkos_flag` in `src/fortran_iface/part_flags.cpp`, resolves both for the Fortran and the diff --git a/kokkos/src/fortran_iface/presets.txt b/kokkos/src/fortran_iface/presets.txt index 4a3985d9..4cf81d79 100644 --- a/kokkos/src/fortran_iface/presets.txt +++ b/kokkos/src/fortran_iface/presets.txt @@ -3,6 +3,7 @@ # scripts/ww3-gpu.sh --kokkos and the matrix's WW3GPU_KOKKOS= read it, so the three # cannot drift. Every part is a PART of parts.d. # -# resident: W3SRCE batched and the PR3 propagation, W3OUTG, the point output's read and W3UICE's -# write of VA on the resident spectrum (WFIP-0005; PATCH.md sections 11 and 15 to 18). -PRESET resident SRCE,XYP3,KTP3,OUTG,IOPE,UPDV +# resident: W3SRCE batched and the PR3 propagation, W3OUTG, the point output's read, W3UICE's +# write of VA and W3CPRT's partitioning on the resident spectrum (WFIP-0005; PATCH.md sections 11, +# 15 to 18, 27 and 28). CPRT refuses no switch OUTG accepts: both need SHRD and refuse MPI and T. +PRESET resident SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT diff --git a/regtests/CMakeLists.txt b/regtests/CMakeLists.txt index 5cdd7a98..53c7faec 100644 --- a/regtests/CMakeLists.txt +++ b/regtests/CMakeLists.txt @@ -24,7 +24,7 @@ # WW3GPU_KOKKOS empty (the default): the Kokkos run sets WW3GPU_ROUTINES as above. Otherwise # the flags of scripts/ww3-gpu.sh --kokkos, so a regtest and a timed run share # them: a preset of kokkos/src/fortran_iface/presets.txt (resident: -# SRCE,XYP3,KTP3,OUTG,IOPE,UPDV), all (WW_KOKKOS_ALL=1), none, +# SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT), all (WW_KOKKOS_ALL=1), none, # or a comma list of the parts of kokkos/src/fortran_iface/parts.d; the # selection is then WW3GPU_SWITCH_WORDS alone, and the Kokkos run must show # each part as =K in W3INIT's parts line diff --git a/regtests/cmake/selftest.cmake b/regtests/cmake/selftest.cmake index cc85355a..10e681e8 100644 --- a/regtests/cmake/selftest.cmake +++ b/regtests/cmake/selftest.cmake @@ -164,11 +164,11 @@ endforeach() # --- WW3GPU_KOKKOS: parts.d, the preset, the run environment, the parts line, the words ------ file(MAKE_DIRECTORY "${t}/parts.d") file(WRITE "${t}/parts.d/a.txt" "# a unit\nPART SRCE\nCOUNT WW_KOKKOS_SRCE_COUNT\n\nFILE model/src/w3srcemd.F90\n") -file(WRITE "${t}/parts.d/b.txt" "PART XYP3\nPART KTP3\nPART OUTG\nPART IOPE\nPART UPDV\nCOUNT WW_KOKKOS_PROP3_COUNT\n") +file(WRITE "${t}/parts.d/b.txt" "PART XYP3\nPART KTP3\nPART OUTG\nPART IOPE\nPART UPDV\nPART CPRT\nCOUNT WW_KOKKOS_PROP3_COUNT\n") ww3gpu_parts_d(pp pc "${t}/parts.d") -expect("${pp}|${pc}" "SRCE;XYP3;KTP3;OUTG;IOPE;UPDV|WW_KOKKOS_SRCE_COUNT;WW_KOKKOS_PROP3_COUNT" "parts.d") +expect("${pp}|${pc}" "SRCE;XYP3;KTP3;OUTG;IOPE;UPDV;CPRT|WW_KOKKOS_SRCE_COUNT;WW_KOKKOS_PROP3_COUNT" "parts.d") ww3gpu_kokkos_preset(f resident "${pp}") -expect("${f}" "SRCE;XYP3;KTP3;OUTG;IOPE;UPDV" "preset resident") +expect("${f}" "SRCE;XYP3;KTP3;OUTG;IOPE;UPDV;CPRT" "preset resident") ww3gpu_kokkos_preset(f all "${pp}") expect("${f}" "ALL" "preset all") ww3gpu_kokkos_preset(f none "${pp}") diff --git a/scripts/ww3-gpu.sh b/scripts/ww3-gpu.sh index b05da369..c8400cf8 100755 --- a/scripts/ww3-gpu.sh +++ b/scripts/ww3-gpu.sh @@ -75,10 +75,11 @@ # tables for SRCE alone (WW3/model/src/w3initmd.F90:759,771). --kokkos resident adds the PR3/UQ # propagation (XYP3, KTP3, opt-in, kokkos/src/fortran_iface/PATCH.md §11 and §13) and W3OUTG (OUTG, # opt-in, §15), the partial read before the point output (IOPE, opt-in, §16) and W3UICE's zeroing -# of VA repeated on the device (UPDV, opt-in, §17): with SRCE on the resident VA, the spectrum then +# of VA repeated on the device (UPDV, opt-in, §17) and W3CPRT's partitioning (CPRT, opt-in, §27): +# with SRCE on the resident VA, the spectrum then # crosses only for the host readers of VA at output times, not for the gridded fields, the point # output copies back only the spectra it interpolates, and an ice update copies none (WFIP-0005 -# units 1 to 5). It is not a default: XYP3, KTP3, OUTG, IOPE and UPDV are not regtested. --kokkos gpu is the default of --gpu when +# units 1 to 5). It is not a default: XYP3, KTP3, OUTG, IOPE, UPDV and CPRT are not regtested. --kokkos gpu is the default of --gpu when # neither --kokkos nor WW3GPU_ROUTINES is given. The Fortran run of the pair is unchanged. # # --cpu is Kokkos' OpenMP backend, --gpu its CUDA backend. The script enters the pinned toolchain @@ -259,7 +260,7 @@ if [ "$mode" = self-test ]; then [ "$(kokkos_env 0 all)" = "WW_KOKKOS_ALL=0" ] || { echo "ww3-gpu: the Fortran run of --kokkos all: $(kokkos_env 0 all)" >&2; exit 1; } [ "$(kokkos_env 1 "$(kokkos_preset gpu)")" = "WW_KOKKOS_ALL=0 WW_KOKKOS_SRCE=1" ] \ || { echo "ww3-gpu: --kokkos gpu: $(kokkos_env 1 "$(kokkos_preset gpu)")" >&2; exit 1; } - [ "$(kokkos_env 1 "$(kokkos_preset resident)")" = "WW_KOKKOS_ALL=0 WW_KOKKOS_SRCE=1 WW_KOKKOS_XYP3=1 WW_KOKKOS_KTP3=1 WW_KOKKOS_OUTG=1 WW_KOKKOS_IOPE=1 WW_KOKKOS_UPDV=1" ] \ + [ "$(kokkos_env 1 "$(kokkos_preset resident)")" = "WW_KOKKOS_ALL=0 WW_KOKKOS_SRCE=1 WW_KOKKOS_XYP3=1 WW_KOKKOS_KTP3=1 WW_KOKKOS_OUTG=1 WW_KOKKOS_IOPE=1 WW_KOKKOS_UPDV=1 WW_KOKKOS_CPRT=1" ] \ || { echo "ww3-gpu: --kokkos resident: $(kokkos_env 1 "$(kokkos_preset resident)")" >&2; exit 1; } [ "$(kokkos_preset SNL1,GATH)" = "SNL1,GATH" ] || { echo "ww3-gpu: kokkos_preset changed a part list" >&2; exit 1; } # cmp_files: a point output or restart that differs, or exists on one side only, fails; the From 52de560a9ed6a22b5fab7f7139cd7cc09788b1bf Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 14:19:15 -0300 Subject: [PATCH 21/27] PATCH.md section 28: the integration fixes on ww3-gpu/integ-fixes A fork-table row for ww3-gpu/integ-fixes (711afb80 on b733066b) and a section for the three integration tasks: the two W3KOKKOS_CPRT fallbacks now report to WW_KOKKOS_STRICT through KOKKOS_FALLBACK; CPRT in the resident preset refuses no switch OUTG did not refuse already; and the KOKKOS build without PR3 needed no fork change, since 31a09522 already guards the PR3-only USE of w3kokkosmd.F90:3183 at 80644ba9 and b733066b contains it. It records ww3_tnc1 (SHRD PR0, partition fields) through the matrix and the smoke and full pairs at 711afb80, and the preset gap it found: without PR3 and UQ, XYP3 and KTP3 never reach KOKKOS_PART, so only the matrix's parts-line check reports them. Goal abbb0456-6043. Tested: matrix entry 38 ww3_tnc1 (WW3GPU_LAB_SWITCH=ON, WW3GPU_STRICT=ON, WW3GPU_SWITCH_WORDS=PR0;SHRD, WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1) with WW3GPU_KOKKOS=resident (test.comp 0/30, parts-line check fails on XYP3,KTP3) and SRCE,OUTG,IOPE,UPDV,CPRT (passes, test.comp 0/30, whole-VA D2H 0); bash scripts/ww3-gpu.sh --ww3-gpu smoke|full --cpu --kokkos resident with the same two variables at WW3@711afb80 (byte-identical to the Fortran and to --ww3, nccmp-tol PASS, whole-VA D2H 37 and 0); scripts/vale.sh kokkos/src/fortran_iface/PATCH.md (0 errors) Co-Authored-By: Claude Opus 5.5 --- kokkos/src/fortran_iface/PATCH.md | 54 +++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/kokkos/src/fortran_iface/PATCH.md b/kokkos/src/fortran_iface/PATCH.md index f139a95d..da727866 100644 --- a/kokkos/src/fortran_iface/PATCH.md +++ b/kokkos/src/fortran_iface/PATCH.md @@ -7,6 +7,7 @@ Read with `git -C WW3 fetch origin && git -C WW3 branch -r` and `git rev-list -- | h0ffmann/WW3 branch | Head | Against the `WW3/` pin | What it wires | |---|---|---|---| +| `ww3-gpu/integ-fixes` | `711afb80` | one commit on `ww3-gpu/integ-abbb0456` (`b733066b`, the merge of `u1-multi-va`, `u2-strict-preset`, `u8-out-readers` and `u9-cprt` onto `80644ba9`), pushed 2026-10-18 (v, `ls-remote`) | `W3KOKKOS_CPRT`'s two fallbacks to the Fortran `W3CPRT` report to `WW_KOKKOS_STRICT` ([section 28](#28-the-integration-fixes-ww3-gpuinteg-fixes)) | | `ww3-gpu/u1-multi-va` | `f330dc53` | five commits on `80644ba9`, pushed 2026-10-18 (v, `ls-remote`) | `ww3_multi`'s exchanges sync only the columns of `VA` they read or write (`W3KOKKOS_VA_READ_COLS`, `W3KOKKOS_VA_WRITE_COLS`, `ww_va_host_wrote_cols`), no read at `W3WAVE`'s return, the ABI `0.10` ([section 26](#26-ww3_multi-and-the-resident-va-column-syncs-no-read-at-w3waves-return-ww3-gpuu1-multi-va)) | | `ww3-gpu/kokkos-outg` | `e0cc8ee9` | the `WW3/` pin of h0ffmann/ww3-gpu `goal/42-gpu-step-units45`; one commit on `ww3-gpu/kokkos-prop3dev`, pushed 2026-10-17 (v) | `W3OUTG`'s integrals on the device copy of `VA` (`ww_outg_va`, `W3KOKKOS_OUTGMD`), the host read of `VA` at an output time split per reader, the ABI `0.7` ([section 15](#15-w3outg-on-the-resident-spectrum-ww3-gpukokkos-outg)) | | `ww3-gpu/kokkos-prop3dev` | `90631fad` | the parent of `e0cc8ee9`, the `WW3/` pin of h0ffmann/ww3-gpu `goal/42-srce-prop3dev`; one commit on `ww3-gpu/kokkos-resident`, pushed 2026-10-17 (v) | `W3XYP3` and `W3KTP3` batched on the device copy of `VA` (`ww_xyp3_batch_va`, `ww_ktp3_batch_va`), their host syncs only on the Fortran path, the ABI `0.6` ([section 13](#13-the-propagation-on-the-resident-spectrum-ww3-gpukokkos-prop3dev)) | @@ -2162,3 +2163,56 @@ throwaway `bench_small` variants with partitioning on and the numbers are in staged switch for its `PR0` ⚠, because the fork's `KOKKOS` build needs `PR3`. No `matrix.base` entry with partitioning builds with `KOKKOS` (`ww3_tnc1` is `PR0`), so the matrix proper did not run this part ⚠. + +## 28. The integration fixes, `ww3-gpu/integ-fixes` + +One fork commit, `711afb80`, on `b733066b` of `ww3-gpu/integ-abbb0456` (sections 23 to 27 merged +onto `80644ba9`), pushed to h0ffmann/WW3 on 2026-10-18; not at the pin until the integrator moves +it. No ABI change. + +- **The `CPRT` fallbacks report to strict mode.** Section 27 left them out of section 24's count. + `W3KOKKOS_CPRT` now calls `KOKKOS_FALLBACK ( 'CPRT', REASON )` on both branches that run the + Fortran `W3CPRT` (`w3kokkos_cprtmd.F90:167-175` at `711afb80`): a sea point with a NaN depth, and + a spectrum `PTMEAN` cannot take (the kernel's `FALLBACK` flag). With `WW_KOKKOS_STRICT=1` the + model stops on `... PART CPRT WOULD RUN ON THE FORTRAN: `; without it the call is counted, + one per output time that fell back. `W3KOKKOS_CPRT_SETUP` makes the counter when the part is on + (`:78-81`), so the line at exit shows `CPRT=0` when nothing fell back. The path that runs the + kernel adds one `LOGICAL` store per call and nothing per point. Neither fallback was made to + happen in a run: no case here has a NaN depth or a spectrum that trips `PTMEAN` ⚠. +- **`CPRT` joins the `resident` preset** ([`presets.txt`](presets.txt): + `SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT`). Its consumers changed with it: the shim compiles the file + (`just kokkos-test serial-debug` and `openmp-release`), `scripts/ww3-gpu.sh --self-test` checks + the seven flags, and `regtests/cmake/selftest.cmake` has `CPRT` in its fixture `parts.d` and in + the expected list. The selection words stay `ST4;NL1;PR3;UQ;SHRD;!PDLIB;!MGP;!MGG`: under a set + flag `KOKKOS_PART` stops on a switch the part does not cover, and `CPRT` refuses a switch without + `SHRD` or with `MPI` or `T` (`w3kokkos_cprtmd.F90:70-76`), a subset of what `OUTG`, already in + the preset, refuses (`SHRD` missing, `MPI`, `IG1`, `O8`, `O9`, `T`, `w3kokkos_outgmd.F90:74-80`). + So the preset stops on no switch it ran before (v, by reading both set-ups). +- **The `KOKKOS` build without `PR3`.** The `PR3`-only `USE` at `w3kokkosmd.F90:3183` of + `80644ba9` is the one `31a09522` (section 23) put under `W3KOKKOS_HAS_PR3`, and `b733066b` + contains that commit (`git merge-base --is-ancestor`, v). Section 27's note that the fork's + `KOKKOS` build needs `PR3` was written on `u9-cprt`, which branched before it. No fork change + was needed: `ww3_tnc1` (`SHRD PR0`) builds with `KOKKOS` at `b733066b` and at `711afb80` (v, + below). + +`ww3_tnc1`, matrix entry 38, the serial `matrix.base` entry that writes partition fields (`PHS +PTP`, `FIELD%PARTITION = '0 1 2'`), on 2026-10-18 (OpenMP backend, gfortran 16.2.0, i9-14900 +shared with other agents, `OMP_NUM_THREADS=4`; never `--gpu`), with `WW3GPU_LAB_SWITCH=ON`, +`WW3GPU_STRICT=ON`, `WW3GPU_SWITCH_WORDS=PR0;SHRD`, `WW_KOKKOS_VA_POISON=1` and +`WW_KOKKOS_VA_COUNT=1`: + +| `WW3GPU_KOKKOS` | Result | +|---|---| +| `resident` | builds; `CPRT=K OUTG=K IOPE=K UPDV=K SRCE=K`, 3 `W3CPRT` Kokkos calls on 243 sea points and 3 partition-field calls of `W3OUTG`; test.comp 0/30 files different; the entry fails its parts-line check, `XYP3,KTP3` not on Kokkos (v) | +| `SRCE,OUTG,IOPE,UPDV,CPRT` | passes: test.comp 0/30, `CPRT` fallbacks 0, whole-`VA` D2H 0, `calls left on the Fortran ...: SRCE=0 CPRT=0` (v) | + +The first row shows a gap of the preset that `CPRT` does not cause: without `PR3` and `UQ`, +`W3KOKKOS_PROP3_SETUP` compiles its body out (`w3kokkos_prop3md.F90:111-138`), so `XYP3` and `KTP3` +never reach `KOKKOS_PART`, the run neither stops nor counts them, and only the matrix's parts-line +check reports it. The selection words keep every `PR0` entry out of a `resident` matrix run. + +`bash scripts/ww3-gpu.sh --ww3-gpu smoke|full --cpu --kokkos resident` with `WW_KOKKOS_VA_POISON=1 +WW_KOKKOS_VA_COUNT=1`, at `711afb80` (2026-10-18, same machine, one run each): every model file +byte-identical to the Fortran of the same executable and every file but the NetCDF to `--ww3`'s, +`nccmp-tol` PASS; whole-`VA` D2H 37 on smoke and 0 on full, as in section 26; `CPRT=K` with 0 calls, +since neither case partitions (v). From 911981f459a00b586a672ff08933e2b90810c65c Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 14:24:41 -0300 Subject: [PATCH 22/27] PATCH.md 28: XYP3 and KTP3 stop or count without PR3 (fork a304018c) Agent H found that on a switch without PR3 and UQ the PROP3 setup compiled to nothing, so the resident preset ran XYP3 and KTP3 on the Fortran without a word, under strict mode too. The fork commit a304018c asks KOKKOS_PART with SUPPORTED false there. Tested: gfortran -E -P of w3kokkos_prop3md.F90, switch_lab_shrd and switch_lab_ompg plus KOKKOS, identical apart from blank lines; ww3_tnc1 matrix entry 38, WW3GPU_KOKKOS=resident, WW3GPU_LAB_SWITCH=ON, words PR0;SHRD, at a304018c: the Kokkos run stops in W3INIT naming XYP3; nix shell nixpkgs#vale -c scripts/vale.sh PATCH.md, rc 0 Co-Authored-By: Claude Opus 5.5 --- kokkos/src/fortran_iface/PATCH.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kokkos/src/fortran_iface/PATCH.md b/kokkos/src/fortran_iface/PATCH.md index da727866..6a5534dc 100644 --- a/kokkos/src/fortran_iface/PATCH.md +++ b/kokkos/src/fortran_iface/PATCH.md @@ -2211,6 +2211,16 @@ The first row shows a gap of the preset that `CPRT` does not cause: without `PR3 never reach `KOKKOS_PART`, the run neither stops nor counts them, and only the matrix's parts-line check reports it. The selection words keep every `PR0` entry out of a `resident` matrix run. +Closed by a second fork commit, `a304018c` on `711afb80` (the integrator, 2026-10-18): +`W3KOKKOS_PROP3_SETUP` gains an `#else` branch for a switch without `PR3` and `UQ` that asks +`KOKKOS_PART ( 'XYP3', .FALSE. )` and `( 'KTP3', .FALSE. )` and registers both. A set flag or a +preset now stops, and `WW_KOKKOS_ALL=1` counts or, under `WW_KOKKOS_STRICT=1`, stops. On +`switch_lab_shrd` and `switch_lab_ompg` the preprocessed module (`gfortran -E -P`, `#ww3` shell) is +the same as before apart from blank lines (v). `ww3_tnc1` with `WW3GPU_KOKKOS=resident`, as above, +at `a304018c`: the Kokkos run stops in `W3INIT` with `WW_KOKKOS_XYP3=1, BUT ITS KOKKOS TRANSLATION +DOES NOT COVER THIS SWITCH FILE`, the row reads `ran fortran-only` (v), where at `711afb80` it ran +with both parts silently on the Fortran. + `bash scripts/ww3-gpu.sh --ww3-gpu smoke|full --cpu --kokkos resident` with `WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1`, at `711afb80` (2026-10-18, same machine, one run each): every model file byte-identical to the Fortran of the same executable and every file but the NetCDF to `--ww3`'s, From f905fbc3a5f1a3f856caf79adaa2a767cc14701b Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 14:25:30 -0300 Subject: [PATCH 23/27] WW3: pin a304018c, the fork with U1, U2, U8, U9 and the integration fixes One pin bump for goal abbb0456-6043: the fork commit a304018c (branch ww3-gpu/integ-fixes of h0ffmann/WW3) merges ww3-gpu/u1-multi-va, u2-strict-preset, u8-out-readers and u9-cprt onto 80644ba9, plus the CPRT strict fallbacks (711afb80) and the PROP3 setup without PR3 (a304018c). The WMIOHG, WMIOEG and W3CPRT rows of kokkos/ports.tsv are wired at this pin; the port score is regenerated. Tested: python3 scripts/port_score.py --write, then --check, current; the fork commits' own evidence is in PATCH.md sections 23 to 28 Co-Authored-By: Claude Opus 5.5 --- WW3 | 2 +- kokkos/PORT_STATUS.md | 2 +- kokkos/PORT_TABLE.tsv | 6 +++--- kokkos/ports.tsv | 6 +++--- site/data/port_score.tsv | 4 ++-- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/WW3 b/WW3 index 80644ba9..a304018c 160000 --- a/WW3 +++ b/WW3 @@ -1 +1 @@ -Subproject commit 80644ba9fcb0498f8ef2cc15eff70cb2bd12b60c +Subproject commit a304018c3453133293b9c708353e7bf6424e271c diff --git a/kokkos/PORT_STATUS.md b/kokkos/PORT_STATUS.md index 12eff9c8..cf98daf6 100644 --- a/kokkos/PORT_STATUS.md +++ b/kokkos/PORT_STATUS.md @@ -46,7 +46,7 @@ Profile: full (bench_small, 120 x 80, 9 520 sea points, 12 h), Release -pg, swit | level | run time | lines | routines | files | what it counts (WW3@03adda91) | |---|---|---|---|---|---| | translated | **99.3 %** | 67947 of 263375 (25.80 %) | 313 of 1023 (30.60 %) | 65 of 147 | a Kokkos translation exists in kokkos/src | -| ported | **99.3 %** | 17022 of 263375 (6.46 %) | 60 of 1023 (5.87 %) | 23 of 147 | WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag | +| ported | **99.3 %** | 17270 of 263375 (6.56 %) | 63 of 1023 (6.16 %) | 25 of 147 | WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag | | regtested | **48.6 %** | 2689 of 263375 (1.02 %) | 12 of 1023 (1.17 %) | 6 of 147 | WW3's regtests ran with it on Kokkos and agreed with the Fortran | The port itself: 59232 lines in 350 files, the C++ under kokkos/src (every .cpp and .hpp line). diff --git a/kokkos/PORT_TABLE.tsv b/kokkos/PORT_TABLE.tsv index 26798ec0..3953338c 100644 --- a/kokkos/PORT_TABLE.tsv +++ b/kokkos/PORT_TABLE.tsv @@ -168,7 +168,7 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3iopomd:w3iopo model/src/w3iopomd.F90:2131-2626 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/iopo_w3iopo.hpp,kokkos/src/ww_kokkos/iopo_state.hpp no: header-only host driver; no shim and no W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3IOPO_KOKKOS.md — w3iopomd:w3iopp model/src/w3iopomd.F90:242-859 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/iopo_w3iopp.hpp,kokkos/src/ww_kokkos/iopo_state.hpp no: header-only host driver; no shim and no W3INIT hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3IOPP_KOKKOS.md — w3iorsmd:w3iors model/src/w3iorsmd.F90:113-1600 0.00 full translated — yes: kokkos/src/ww_kokkos/iors_w3iors.hpp,kokkos/src/ww_kokkos/iors_stream.hpp no: header-only host driver; no shim and no W3INIT or W3WAVE hunk yet pending no — — none: ADR-0010 kokkos/port_report/W3IORS_KOKKOS.md -— w3iosfmd:w3cprt model/src/w3iosfmd.F90:109-344 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/iosf_w3cprt.hpp, kokkos/src/fortran_iface/cprt_shim.cpp no: wired on the fork branch ww3-gpu/u9-cprt (W3KOKKOS_CPRT under WW_KOKKOS_CPRT=1), not at the WW3/ pin until the integrator merges it pending no — — none: ADR-0010 kokkos/port_report/W3CPRT_KOKKOS.md +— w3iosfmd:w3cprt model/src/w3iosfmd.F90:109-344 0.00 pdlib wired — yes: kokkos/src/ww_kokkos/iosf_w3cprt.hpp, kokkos/src/fortran_iface/cprt_shim.cpp yes: WW3/ pin a304018c, w3wavemd.F90:2692 (W3KOKKOS_CPRT under WW_KOKKOS_CPRT=1 and in the resident preset) pending no — — none: ADR-0010 kokkos/port_report/W3CPRT_KOKKOS.md — w3parall:get_jsea_ibelong model/src/w3parall.F90:1265-1384 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/parall_get_jsea_ibelong.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/GET_JSEA_IBELONG_KOKKOS.md — w3parall:init_get_jsea_isproc model/src/w3parall.F90:1152-1252 0.00 pdlib translated — yes: kokkos/src/ww_kokkos/parall_init_get_jsea_isproc.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/INIT_GET_JSEA_ISPROC_KOKKOS.md — w3parall:set_up_nseal_nsealm model/src/w3parall.F90:1030-1139 0.00 full translated — yes: kokkos/src/ww_kokkos/parall_set_up_nseal_nsealm.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/SET_UP_NSEAL_NSEALM_KOKKOS.md @@ -408,5 +408,5 @@ priority routine ww3_file self_pct profile state claim translated wired checkpoi — w3wavset:trig_compute_lh_stress model/src/w3wavset.F90:448-584 — — translated — yes: kokkos/src/ww_kokkos/wavset_trig_compute_lh_stress.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/TRIG_COMPUTE_LH_STRESS_KOKKOS.md — w3wavset:trig_wave_setup_apply_fct model/src/w3wavset.F90:1020-1107 — — translated — yes: kokkos/src/ww_kokkos/wavset_trig_wave_setup_apply_fct.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/TRIG_WAVE_SETUP_APPLY_FCT_KOKKOS.md — w3wavset:trig_wave_setup_apply_precond model/src/w3wavset.F90:892-1005 — — translated — yes: kokkos/src/ww_kokkos/wavset_trig_wave_setup_apply_precond.hpp no: header-only; no shim and no caller hunk yet pending no — — none: ADR-0010 kokkos/port_report/TRIG_WAVE_SETUP_APPLY_PRECOND_KOKKOS.md -— wminiomd:wmioeg model/src/wminiomd.F90:3258-3262 — — regtested — yes: kokkos/src/fortran_iface/resident_shim.cpp no: called on the fork branch ww3-gpu/u1-multi-va (wminiomd.F90:3341-3363, :3457-3463 at f330dc53); the WW3/ pin moves with the integrator passed yes not recorded — none: ADR-0010 kokkos/port_report/WMIOEG_KOKKOS.md -— wminiomd:wmiohg model/src/wminiomd.F90:2260-2266 — — regtested — yes: kokkos/src/fortran_iface/resident_shim.cpp no: called on the fork branch ww3-gpu/u1-multi-va (wminiomd.F90:2318-2327 at f330dc53); the WW3/ pin moves with the integrator passed yes not recorded — none: ADR-0010 kokkos/port_report/WMIOHG_KOKKOS.md +— wminiomd:wmioeg model/src/wminiomd.F90:3258-3262 — — regtested — yes: kokkos/src/fortran_iface/resident_shim.cpp yes: WW3/ pin a304018c, wminiomd.F90:3341-3363 and :3457-3463 (column read before 2.a, column write after 2.b, under the resident VA) passed yes not recorded — none: ADR-0010 kokkos/port_report/WMIOEG_KOKKOS.md +— wminiomd:wmiohg model/src/wminiomd.F90:2260-2266 — — regtested — yes: kokkos/src/fortran_iface/resident_shim.cpp yes: WW3/ pin a304018c, wminiomd.F90:2318-2327 (W3KOKKOS_VA_WRITE_COLS after the store loop, under the resident VA) passed yes not recorded — none: ADR-0010 kokkos/port_report/WMIOHG_KOKKOS.md diff --git a/kokkos/ports.tsv b/kokkos/ports.tsv index 34706985..048c4d2c 100644 --- a/kokkos/ports.tsv +++ b/kokkos/ports.tsv @@ -81,7 +81,7 @@ w3partmd:ptsort model/src/w3partmd.F90:511-613 103 ported translated 2026-10-16; w3partmd:ptmean model/src/w3partmd.F90:1175-1538 364 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4, every sum in the Fortran's order, the nine MAX/MIN sites in the object's operand order, SIGN folds, constants and the SIGP**(-5) power tree; a fresh review found no defect and corrected comments; WW3's out-of-bounds read when IFPMAX stays 0 (a NaN spectrum only) returns kPtmeanNoPeak so the shim calls the Fortran there (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_ptmean.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it w3snl3md:insnl3 model/src/w3snl3md.F90:788-1649 862 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_glwu (NOGRB) and switch_UoM_nl3; the four LOG(AUXFR) of :1046 and :1273 follow the object's libmvec/scalar split; NKD is 1300 under ST4 by use association and 275 without, kept as a flag; a fresh review found no defect and corrected a call count in the report (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3IOGR hunk kokkos/src/ww_kokkos/snl3_tables.hpp no: header-only; no shim and no W3IOGR hunk yet w3partmd:w3part model/src/w3partmd.F90:138-497 360 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4 (UPAR maxss order, MINVAL/MAXVAL/MAXLOC NaN seeds, NINT as lroundf with the IHMAX clamp, the wind-sea sort and combination in the Fortran's order); calls ptsort, pt_fld, ptmean and the PtnghbTable, none re-translated; a fresh review found no defect (NINT range-guarded and equal to lroundf for every float, the CUDA FFMA are libdevice and IEEE division only) and fixed the shim contract on kPtmeanNoPeak (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_w3part.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it -w3iosfmd:w3cprt model/src/w3iosfmd.F90:109-344 236 ported translated and wired 2026-10-18 on the resident VA (kokkos/src/fortran_iface/PATCH.md section 27); static parity by -O3 GIMPLE of w3iosfmd.F90 and of the shim under switch_lab_shrd: UABS, UDIR with the same folded RADE, FACT = (SIG*TPI)/CG, E2 = VA*FACT, no FMA, W3PART called unchanged, ICPRT and DTPRT integer placement (port report) no: regtest matrix pending; the CPU replay ran it on 2026-10-18: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT on two partition variants of bench_small (switch_lab_shrd, and switch_lab_ompg for the fields variant), WW_KOKKOS_VA_POISON=1: every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 (docs/log/2026-10-18-cprt-resident.md) kokkos/src/ww_kokkos/iosf_w3cprt.hpp, kokkos/src/fortran_iface/cprt_shim.cpp no: wired on the fork branch ww3-gpu/u9-cprt (W3KOKKOS_CPRT under WW_KOKKOS_CPRT=1), not at the WW3/ pin until the integrator merges it +w3iosfmd:w3cprt model/src/w3iosfmd.F90:109-344 236 ported translated and wired 2026-10-18 on the resident VA (kokkos/src/fortran_iface/PATCH.md section 27); static parity by -O3 GIMPLE of w3iosfmd.F90 and of the shim under switch_lab_shrd: UABS, UDIR with the same folded RADE, FACT = (SIG*TPI)/CG, E2 = VA*FACT, no FMA, W3PART called unchanged, ICPRT and DTPRT integer placement (port report) no: regtest matrix pending; the CPU replay ran it on 2026-10-18: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT on two partition variants of bench_small (switch_lab_shrd, and switch_lab_ompg for the fields variant), WW_KOKKOS_VA_POISON=1: every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 (docs/log/2026-10-18-cprt-resident.md) kokkos/src/ww_kokkos/iosf_w3cprt.hpp, kokkos/src/fortran_iface/cprt_shim.cpp yes: WW3/ pin a304018c, w3wavemd.F90:2692 (W3KOKKOS_CPRT under WW_KOKKOS_CPRT=1 and in the resident preset) w3snl3md:w3snl3 model/src/w3snl3md.F90:181-777 597 ported translated 2026-10-16 (non-PDLIB WAVNU1 variant, with EXPAND and EXPND2; no NL3 switch has PDLIB); static parity by -O3 GIMPLE and object comparison under switch_NCEP_glwu (NCEP2 -> NOGRB, SCRIP/SCRIPNC dropped) and switch_UoM_nl3, including the libmvec powf grouping of the SCALE loop; a fresh review found no defect and corrected two statements in the report (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3SRCE hunk kokkos/src/ww_kokkos/snl3_gmd.hpp no: header-only; no shim and no W3SRCE hunk yet w3src0md:w3spr0 model/src/w3src0md.F90:81-231 151 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd with ST0 FLX2; a fresh review found no arithmetic defect and corrected where the report placed the reversed NaN-payload operands; compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests kokkos/src/ww_kokkos/srce_st0_spr0.hpp yes: WW_KOKKOS_SPR0, w3srcemd.F90:1086, :2201 at 357f575c w3src1md:w3spr1 model/src/w3src1md.F90:87-240 154 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd with ST1 FLX1; a fresh review found no defect; compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests kokkos/src/ww_kokkos/srce_st1_spr1.hpp yes: WW_KOKKOS_SPR1, w3srcemd.F90:1098, :2212 at 357f575c @@ -334,5 +334,5 @@ w3pro1md:w3ktp1 model/src/w3pro1md.F90:900-1230 331 ported translated 2026-10-16 w3pro2md:w3xyp2 model/src/w3pro2md.F90:508-1264 757 ported translated 2026-10-16 under switch_lab_shrd with PR2 for PR3; static parity by -O3 GIMPLE and assembly comparison (literals, MIN/MAX and the CGN tree, NINT as lroundf, W3QCK3 calls); a fresh reviewer found no code defect and fixed the report's pin; compiled on serial-debug, never run no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist kokkos/src/ww_kokkos/prop_xyp2.hpp,kokkos/src/ww_kokkos/prop_params2.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_XYP2=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) w3pro2md:w3ktp2 model/src/w3pro2md.F90:1306-1710 405 ported translated 2026-10-16 under switch_lab_shrd with PR2 for PR3; static parity by -O3 GIMPLE and assembly comparison (FDDMAX four-lane grouping, MIN/MAX operand order, W3QCK1/W3QCK2 arguments); a fresh reviewer found no arithmetic defect and fixed two citations; NaN-payload order (port report); compiled on serial-debug, never run no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once a shim and the W3WAVE wiring exist kokkos/src/ww_kokkos/prop_ktp2.hpp no: header-only device function; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_KTP2=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) w3profsmd:w3xypug model/src/w3profsmd.F90:69-272 204 ported translated 2026-10-16 under the switch of ww3_tp2.6 (SHRD PR3, no PDLIB); static parity by -O3 GIMPLE and object comparison (final MAX as maxss with a +0 source, no FMA); the schemes W3XYPFSN2, W3XYPFSPSI2, W3XYPFSFCT2, W3XYPFSNIMP and DSEC21 are host hooks, SETDEPTH called ported; a fresh reviewer found no arithmetic defect and fixed one citation; compiled on serial-debug, never run no: translated only, as the owner asked on 2026-10-16 (no test run in this round); the propagation checkpoint (ADR-0011) runs it once the four scheme hooks, a shim and the W3WAVE wiring exist kokkos/src/ww_kokkos/prop_xypug.hpp no: header-only host function; the four schemes are hooks; no shim and no W3WAVE hunk yet; opt-in only once wired (WW_KOKKOS_XYPUG=1, ignored by WW_KOKKOS_ALL, part_flags.cpp) -wminiomd:wmiohg model/src/wminiomd.F90:2260-2266 7 ported the store loop 2.e repeated on the resident VA as a column scatter (ww_va_host_wrote_cols, kernel ww_va.scatter_cols), 2026-10-18; no arithmetic on the device, the Fortran statement runs unchanged on the host (PATCH.md section 26, port report) yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), WW3GPU_LAB_SWITCH=ON matrix entries mww3_test_01 to _04 (resident) and mww3_test_05 (SRCE,OUTG,IOPE,UPDV), test.comp 0 files different, plain and with WW_KOKKOS_VA_POISON=1 (docs/log/2026-10-18-ww3-multi-resident-va.md) kokkos/src/fortran_iface/resident_shim.cpp no: called on the fork branch ww3-gpu/u1-multi-va (wminiomd.F90:2318-2327 at f330dc53); the WW3/ pin moves with the integrator -wminiomd:wmioeg model/src/wminiomd.F90:3258-3262 5 ported the stores of 2.a (and of 2.b, :3278, :3326, :3330) repeated on the resident VA: column gather before, column scatter after (ww_va_host_read_cols, ww_va_host_wrote_cols), 2026-10-18; no arithmetic on the device (PATCH.md section 26, port report) yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), the entries of WMIOHG, test.comp 0 files different; whether they reach 2.b was not checked kokkos/src/fortran_iface/resident_shim.cpp no: called on the fork branch ww3-gpu/u1-multi-va (wminiomd.F90:3341-3363, :3457-3463 at f330dc53); the WW3/ pin moves with the integrator +wminiomd:wmiohg model/src/wminiomd.F90:2260-2266 7 ported the store loop 2.e repeated on the resident VA as a column scatter (ww_va_host_wrote_cols, kernel ww_va.scatter_cols), 2026-10-18; no arithmetic on the device, the Fortran statement runs unchanged on the host (PATCH.md section 26, port report) yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), WW3GPU_LAB_SWITCH=ON matrix entries mww3_test_01 to _04 (resident) and mww3_test_05 (SRCE,OUTG,IOPE,UPDV), test.comp 0 files different, plain and with WW_KOKKOS_VA_POISON=1 (docs/log/2026-10-18-ww3-multi-resident-va.md) kokkos/src/fortran_iface/resident_shim.cpp yes: WW3/ pin a304018c, wminiomd.F90:2318-2327 (W3KOKKOS_VA_WRITE_COLS after the store loop, under the resident VA) +wminiomd:wmioeg model/src/wminiomd.F90:3258-3262 5 ported the stores of 2.a (and of 2.b, :3278, :3326, :3330) repeated on the resident VA: column gather before, column scatter after (ww_va_host_read_cols, ww_va_host_wrote_cols), 2026-10-18; no arithmetic on the device (PATCH.md section 26, port report) yes: on the fork branch ww3-gpu/u1-multi-va (f330dc53), the entries of WMIOHG, test.comp 0 files different; whether they reach 2.b was not checked kokkos/src/fortran_iface/resident_shim.cpp yes: WW3/ pin a304018c, wminiomd.F90:3341-3363 and :3457-3463 (column read before 2.a, column write after 2.b, under the resident VA) diff --git a/site/data/port_score.tsv b/site/data/port_score.tsv index 759ca9e1..1767c729 100644 --- a/site/data/port_score.tsv +++ b/site/data/port_score.tsv @@ -2,9 +2,9 @@ # WW3@03adda91. The website reads this file (site/content). # name share count what of the run time, translated 99.3 % 313 of 1 023 routines a Kokkos translation exists in kokkos/src; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) -of the run time, ported 99.3 % 60 of 1 023 routines WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) +of the run time, ported 99.3 % 63 of 1 023 routines WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) of the run time, regtested 48.6 % 12 of 1 023 routines WW3's regtests ran with it on Kokkos and agreed with the Fortran; the profile's self time, full (bench_small, 120 x 80, 9 520 sea points, 12 h) of the lines of code, translated 25.80 % 67 947 of 263 375 Fortran lines of the translated routines, of every line in WW3's model/src, in 65 of 147 files -of the lines of code, ported 6.46 % 17 022 of 263 375 Fortran lines of the ported routines, of every line in WW3's model/src, in 23 of 147 files +of the lines of code, ported 6.56 % 17 270 of 263 375 Fortran lines of the ported routines, of every line in WW3's model/src, in 25 of 147 files of the lines of code, regtested 1.02 % 2 689 of 263 375 Fortran lines of the regtested routines, of every line in WW3's model/src, in 6 of 147 files lines of Kokkos C++ 59 232 in 350 files under kokkos/src: the port itself From f304eb5db1d57a0cd4e90e15606ef7d4ba9b0df5 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 14:46:40 -0300 Subject: [PATCH 24/27] WW3: pin 50a10730, the ESMF cap reads the resident VA before its exports The fork commit 50a10730 (ww3-gpu/integ-fixes) adds one host read of VA in WMESMFMD's SetExport: with no read at W3WAVE's return, the ESMF cap's export fields would have read a stale host VA. wmesmfmd.F90 joins the FILE lines of parts.d/resident.txt, so --ww3's fork check accepts it, and PATCH.md section 29 records it. The file is compiled only with MULTI_ESMF, which no regtest here builds, so the evidence at a304018c holds. Tested: gfortran -E -P of wmesmfmd.F90 with switch_lab_shrd, without KOKKOS identical to WW3-noaa's apart from __LINE__ in ESMF messages and the include-path comment; bash scripts/ww3-gpu.sh --self-test ok; nix shell nixpkgs#vale -c scripts/vale.sh PATCH.md, rc 0; not compiled (no ESMF library) Co-Authored-By: Claude Opus 5.5 --- WW3 | 2 +- kokkos/src/fortran_iface/PATCH.md | 23 +++++++++++++++++++ kokkos/src/fortran_iface/parts.d/resident.txt | 1 + 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/WW3 b/WW3 index a304018c..50a10730 160000 --- a/WW3 +++ b/WW3 @@ -1 +1 @@ -Subproject commit a304018c3453133293b9c708353e7bf6424e271c +Subproject commit 50a107307aad05207f751f399f5f6481becfc2eb diff --git a/kokkos/src/fortran_iface/PATCH.md b/kokkos/src/fortran_iface/PATCH.md index 6a5534dc..7d095976 100644 --- a/kokkos/src/fortran_iface/PATCH.md +++ b/kokkos/src/fortran_iface/PATCH.md @@ -2226,3 +2226,26 @@ WW_KOKKOS_VA_COUNT=1`, at `711afb80` (2026-10-18, same machine, one run each): e byte-identical to the Fortran of the same executable and every file but the NetCDF to `--ww3`'s, `nccmp-tol` PASS; whole-`VA` D2H 37 on smoke and 0 on full, as in section 26; `CPRT=K` with 0 calls, since neither case partitions (v). + +## 29. The ESMF cap reads the resident `VA` before its exports, `ww3-gpu/integ-fixes` + +Fork commit `50a10730` on `a304018c`, 2026-10-18. Section 26 removed the read at `W3WAVE`'s return +and synced every host use of `VA` in `ww3_shel` and `ww3_multi`. `wmesmfmd.F90`, the ESMF cap of +`ww3_multi` (built only with `MULTI_ESMF`, `model/src/CMakeLists.txt:193-199`), reads the export +grid's `VA` on the host after `WMWAVE` in `SetExport` (`wmesmfmd.F90:2824-3046` at `a304018c`): +`W3OUTG` under `USE_W3OUTG_FOR_EXPORT`, `CalcCharnk` and `CalcRoughl` (`W3SPR3`/`W3SPR4` on +`va(:,jsea)`), `CalcStokes3D`, `CalcPStokes`, `CalcBotcur`, `CalcRadstr2D`. Every one reads the +whole field, so one `W3KOKKOS_VA_HOST_READ` after the grid's `w3set*` calls covers them; the +"Resident spectrum" rule allows a whole-field read before an output that needs every point. Its +`w3uini ( va )` at `:2180` runs in `InitializeP3` before the resident `VA` is bound (the C++ side +binds it on the first sync, `w3kokkos_residentmd.F90:68-70`), so it needs no sync. + +- Without `KOKKOS`, `gfortran -E -P` with `switch_lab_shrd` gives `WW3-noaa/`'s file apart from the + `__LINE__` values `ESMF_LOGERR_PASSTHRU` writes into error messages and the include-path comment + (v); with `KOKKOS`, only the `USE` and the read are added (v). +- Not compiled and not run: there is no ESMF library here ⚠. No regtest of this repository builds + `MULTI_ESMF`, so the matrix evidence of section 26 and of U3 holds at this commit. +- Not covered: the NUOPC cap of the UFS (`wav_comp_nuopc.F90`, `wav_import_export.F90`, + `wav_restart_mod.F90`), which is not in the fork at `a304018c`; `production/GFS.v17` has it, with + 16 and 13 uses of `VA` in the last two (v, `git show`). The pin move to that branch + ([plan](../../../docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md)) lists and syncs them. diff --git a/kokkos/src/fortran_iface/parts.d/resident.txt b/kokkos/src/fortran_iface/parts.d/resident.txt index c7cb1011..630abb49 100644 --- a/kokkos/src/fortran_iface/parts.d/resident.txt +++ b/kokkos/src/fortran_iface/parts.d/resident.txt @@ -3,3 +3,4 @@ COUNT WW_KOKKOS_VA_COUNT FILE model/src/wminiomd.F90 FILE model/src/wmwavemd.F90 +FILE model/src/wmesmfmd.F90 From 45fd2fbb86dbcd37831e630fd75b8b86725074db Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 14:47:50 -0300 Subject: [PATCH 25/27] kokkos: port score after the rebase onto main with #193 Tested: python3 scripts/port_score.py --write, then --check, current Co-Authored-By: Claude Opus 5.5 --- kokkos/PORT_STATUS.md | 2 +- site/data/port_score.tsv | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/kokkos/PORT_STATUS.md b/kokkos/PORT_STATUS.md index cf98daf6..d44e49c7 100644 --- a/kokkos/PORT_STATUS.md +++ b/kokkos/PORT_STATUS.md @@ -49,7 +49,7 @@ Profile: full (bench_small, 120 x 80, 9 520 sea points, 12 h), Release -pg, swit | ported | **99.3 %** | 17270 of 263375 (6.56 %) | 63 of 1023 (6.16 %) | 25 of 147 | WW3 at the WW3/ pin calls the Kokkos code, behind its run-time flag | | regtested | **48.6 %** | 2689 of 263375 (1.02 %) | 12 of 1023 (1.17 %) | 6 of 147 | WW3's regtests ran with it on Kokkos and agreed with the Fortran | -The port itself: 59232 lines in 350 files, the C++ under kokkos/src (every .cpp and .hpp line). +The port itself: 59511 lines in 350 files, the C++ under kokkos/src (every .cpp and .hpp line). On a GPU: none yet. The CUDA build links; the RTX 4090 run, `just smoke-ww3-gpu gpu`, is the owner's, supervised. diff --git a/site/data/port_score.tsv b/site/data/port_score.tsv index 1767c729..6bf4ec66 100644 --- a/site/data/port_score.tsv +++ b/site/data/port_score.tsv @@ -7,4 +7,4 @@ of the run time, regtested 48.6 % 12 of 1 023 routines WW3's regtests ran with i of the lines of code, translated 25.80 % 67 947 of 263 375 Fortran lines of the translated routines, of every line in WW3's model/src, in 65 of 147 files of the lines of code, ported 6.56 % 17 270 of 263 375 Fortran lines of the ported routines, of every line in WW3's model/src, in 25 of 147 files of the lines of code, regtested 1.02 % 2 689 of 263 375 Fortran lines of the regtested routines, of every line in WW3's model/src, in 6 of 147 files -lines of Kokkos C++ 59 232 in 350 files under kokkos/src: the port itself +lines of Kokkos C++ 59 511 in 350 files under kokkos/src: the port itself From daaef07d357f95e00ad7d0a9134f9338ea421818 Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 14:49:10 -0300 Subject: [PATCH 26/27] docs/compound: plans 003 and 004 after #192, #193 and #194 merged Plan 003: #193 is merged; the CUDA Docker image of #192 builds with WW_KOKKOS_LIBM=OFF, so it calls CUDA's libm and stays out of the first GPU run; #194's KOKKOS_LIBM_ON_VS_OFF.md holds the owner's decision on that option, and A3 records the value the run's build has. Plan 004: the UFS NUOPC cap on production/GFS.v17 reads VA after W3WAVE, so its host uses are listed and synced, and one coupled case runs with the VA poison, as section 29 did for the ESMF cap. Tested: nix shell nixpkgs#vale -c scripts/vale.sh on both plans, rc 0 Co-Authored-By: Claude Opus 5.5 --- .../2026-10-18-003-gpu-first-supervised-run-plan.md | 10 +++++++--- .../compound/plans/2026-10-18-004-pin-gfs-v17-plan.md | 11 ++++++++++- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md index 60f6c2d6..39474216 100644 --- a/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md +++ b/docs/compound/plans/2026-10-18-003-gpu-first-supervised-run-plan.md @@ -103,7 +103,7 @@ Not ranked: skipping `W3SDS4`'s zero-weight terms (12 of 23 products, profile lo result for an infinite or NaN input, so it is the owner's decision and its own PR. **Work in progress elsewhere (2026-10-18).** PR #193 (branch -`goal/eb10363f-4821-noaa-progress-record`, another session) builds the `W3SDS4` saturation-integral loop +`goal/eb10363f-4821-noaa-progress-record`, another session, merged 2026-10-18) builds the `W3SDS4` saturation-integral loop interchange (`kokkos/src/ww_kokkos/st4_sds4.hpp`), a DIA direction-halo layout that turns the index-table gathers into contiguous loads (`snl1_team.hpp`, `snl1_tables.{hpp,cpp}`), and `-fno-math-errno` for the host build (`kokkos/src/ww_kokkos/CMakeLists.txt`). The halo layout @@ -202,8 +202,12 @@ of three; one run on a many-core EPYC node if the owner has one. `build/ww3` also rebuilds inside `#cuda` and again on the next CPU run (`:480-487`). - `run_dir`s are deleted at every start (`build/runs/ww3-gpu--cuda/`); `WW3GPU_RUNS` moves them. -- Docker's CUDA image is another toolchain (CUDA 13.4, no OpenMP, counts check skipped): not - used for this run. +- Docker's CUDA image is another toolchain (CUDA 13.4, no OpenMP, counts check skipped) and, since + PR #192, builds with `WW_KOKKOS_LIBM=OFF` (`docker/Dockerfile.cuda`), so its kernels call CUDA's + libm, not the device copies of glibc's functions: not used for this run. Which value each kind of + build should use is the owner's decision in + [`KOKKOS_LIBM_ON_VS_OFF.md`](../../KOKKOS_LIBM_ON_VS_OFF.md) (PR #194); A3 records the value the + run's build has (`WW_KOKKOS_LIBM` in `CMakeCache.txt`). ### Phase A: preparation by agents (next session, no device run) diff --git a/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md b/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md index 99602044..0d4ed186 100644 --- a/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md +++ b/docs/compound/plans/2026-10-18-004-pin-gfs-v17-plan.md @@ -65,7 +65,16 @@ resident wall time, did not change (v, absent from the name list). `wminiomd.F90 `WW_KOKKOS_VA_POISON=1` and `WW_KOKKOS_VA_COUNT=1`. Then `port_score.py --write`, and every page that cites `model/src/.F90:` at `80644ba9` or `03adda91` updated to the new pin (AGENTS.md says the pin's commit at the time of writing). -5. **Docs:** AGENTS.md, CONTRIBUTING.md and `docs/TOOLCHAIN.md` name the new pin and why it is +5. **The coupling caps.** `production/GFS.v17` carries the UFS NUOPC cap, which the fork at + `50a10730` does not: `wav_comp_nuopc.F90` calls `W3WAVE`, and `wav_import_export.F90` and + `wav_restart_mod.F90` use `VA` 16 and 13 times (v, `git show` at `7a23cd77`). Since PR #191, + `W3WAVE` returns with the resident `VA` possibly newer on the device, so every host use of `VA` + in those files is listed with its file:line and synced where it is (columns where the cap reads + columns, a whole-field read only before an export or a restart that needs every point), as + PATCH.md section 29 did for the ESMF cap. Then one coupled case with + `WW_KOKKOS_VA_POISON=1`, which fills a stale host `VA` with NaN, so a missed sync shows in the + exports; which UFS case runs on this machine is not known ⚠. +6. **Docs:** AGENTS.md, CONTRIBUTING.md and `docs/TOOLCHAIN.md` name the new pin and why it is a production branch; `README.md` and `README.pt-BR.md` together. ## Risks From 0cfd336ea37b0cab1c546a74f0a8e9632938a08a Mon Sep 17 00:00:00 2001 From: "M.Hoffmann" Date: Sun, 18 Oct 2026 15:42:39 -0300 Subject: [PATCH 27/27] U3 regtests on the integrated pin: resident serial, OMPG, ww3_multi, ww3_tnc1 Goal abbb0456-6043, unit U3 of docs/compound/plans/2026-10-18-002-production-single-node-plan.md, PR #191. On daaef07 with WW3 at 50a10730 and WW3GPU_LAB_SWITCH=ON, WW3GPU_STRICT=ON, every Kokkos run that ran gave test.comp 0 files different, with WW_KOKKOS_VA_POISON=1: the resident serial selection (the owner's run and the agent's), the ten ST4 OMPG entries with SRCE,OUTG,IOPE,UPDV,CPRT, agent A's ww3_multi entries and ww3_tnc1 with CPRT. ww3_tp2.6 (both) and ww3_tnc1 under resident stop in W3INIT by design (XYP3 refuses an unstructured grid or a switch without PR3 UQ) and stay FAIL rows, as the owner decided; mww3_test_07 fails in the Fortran's ww3_grid too. The summaries go to docs/data/, the record to docs/log/, the CPRT row of kokkos/ports.tsv names the matrix record and stays "no" (the owner decides on lab-switch evidence), and gaps 2 and 10 are updated. No matrix.base restart pair is SHRD, so scripts/restart_pair.sh is the smallest equivalent on the full case. It is new, was not run (the owner stopped new long runs), changes no gate, and needs a person's review before it merges. Tested: WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 4 serial (with WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1): 5 pass, ww3_tp2.6 x2 FAIL by design Tested: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV,CPRT WW3GPU_SWITCH_WORDS='ST4,SHRD,!PDLIB' WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 4 omp: 10/10 entries 0 files different; selection:count fails (no expected count for this word list) Tested: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,!PDLIB,!MGP,!MGG' ... just ww3-gpu-matrix 2 serial -R '^0(33|34|38|43|61):' and -R '^061:': 4 pass, 61 fails in the Fortran ww3_grid Tested: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,MGP,!PDLIB' ... just ww3-gpu-matrix 2 omp -R '^00(2|8):': 2 pass Tested: WW3GPU_SWITCH_WORDS='PR0;SHRD' just ww3-gpu-matrix 1 serial -R '^038:' with SRCE,OUTG,IOPE,UPDV,CPRT (pass, 0/30) and resident (stops in W3INIT naming XYP3) Tested: bash -n, shellcheck and --self-test of scripts/restart_pair.sh; python3 scripts/port_score.py --write and --check; scripts/vale.sh on the changed Markdown (0 errors) Co-Authored-By: Claude Opus 5.5 --- docs/PRODUCTION_READINESS_202610.md | 21 ++- ...u_matrix_multi_pin-50a10730_2026-10-18.tsv | 44 +++++ ...gpu_matrix_omp_pin-50a10730_2026-10-18.tsv | 24 +++ ..._matrix_serial_pin-50a10730_2026-10-18.tsv | 42 +++++ ...pu_matrix_tnc1_pin-50a10730_2026-10-18.tsv | 27 +++ docs/log/2026-10-18-u3-regtests-v2.md | 163 ++++++++++++++++++ docs/log/README.md | 1 + kokkos/PORT_RANKING.md | 2 +- kokkos/ports.tsv | 2 +- scripts/restart_pair.sh | 133 ++++++++++++++ 10 files changed, 455 insertions(+), 4 deletions(-) create mode 100644 docs/data/ww3gpu_matrix_multi_pin-50a10730_2026-10-18.tsv create mode 100644 docs/data/ww3gpu_matrix_omp_pin-50a10730_2026-10-18.tsv create mode 100644 docs/data/ww3gpu_matrix_serial_pin-50a10730_2026-10-18.tsv create mode 100644 docs/data/ww3gpu_matrix_tnc1_pin-50a10730_2026-10-18.tsv create mode 100644 docs/log/2026-10-18-u3-regtests-v2.md create mode 100755 scripts/restart_pair.sh diff --git a/docs/PRODUCTION_READINESS_202610.md b/docs/PRODUCTION_READINESS_202610.md index 66e6dc88..52b3d126 100644 --- a/docs/PRODUCTION_READINESS_202610.md +++ b/docs/PRODUCTION_READINESS_202610.md @@ -38,7 +38,7 @@ regtest matrix yet"). 10 of the 310 rows of `kokkos/ports.tsv` are regtested and | # | Gap | (a) | (b) | Smallest next step | |---|---|---|---|---| | 1 | No regtest switch has the source-term set the Kokkos `W3SRCE` translates; NOAA's UFS regtests miss it too | Blocker | Blocker | Read GFS-Wave's operational switch file at its tag; run the `WW3GPU_LAB_SWITCH` matrix on the MPI and `ww3_multi` entries | -| 2 | `ww3_multi` writes `VA` on the host between `W3WAVE` calls with no sync; fixed under `SHRD` on 2026-10-18 by column syncs at the exchanges (section 2) | Minor ⚠ | Minor ⚠ | Move the pin to the fork branch; `mww3_test_08` under MPI when the owner runs MPI | +| 2 | `ww3_multi` writes `VA` on the host between `W3WAVE` calls with no sync; fixed under `SHRD` on 2026-10-18 by column syncs at the exchanges, passed again at the integrated pin `50a10730` (section 2) | Minor ⚠ | Minor ⚠ | `mww3_test_08` under MPI when the owner runs MPI | | 3 | MPI: four of the six resident parts are `SHRD` only, the preset stops on `DIST MPI`, the communicator is ignored, the device id is not per rank | Blocker | Major | A resident preset without `XYP3`, `OUTG`, `IOPE`, `UPDV`; one run on `switch_lab_mpi` | | 4 | Bit parity measured with gfortran and g++ only; the operational compiler and the GPU are not measured | Blocker | Major | Static argument for the Intel build's math calls; one smoke with it | | 5 | Build and packaging: a separate static library, C++20 and CMake 3.25; the install-only configure and the exact Kokkos check are done (U6) | Major | Major | A shared-library build and a recipe for the module systems of an operational centre | @@ -46,7 +46,7 @@ regtest matrix yet"). 10 of the 310 rows of `kokkos/ports.tsv` are regtested and | 7 | Configuration is 61 environment variables, no namelist; the preset lives in a lab script | Major | Major | Smallest form done on `ww3-gpu/u2-strict-preset`: `WW_KOKKOS_PRESET=resident` from one table, `presets.txt`; next, a namelist group read by `W3INIT` | | 8 | Observability: log lines on every rank, counters at exit to `stderr`, no provenance in the output files | Major | Minor | Gate the lines by `NAPLOG`; one NetCDF global attribute | | 9 | OpenMP inside MPI: the per-point parts are serialised by `!$OMP CRITICAL`; the thread budget per rank is not set by the shim | Major | Minor | Measure the hybrid entries of `ww3_ufs1.1` before claiming a speed for them | -| 10 | Restart with the resident spectrum: covered by the sync calls, never regtested | Major | Minor | The restart-reproduction pair of `ww3_ufs1.1` (`work_c` against `work_d`) under the lab switch | +| 10 | Restart with the resident spectrum: covered by the sync calls, never regtested; every `matrix.base` restart pair is `MPI` | Major | Minor | Run `scripts/restart_pair.sh` (a `SHRD` restart pair on `full`, written 2026-10-18, not run) | | 11 | `PDLIB` and unstructured grids: translated, not wired, gated out | Minor ⚠ | Major | None for GFS-Wave v16; a wiring unit for `PDLIB` when the owner asks | | 12 | The fork patch: 4 082 lines of new Fortran, 819 changed lines in three WW3 files, 58 218 lines of C++ in another repository, most of it LGPL-3.0-or-later, some MIT, 76 files with no licence line | Minor | Blocker | ADR-0018 (Proposed) names the three places `ww_kokkos` could live; the owner chooses, after a legal reading ⚠ | | 13 | A stale contract remark: "one spectral grid per process"; edited on the fork branch `ww3-gpu/u1-multi-va` (section 13) | Minor | Minor | Move the pin | @@ -149,6 +149,15 @@ reading ⚠, not run (ADR-0017); the fork branch waits for the pin bump. - **(a), (b) now Minor** under `SHRD` ⚠: shown on the four `PR3 UQ` `grdset_a`-type entries and one moving grid, with the lab switch's source terms, not on GFS-Wave's grids. +**Update, 2026-10-18 (plan 002, U3).** On the integrated pin (`WW3/` `50a10730`, `CPRT` in the +preset), agent A's commands again gave `test.comp` 0 files different with the host copy poisoned +on `mww3_test_01` to `_04` (resident) and on both `mww3_test_05` moving-grid entries +(`SRCE,OUTG,IOPE,UPDV`), and the five `mww3_test_05` `ST4` OMPG entries passed with `CPRT` too (v, +[log](log/2026-10-18-u3-regtests-v2.md)). Whole-`VA` copies device to host: `_02` 8 → 0, `_03` +216 → 3, the others unchanged (v). `mww3_test_01` to `_04` have `FLSOU` false, so they test the +exchanges and the propagation, not `W3SRCE` (v). `mww3_test_07` fails in the Fortran run as well +(its `SDS4` namelist under the lab switch). `mww3_test_08` (MPI) is still not run. + ## 3. MPI *Scope, 2026-10-18:* out of scope by decision. The port is single node, and an MPI build keeps @@ -373,6 +382,14 @@ the UFS case (`work_c`, then `work_d` from its restart, compared by `test.comp`; - **(b) Minor.** - **Next step.** That pair under `WW3GPU_LAB_SWITCH=ON` with the per-point parts and `SRCE`. +**Update, 2026-10-18 (plan 002, U3).** `ww3_ufs1.1` is an `MPI` case, as is every other +`rstrt_b4b` pair of `matrix.base` (`ww3_tp2.17` with `PDLIB`; v, `matrix.base:2048-2067`, +`:2171-2205` at `50a10730`), and the resident preset stops under `MPI` (ADR-0017). No entry +restarts a `SHRD` case. [`scripts/restart_pair.sh`](../scripts/restart_pair.sh) is the smallest +equivalent on `full` with `switch_lab_shrd`: a 00:00 to 12:00 run and a 06:00 to 12:00 run from +its own 06:00 restart, for the Fortran and the resident Kokkos of one executable, compared with +`cmp`. It was not run ⚠ ([log](log/2026-10-18-u3-regtests-v2.md)); the gap stays Major for (a). + ## 11. `PDLIB` and unstructured grids 29 rows of `kokkos/ports.tsv` translate `PDLIB` routines (`pdlib_w3profsmd`, `w3parall`); none diff --git a/docs/data/ww3gpu_matrix_multi_pin-50a10730_2026-10-18.tsv b/docs/data/ww3gpu_matrix_multi_pin-50a10730_2026-10-18.tsv new file mode 100644 index 00000000..e4a367fa --- /dev/null +++ b/docs/data/ww3gpu_matrix_multi_pin-50a10730_2026-10-18.tsv @@ -0,0 +1,44 @@ +# Unit U3, item 3: agent A's ww3_multi commands of docs/log/2026-10-18-ww3-multi-resident-va.md on the integrated pin; record: docs/log/2026-10-18-u3-regtests-v2.md. +# Block 1: the SHRD serial entries, resident preset. Block 2: entry 61 (mww3_test_07) alone, to keep its ww3_grid error. Block 3: the moving-grid OMPG entries of mww3_test_05. +# Entry 61 fails in the Fortran run too: ww3_grid stops with "ERROR IN READING SDS4 FROM FILE, IOSTAT = 5010" (work_F_61/ww3_grid_refug.out), the entry's namelist against the lab switch's ST4. +# The VA diagnostics were set in the environment of the U3 agent's runs and the summary does not record them: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1. +# date: 2026-10-18T18:31:52Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 serial -R ^0(33|34|38|43|61): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +33 mww3_test_01 PR3_UQ yes yes yes 0/22 PASS hs 107 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ +34 mww3_test_02 PR3_UQ yes yes yes 0/23 PASS dp,fp,hs 127 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +38 mww3_test_03 PR3_UQ yes yes yes 0/34 PASS dp,fp,hs 126 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +43 mww3_test_04 PR3_UQ yes yes yes 0/25 PASS dir,dp,fp,hs 201 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s PR3_UQ -w work_PR3_UQ_a -m grdset_a +61 mww3_test_07 default yes no no n/a n/a - 90 0 - -w work_PR3_UQ -m grdset +# +# date: 2026-10-18T18:38:02Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,!PDLIB,!MGP,!MGG WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 1 serial -R ^061: +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +61 mww3_test_07 default yes no no n/a n/a - 60 0 - -w work_PR3_UQ -m grdset +# +# date: 2026-10-18T18:32:18Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV WW3GPU_SWITCH_WORDS=PR3,UQ,SHRD,MGP,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 2 omp -R ^00(2|8): +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words PR3 UQ SHRD MGP !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV = SRCE,OUTG,IOPE,UPDV; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +2 mww3_test_05 ST1_PR3_UQ_OMP yes yes yes 0/27 PASS hs 352 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST1_PR3_UQ_OMP -w work_ST1_PR3_UQ_OMP -m grdset_a -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes yes yes 0/27 PASS hs 351 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np diff --git a/docs/data/ww3gpu_matrix_omp_pin-50a10730_2026-10-18.tsv b/docs/data/ww3gpu_matrix_omp_pin-50a10730_2026-10-18.tsv new file mode 100644 index 00000000..60f2992e --- /dev/null +++ b/docs/data/ww3gpu_matrix_omp_pin-50a10730_2026-10-18.tsv @@ -0,0 +1,24 @@ +# Unit U3, item 2; record: docs/log/2026-10-18-u3-regtests-v2.md. The U3 agent's run, as written by record:summary. +# The OMPG entries of matrix.base with ST4 and SHRD, without PDLIB; XYP3 and KTP3 left out (they refuse a moving grid, w3kokkos_prop3md.F90:123-125 at 50a10730). +# selection:count failed: CMakeLists.txt has no expected count for this word list, so the check has nothing to compare; every entry ran and passed. +# The VA diagnostics were set in the environment of the U3 agent's runs and the summary does not record them: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1. +# date: 2026-10-18T18:24:51Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV,CPRT WW3GPU_SWITCH_WORDS=ST4,SHRD,!PDLIB WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 4 omp +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: omp, switch with the words ST4 SHRD !PDLIB and OMPG or OMPH, no MPI; $np = WW3GPU_OMP_THREADS; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV,CPRT = SRCE,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +1 ww3_ts3 ST4_PR1_OMP yes yes yes 0/20 PASS hs 217 SRCE:42025,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR1_OMP -w work_ST4_PR1_OMP -f -O -n $np +2 mww3_test_05 ST4_PR1_OMP yes yes yes 0/27 PASS hs 395 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR1_OMP -w work_ST4_PR1_OMP -m grdset_a -f -O -n $np +3 ww3_ts3 ST4_PR2_UQ_OMP yes yes yes 0/20 PASS hs 213 SRCE:42025,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR2_UQ_OMP -w work_ST4_PR2_UQ_OMP -f -O -n $np +4 mww3_test_05 ST4_PR2_UQ_OMP yes yes yes 0/27 PASS hs 413 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR2_UQ_OMP -w work_ST4_PR2_UQ_OMP -m grdset_a -f -O -n $np +5 ww3_ts3 ST4_PR2_UNO_OMP yes yes yes 0/20 PASS hs 215 SRCE:42025,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR2_UNO_OMP -w work_ST4_PR2_UNO_OMP -f -O -n $np +6 mww3_test_05 ST4_PR2_UNO_OMP yes yes yes 0/27 PASS hs 373 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR2_UNO_OMP -w work_ST4_PR2_UNO_OMP -m grdset_a -f -O -n $np +7 ww3_ts3 ST4_PR3_UQ_OMP yes yes yes 0/20 PASS hs 183 SRCE:42025,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -f -O -n $np +8 mww3_test_05 ST4_PR3_UQ_OMP yes yes yes 0/27 PASS hs 295 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ_OMP -w work_ST4_PR3_UQ_OMP -m grdset_a -f -O -n $np +9 ww3_ts3 ST4_PR3_UNO_OMP yes yes yes 0/20 PASS hs 185 SRCE:42025,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UNO_OMP -w work_ST4_PR3_UNO_OMP -f -O -n $np +10 mww3_test_05 ST4_PR3_UNO_OMP yes yes yes 0/27 PASS hs 215 SRCE:300876,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UNO_OMP -w work_ST4_PR3_UNO_OMP -m grdset_a -f -O -n $np diff --git a/docs/data/ww3gpu_matrix_serial_pin-50a10730_2026-10-18.tsv b/docs/data/ww3gpu_matrix_serial_pin-50a10730_2026-10-18.tsv new file mode 100644 index 00000000..c9fd897d --- /dev/null +++ b/docs/data/ww3gpu_matrix_serial_pin-50a10730_2026-10-18.tsv @@ -0,0 +1,42 @@ +# Unit U3 of docs/compound/plans/2026-10-18-002-production-single-node-plan.md; record: docs/log/2026-10-18-u3-regtests-v2.md. +# Block 1: the owner's run, in the worktree ~/code/ww3-gpu-pr191 at daaef07d, as written by record:summary. Block 2: the same command by the U3 agent in its worktree, with OMP_NUM_THREADS=4. +# Rows 1 and 2 (ww3_tp2.6 ST4, ST4_ASCII) are FAIL in both blocks: the Kokkos run stops in W3INIT with "WW_KOKKOS_XYP3=1, BUT ITS KOKKOS TRANSLATION DOES NOT COVER THIS SWITCH FILE" +# (logs/1.K.log:160 of the owner's build): LIMON is an unstructured grid, which W3KOKKOS_PROP3_SETUP refuses (WW3/model/src/w3kokkos_prop3md.F90:122,134 at 50a10730), as docs/log/2026-10-18-resident-regtest-lab-switch.md:51 records. +# The owner decided on 2026-10-18 to keep both as FAIL rows, with no change to the selection, the verdict or the parts; issue #196 is the port of the unstructured path. +# +# date: 2026-10-18T18:29:19Z +# ww3-gpu commit: daaef07d +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=8 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON just ww3-gpu-matrix 4 serial +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words ST4 NL1 PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +1 ww3_tp2.6 ST4 yes fortran-only no n/a n/a - 202 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s ST4 -w work_ST4 +2 ww3_tp2.6 ST4_ASCII yes fortran-only no n/a n/a - 197 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s ST4_ASCII -w work_ST4_ASCII +3 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 159 SRCE:6900,SDS4:0,SIN4:0,SPR4:0 - -w work_PR3_UQ +4 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 979 SRCE:137000,SDS4:0,SIN4:0,SPR4:0 - -w work_5km -g 5km +5 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 141 SRCE:6624,SDS4:0,SIN4:0,SPR4:0 - -i input_const -w work_PR3_UQ_CONST +6 ww3_tp2.15 default yes yes yes 0/25 PASS dir,dp,fp,hs 153 SRCE:6900,SDS4:0,SIN4:0,SPR4:0 - -i input_rho -w work_PR3_UQ_RHO +7 ww3_ts2 ST4_PR3_UQ yes yes yes 0/22 n/a - 117 SRCE:4632,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ -w work_ST4_PR3_UQ +# +# The VA diagnostics were set in the environment of the U3 agent's runs and the summary does not record them: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1. +# date: 2026-10-18T18:09:13Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 4 serial +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words ST4 NL1 PR3 UQ SHRD !PDLIB !MGP !MGG, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +1 ww3_tp2.6 ST4 yes fortran-only no n/a n/a - 127 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s ST4 -w work_ST4 +2 ww3_tp2.6 ST4_ASCII yes fortran-only no n/a n/a - 132 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -s ST4_ASCII -w work_ST4_ASCII +3 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 95 SRCE:6900,SDS4:0,SIN4:0,SPR4:0 - -w work_PR3_UQ +4 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 545 SRCE:137000,SDS4:0,SIN4:0,SPR4:0 - -w work_5km -g 5km +5 ww3_tp2.15 default yes yes yes 0/23 PASS dir,dp,fp,hs 87 SRCE:6624,SDS4:0,SIN4:0,SPR4:0 - -i input_const -w work_PR3_UQ_CONST +6 ww3_tp2.15 default yes yes yes 0/25 PASS dir,dp,fp,hs 82 SRCE:6900,SDS4:0,SIN4:0,SPR4:0 - -i input_rho -w work_PR3_UQ_RHO +7 ww3_ts2 ST4_PR3_UQ yes yes yes 0/22 n/a - 72 SRCE:4632,SDS4:0,SIN4:0,SPR4:0 - -s ST4_PR3_UQ -w work_ST4_PR3_UQ diff --git a/docs/data/ww3gpu_matrix_tnc1_pin-50a10730_2026-10-18.tsv b/docs/data/ww3gpu_matrix_tnc1_pin-50a10730_2026-10-18.tsv new file mode 100644 index 00000000..3abd0eab --- /dev/null +++ b/docs/data/ww3gpu_matrix_tnc1_pin-50a10730_2026-10-18.tsv @@ -0,0 +1,27 @@ +# Unit U3, item 4: ww3_tnc1 (PR0, partition fields); record: docs/log/2026-10-18-u3-regtests-v2.md. +# Block 1: SRCE,OUTG,IOPE,UPDV,CPRT. Block 2: resident, which stops in W3INIT by design ("WW_KOKKOS_XYP3=1, BUT ITS KOKKOS TRANSLATION DOES NOT COVER THIS SWITCH FILE", +# KOKKOS_PART at WW3/model/src/w3kokkosmd.F90:1244-1245, asked from the no-PR3 branch at w3kokkos_prop3md.F90:143 at 50a10730). +# The VA diagnostics were set in the environment of the U3 agent's runs and the summary does not record them: WW_KOKKOS_VA_POISON=1 WW_KOKKOS_VA_COUNT=1. +# date: 2026-10-18T18:34:09Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV,CPRT WW3GPU_SWITCH_WORDS=PR0;SHRD WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 1 serial -R ^038: +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR0 SHRD, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): SRCE,OUTG,IOPE,UPDV,CPRT = SRCE,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +38 ww3_tnc1 default yes yes yes 0/30 n/a - 57 SRCE:405,SDS4:0,SIN4:0,SPR4:0 - -N +# +# date: 2026-10-18T18:36:08Z +# ww3-gpu commit: daaef07 +# WW3 commit: 50a10730 (origin/ww3-gpu/integ-fixes) +# machine: Intel(R) Core(TM) i9-14900, 32 hardware threads (nproc --all), OMP_NUM_THREADS=4 per run, WW3GPU_OMP_NESTED=OFF +# command: WW3GPU_KOKKOS=resident WW3GPU_SWITCH_WORDS=PR0;SHRD WW3GPU_LAB_SWITCH=ON WW3GPU_OMP_THREADS=4 just ww3-gpu-matrix 1 serial -R ^038: +# data: /nix/store/97lk2yrkjdjxaniq48mq2p6ciw9y79js-ww3-regtest-data-7.14.2; SCOTCH_PATH: /nix/store/v6003b5s6nr5by0zhncxkg7h4n06gvyq-ptscotch-static-7.0.12; METIS_PATH: /nix/store/2sxmb2qk62pv3f73psif7cc1plwcldns-ww3-metis +# selection: serial, switch with the words PR0 SHRD, none of MPI, OMPG, OMPH; WW3GPU_STRICT=ON +# switch: each entry's source-term words replaced by switches/switch_lab_shrd's (WW3GPU_LAB_SWITCH=ON) +# Kokkos flags (WW3GPU_KOKKOS, as scripts/ww3-gpu.sh --kokkos): resident = SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT; the Fortran runs WW_KOKKOS_ALL=0, every part unset +n test switch built ran kokkos_log test_comp nccmp_tol judged wall_s kokkos_spectra ww3_self options +38 ww3_tnc1 default yes fortran-only no n/a n/a - 70 SRCE:0,SDS4:0,SIN4:0,SPR4:0 - -N diff --git a/docs/log/2026-10-18-u3-regtests-v2.md b/docs/log/2026-10-18-u3-regtests-v2.md new file mode 100644 index 00000000..27dce2c5 --- /dev/null +++ b/docs/log/2026-10-18-u3-regtests-v2.md @@ -0,0 +1,163 @@ +# U3 on the integrated pin: the resident serial, OMPG, `ww3_multi` and `ww3_tnc1` entries; the restart pair designed, not run + +2026-10-18, the U3 agent for M. Hoffmann (goal `abbb0456-6043`), unit U3 of +[the production single-node plan](../compound/plans/2026-10-18-002-production-single-node-plan.md), +PR #191; gaps 2 and 10 of [`PRODUCTION_READINESS_202610.md`](../PRODUCTION_READINESS_202610.md). +The owner ran item 1 himself. + +## Question + +On the branch rebased onto `main` (`daaef07`, `WW3/` at `50a10730`, with #193's DIA-halo and +`W3SDS4` kernel changes and `CPRT` in the resident preset), does every Kokkos run of WW3's +regtests that the preset can select give the Fortran's bytes, with the host copy of `VA` poisoned? +And is there a restart-reproduction pair for a `SHRD` case? + +## What was done + +The owner's Intel Core i9-14900 (32 hardware threads, shared with the integrator's jobs: load +average from 2.4 to 17.0 over the runs, read with `uptime` at each start and end), the pinned `ww3` +shell (gfortran and g++ 16.2.0), Kokkos' OpenMP backend, never `--gpu`. Every run had +`WW3GPU_LAB_SWITCH=ON` and `WW3GPU_STRICT=ON` (the default); the matrix's Kokkos run sets +`WW_KOKKOS_STRICT=1` itself. The U3 agent's runs also had `WW_KOKKOS_VA_POISON=1` and +`WW_KOKKOS_VA_COUNT=1` in the environment and at most 4 jobs of 4 threads. + +1. **Resident, serial**, run by the owner in `~/code/ww3-gpu-pr191` at `daaef07d` + (2026-10-18T18:29:19Z, `OMP_NUM_THREADS=8`, 1012 s), and by the U3 agent in its worktree with + `WW3GPU_OMP_THREADS=4` and the VA diagnostics: + + ```bash + WW3GPU_KOKKOS=resident WW3GPU_LAB_SWITCH=ON just ww3-gpu-matrix 4 serial + ``` + + `just ww3-gpu-matrix-list serial` with the same variables lists 7 entries, as the selftest + expects. +2. **OMPG, no MPI**, parts without `XYP3` and `KTP3`, which refuse a moving grid + (`w3kokkos_prop3md.F90:123-125` at `50a10730`): + + ```bash + WW3GPU_OMP_THREADS=4 WW3GPU_LAB_SWITCH=ON WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV,CPRT \ + WW3GPU_SWITCH_WORDS='ST4,SHRD,!PDLIB' just ww3-gpu-matrix 4 omp + ``` + + With `SHRD,!PDLIB` alone the list has 50 entries: `ww3_ts3` and `mww3_test_05`, each with five + propagation schemes (`PR1`, `PR2_UQ`, `PR2_UNO`, `PR3_UQ`, `PR3_UNO`) and five source-term sets + (`ST1`, `ST2`, `ST3`, `ST4`, `ST6`). The source-term sets differ only in the words + `ww3gpu_lab_switch` replaces (`ST`, `FLX`, `STAB`; v, a `diff` of the sorted switch words of + every `ST4` file against its four siblings, `regtests/cmake/ww3gpu_common.cmake:224-247`), so + under the lab switch the 40 others build the same 10 switches. `ST4` keeps one of each: the 10 + entries cover every propagation scheme on both tests. +3. **`ww3_multi`, `SHRD`**, agent A's commands of + [the `ww3_multi` log](2026-10-18-ww3-multi-resident-va.md): + + ```bash + WW3GPU_OMP_THREADS=4 WW3GPU_LAB_SWITCH=ON WW3GPU_KOKKOS=resident \ + WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,!PDLIB,!MGP,!MGG' just ww3-gpu-matrix 2 serial -R '^0(33|34|38|43|61):' + WW3GPU_OMP_THREADS=4 WW3GPU_LAB_SWITCH=ON WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV \ + WW3GPU_SWITCH_WORDS='PR3,UQ,SHRD,MGP,!PDLIB' just ww3-gpu-matrix 2 omp -R '^00(2|8):' + ``` + + then entry 61 alone (`-R '^061:'`, 1 job) to keep its `ww3_grid` output. +4. **`ww3_tnc1`**, entry 38 with `WW3GPU_SWITCH_WORDS='PR0;SHRD'`, 1 job, with + `WW3GPU_KOKKOS=SRCE,OUTG,IOPE,UPDV,CPRT` and then with `resident`. The first `resident` try ran + at the same time as the other set and its `setup:kokkos` failed ("No known features for CXX + compiler"), two configures of `kokkos/build/openmp-release` at once, as `regtests/README.md` + warns; it was run again alone. +5. **Restart pair.** `grep -n rstrt_b4b WW3/regtests/bin/matrix.base`: every restart-reproduction + entry is `MPI` (`ww3_tp2.17` `work_ma1`, `work_mc1`, `work_ice_restart`, `work_deep_restart` + with `PDLIB`; `ww3_ufs1.1` `work_d`, `work_b`; `matrix.base:2048-2067`, `:2171-2205` at + `50a10730`), where the resident preset stops (ADR-0017). No entry restarts a `SHRD` case. The + equivalent is written as [`scripts/restart_pair.sh`](../../scripts/restart_pair.sh) and was not + run: the owner stopped every new long run on 2026-10-18 to run them by hand. + +The VA counters are the `ww_kokkos: VA ...` lines of each entry's `logs/.K.log`, summed over +the processes; the parts line is the `Kokkos parts` line of `work_K_/log.*`. + +## Result + +The summaries, with their headers: +[serial](../data/ww3gpu_matrix_serial_pin-50a10730_2026-10-18.tsv), +[omp](../data/ww3gpu_matrix_omp_pin-50a10730_2026-10-18.tsv), +[multi](../data/ww3gpu_matrix_multi_pin-50a10730_2026-10-18.tsv), +[tnc1](../data/ww3gpu_matrix_tnc1_pin-50a10730_2026-10-18.tsv). `ww3_self` is `-` on every row: +no `test.comp` difference, so no second Fortran run was needed. "D2H" is the whole-`VA` +device-to-host copies of the Kokkos run (`WW_KOKKOS_VA_COUNT=1`, the U3 agent's runs). + +| Item | Entry | Parts | test.comp | nccmp_tol | judged | D2H | Spectra, `CPRT` calls | +|---|---|---|---|---|---|---|---| +| 1 | 1 `ww3_tp2.6` `ST4` | resident | n/a: FAIL, `ran` fortran-only | n/a | - | 0 | 0 | +| 1 | 2 `ww3_tp2.6` `ST4_ASCII` | resident | n/a: FAIL, `ran` fortran-only | n/a | - | 0 | 0 | +| 1 | 3 `ww3_tp2.15` `PR3_UQ` | resident | 0/23 | PASS | dir,dp,fp,hs | 0 | 6 900, 0 | +| 1 | 4 `ww3_tp2.15` `-g 5km` | resident | 0/23 | PASS | dir,dp,fp,hs | 0 | 137 000, 0 | +| 1 | 5 `ww3_tp2.15` `input_const` | resident | 0/23 | PASS | dir,dp,fp,hs | 0 | 6 624, 0 | +| 1 | 6 `ww3_tp2.15` `input_rho` | resident | 0/25 | PASS | dir,dp,fp,hs | 0 | 6 900, 0 | +| 1 | 7 `ww3_ts2` `ST4_PR3_UQ` | resident | 0/22 | n/a | - | 0 | 4 632, 0 | +| 2 | 1, 3, 5, 7, 9 `ww3_ts3` `ST4_{PR1,PR2_UQ,PR2_UNO,PR3_UQ,PR3_UNO}_OMP` | `SRCE,OUTG,IOPE,UPDV,CPRT` | 0/20 each | PASS | hs | 24 each | 42 025, 0 | +| 2 | 2, 4, 6, 8, 10 `mww3_test_05` `ST4_*_OMP`, the same five | `SRCE,OUTG,IOPE,UPDV,CPRT` | 0/27 each | PASS | hs | 153 each | 300 876, 39 | +| 3 | 33 `mww3_test_01` `PR3_UQ` | resident | 0/22 | PASS | hs | 12 | 0, 0 | +| 3 | 34 `mww3_test_02` `PR3_UQ` `grdset_a` | resident | 0/23 | PASS | dp,fp,hs | 0 | 0, 0 | +| 3 | 38 `mww3_test_03` `PR3_UQ` `grdset_a` | resident | 0/34 | PASS | dp,fp,hs | 3 | 0, 219 | +| 3 | 43 `mww3_test_04` `PR3_UQ` `grdset_a` | resident | 0/25 | PASS | dir,dp,fp,hs | 0 | 0, 0 | +| 3 | 61 `mww3_test_07` `default` | resident | n/a: FAIL, `ran` no | n/a | - | 0 | 0 | +| 3 | 2, 8 `mww3_test_05` `ST1`/`ST4_PR3_UQ_OMP` | `SRCE,OUTG,IOPE,UPDV` | 0/27 each | PASS | hs | 156 each | 300 876, 0 | +| 4 | 38 `ww3_tnc1` | `SRCE,OUTG,IOPE,UPDV,CPRT` | 0/30 | n/a | - | 0 | 405, 3 | +| 4 | 38 `ww3_tnc1` | resident | n/a: FAIL, `ran` fortran-only | n/a | - | 0 | 0 | + +- (v) Every entry whose Kokkos run ran gave `test.comp` 0 files different with the host copy of + `VA` poisoned, every asked part `=K` in the parts line, and `calls left on the Fortran ...: + SRCE=0 CPRT=0`. The owner's item 1 gave the same verdicts (rows 3 to 7: 0/23, 0/23, 0/23, 0/25, + 0/22). +- (v) Rows 1 and 2 of item 1 stop in `W3INIT`: "WW_KOKKOS_XYP3=1, BUT ITS KOKKOS TRANSLATION DOES + NOT COVER THIS SWITCH FILE" (`logs/1.K.log:160` of the owner's build), since `W3KOKKOS_PROP3_SETUP` + sets `XYP3` off on a grid that is neither rectilinear nor curvilinear + (`w3kokkos_prop3md.F90:122`, `:134` at `50a10730`); `ww3_tp2.6` reads the unstructured mesh + `limon_ll.msh`. The `ww3_ounf` error after it is a downstream effect. Refused by design, as + [the lab-switch log](2026-10-18-resident-regtest-lab-switch.md) (line 51) records. The owner + decided on 2026-10-18 to keep both as FAIL rows, with no change to the selection, the verdict or + the parts; issue #196 is the port of the unstructured path. +- (v) `ww3_tnc1` with `resident` stops in `W3INIT` the same way: without `PR3` and `UQ` the set-up + asks `KOKKOS_PART ( 'XYP3', .FALSE. )` (`w3kokkos_prop3md.F90:143`), whose message is + `w3kokkosmd.F90:1244-1245`. Refused by design (`PATCH.md` section 28). +- (v) `mww3_test_07` fails in the Fortran run too: its `ww3_grid` stops with "ERROR IN READING SDS4 + FROM FILE, IOSTAT = 5010" (`work_F_61/ww3_grid_refug.out`), the entry's namelist read against the + lab switch's `ST4`. A WW3 input failure of both runs under the lab switch, as agent A found; the + row stays FAIL. +- (v) The `selection:count` test of item 2 failed: `regtests/CMakeLists.txt` has no expected count + for the word list `ST4;SHRD;!PDLIB` ("gives 10 omp ANY ST4;SHRD;!PDLIB entries, not ;"), so the + check had nothing to compare. All 10 entries ran and passed. The check was not changed. +- (v) `mww3_test_01` to `_04` set `FLSOU` false in every `ww3_grid*.inp`, the sixth flag of its `FLDRY ... FLSOU` line + (`F T T T F F` and similar; `mww3_test_05`'s are `F T T T F T`), so `SRCE=K` there computes no spectrum: these entries test the propagation, the + outputs and the `ww3_multi` exchanges on the resident `VA`, not `W3SRCE`. +- (v) Whole-`VA` copies against agent A's after-run on `80644ba9`'s fork branch (the `ww3_multi` log): + `mww3_test_01` 12 → 12, `_02` 8 → 0, `_03` 216 → 3, `_04` 0 → 0, `_05` 156 → 156 without `CPRT` + and 153 with it. None went up. `_03` lost its `W3OUTG` fallback: with `CPRT` in the preset, + `W3OUTG`'s partition fields run on Kokkos (219 `W3CPRT` calls). The 24 per run of `ww3_ts3` and + the 153 to 156 of `mww3_test_05` are the Fortran propagation's host write per step, `XYP3` and + `KTP3` being off on a moving grid. +- ⚠ No wall time here is a timing: the machine was shared, and none enters a table. + +**The restart pair (gap 10), designed and not run.** `scripts/restart_pair.sh`, on the `full` case +(`kokkos/tests/fixtures/bench_small`) built with `switch_lab_shrd`: + +```bash +WW3GPU_SWITCH=switch_lab_shrd bash scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos resident --build-only +WW_KOKKOS_VA_POISON=1 OMP_NUM_THREADS=4 nix develop picasso/labs/pratico#ww3 --command bash scripts/restart_pair.sh +``` + +It runs, with one executable, `fortran/a` and `kokkos/a` from 00:00 to 12:00 (nine fields every +hour from 06:00, a restart at 06:00 and 12:00) and `fortran/b` and `kokkos/b` from 06:00, each +from its own `a` run's 06:00 restart copied to `restart.ww3`. It compares with `cmp` `out_grd.ww3` +and the 12:00 restart of `b` against `a` in each mode, and every file of `kokkos/x` against +`fortran/x`; a Kokkos difference that the Fortran pair shows too is printed as WW3's own. Checked: +`bash -n`, `shellcheck` (clean) and `--self-test` (the namelist edits), nothing more ⚠. + +## What it changes + +- Gap 2 under `SHRD` holds on the integrated pin: the six `ww3_multi` entries that run give the + Fortran's bytes with the host copy poisoned, now with `CPRT` in the preset. `mww3_test_08` (MPI) + stays unrun (ADR-0017). +- Gap 10 stays open: no matrix entry restarts a `SHRD` case, and the pair above waits for the owner. +- `kokkos/ports.tsv`: the `w3iosfmd:w3cprt` cell names this matrix record and stays "no": whether + lab-switch evidence moves a row to "yes" is the owner's step of plan 002. No other cell changed. +- `scripts/restart_pair.sh` is new and changes no gate; a person reviews it before it merges + (`AGENTS.md`, "Never cheat a regtest tool"). The `selection:count` gap for custom word lists is + for the owner. diff --git a/docs/log/README.md b/docs/log/README.md index 970d8e9e..62dc56a1 100644 --- a/docs/log/README.md +++ b/docs/log/README.md @@ -71,3 +71,4 @@ Date, author, issue/PR/WFIP. | 2026-10-18 | [The boundary and track outputs read only their columns of the resident `VA`](2026-10-18-resident-output-readers.md) | #42, WFIP-0005 | | 2026-10-18 | [`ww3_multi` and the resident `VA`: the defect reproduced, column syncs at the exchanges, no read at `W3WAVE`'s return](2026-10-18-ww3-multi-resident-va.md) | plan 2026-10-18-002 U1 + U7, WFIP-0005 | | 2026-10-18 | [`W3CPRT` on the resident `VA`: partitioning without copying the spectrum back](2026-10-18-cprt-resident.md) | plan 2026-10-18-002 U9, WFIP-0005 | +| 2026-10-18 | [U3 on the integrated pin: the resident serial, OMPG, `ww3_multi` and `ww3_tnc1` entries; the restart pair designed, not run](2026-10-18-u3-regtests-v2.md) | plan 2026-10-18-002 U3, #191 | diff --git a/kokkos/PORT_RANKING.md b/kokkos/PORT_RANKING.md index f313cd1e..8a8a481b 100644 --- a/kokkos/PORT_RANKING.md +++ b/kokkos/PORT_RANKING.md @@ -132,7 +132,7 @@ The record is the `regtested` column of `kokkos/ports.tsv`. | `w3partmd:ptmean` | ported | `part_ptmean.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | | `w3snl3md:insnl3` | ported | `snl3_tables.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3IOGR hunk | | `w3partmd:w3part` | ported | `part_w3part.hpp` | no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) | -| `w3iosfmd:w3cprt` | ported | `iosf_w3cprt.hpp`, `cprt_shim.cpp` | no: regtest matrix pending; the CPU replay ran it on 2026-10-18: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT on two partition variants of bench_small (switch_lab_shrd, and switch_lab_ompg for the fields variant), WW_KOKKOS_VA_POISON=1: every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 (docs/log/2026-10-18-cprt-resident.md) | +| `w3iosfmd:w3cprt` | ported | `iosf_w3cprt.hpp`, `cprt_shim.cpp` | no: the owner decides whether lab-switch evidence counts (plan 2026-10-18-002); the WW3 regtest matrix ran it on 2026-10-18 at WW3 50a10730 with WW3GPU_LAB_SWITCH=ON and WW_KOKKOS_VA_POISON=1: ww3_tnc1 (SRCE,OUTG,IOPE,UPDV,CPRT) test.comp 0/30, 3 Kokkos calls; mww3_test_03 grdset_a (resident) 0/34, 219 calls; the five mww3_test_05 ST4 OMPG entries (SRCE,OUTG,IOPE,UPDV,CPRT) 0/27 each, 39 calls each (docs/log/2026-10-18-u3-regtests-v2.md, docs/data/ww3gpu_matrix_*_pin-50a10730_2026-10-18.tsv); before that the CPU replay ran it on 2026-10-18: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT on two partition variants of bench_small (switch_lab_shrd, and switch_lab_ompg for the fields variant), WW_KOKKOS_VA_POISON=1: every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 (docs/log/2026-10-18-cprt-resident.md) | | `w3snl3md:w3snl3` | ported | `snl3_gmd.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3SRCE hunk | | `w3src0md:w3spr0` | ported | `srce_st0_spr0.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests | | `w3src1md:w3spr1` | ported | `srce_st1_spr1.hpp` | no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests | diff --git a/kokkos/ports.tsv b/kokkos/ports.tsv index 048c4d2c..c84e17bb 100644 --- a/kokkos/ports.tsv +++ b/kokkos/ports.tsv @@ -81,7 +81,7 @@ w3partmd:ptsort model/src/w3partmd.F90:511-613 103 ported translated 2026-10-16; w3partmd:ptmean model/src/w3partmd.F90:1175-1538 364 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4, every sum in the Fortran's order, the nine MAX/MIN sites in the object's operand order, SIGN folds, constants and the SIGP**(-5) power tree; a fresh review found no defect and corrected comments; WW3's out-of-bounds read when IFPMAX stays 0 (a NaN spectrum only) returns kPtmeanNoPeak so the shim calls the Fortran there (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_ptmean.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it w3snl3md:insnl3 model/src/w3snl3md.F90:788-1649 862 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_glwu (NOGRB) and switch_UoM_nl3; the four LOG(AUXFR) of :1046 and :1273 follow the object's libmvec/scalar split; NKD is 1300 under ST4 by use association and 275 without, kept as a flag; a fresh review found no defect and corrected a call count in the report (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3IOGR hunk kokkos/src/ww_kokkos/snl3_tables.hpp no: header-only; no shim and no W3IOGR hunk yet w3partmd:w3part model/src/w3partmd.F90:138-497 360 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_NCEP_st4 (UPAR maxss order, MINVAL/MAXVAL/MAXLOC NaN seeds, NINT as lroundf with the IHMAX clamp, the wind-sea sort and combination in the Fortran's order); calls ptsort, pt_fld, ptmean and the PtnghbTable, none re-translated; a fresh review found no defect (NINT range-guarded and equal to lroundf for every float, the CUDA FFMA are libdevice and IEEE division only) and fixed the shim contract on kPtmeanNoPeak (port report); compiled for CUDA, never run no: regtest matrix pending; run inside WW3 through W3KOKKOS_CPRT (W3CPRT on the resident VA) on 2026-10-18, every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 on two partition variants of bench_small (kokkos/port_report/W3CPRT_KOKKOS.md) kokkos/src/ww_kokkos/part_w3part.hpp no: called by W3KOKKOS_CPRT on the fork branch ww3-gpu/u9-cprt under WW_KOKKOS_CPRT=1, not at the WW3/ pin until the integrator merges it -w3iosfmd:w3cprt model/src/w3iosfmd.F90:109-344 236 ported translated and wired 2026-10-18 on the resident VA (kokkos/src/fortran_iface/PATCH.md section 27); static parity by -O3 GIMPLE of w3iosfmd.F90 and of the shim under switch_lab_shrd: UABS, UDIR with the same folded RADE, FACT = (SIG*TPI)/CG, E2 = VA*FACT, no FMA, W3PART called unchanged, ICPRT and DTPRT integer placement (port report) no: regtest matrix pending; the CPU replay ran it on 2026-10-18: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT on two partition variants of bench_small (switch_lab_shrd, and switch_lab_ompg for the fields variant), WW_KOKKOS_VA_POISON=1: every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 (docs/log/2026-10-18-cprt-resident.md) kokkos/src/ww_kokkos/iosf_w3cprt.hpp, kokkos/src/fortran_iface/cprt_shim.cpp yes: WW3/ pin a304018c, w3wavemd.F90:2692 (W3KOKKOS_CPRT under WW_KOKKOS_CPRT=1 and in the resident preset) +w3iosfmd:w3cprt model/src/w3iosfmd.F90:109-344 236 ported translated and wired 2026-10-18 on the resident VA (kokkos/src/fortran_iface/PATCH.md section 27); static parity by -O3 GIMPLE of w3iosfmd.F90 and of the shim under switch_lab_shrd: UABS, UDIR with the same folded RADE, FACT = (SIG*TPI)/CG, E2 = VA*FACT, no FMA, W3PART called unchanged, ICPRT and DTPRT integer placement (port report) no: the owner decides whether lab-switch evidence counts (plan 2026-10-18-002); the WW3 regtest matrix ran it on 2026-10-18 at WW3 50a10730 with WW3GPU_LAB_SWITCH=ON and WW_KOKKOS_VA_POISON=1: ww3_tnc1 (SRCE,OUTG,IOPE,UPDV,CPRT) test.comp 0/30, 3 Kokkos calls; mww3_test_03 grdset_a (resident) 0/34, 219 calls; the five mww3_test_05 ST4 OMPG entries (SRCE,OUTG,IOPE,UPDV,CPRT) 0/27 each, 39 calls each (docs/log/2026-10-18-u3-regtests-v2.md, docs/data/ww3gpu_matrix_*_pin-50a10730_2026-10-18.tsv); before that the CPU replay ran it on 2026-10-18: scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT on two partition variants of bench_small (switch_lab_shrd, and switch_lab_ompg for the fields variant), WW_KOKKOS_VA_POISON=1: every model file and partition.ww3 byte-identical to the Fortran of the same executable and to --ww3 (docs/log/2026-10-18-cprt-resident.md) kokkos/src/ww_kokkos/iosf_w3cprt.hpp, kokkos/src/fortran_iface/cprt_shim.cpp yes: WW3/ pin a304018c, w3wavemd.F90:2692 (W3KOKKOS_CPRT under WW_KOKKOS_CPRT=1 and in the resident preset) w3snl3md:w3snl3 model/src/w3snl3md.F90:181-777 597 ported translated 2026-10-16 (non-PDLIB WAVNU1 variant, with EXPAND and EXPND2; no NL3 switch has PDLIB); static parity by -O3 GIMPLE and object comparison under switch_NCEP_glwu (NCEP2 -> NOGRB, SCRIP/SCRIPNC dropped) and switch_UoM_nl3, including the libmvec powf grouping of the SCALE loop; a fresh review found no defect and corrected two statements in the report (port report); compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs a shim and the W3SRCE hunk kokkos/src/ww_kokkos/snl3_gmd.hpp no: header-only; no shim and no W3SRCE hunk yet w3src0md:w3spr0 model/src/w3src0md.F90:81-231 151 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd with ST0 FLX2; a fresh review found no arithmetic defect and corrected where the report placed the reversed NaN-payload operands; compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests kokkos/src/ww_kokkos/srce_st0_spr0.hpp yes: WW_KOKKOS_SPR0, w3srcemd.F90:1086, :2201 at 357f575c w3src1md:w3spr1 model/src/w3src1md.F90:87-240 154 ported translated 2026-10-16; static parity by -O3 GIMPLE and object comparison under switch_lab_shrd with ST1 FLX1; a fresh review found no defect; compiled for CUDA, never run no: translated only, as the owner asked on 2026-10-16 (no test run); its checkpoint (ADR-0011) needs the fork branch pinned and WW3's regtests kokkos/src/ww_kokkos/srce_st1_spr1.hpp yes: WW_KOKKOS_SPR1, w3srcemd.F90:1098, :2212 at 357f575c diff --git a/scripts/restart_pair.sh b/scripts/restart_pair.sh new file mode 100755 index 00000000..c04018e1 --- /dev/null +++ b/scripts/restart_pair.sh @@ -0,0 +1,133 @@ +#!/usr/bin/env bash +# restart_pair.sh: does a run restarted from its own restart file give the bytes of the run it +# was cut from, with the resident Kokkos preset, on a SHRD case? Gap 10 of +# docs/PRODUCTION_READINESS_202610.md. Every restart-reproduction entry of WW3's matrix.base at +# the WW3/ pin (rstrt_b4b: ww3_tp2.17, ww3_ufs1.1) is MPI, where no resident part runs +# (ADR-0017), so this is the smallest equivalent on the full case of scripts/ww3-gpu.sh. +# +# scripts/restart_pair.sh run the four runs below and compare +# scripts/restart_pair.sh --self-test check the namelist edits on a scratch copy; runs no model +# +# It needs the executable that `WW3GPU_SWITCH=switch_lab_shrd bash scripts/ww3-gpu.sh --ww3-gpu +# full --cpu --kokkos resident --build-only` builds in build/ww3-gpu-openmp/ww3/bin, and stops +# when that build's switch is not SHRD with KOKKOS. Inside the pinned ww3 shell (just ww3). +# +# Case: kokkos/tests/fixtures/bench_small (120 x 80, 9 520 sea points, 12 hours), with nine +# fields every hour from 06:00 to 12:00 and a restart at 06:00 and at 12:00. Runs, one executable: +# fortran/a, kokkos/a 00:00 to 12:00 from the cold start +# fortran/b, kokkos/b 06:00 to 12:00 from /a's 06:00 restart, copied to restart.ww3 +# The Fortran runs set WW_KOKKOS_ALL=0; the Kokkos runs WW_KOKKOS_PRESET=resident, +# WW_KOKKOS_STRICT=1 and every *_COUNT=1. Both keep WW_KOKKOS_VA_POISON and WW_KOKKOS_VA_CHECK +# from the caller. Compared with cmp: out_grd.ww3 and the 12:00 restart of b against a (restart +# reproduction), and every file of kokkos/x against fortran/x (bit for bit). +# +# Exit codes: 0 every comparison equal; 1 a run failed or a file differs; 3 no SHRD KOKKOS build; +# 64 a bad argument. A Kokkos difference that the Fortran pair shows too is printed as WW3's own. +set -euo pipefail + +root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +case_in="$root/kokkos/tests/fixtures/bench_small" +bdir="${WW3GPU_RESTART_BUILD:-$root/build/ww3-gpu-openmp}" +runs="${WW3GPU_RUNS:-$root/build/runs}/restart-pair" +fields="HS LM T02 T0M1 T01 FP DIR SPR DP" + +# edit_case : the fixture's ww3_shel.nml with , the fields and the restarts. +edit_case() { + local d="$1" start="$2" + cp "$case_in"/* "$d"/ + sed -i -e "s#^ DOMAIN%START = .*# DOMAIN%START = '20200101 $start'#" \ + -e "s#^ TYPE%FIELD%LIST = .*# TYPE%FIELD%LIST = '$fields'#" \ + -e "s#^ DATE%FIELD = .*# DATE%FIELD = '20200101 060000' '3600' '20200101 120000'\n DATE%RESTART = '20200101 060000' '21600' '20200101 120000'#" \ + "$d/ww3_shel.nml" +} + +if [ "${1:-}" = --self-test ]; then + t="$(mktemp -d)" + trap 'rm -rf -- "$t"' EXIT + edit_case "$t" 060000 + for want in " DOMAIN%START = '20200101 060000'" " TYPE%FIELD%LIST = '$fields'" \ + " DATE%FIELD = '20200101 060000' '3600' '20200101 120000'" \ + " DATE%RESTART = '20200101 060000' '21600' '20200101 120000'" " DOMAIN%STOP = '20200101 120000'"; do + grep -qxF "$want" "$t/ww3_shel.nml" || { echo "restart_pair: self-test: no line '$want'" >&2; exit 1; } + done + [ "$(grep -c 'DATE%' "$t/ww3_shel.nml")" = 2 ] || { echo "restart_pair: self-test: DATE lines" >&2; exit 1; } + echo "restart_pair: self-test passed" + exit 0 +fi +[ $# = 0 ] || { sed -n '2,/^set -euo/{/^set -euo/!p}' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 64; } + +bin="$bdir/ww3/bin" +sw="$(cat "$bdir/switch" 2>/dev/null || true)" +[ -x "$bin/ww3_shel" ] && [[ " $sw " == *" SHRD "* ]] && [[ " $sw " == *" KOKKOS "* ]] \ + && [[ " $sw " != *" OMPG "* ]] && [[ " $sw " != *" MPI "* ]] || { + echo "restart_pair: $bin is not a SHRD KOKKOS build; run WW3GPU_SWITCH=switch_lab_shrd bash scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos resident --build-only" >&2 + exit 3 +} + +unset_parts=(-u WW_KOKKOS_PRESET -u WW_KOKKOS_STRICT -u WW_KOKKOS_ALL) +counts=() +for f in "$root"/kokkos/src/fortran_iface/parts.d/*.txt; do + while read -r key val _; do + case "$key" in PART) unset_parts+=(-u "WW_KOKKOS_$val") ;; COUNT) counts+=("$val=1") ;; esac + done < "$f" +done +preset="$(awk '$1 == "PRESET" && $2 == "resident" { print $3 }' "$root/kokkos/src/fortran_iface/presets.txt")" +[ -n "$preset" ] || { echo "restart_pair: no resident preset in presets.txt" >&2; exit 1; } + +run() { # + local d="$runs/$1/$2" start=000000 + local -a flags=(WW_KOKKOS_ALL=0) + [ "$1" = fortran ] || flags=(WW_KOKKOS_PRESET=resident WW_KOKKOS_STRICT=1 "${counts[@]}") + mkdir -p "$d" + if [ "$2" = b ]; then + start=060000 + cp "$runs/$1/a/restart001.ww3" "$d/restart.ww3" + fi + edit_case "$d" "$start" + (cd "$d" && "$bin/ww3_grid" > ww3_grid.out 2>&1) || { tail -20 "$d/ww3_grid.out" >&2; return 1; } + (cd "$d" && env "${unset_parts[@]}" OMP_PROC_BIND="${OMP_PROC_BIND:-false}" "${flags[@]}" \ + "$bin/ww3_shel" > ww3_shel.out 2>&1) || { tail -20 "$d/ww3_shel.out" >&2; return 1; } + grep -q "End of program" "$d/ww3_shel.out" || { echo "restart_pair: ww3_shel did not finish in $d" >&2; return 1; } + if [ "$1" = kokkos ]; then + local p line + line="$(grep -h "Kokkos parts" "$d/ww3_shel.out" | head -1)" + for p in ${preset//,/ }; do + [[ "$line" == *"$p=K"* ]] || { echo "restart_pair: $d: $p is not K in '$line'" >&2; return 1; } + done + fi + echo ">> $1/$2: ww3_shel from 20200101 $start finished; $(cd "$d" && find . -maxdepth 1 -name 'restart0*.ww3' -printf '%f ' | sort)" +} + +same() { #