Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/GLOSSARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -704,7 +704,7 @@ and [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md).
| resident VA / `ww_va_host_read` / `ww_va_host_write` / `ww_va_device` / `ww_va_report` / `ww_srce_batch_va` | the device copy of the spectrum `VA(NSPEC,0:NSEALM)` that lives for the run (WFIP-0005), with a state per grid saying which copy holds the newer bits (`HOST`, `DEVICE`, `BOTH`); the C calls W3WAVE makes before a host read or write of `VA`, the one a kernel shim makes before reading the device copy, the traffic report, and the batched `W3SRCE` that works in place on the device copy `(v)` [kokkos/src/fortran_iface/resident_c.hpp](../kokkos/src/fortran_iface/resident_c.hpp) | The protocol of `Kokkos::DualView` (modify, sync) over WW3's own array; with no kernel on the device copy no call copies anything. Wrapped by `W3KOKKOS_VA_HOST_READ` / `_WRITE` in `w3kokkos_residentmd.F90` on the fork. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) |
| `WW_KOKKOS_VA_RESIDENT` / `WW_KOKKOS_VA_COUNT` | diagnostics of the resident VA, ignoring `WW_KOKKOS_ALL`: `=1` makes every host sync call push a host-newer `VA` to the device and copy it back, so a host write without its sync call shows in the output; the count prints the traffic line and the copy counters at exit `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset. On the CPU both copies are `memcpy`, so the bits cannot change. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md) |
| `WW_KOKKOS_VA_CHECK` / `WW_KOKKOS_VA_POISON` | diagnostics of the resident VA, read once, ignoring `WW_KOKKOS_ALL`, which compare or overwrite only the stale copy: `VA_CHECK=1` makes every sync call check what the state claims before it copies (unless `HOST`, the host `VA` still holds the bits of the last sync, so a host write without its `W3KOKKOS_VA_HOST_WRITE` stops the model; with `BOTH`, `d_va` holds them too, so a kernel that wrote `d_va` without marking it does); `VA_POISON=1` fills the host `VA` with a quiet NaN after every kernel write of `d_va`, so a host read without its sync, or a column the partial read misses, puts NaN in the output `(v)` [kokkos/src/fortran_iface/resident_shim.cpp](../kokkos/src/fortran_iface/resident_shim.cpp) | Default unset; a correct run writes the same bits with either. Added by the mutation audit of 2026-10-17. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) |
| `WW_KOKKOS_SRCE_LANES` | sea points per team of the batched `W3SRCE` launch on a host backend, 1 to 8, read once; each point runs the statements it runs alone, the points of a team take turns `(v)` [kokkos/src/fortran_iface/srce_shim.cpp](../kokkos/src/fortran_iface/srce_shim.cpp) | Default 1, today's one point per team; a device always uses 1. The lane layer of plan 005: later units vectorise across the points of a team. | [docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md](compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md) |
| `WW_KOKKOS_SRCE_LANES` | the most sea points per team of the batched `W3SRCE` launch on a host backend, 1 to 8, read once; each point runs the statements it runs alone, the points of a team take turns `(v)` [kokkos/src/fortran_iface/srce_shim.cpp](../kokkos/src/fortran_iface/srce_shim.cpp) | Default 8 (the owner, 2026-10-18), 1 for one point per team; a launch with fewer points than lanes times threads uses 1, and a device always uses 1. The lane layer of plan 005, which `W3SIN4`'s stress scan vectorises across (L2). | [docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md](compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md) |
| `WW_KOKKOS_OUTG_STAGED` | a diagnostic of the outg unit, read once: `=1` makes a host backend take the device's path of `ww_outg_va`, the inputs copied into the shim's own buffers, the fields computed into buffers, and only the fields a host reader needs copied back, so a field the copy-back selection misses keeps its old bits in WW3's array and shows in `out_grd.ww3` `(v)` [kokkos/src/fortran_iface/outg_shim.cpp](../kokkos/src/fortran_iface/outg_shim.cpp) | Default unset: on Serial and OpenMP the kernel writes WW3's arrays in place. | [docs/log/2026-10-17-mutation-audit.md](log/2026-10-17-mutation-audit.md) |
| `WW_KOKKOS_STRICT` | environment variable read once by `ww_kokkos_strict()`: `=1` makes every place where a part asked on Kokkos would run the Fortran stop the model with `EXTCDE` and a `NDSE` line naming the part and the reason: `KOKKOS_PART` under `WW_KOKKOS_ALL=1`, the two `W3SRCE` set-up fallbacks and the per-step `W3SRCE` gate (`W3KOKKOS_SRCE_STEP`). Unset, those places run the Fortran as before and the shim prints one line at exit, `ww_kokkos: calls left on the Fortran (...): SRCE=<n> ...` `(v)` [kokkos/src/fortran_iface/part_flags.cpp](../kokkos/src/fortran_iface/part_flags.cpp) | Default unset. The regtest matrix's Kokkos run sets it, its Fortran runs unset it, so a pair cannot compare the Fortran with itself unseen. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 24 |
| `WW_KOKKOS_PRESET` / `presets.txt` | environment variable naming a line `PRESET <name> <PART,...>` of `kokkos/src/fortran_iface/presets.txt`: `ww_kokkos_flag` turns its parts on as if each `WW_KOKKOS_<PART>` were `1`; a set `WW_KOKKOS_<PART>` still wins, and `W3INIT` stops on a name the file lacks. `resident` is the only preset (`SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT`) `(v)` [kokkos/src/fortran_iface/presets.txt](../kokkos/src/fortran_iface/presets.txt) | The file is the one list: CMake compiles it into the shim, `scripts/ww3-gpu.sh --kokkos resident` and the matrix's `WW3GPU_KOKKOS=resident` read it. | [kokkos/src/fortran_iface/PATCH.md](../kokkos/src/fortran_iface/PATCH.md), section 25 |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,8 @@ change a bit.
| Unit | What | Done when |
|---|---|---|
| S0 | One-thread profile of today's batch, libm separated from the rest | the shares below are measured; the go/no-go of S0 is applied |
| L1 | The lane layer: teams of `W` points (`WW_KOKKOS_SRCE_LANES`, 1 to 8, 1 reproduces today), each section run for the team's points in turn with today's scalar routines | **Done** 2026-10-18: `full` and `smoke` byte-identical with 1 and 8 (`kokkos/port_report/W3SRCE_KOKKOS.md`, "Points per team"). The point-minor scratch moves to L2, the first unit that reads it |
| L2 | `W3SIN4` on lanes (its stress loop, `st4_sin4.hpp:198-249`) | byte-identical; kernel time of `W3SRCE` measured with and without, 1 thread |
| L1 | The lane layer: teams of `W` points (`WW_KOKKOS_SRCE_LANES`, 1 to 8, 1 reproduces today, default 8 since L2), each section run for the team's points in turn with today's scalar routines | **Done** 2026-10-18: `full` and `smoke` byte-identical with 1 and 8 (`kokkos/port_report/W3SRCE_KOKKOS.md`, "Points per team"). The point-minor scratch moves to L2, the first unit that reads it |
| L2 | `W3SIN4` on lanes (its stress loop, `st4_sin4.hpp:198-249`) | **Done** 2026-10-18: byte-identical on `smoke` and `full`; one-thread `W3SRCE` kernel 105.17 s to 99.77 s, 5.1 % ([log](../../log/2026-10-18-srce-lanes-sin4.md)) |
| L3 | `W3SDS4` on lanes, the libmvec sites left as lane loops | the same |
| L4 | The integration and the initialisation (`srce_kernel.hpp:177-400`) | the same |
| L5 | `W3SPR4`, `W3SLN1`, `W3SDB1`, `W3SBT1` | the same |
Expand Down
66 changes: 66 additions & 0 deletions docs/log/2026-10-18-goal-c3e7eecf-handover.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Handover of goal `c3e7eecf`: the NOAA record, the ten-run median, points in lanes up to `W3SIN4`

2026-10-18, an agent for M. Hoffmann (Claude Code session `ec4548a3-46a8-46e2-be0a-2d74c3e7eecf`).
What this session finished, what is open, and what it learned, for whoever continues. It picks up
where [the `abbb0456` handover](2026-10-18-goal-abbb0456-handover.md) stops; every claim names its
record, and nothing here was re-run for this entry.

## What worked (merged)

| Result | Record |
|---|---|
| The NOAA progress record after #191: pin `50a10730`, seven preset parts with `CPRT`, strict mode, the NaN fill, one copy of `VA` to the device on `full`, 22 regtest entries with 0 files different and the four FAIL rows kept, a status table up front; the READMEs link it; gaps 6, 7 and 13 of the readiness list marked at the pin | PR #197; `docs/NOAA_PROGRESS_RECORD_202610.md`, `docs/PRODUCTION_READINESS_202610.md` |
| The record's speed figure: median of ten alternated runs on `main`'s code (`5cd0140`), `full`, 32 threads, NOAA WW3 26.18 s, Kokkos resident 15.84 s, 1.65 times less wall time, every output byte-identical to run 1's; regtest pending ⚠ (22 entries, not the whole matrix) | PR #197; the record, section 3 |
| Plan 005, sea points in the vector lanes of the batched `W3SRCE`, with its per-lane parity argument and units S0 to L7 | `docs/compound/plans/2026-10-18-005-srce-points-in-lanes-plan.md` |
| S0: one thread of the resident batch is 51.1 % scalar per-point code lanes can reach, 39.0 % DIA, 8.9 % libm and libmvec | `docs/log/2026-10-18-srce-lanes-s0-profile.md` |
| L1: `SrcePoint` sections with the kernel's bodies moved verbatim, `w3srce_lanes`, `WW_KOKKOS_SRCE_LANES` (1 to 8, host only; default 1 in #197, 8 since #199); `smoke` and `full` byte-identical with 1 and 8 | `kokkos/port_report/W3SRCE_KOKKOS.md`, "Points per team" |
| PR #198's conflict with `main` resolved by a merge (no history rewritten) | merge commit `41f4197` |

## Open work

| Item | State | Next step |
|---|---|---|
| PR #199, plan 005 L2: `W3SIN4`'s stress scan on 8-wide vectors | `smoke`, `full` byte-identical with 8 points per team; one-thread kernel 105.17 s to 99.77 s (5.1 %); CUDA `--build-only` clean; merged with `main` | the regtest matrix with `WW_KOKKOS_SRCE_LANES=8` (the matrix passes the variable through to its Kokkos runs, `regtests/cmake/ww3gpu_common.cmake`, `ww3gpu_kokkos_env`); the owner's review of the loop nesting and of the explicit vector types (the PR body) |
| Plan 005 L3: `W3SDS4` on lanes | not started; 24.8 % of the batch in S0, the largest unit left | split `w3sds4_point` as L2 split `W3SIN4`; the five libmvec sites (`st4_libmvec.hpp`) stay a loop over the lanes, each lane calling today's grouped code; time it pinned (`taskset -c 2`) as L2 was |
| Plan 005 L4 to L7 | not started | the integration and `w3srce_team` (9.9 %), `W3SPR4` and the small terms, the DIA layout, then a median of three at 32 threads on an idle machine |
| The opportunity and GFS-Wave analysis of 2026-10-18 | written by a sub-agent into the session's scratch directory, not committed; three of its claims checked (the `Kokkos::single` sites, the batched `W3SRCE` switch gate refusing `FLD2` and `SEED` and needing `LN1`, `7a23cd77` being plan 004's GFS.v17 commit) ⚠ the rest not checked | if wanted, redo it into `docs/compound/plans/`; its order for GFS-Wave was: read GFS.v17's switch, grid and compiler from the UFS weather model; plan 004; `W3FLD2` and the `SEED` and no-`LN1` paths in the batched `W3SRCE`; a `SHRD` build of `ww3_ufs1.1`; profile, restart pair, the Intel compiler; an ADR before any MPI work (ADR-0017) |

## Waiting on the owner

- The decisions plan 005 names: whether the host may call the transcribed vector libm
(`libm_device.hpp`) inside lane loops, and whether a sampled check is enough for `powf` and
`atan2f` (`docs/GPU_LIBM_BITS_202610.md`); whether to skip `W3SDS4`'s zero weights (not
bit-neutral: `0 x Inf` is NaN).
- The restart pair (`scripts/restart_pair.sh`), still never run.
- The `bench/results/` record of the ten-run median, once WW3's regtests have passed on `5cd0140`
beyond the 22 entries.
- Whether the regtests of #191 move the `regtested` cells of `kokkos/ports.tsv` for the resident
parts to "yes"; the ledger still says "no".
- From the `abbb0456` handover, still open: the strict-OUTG/POISON follow-up branch
(`goal/abbb0456-8241-strict-outg-poison`, no PR), the Fortran A/B harness branch, #196.

## Left on disk, not in the repository

- `~/code/ww3-gpu-2/noaa-timing.sh` (untracked): the timing of the record, `bash noaa-timing.sh
[N]`, N alternated runs each, the load printed at the start and the end. It moves `WW3/` to the
checked-out commit's pin first.
- `~/code/ww3-gpu-2` is on `goal/c3e7eecf-6142-srce-lanes-sin4`, `WW3/` at `50a10730`.

## Traps met (for whoever continues)

- **A one-thread timing on the i9-14900 is no measure unless pinned.** The same code took 113 s to
155 s in four unpinned runs as the thread moved between performance and efficiency cores; pinned
to CPU 2 (`taskset -c 2`), three runs agreed within 0.1 s.
- **The load average counts the timed job's own threads.** A 32-thread run shows a load of 25 to 27
with nothing else on the machine; read the load before the job starts.
- **g++ leaves a lane loop with selects scalar under its default `-ftrapping-math`**: it sinks the
values only one side of a select uses into a branch and will not speculate them back
("unsupported control flow in loop"). Plan 005 uses GCC vector types instead of changing that
flag for every kernel; `-fopt-info-vec-all` on the shim's translation unit, inside the pinned
shell, tells which loops vectorised.
- **`serial-debug` has no `-march`**: a function that returns a 32-byte vector by value warns
`-Wpsabi` there; pass vectors by reference.
- **`pkill -f <pattern>` from a tool shell kills that shell too** when the pattern is in its own
command line; kill by PID.
- **Merged means finished**: work after a merged PR goes on a new `goal/` branch from `main`, as L2
did after #197 merged.
Loading
Loading