Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,15 @@
// over https. CPU only; a GPU gate (`just kokkos-cuda-test`) needs a host with a driver. The
// first `nix develop` downloads the toolchain; later ones hit the store. Open a full clone, not a
// git worktree: a worktree's .git points into a repository the container does not mount.
// cacheFrom reuses the published dev image (.github/workflows/docker-dev.yml, pushed by hand
// only), so Kokkos is not rebuilt; while none is pushed the build runs in full.
{
"name": "ww3-gpu",
"build": {
"dockerfile": "../docker/Dockerfile",
"context": "..",
"target": "dev"
"target": "dev",
"cacheFrom": "ghcr.io/h0ffmann/ww3-gpu-dev:latest"
},
"remoteUser": "ubuntu",
"postCreateCommand": "bash scripts/agent_env.sh",
Expand Down
60 changes: 60 additions & 0 deletions .github/workflows/docker-dev.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# docker-dev — the development image: the `dev` stage of docker/Dockerfile (Ubuntu 26.04, Ubuntu's
# compilers and netCDF, Kokkos 5.2.2, the gate tools, Nix for picasso), the image
# .devcontainer/devcontainer.json builds. It holds no source: the checkout is mounted
# (scripts/docker.sh run-dev), so working on ww3-gpu needs only Docker.
#
# Manual only: no push, pull request or schedule builds it (the owner, 18 October 2026). A run
# started from Actions → docker-dev → Run workflow builds the image and checks its tools; with
# `push` ticked it also pushes ghcr.io/<owner>/ww3-gpu-dev, and `latest` moves only when that run
# is on main.
name: docker-dev

on:
workflow_dispatch:
inputs:
push:
description: Push the image to ghcr.io (unticked, the run only builds and checks it)
type: boolean
default: false
tag:
description: Image tag (empty, sha-<short commit>)
type: string
default: ""

permissions:
contents: read

jobs:
# The environment `ghcr` is where the owner can require an approval before each run
# (Settings → Environments).
image:
runs-on: ubuntu-latest
timeout-minutes: 60
environment: ghcr
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v7
with:
submodules: false
- name: Build the dev stage (scripts/docker.sh build-dev)
run: bash scripts/docker.sh build-dev ci
- name: Its tools answer
run: docker run --rm ww3-gpu-dev:ci bash -c 'gfortran --version | head -1 && cmake --version | head -1 && just --version && shellcheck --version | head -2 && nix --version && test -d /opt/kokkos/include'
- name: Push to ghcr.io
if: inputs.push
env:
TAG: ${{ inputs.tag }}
OWNER: ${{ github.repository_owner }}
GH_TOKEN: ${{ github.token }}
run: |
tag="${TAG:-sha-${GITHUB_SHA::7}}"
[[ "$tag" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$ ]] || { echo "bad tag: $tag" >&2; exit 64; }
[ "$tag" != latest ] || { echo "latest is set only by a run on main" >&2; exit 64; }
image="ghcr.io/${OWNER,,}/ww3-gpu-dev"
refs=("$image:$tag")
[ "$GITHUB_REF" = refs/heads/main ] && refs+=("$image:latest")
echo "$GH_TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdin
bash scripts/docker.sh push ww3-gpu-dev:ci "${refs[@]}"
printf 'Pushed:\n' >> "$GITHUB_STEP_SUMMARY"; printf -- '- %s\n' "${refs[@]}" >> "$GITHUB_STEP_SUMMARY"
85 changes: 85 additions & 0 deletions .github/workflows/docker-dist.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# docker-dist — the distribution images, binaries only: NOAA's WW3 at the WW3-noaa/ pin (--ww3)
# and ww3-gpu (--ww3-gpu, the fork with the Kokkos port) compiled side by side, so a user can run a
# case with both and compare the outputs pair to pair (scripts/ww3-gpu.sh is the entry point).
# cpu docker/Dockerfile ghcr.io/<owner>/ww3-gpu Kokkos on OpenMP, plus WW4
# cuda docker/Dockerfile.cuda ghcr.io/<owner>/ww3-gpu-cuda Kokkos on CUDA, --ww3 on the CPU
# The final stage of each keeps the compiled programs, their runtime libraries and the case
# inputs; `scripts/docker.sh check` fails when a source file or either WW3 is missing. The labels
# io.github.h0ffmann.ww3-gpu.ww3-noaa and .ww3 name the two WW3 commits the binaries came from.
#
# Manual only: no push, pull request or schedule builds them (the owner, 18 October 2026). A run
# started from Actions → docker-dist → Run workflow builds and checks the chosen images (nothing
# runs on a GPU: the CUDA image is only compiled); with `push` ticked it also pushes them, and
# `latest` moves only when that run is on main. The CPU image's vulnerability scan is
# docker-scan.yml, manual too.
name: docker-dist

on:
workflow_dispatch:
inputs:
flavour:
description: Which image
type: choice
options: [both, cpu, cuda]
default: both
push:
description: Push to ghcr.io (unticked, the run only builds and checks)
type: boolean
default: false
tag:
description: Image tag, e.g. a release 0.5.0 (empty, sha-<short commit>)
type: string
default: ""

permissions:
contents: read

jobs:
# The environment `ghcr` is where the owner can require an approval before each run
# (Settings → Environments).
image:
runs-on: ubuntu-latest
timeout-minutes: 150
environment: ghcr
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
flavour: ${{ fromJSON(inputs.flavour == 'both' && '["cpu","cuda"]' || format('["{0}"]', inputs.flavour)) }}
steps:
- uses: actions/checkout@v7
with:
submodules: false
- name: Free disk space
run: sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL && df -h /
- name: Fetch WW3, WW3-noaa and WW4 at their pins
run: bash scripts/docker.sh submodules
- name: Build
env:
FLAVOUR: ${{ matrix.flavour }}
run: |
if [ "$FLAVOUR" = cuda ]; then bash scripts/docker.sh build-cuda ci; else bash scripts/docker.sh build ci; fi
- name: Binaries only, both WW3s present
env:
FLAVOUR: ${{ matrix.flavour }}
run: bash scripts/docker.sh check "$([ "$FLAVOUR" = cuda ] && echo ww3-gpu-cuda || echo ww3-gpu):ci"
- name: Push to ghcr.io
if: inputs.push
env:
FLAVOUR: ${{ matrix.flavour }}
TAG: ${{ inputs.tag }}
OWNER: ${{ github.repository_owner }}
GH_TOKEN: ${{ github.token }}
run: |
name=ww3-gpu; [ "$FLAVOUR" = cuda ] && name=ww3-gpu-cuda
tag="${TAG:-sha-${GITHUB_SHA::7}}"
[[ "$tag" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$ ]] || { echo "bad tag: $tag" >&2; exit 64; }
[ "$tag" != latest ] || { echo "latest is set only by a run on main" >&2; exit 64; }
image="ghcr.io/${OWNER,,}/$name"
refs=("$image:$tag")
[ "$GITHUB_REF" = refs/heads/main ] && refs+=("$image:latest")
echo "$GH_TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdin
bash scripts/docker.sh push "$name:ci" "${refs[@]}"
printf 'Pushed:\n' >> "$GITHUB_STEP_SUMMARY"; printf -- '- %s\n' "${refs[@]}" >> "$GITHUB_STEP_SUMMARY"
11 changes: 3 additions & 8 deletions .github/workflows/docker-scan.yml
Original file line number Diff line number Diff line change
@@ -1,17 +1,12 @@
# docker-scan — builds docker/Dockerfile, the image this repository distributes, and scans it with
# Trivy (free, open source): the job fails on any HIGH or CRITICAL vulnerability that has a fix,
# in an Ubuntu package or in a binary of the image. Unfixed ones are reported, not fatal: there
# is nothing to upgrade to. Weekly too, since new CVEs land without any change here.
# is nothing to upgrade to. Manual only (Actions → docker-scan → Run workflow): no Docker image
# is built by a push, a pull request or a schedule (the owner, 18 October 2026); `just dscan`
# runs the same scan locally.
name: docker-scan

on:
pull_request:
paths: [docker/**, .dockerignore, scripts/ww3-gpu.sh, scripts/docker.sh, kokkos/**, WW3, WW4, .github/workflows/docker-scan.yml]
push:
branches: [main]
paths: [docker/**, .dockerignore, scripts/ww3-gpu.sh, scripts/docker.sh, kokkos/**, WW3, WW4, .github/workflows/docker-scan.yml]
schedule:
- cron: "17 6 * * 1"
workflow_dispatch:

permissions:
Expand Down
42 changes: 40 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -472,11 +472,11 @@ become `dsmoke`, `dsmoke-ww3`, and so on. The image needs no Nix and no picasso:
installs Ubuntu's compilers and libraries, compiles Kokkos 5.2.2 from its release tarball (Ubuntu
packages an older one), and the final stage copies only the compiled programs.

CI scans the image with Trivy on every change to it and weekly (`.github/workflows/docker-scan.yml`): a HIGH or CRITICAL vulnerability with a fix fails the job. `just dscan` runs the same scan here.
No GitHub Actions workflow builds a Docker image on its own: every one starts by hand from Actions (Run workflow). `.github/workflows/docker-scan.yml` scans the image with Trivy when run: a HIGH or CRITICAL vulnerability with a fix fails the job. `just dscan` runs the same scan here.

The Kokkos code on the GPU has its own image, `docker/Dockerfile.cuda`: NVIDIA's CUDA 13.4 on
Ubuntu 26.04, Kokkos 5.2.2 built with CUDA for compute capability 8.9 (the RTX 4090; `KOKKOS_ARCH`
picks another), and only `--ww3-gpu --gpu` inside. The host needs the NVIDIA driver and the
picks another), with `--ww3-gpu --gpu` and, as its reference on the CPU, `--ww3`. Its kernels call CUDA's libdevice, not the device copies of glibc's libm (`WW_KOKKOS_LIBM=OFF`: Ubuntu ships libmvec stripped, and the copies need a symbol of it), so a GPU run in the image is judged by tolerance ([ADR-0014](docs/ADRs/ADR-0014-gpu-runs-judged-by-tolerance.md)). The host needs the NVIDIA driver and the
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
for `docker run --gpus all`. As on the host, every `--gpu` run asks the supervising person for
`yes` at the terminal ([Running on the GPU](AGENTS.md#running-on-the-gpu-hard-rule)).
Expand All @@ -501,6 +501,26 @@ Ubuntu's: `--ww3-gpu` still compares the Kokkos and the Fortran DIA inside one e
bit-for-bit verdict holds in the image; a number from the image is not a number from the pinned
toolchain of [`docs/TOOLCHAIN.md`](docs/TOOLCHAIN.md), which uses another gfortran.

**Published images.** `.github/workflows/docker-dist.yml` runs only when a person starts it from
Actions (Run workflow); no push, pull request or schedule triggers it. It builds the chosen
runtime images and checks, without running a case, that each holds NOAA's WW3 (`--ww3`, compiled
from `WW3-noaa/` at its pin) and ww3-gpu (`--ww3-gpu`) and no source file (`just dcheck` runs the
same check). With `push` ticked it also pushes `ghcr.io/h0ffmann/ww3-gpu:<tag>` (CPU) and
`ghcr.io/h0ffmann/ww3-gpu-cuda:<tag>`, and `latest` too when the run is on main. The labels
`io.github.h0ffmann.ww3-gpu.ww3-noaa` and `io.github.h0ffmann.ww3-gpu.ww3` name the two WW3 commits
the binaries came from. With a published image, anyone can run NOAA's WW3 and ww3-gpu on the same
case and compare the outputs with only Docker:

```console
img=ghcr.io/h0ffmann/ww3-gpu:<tag>
docker inspect "$img" --format '{{json .Config.Labels}}' # which WW3 commits
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/runs:/work/runs" "$img" --ww3 smoke
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/runs:/work/runs" "$img" --ww3-gpu smoke
```

(v) 2026-10-18: `scripts/docker.sh build` and `check` passed on the CPU image (WW3-noaa `03adda9`, WW3 `80644ba`); the CUDA image was not built here ⚠. No image has been pushed yet ⚠. GHCR may create the first push of a package as private; the owner
then makes it public in the package's settings.

### 3 · the dev container

`.devcontainer/devcontainer.json` builds the `dev` stage of `docker/Dockerfile`: Ubuntu 26.04 with
Expand All @@ -521,6 +541,24 @@ npx --yes @devcontainers/cli@0.89.0 exec --workspace-folder . just smoke-ww3-gpu
- `/nix` lives in the container, so a rebuilt container downloads picasso's toolchain again
(about 3.1 GB, about a minute on the lab's link).

Without VS Code or the Dev Containers CLI, the dev image alone is enough. `just dbuild-dev` builds
it, or pull `ghcr.io/h0ffmann/ww3-gpu-dev`, which `.github/workflows/docker-dev.yml` builds and,
with `push` ticked, pushes, only when a person starts it from Actions. `just ddev`
then opens a shell with this checkout at `/workspaces/ww3-gpu`, and `just ddev <command>` runs one
command (`DOCKER_IMAGE=ghcr.io/h0ffmann/ww3-gpu-dev:latest` picks the published image). Without
just:

```console
docker run --rm -it -v "$PWD:/workspaces/ww3-gpu" -w /workspaces/ww3-gpu ghcr.io/h0ffmann/ww3-gpu-dev:latest
```

The user inside is `ubuntu` (uid 1000), so files it writes belong to uid 1000 on the host. Run
`bash scripts/agent_env.sh` once inside when `picasso/` is not checked out. `devcontainer.json`
takes the published image as its build cache (`cacheFrom`), so Kokkos is not compiled again.
(v) 2026-10-18: `just dbuild-dev` built, and `docker-dev.yml`'s check found gfortran 15.2.0,
CMake 4.2.3, just 1.45.0, ShellCheck 0.11.0, Nix 2.34.3 and Kokkos in `/opt/kokkos`. ⚠ `just ddev
just kokkos-test serial-debug` and the cache from a pushed image are not run yet.

(v) 2026-10-15, CLI 0.89.0, Docker 29.9.1, `main` at `78e5a33`, full clone: `devcontainer up`
passed, `just kokkos-test serial-debug` passed 10 of 10, `just smoke-ww3-gpu` gave byte-identical
Fortran and Kokkos outputs, and every file the runs wrote belongs to the host user.
Expand Down
44 changes: 42 additions & 2 deletions README.pt-BR.md
Original file line number Diff line number Diff line change
Expand Up @@ -469,11 +469,11 @@ Cada receita que roda o `scripts/ww3-gpu.sh` no host tem seu par na imagem, com
build instala os compiladores e as bibliotecas do Ubuntu, compila o Kokkos 5.2.2 a partir do tarball
da versão (o Ubuntu empacota uma mais antiga), e o estágio final copia só os programas compilados.

O CI varre a imagem com o Trivy a cada mudança nela e toda semana (`.github/workflows/docker-scan.yml`): uma vulnerabilidade HIGH ou CRITICAL com correção reprova o job. `just dscan` roda a mesma varredura aqui.
Nenhum workflow do GitHub Actions constrói uma imagem Docker sozinho: todos começam à mão pelo Actions (Run workflow). O `.github/workflows/docker-scan.yml` varre a imagem com o Trivy quando disparado: uma vulnerabilidade HIGH ou CRITICAL com correção reprova o job. `just dscan` roda a mesma varredura aqui.

O código Kokkos na GPU tem sua própria imagem, `docker/Dockerfile.cuda`: o CUDA 13.4 da NVIDIA
sobre Ubuntu 26.04, o Kokkos 5.2.2 compilado com CUDA para a capacidade de computação 8.9 (a RTX
4090; `KOKKOS_ARCH` escolhe outra), e só o `--ww3-gpu --gpu` dentro. O host precisa do driver da
4090; `KOKKOS_ARCH` escolhe outra), com o `--ww3-gpu --gpu` e, como referência na CPU, o `--ww3`. Os kernels chamam a libdevice do CUDA, não as cópias da libm da glibc no dispositivo (`WW_KOKKOS_LIBM=OFF`: o Ubuntu distribui a libmvec sem a tabela de símbolos, e as cópias precisam de um símbolo dela), então uma rodada na GPU dentro da imagem é julgada por tolerância ([ADR-0014](docs/ADRs/ADR-0014-gpu-runs-judged-by-tolerance.md)). O host precisa do driver da
NVIDIA e do [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
para `docker run --gpus all`. Como no host, toda rodada `--gpu` pede um `yes` no terminal a quem
supervisiona ([Running on the GPU](AGENTS.md#running-on-the-gpu-hard-rule)).
Expand All @@ -499,6 +499,27 @@ dentro de um mesmo executável, então o veredito bit a bit vale na imagem; um n
imagem não é um número do toolchain fixado de [`docs/TOOLCHAIN.md`](docs/TOOLCHAIN.md), que usa
outro gfortran.

**Imagens publicadas.** O `.github/workflows/docker-dist.yml` só roda quando uma pessoa o dispara
pelo Actions (Run workflow); nenhum push, pull request ou agendamento o aciona. Ele constrói as
imagens de execução escolhidas e confere, sem rodar caso nenhum, que cada uma traz o WW3 da NOAA
(`--ww3`, compilado do `WW3-noaa/` no seu pin) e o ww3-gpu (`--ww3-gpu`) e nenhum arquivo-fonte (o
`just dcheck` faz a mesma conferência). Com `push` marcado, ele também publica
`ghcr.io/h0ffmann/ww3-gpu:<tag>` (CPU) e `ghcr.io/h0ffmann/ww3-gpu-cuda:<tag>`, e também `latest`
quando a rodada é na main. Os rótulos
`io.github.h0ffmann.ww3-gpu.ww3-noaa` e `io.github.h0ffmann.ww3-gpu.ww3` dizem de quais commits do
WW3 vieram os binários. Com uma imagem publicada, qualquer pessoa roda o WW3 da NOAA e o ww3-gpu no
mesmo caso e compara as saídas só com o Docker:

```console
img=ghcr.io/h0ffmann/ww3-gpu:<tag>
docker inspect "$img" --format '{{json .Config.Labels}}' # quais commits do WW3
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/runs:/work/runs" "$img" --ww3 smoke
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/runs:/work/runs" "$img" --ww3-gpu smoke
```

(v) 18/10/2026: o `scripts/docker.sh build` e o `check` passaram na imagem de CPU (WW3-noaa `03adda9`, WW3 `80644ba`); a imagem CUDA não foi construída aqui ⚠. Nenhuma imagem foi publicada ainda ⚠. O GHCR pode criar como privada a primeira publicação de um
pacote; o dono então a torna pública nas configurações do pacote.

### 3 · o dev container

O `.devcontainer/devcontainer.json` constrói o estágio `dev` do `docker/Dockerfile`: Ubuntu 26.04
Expand All @@ -520,6 +541,25 @@ npx --yes @devcontainers/cli@0.89.0 exec --workspace-folder . just smoke-ww3-gpu
- O `/nix` fica dentro do container, então um container reconstruído baixa de novo o toolchain do
picasso (cerca de 3,1 GB, cerca de um minuto na rede do laboratório).

Sem o VS Code nem a Dev Containers CLI, a imagem de desenvolvimento basta. O `just dbuild-dev` a
constrói, ou baixe a `ghcr.io/h0ffmann/ww3-gpu-dev`, que o `.github/workflows/docker-dev.yml`
constrói e, com `push` marcado, publica, só quando uma pessoa o dispara pelo Actions.
O `just ddev` abre então um shell com este clone em `/workspaces/ww3-gpu`, e `just ddev <comando>`
roda um comando (`DOCKER_IMAGE=ghcr.io/h0ffmann/ww3-gpu-dev:latest` escolhe a imagem publicada).
Sem o just:

```console
docker run --rm -it -v "$PWD:/workspaces/ww3-gpu" -w /workspaces/ww3-gpu ghcr.io/h0ffmann/ww3-gpu-dev:latest
```

O usuário lá dentro é o `ubuntu` (uid 1000), então os arquivos que ele escreve pertencem ao uid 1000
no host. Rode `bash scripts/agent_env.sh` uma vez lá dentro quando o `picasso/` não estiver no
clone. O `devcontainer.json` usa a imagem publicada como cache de build (`cacheFrom`), então o
Kokkos não é compilado de novo. (v) 18/10/2026: o `just dbuild-dev` construiu, e a conferência do
`docker-dev.yml` achou gfortran 15.2.0, CMake 4.2.3, just 1.45.0, ShellCheck 0.11.0, Nix 2.34.3 e o
Kokkos em `/opt/kokkos`. ⚠ O `just ddev just kokkos-test serial-debug` e o cache a partir de uma
imagem publicada ainda não rodaram.

(v) 15/10/2026, CLI 0.89.0, Docker 29.9.1, `main` em `78e5a33`, clone completo: o `devcontainer up`
passou, o `just kokkos-test serial-debug` passou 10 de 10, o `just smoke-ww3-gpu` deu saídas em
Fortran e em Kokkos idênticas byte a byte, e todo arquivo que as rodadas escreveram pertence ao
Expand Down
2 changes: 2 additions & 0 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,8 @@ RUN useradd --create-home --shell /bin/bash wave
COPY --chmod=755 scripts/ww3-gpu.sh /opt/ww3-gpu/scripts/ww3-gpu.sh
# Its list of Kokkos parts, one file per wiring unit.
COPY kokkos/src/fortran_iface/parts.d/ /opt/ww3-gpu/kokkos/src/fortran_iface/parts.d/
# The switch the programs were built with: ww3-gpu.sh reads it before any run (WW3GPU_SWITCH).
COPY switches/switch_lab_ompg /opt/ww3-gpu/switches/switch_lab_ompg
ENV WW3GPU_PREBUILT=/opt/ww3-gpu/prebuilt \
WW3GPU_RUNS=/work/runs \
OMP_PROC_BIND=false
Expand Down
Loading
Loading