NOAA/NASA Domain Specific Language middleware

NDSL is a middleware for climate and weather modelling developed jointly by NOAA and NASA. The middleware brings together GT4Py (the cartesian flavor), ETH CSCS's stencil DSL, and DaCe, ETH SPCL's data flow framework, both developed for high-performance and portability. On top of those pillars, NDSL deploys a series of optimized APIs for common operations (Halo exchange, domain decomposition, MPI, ...), a set of bespoke optimizations for the models targeted by the middleware and tools to port existing models.

Batteries-included for FV-based models

Historically, NDSL was developed to port the FV3 dynamical core on the cubed-sphere. Therefore, the middleware ships with ready-to-execute specialization for models based on cubed-sphere grids and FV-based models in particular.

Quickstart

Currently, NDSL requires Python version 3.11 or 3.12, a GNU compiler and MPI installed. All other dependencies installed during package installation. We recommend using virtual (or conda) environment.

# We have submodules for GT4Py and DaCe. Don't forget to pull them
git clone --recurse-submodules git@github.com:NOAA-GFDL/NDSL.git

cd NDSL/

# We strongly recommend using a virtual environment (or conda)
python -m venv .venv/
source ./venv/bin/activate

# Choose pip install -e .[dev] if you'd like to contribute
pip install .[demos]

Now, checkout examples/NDSL and run through the Jupyter notebooks. Note that you have to install NDSL from GitHub, as it is not available on pypi.

The slightly longer version

NDSL is under active development and may only work with specific setups. This is what we know works for us.

Requirements and supported compilers

The run the CPU backends you will need:

• Python: 3.11 or 3.12
• CXX compiler: GNU 11.2+
• Libraries: MPI

To run the GPU backends, you'll need:

• Python: 3.11 or 3.12
• CXX compiler: GNU 11.2+
• Libraries: MPI compiled with CUDA support
• CUDA 11.2+
• Python package:
  • cupy (latest with proper driver support see install notes)

A simple way to install MPI is using pre-built wheels, e.g.

# See "quickstart" above how to setup a virtual environment
cd NDSL/
source ./venv/bin/activate

# Install MPI into your virtual environment
pip install openmpi

A note on the compiler: NDSL currently only works with the GNU compiler. Using clang will result in errors related to undefined OpenMP flags. For MacOS users, we know that gcc version 14 from homebrew works.

Porting workflows might depend on serialbox, e.g. the script ndsl-serialbox_to_netcdf depends on it. Serialbox is an optional dependency. If you install the serialbox extra, you'll need the Boost library and development headers.

Installation options

See quickstart above on how to pull and setup a virtual environment. The packages has a few options:

• ndsl[test]: extra dependencies to run tests (based on pytest)
• ndsl[demos]: extra dependencies to run NDSL examples
• ndsl[docs]: extra dependencies to build the docs
• ndsl[dev]: installs tools for development, docs, and tests.
• ndsl[serialbox]: installs serialbox, which is used in porting workflows

Running tests

Tests are available via pytest (don't forget to install the test or dev extras).

To run serial tests on CPU (GPU tests also run if cupy is available)

pytest -m "not parallel and not gpu and not zarr" tests/

To run parallel tests on CPU (GPU tests also run if cupy is available)

mpirun -np 6 pytest -m "parallel and not gpu" tests/

Development

Code/contribution guidelines

1. Code quality is enforced by pre-commit (which is part of the "dev" extra). Run pre-commit install to install the pre-commit hooks locally or make sure to run pre-commit run -a before submitting a pull request.
2. While we don't strictly enforce type hints, we add them on new code.
3. Pull requests have to merged as "squash merge" to keep the git history clean.

Documentation

Documentation is available online. We are using Material for MkDocs, which allows us to write the docs in Markdown files and serve it as a static site.

To view the documentation locally, install NDSL with the docs or dev extras. Then run the following:

mkdocs serve

Contributing to the documentation is straight forward:

1. Add and/or change files in the docs/ folder as necessary.
2. [Optional] If you have changes to the navigation, modify mkdocs.yml.
3. [Optional] Start the development server and look how your changes are rendered.
4. Submit a pull request with your changes.

Points of contact

• NOAA: Rusty Benson: rusty.benson -at- noaa.gov
• NASA: Florian Deconinck florian.g.deconinck -at- nasa.gov