Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
efabf3f
Add optional native C++/CUDA radio backend for xBeam
emcastillo Nov 16, 2026
36f9146
Use neutral visibility names and synthetic native fixtures
emcastillo Nov 16, 2026
6457e28
Reject unsupported native radio metadata
tpn Nov 16, 2026
293245c
Require an explicit native correlation profile
tpn Nov 16, 2026
4cdf659
Publish native completion reports after checked writes
tpn Nov 16, 2026
1b87b1c
Rebuild native bindings when execution inputs change
tpn Nov 16, 2026
f25a4d1
Supply Cython in native distribution build environments
tpn Nov 16, 2026
cd4b2c5
Keep fractional timestamp precision in native codec coverage
tpn Nov 16, 2026
f601626
Include native correlation in the shared CLI contract
tpn Nov 16, 2026
8458d34
Validate native scan metadata and execution sidecars
tpn Nov 16, 2026
4cc08a2
Preserve native CUDA resources through exceptional cleanup
tpn Nov 16, 2026
7a41b0b
Align native execution arguments and reject invalid inputs
tpn Nov 16, 2026
9c0f988
Ship native smoke sources and use locked CI build inputs
tpn Nov 16, 2026
f7ab27d
Document native binding and dynamic worker requirements
tpn Nov 16, 2026
28f5466
Refresh developer lock metadata for native build inputs
tpn Nov 16, 2026
dc752df
Support native CUDA correlation of contiguous HDF5 time series
emcastillo Nov 16, 2026
88fbcd0
Invalidate native binding builds when dependency contents change
emcastillo Nov 16, 2026
fdf1ee3
Scale native FP64 comparisons to each signal pair
tpn Nov 16, 2026
dcda861
Expand home-relative paths before native job dispatch
tpn Nov 16, 2026
ab5939b
Reject ambiguous native job metadata and unchecked extents
tpn Nov 16, 2026
3221f47
Read sidecar entries through bounded owned buffers
tpn Nov 16, 2026
4e675fb
Preserve prepared products when reconfiguration fails
tpn Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,27 @@ jobs:
uv sync --locked --python "${PYTHON_VERSION}"
--extra dev --extra torch --extra viz --extra photometry --extra ximage --extra rubin --extra catalog
- run: make ci-test-cpu
- name: Build and test native xBeam CPU backend
if: matrix.name == 'linux-x64' && matrix.python == '3.12'
run: |
cmake -S src/cuphoton/xbeam/native -B build/xbeam-native -DCMAKE_BUILD_TYPE=Release
cmake --build build/xbeam-native -j2
uv sync --locked --group native-build --inexact
CUPHOTON_XBEAM_BUILD_EXT=1 uv run --no-sync \
python setup.py build_ext --build-lib build/native-binding
XBEAM_NATIVE_BINDING_DIR=build/native-binding uv run --no-sync pytest -q -rs \
--junitxml="${RUNNER_TEMP}/xbeam-native-cpu.xml" tests/xbeam/native
uv run --no-sync python - <<'PYTEST'
import os
from pathlib import Path
import xml.etree.ElementTree as ET

report = ET.parse(Path(os.environ["RUNNER_TEMP"]) / "xbeam-native-cpu.xml")
assert report.findall(".//testcase"), "Native xBeam tests must execute"
assert not report.findall(".//skipped"), "Native xBeam tests must not skip"
assert not report.findall(".//failure")
assert not report.findall(".//error")
PYTEST
- name: Build native xRay CPU extension
if: matrix.name == 'linux-x64' && matrix.python == '3.12'
env:
Expand Down
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ repos:
rev: v22.1.8
hooks:
- id: clang-format
exclude: ^src/cuphoton/xbeam/native/third_party/
# identify classifies both .cu and .cuh as cuda.
types_or: [c++, c, cuda]

Expand Down
3 changes: 3 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,11 @@ prune dev
recursive-include examples *.py *.yaml
recursive-include examples/distributed-pipeline *.json *.txt *.license
recursive-include tests *.py
recursive-include tests/xbeam/native *.cpp
recursive-include src/cuphoton/xdr/src *.cpp *.h *.sh
recursive-include src/cuphoton/xfit/src *.cpp *.cu *.cuh *.h
recursive-include src/cuphoton/core/src *.cpp *.cu *.h
recursive-include src/cuphoton/xray/src *.cpp *.cu *.h
recursive-include src/cuphoton/xbeam/schemas *.json

recursive-include src/cuphoton/xbeam/native *.pyi *.pyx *.hpp *.cpp *.cu *.py CMakeLists.txt LICENSE.MIT
7 changes: 7 additions & 0 deletions THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -465,3 +465,10 @@ DragonHPC, `mpi4py`, MPI, and CUDA runtime libraries are installed separately.
A development source build links against the available CFITSIO and CUDA
libraries. Distributors must retain the licenses and notices for the libraries
included in their artifacts.

## nlohmann/json (native xBeam)

Version 3.12.0, copyright Niels Lohmann, distributed under the MIT License.
The pinned single-header source and complete license are included under
`src/cuphoton/xbeam/native/third_party/nlohmann/`.
Upstream: https://github.com/nlohmann/json/tree/v3.12.0
1 change: 1 addition & 0 deletions dev/uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 7 additions & 1 deletion docs/xbeam/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ lag conventions, validity masks and independent result shards.

A radio job describes station clocks, spectral bands, baseline products,
scans and their delay models. The radio adapter reads two-bit, non-legacy
VDIF files. Pass a normalized JSON job or supported DiFX `.input` metadata.
VDIF files. Pass a normalized JSON job or supported text `.input` metadata.
Correlation reads each recording through a header index
(a sidecar). Build a sidecar for every recording first with
`build-vdif-sidecar`, or pass `--assume-ordered` for recordings you have
Expand Down Expand Up @@ -133,3 +133,9 @@ batches, with asynchronous uploads for GPU execution. Automatic limits preserve
the backend defaults.
The radio command still accepts `--vdif-upload-workers` as an alias.
See [time-series input](time-series.md) for HDF5 queue and worker bounds.

## Experimental native radio backend

An optional [C++/CUDA backend](native.md) provides standalone radio execution
and a Cython entry point. It is built separately; the existing Python backends
and their defaults are unchanged.
233 changes: 233 additions & 0 deletions docs/xbeam/native.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,233 @@
# Experimental native backend

The C++17/CUDA execution library lives under `src/cuphoton/xbeam/native`.
The standalone radio runner needs neither Python nor CuPy at runtime. The
HDF5 adapter uses Python for metadata and result serialization while keeping
raw reads and numerical execution in C++. Support is experimental. The optional
Cython interface exposes the experimental `native-correlate` command;
it is not a production `cuphoton xbeam correlate` backend.

Build the standalone runtime with a CUDA toolkit, OpenSSL development files,
and a C++17 compiler:

```bash
cmake -S src/cuphoton/xbeam/native -B build/xbeam-native \
-DCMAKE_BUILD_TYPE=Release \
-DXBEAM_ENABLE_CUDA=ON \
-DXBEAM_ENABLE_CUFILE=ON \
-DCMAKE_CUDA_ARCHITECTURES=native
cmake --build build/xbeam-native -j
```

Python is used during the build to extract the existing cuPhoton-owned CUDA
kernel sources. Execution uses AOT kernels and cuFFT. With cuFile enabled, its
headers and library must be available in the CUDA toolkit. Omit that option
for a POSIX-only build.

Run an explicitly bounded integration window:

```bash
build/xbeam-native/xbeam-native-correlate JOB.input \
--output NEW_OUTPUT_DIRECTORY \
--index-dir SIDECARS \
--scan SCAN_ID \
--first-integration 0 \
--integration-count 1 \
--fft-tile 400 \
--device 0 \
--vdif-io cufile \
--profile interop-real-usb-buffered-window-replay-v1
```

The runner loads `.input`, `.calc`, and `.im` metadata; validates existing
schema-v4 sidecars; computes buffered-window source selection, timing and
validity; reads physical frames; performs rotation, FFT, channelization and
XMAC; accumulates integrations; and writes a SWIN file plus `native-run.json`.
The output directory must be new. A failed run has no completion report.
A contiguous run writes `visibilities.swin`; dynamically scheduled runs write
`visibilities.chunk-NNNNNN.swin` files, listed in the completion report.
Choose `--vdif-io posix` for reads through pinned host memory. Both delivery
paths use the same GPU frame-payload extraction and correlation kernels.
cuFile sessions, file handles and device registrations persist across reads.
Direct reads and their owning GPU buffer views use 64-KiB alignment. Each
read chunk has its own persistent registration so delivery does not depend
on offsets into a larger registered region.
`--upload-workers` bounds concurrent reads for either I/O mode.
`--read-chunk-bytes` sets their maximum size (16 MiB by default), in
64-KiB multiples. More workers or larger reads can increase contention when
several GPUs share storage; tune these limits for the machine.
The standalone runner also supports `--read-staging shared` for cuFile.
It retains one registered raw staging buffer per upload worker and extracts
frame payloads before reusing that buffer, including payloads split across
read boundaries. The default `per-input` mode retains raw buffers with each
input slot. Both produce the same packed payload and validity buffers.
`--prefetch on` uses two input slots to overlap the next read with current
GPU work; `--prefetch off` provides a synchronous comparison. Both preserve
the same source selection and numerical contract.
XMAC packs products that share spectra, using their addresses and strides;
output positions and per-product normalization and weight rules are retained.
Failed or incomplete reads cannot publish an input buffer. Chunk tasks drain
before their input allocations are released.

Radio execution requires one real-sampled physical recording per datastream,
a shared FFT time grid, whole subintegrations per output integration, and
whole reader requests per tile. The profile remains explicit; unsupported
geometry fails rather than delegating to Python or substituting a profile.
Parallel-hand autocorrelations currently use the interoperability truncation
convention. Additional processing profiles, JSON-job loading, general sampling
geometry and shard publication remain open.
The optional CPU inspection binding builds with `CUPHOTON_XBEAM_BUILD_EXT=1`.
The default cuPhoton package build remains pure Python.

Inspection commands:

```bash
build/xbeam-native/xbeam-native inspect-vdif RECORDING.vdif
build/xbeam-native/xbeam-native inspect-job JOB.input
build/xbeam-native/xbeam-native inspect-index RECORDING.vdif INDEX.vdix
```

Component executables `xbeam-native-station-test` and
`xbeam-native-products-test` check CUDA execution. The bounded
`xbeam-native-station-file-test` diagnoses station frontend output; it does
not generate baseline visibilities. Component checks do not replace full-scan numerical and performance
qualification for a particular input and execution configuration.

To build the optional CUDA Cython interface, set
`CUPHOTON_XBEAM_BUILD_EXT=1 CUPHOTON_XBEAM_NATIVE_CUDA=1` during installation.
This binding links cuFFT and cuFile, so both development libraries are required.
`CUDA_HOME` selects the toolkit, and
`CUPHOTON_XBEAM_CUDA_ARCHITECTURES` can select explicit CMake CUDA targets for
builds without an accessible GPU. Then the same execution library is available
without launching a subprocess:

```python
from cuphoton.xbeam.native import correlate

report = correlate(
"JOB.input", "NEW_OUTPUT_DIRECTORY", "SIDECARS", "SCAN_ID",
first_integration=0, integration_count=1, fft_tile=1250,
device=0, vdif_io="cufile",
profile="interop-real-usb-buffered-window-replay-v1",
)
```

The CPU-only extension explicitly refuses correlation. It never silently
falls back to Python execution.
Python callers must supply `integration_count` and `profile` explicitly.
Input delivery defaults to buffered POSIX reads. Each call runs its selected
window to completion; use bounded windows for interactive applications.

With the CUDA extension installed, the command line calls the same Cython
entry point without entering Python radio loading, planning, or execution:

```bash
cuphoton xbeam native-correlate JOB.input \
--output NEW_OUTPUT_DIRECTORY --index-dir SIDECARS \
--scan SCAN_ID --first-integration 0 --integration-count 8 \
--vdif-io cufile --fft-tile 1250 --upload-workers 2 \
--profile interop-real-usb-buffered-window-replay-v1
```

Use the standalone executable when no Python runtime is desired. Native
outputs currently require separate shard assembly; they do not yet publish
the manifests required by `merge-shards`.

## CPU development checks

The CPU parser, reader planner, codec, and Cython inspection binding can be
checked without a GPU. Install the development extra, CMake, OpenSSL headers,
and a C++17 compiler, then run from a source checkout:

```bash
python -m pip install Cython setuptools-scm
cmake -S src/cuphoton/xbeam/native -B build/xbeam-native \
-DCMAKE_BUILD_TYPE=Release
cmake --build build/xbeam-native -j
CUPHOTON_XBEAM_BUILD_EXT=1 python setup.py build_ext --build-lib build/native-binding
XBEAM_NATIVE_BINDING_DIR=build/native-binding python -m pytest tests/xbeam/native
```

Native binding tests are skipped when the optional binding is not supplied.
The CLI routing tests also run in a pure Python installation. Default package
builds do not require a CUDA toolkit or OpenSSL development files.

## Scheduling and I/O

For buffered POSIX, prefer contiguous integration windows per GPU to preserve
read locality. Budget CPU cores for all GPU workers on the node; limiting
four workers to one small CPU/NUMA subset can dominate input delivery time.
The standalone runner additionally offers `--vdif-io posix-direct`, which
uses aligned CPU `O_DIRECT` reads and pinned host-to-device copies. This is
separate from cuFile/GDS and is not automatically selected for POSIX.

Outputs should be written to node-local scratch before publication to shared
storage. Numerical acceptance should compare the complete selected integration
scope; output assembly and comparison are currently external to this backend.

The standalone runner can distribute one integration scope dynamically with
`--claim-dir DIRECTORY --workers-total N --min-chunk COUNT`. Create an empty
claim directory for that scope, give every worker the same job, scan,
integration range and scheduling options, and give each worker its own new
output directory. Workers claim chunks atomically and list their completed
files in `native-run.json`. Assembly must verify complete, nonoverlapping
coverage across all worker reports.

Claims remain assigned if a worker fails. Treat that campaign as incomplete
and use fresh claim and output directories when rerunning its scope. Claim
directories belong to one campaign and its common configuration.

## HDF5 time series

The native engine also computes generic time-series spectra and lag
correlations. HDF5 metadata is validated by the existing Python adapter;
C++ owns raw file reads, pinned/device buffers, cuFFT plans, cross-products,
validity statistics, and inverse transforms. Python serializes the owned
host results through the existing result writer. This path requires the
CUDA Cython extension and does not require CuPy at runtime.

Use the existing series configuration with `--backend native`:

```bash
cuphoton xbeam correlate-series series.json \
--backend native --output results.h5 --upload-workers 2
```

```json
{
"input": "signals.h5",
"datasets": {"a": "/signals/a", "b": "/signals/b"},
"io_backend": "posix",
"correlation": {
"pairs": [["a", "b"]],
"window_samples": 4096,
"hop_samples": 2048,
"windows_per_result": 16,
"batch_windows": 8,
"output": "both",
"lag_mode": "linear"
}
}
```

Both `posix` and `cufile` use the same native numerical pipeline and the
existing time-series result schema. cuFile requires the existing aligned
HDF5 raw-layout contract. The native path currently requires contiguous,
allocated, native-endian float32/float64 or complex64/complex128 vectors;
optional Boolean validity vectors must also be contiguous. Compressed,
chunked, virtual, externally stored, and integer inputs are rejected rather
than delegated to a Python engine. Each signal retains its required
`sample_rate_hz` and `origin_seconds` attributes.

FP32/FP64 selection, overlapping or separated windows, pair ordering,
linear/circular lags, validity and exposure counts, and result sharding
follow the existing series API. Incomplete result groups are dropped.
The executor captures the CUDA device selected at construction; separate
processes can select GPUs with `CUDA_VISIBLE_DEVICES` and partition result
ordinals using `shard_count` and `shard_index` in `correlation`.

Reads use two reusable slots; `io_queue_depth` must currently be 2.
`read_buffer_bytes` bounds each raw input span, reducing the processing batch
when needed. A single window must fit that bound. `prefetch_batches=0`
disables the overlap. Results are host-owned; no borrowed device arrays
are exposed by this backend.
1 change: 1 addition & 0 deletions packaging/conda/recipe.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ requirements:
- setuptools >=84.2.1
- setuptools-scm ==10.3.4
- wheel
- cython >=3.1,<4
- pybind11 ==3.0.4
- cuda-version ==13.0
- cuda-cudart-dev
Expand Down
5 changes: 4 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# SPDX-License-Identifier: Apache-2.0

[build-system]
requires = ["setuptools>=84.2.1", "setuptools-scm==10.3.4", "wheel"]
requires = ["setuptools>=84.2.1", "setuptools-scm==10.3.4", "wheel", "Cython>=3.1,<4"]
build-backend = "setuptools.build_meta"

[project]
Expand All @@ -18,6 +18,7 @@ license-files = [
"THIRD_PARTY_NOTICES.md",
"licenses/CCCL-3.1.2.txt",
"licenses/CFITSIO-4.7.0-Bison.txt",
"src/cuphoton/xbeam/native/third_party/nlohmann/LICENSE.MIT",
]
authors = [{ name = "NVIDIA Corporation" }]
keywords = ["astronomy", "cuda", "gpu", "image-processing"]
Expand Down Expand Up @@ -132,6 +133,7 @@ native-build = [
"setuptools>=84.2.1",
"setuptools-scm==10.3.4",
"wheel",
"Cython>=3.1,<4",
"pybind11>=3.0,<4",
]
notebook = [
Expand Down Expand Up @@ -163,6 +165,7 @@ where = ["src"]
include = ["cuphoton*"]

[tool.setuptools.package-data]
"cuphoton.xbeam.native" = ["*.pyi"]
"cuphoton.xbeam" = ["schemas/*.json"]
"cuphoton.xlens" = ["assets/*.html", "assets/*.css", "assets/*.js"]
"cuphoton.xpois" = ["assets/*"]
Expand Down
1 change: 1 addition & 0 deletions scripts/wheels/build-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ setuptools==84.2.1
setuptools-scm==10.3.4
vcs-versioning==2.5.0
wheel==0.48.0
Cython==3.3.1
pybind11==3.0.4
libkvikio-cu13==26.6.0
rapids-logger==0.2.3
Expand Down
6 changes: 6 additions & 0 deletions scripts/wheels/check_distributions.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,12 @@
"src/cuphoton/xray/src/structured_roots.cpp",
"src/cuphoton/xray/src/structured_refine.cpp",
"src/cuphoton/xray/src/structured_roots.cu",
"tests/xbeam/native/header_smoke.cpp",
"tests/xbeam/native/index_selection_smoke.cpp",
"tests/xbeam/native/radio_options_smoke.cpp",
"tests/xbeam/native/request_smoke.cpp",
"tests/xbeam/native/work_queue_smoke.cpp",
"tests/xbeam/native/run_report_smoke.cpp",
"scripts/wheels/prepare_cfitsio.sh",
"scripts/wheels/patches/cfitsio-version-string.patch",
"scripts/wheels/install_build_dependencies.sh",
Expand Down
2 changes: 2 additions & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ def _build_enabled(variable, *, default=False):

extensions = []
commands = {}
if _build_enabled("CUPHOTON_XBEAM_BUILD_EXT"):
extensions.extend(_load_build_helpers("xbeam").get_extensions())
if _build_enabled("CUPHOTON_XDR_BUILD_EXT"):
extensions.extend(_load_build_helpers("xdr").get_extensions())
if _build_enabled("CUPHOTON_XRAY_BUILD_EXT"):
Expand Down
Loading
Loading