This page describes the stable concepts a workflow adapter should preserve.
Individual commands may accept additional fields; use <command> help <subcommand> for the complete option contract.
- Array dimensions are listed in storage order.
- Image coordinates use
(y, x)array order unless an option explicitly asks for(x, y)pixel coordinates. - Inputs must be finite unless a command documents a NaN policy.
- Variance arrays use squared image units. Fit weights require positive variance on fitted pixels; xRep also accepts zero variance for propagation.
- Boolean fit masks use
Truefor selected pixels. Instrument bit masks must be translated with the command's mask policy. - Keep units and coordinate frames in FITS headers or adjacent metadata, including metadata accompanying NumPy arrays.
xFit reads pickle-free .npz archives or JSON manifests describing FITS
planes and candidate positions. The archive requires a unique
one-dimensional candidate_id array and floating-point images. Difference
images have shape (batch, y, x); split images have shape
(batch, 3, y, x) with channels ordered as difference, positive, and
negative. Optional initial parameters use (batch, parameter). Optional
mask and variance arrays must align with the fitted image pixels;
variances must be positive wherever the mask selects a pixel. The
sampled-stamp model additionally consumes a numeric stamp_basis array.
For split inputs, (batch, y, x) auxiliaries are per-candidate and broadcast
over all three planes. This interpretation also wins when batch == 3; use
(1, 3, y, x) to express per-plane auxiliaries explicitly.
The command rejects standalone .npy inputs, object arrays, and archives that
require pickle. Callers supply the observational data.
FITS manifests use schema cuphoton.xfit.fits-input/v1. They specify one
difference plane or three split planes in difference, positive, negative
order; all planes share an already aligned pixel grid. Each candidate has a
unique integer or string ID and integer zero-based (x, y) center, with an
odd [height, width] stamp fully inside the image. Paths resolve relative to
the manifest, and HDU indices are zero-based. Optional variance and integer
mask HDUs come from the same file as their image plane. bad_mask_bits
selects excluded bits; without it, any nonzero mask pixel is excluded. Split
mode requires variance for every plane or none. See the
complete FITS manifest example.
A successful fit-dipoles run contains:
| File | Meaning |
|---|---|
summary.json |
requested and resolved model/backend, input and compute dtypes, device, input-archive or manifest hash, input counts, artifact paths, and artifact hashes; FITS inputs also record source hashes and read receipts |
effective-config.yaml |
validated options used for the fit |
fits.parquet |
one row per candidate with an exact input-stamp hash, parameters, status, convergence, valid-pixel coverage, fitted and zero-signal chi-square statistics, and uncertainties |
fit-arrays.npz |
candidate indices and IDs, covariance matrices, and residual arrays without pickled objects |
Covariance uses supplied variances when present and residual scaling otherwise. Non-converged or rank-deficient fits retain status information and mark their uncertainties invalid.
For distributed xFit and xScan inference, summary.json is the terminal
execution summary. Merged scientific artifacts live in scientific/ for
ordinary runs or rounds/<round-id>/scientific/ for benchmark runs. Round
summaries describe individual passes; only the root summary includes the
executor's final lifecycle status. Warmup artifacts are retained.
xScan accepts an HSC NPY directory directly or as HSC_npy beneath a supplied
base directory. The directory contains metadata.json and these arrays:
| File | Axis order | Meaning |
|---|---|---|
images.npy |
(batch, y, x, exposure) |
observed images |
variances.npy |
(batch, y, x, exposure) |
per-pixel variances |
psfs.npy |
(psf_y, psf_x, channel, exposure) |
exposure PSFs |
exp_times.npy |
(exposure,) |
exposure times |
masks.npy |
optional (batch, y, x, mask_class, exposure) |
mask planes |
sky.npy |
optional (batch, y, x, exposure) |
sky estimates |
Spatial dimensions and exposure counts must agree across corresponding arrays. Treat the directory as caller-supplied data and keep it outside the repository.
reference and target are two-dimensional arrays with the same shape. An
optional target variance image, used for fit weights and chi-square, and
input masks must also match that shape. Kernels have odd height and width.
An explicit NPY fit mask is boolean or binary and uses True/1 for pixels
included in the weighted solve.
The spatial ALS Python API and image-pair manifest can instead receive an
explicit (sample, 2) array of zero-based (y, x) positions. In a v2 manifest,
fit_positions names a strict integer NPY file whose coordinates refer to the
post-crop image. Position files require solver=spatial-als and cannot be
combined with a fit mask or automatic stamp selection. Duplicate positions are
retained and therefore increase that pixel's weight. Every explicit position
must reference a finite target, variance, and source footprint; an invalid
explicit position raises an error for the selection. Mask and default
selection use valid pixels. For spatial ALS, Chebyshev coordinates are
normalized over the fitted image axes recorded in the summary's image_shape,
which is the crop when a workflow crop is applied, so saved coefficient fields
and kernel_at use crop coordinates. These positions and all image, variance,
mask, registration, PSF,
and instrument-calibration inputs come from the calling pipeline.
Explicit-position summaries distinguish row count, unique pixel count, and duplicate row count.
A successful fit writes summary.json and these arrays under artifacts/:
| File | Meaning |
|---|---|
kernel.npy |
fitted matching kernel for the constant solver |
matched.npy |
convolved reference plus fitted background |
residual.npy |
target minus matched image |
fit_mask.npy |
pixels used by the solve |
fit_positions.npy |
ordered fit rows, including duplicates, when supplied |
background.npy |
fitted differential background |
Spatial ALS runs save their position-dependent kernel as horizontal and
vertical reference profiles and bases, both coefficient matrices, and a
center-evaluated preview in kernel_center.npy. They also save the
background coefficients and objective history. The summary records the
solver, term order, coordinate normalization, convergence, regularization,
reference-profile scale, center kernel sum, and fit counts needed to
interpret them, plus design_chunk_size, the row-batch cap used for normal
equations and reconstruction. That cap is fixed on CPU and derived from free
device memory on CuPy; different caps can produce floating-point rounding
differences. The vertical_reference_scale field is always saved because it
multiplies the vertical reference when the factors are evaluated. With flux
conservation enabled, the summary also records it as flux_scale, the
position-independent signed kernel sum. With flux conservation disabled, the
summary stores the multiplier as vertical_reference_scale, and the local
photometric scale comes from the sum of kernel_at(y, x). The center kernel
sum is recorded in either case. Spatial ALS dof is the nominal fit-row
count minus parameter count. Estimating effective degrees of freedom for
this regularized nonlinear fit requires a separate statistical analysis.
Spatial Gaussian-polynomial runs save basis_kernels.npy,
shape_coefficients.npy, photometric_coefficients.npy, and
background_coefficients.npy. The spatial_gaussian_polynomial section of
summary.json records the ordered (x_degree, y_degree) Chebyshev terms for
each field and its spatial_domain: the array's parent origin and the
half-open parent bounding box used for normalization. With a CLI crop, this
box retains the original full-image bounds; the coefficients use parent
coordinates. kernel_center.npy previews the kernel at the crop's center,
and photometric_scale.npy contains the local kernel sum at each image pixel.
When target variance is supplied, marginal_residual_variance.npy and
marginal_standardized_residual.npy contain the target-only marginal noise
diagnostics. See the model description
for reconstruction and uncertainty conventions.
Auto-stamp selection also writes metadata describing the selected regions.
Benchmark runs write timings.json, comparisons.json, and each backend's
result arrays as <backend>_<name>.npy. Constant-kernel benchmarks save the
kernel, matched, residual, fit-mask, and background images. Spatial ALS
benchmarks save the matched, residual, fit-mask, and background images, plus
the line references and bases, both coefficient fields, the background
coefficients, flux_scale, the objective history, the spatial and
background term tables, kernel_sample_positions_yx (the four image corners
and the center), and realized_kernels evaluated there; the kernels are
reproducible from the saved factors. With flux conservation enabled,
flux_scale is the signed kernel sum; otherwise it is the vertical
reference multiplier, and local kernel sums must be evaluated from the saved
factors. The summary separates first-solve from warm timings, states each
timing boundary, and reports median speedups against the reference backend,
per-backend solver facts including design_chunk_size, and the CPU thread
environment. CuPy timing rows add CUDA-event intervals and memory-pool
observations. parity.ok gates arrays, scalars, exact fields, and fit
masks; the objective history, condition number, iteration count, and
convergence flag are reported as non-gating diagnostics.
A prepared dataset directory contains:
| File | Shape or format |
|---|---|
search.npy |
(sample, y, x) floating-point stamps |
template.npy |
(sample, y, x) floating-point stamps |
difference.npy |
optional (sample, y, x) triplet channel |
labels.npy |
(sample,) binary labels |
split.npy |
(sample,); train 0, validation 1, test 2 |
metadata.jsonl |
one JSON object per sample when metadata is available |
summary.json |
dataset kind, shapes, split counts, and saved files |
All image arrays have the same shape and all first dimensions equal the label
count. Pair models consume search and template channels; triplet models also
require difference.npy. Train with reviewed labels whose provenance has been
validated. Reserve smoke and unlabeled placeholder rows for workflow checks.
Training runs contain an effective config, checkpoint and metric artifacts, and a summary that records the selected device and split. Review annotations are separate append-only artifacts and should retain reviewer and source-run provenance.
cuphoton xscan data-export-xfit-input writes a pickle-free .npz containing
numeric or Unicode arrays: unique candidate_id and exact float images rows from
difference.npy. It is the default bridge into cuphoton xfit fit-dipoles;
optional --variance and --mask NPY files add xFit's existing variance and
mask members. Each input must match the original difference.npy shape and
row order. The exporter retains each plane's dtype and applies the same
first-occurrence candidate deduplication to every member; duplicate auxiliary
planes must contain equal values, including matching NaNs at excluded pixels.
Nonzero mask values include pixels. The mask must be finite everywhere;
images must be finite and variance must be finite and positive at included
pixels. Variance uses squared image units. The optional --image-unit label
and source-file SHA-256 hashes appear in the emitted JSON summary, outside the
NPZ's numeric-array contract. Redirect that summary to a file to retain it.
Construct an archive manually when initial parameters or a stamp basis are
required.
cuphoton xscan data-build-xfit-features joins a completed difference-mode
xFit run to a prepared xScan dataset by candidate_id and writes a new bundle
directory. It verifies the xFit artifact hashes and binds every matched row to
the exact xScan difference.npy stamp by a dtype-, shape-, and content-aware
hash:
| File | Meaning |
|---|---|
candidate-id.npy |
numeric or Unicode candidate IDs in exact xScan metadata order, loaded with pickle disabled |
features.npy |
row-aligned float32 feature matrix, loaded read-only through a memory map so split datasets and data-loader workers share the same file-backed values |
input-image-sha256.npy |
fixed-width ASCII SHA-256 values for the exact difference stamps, loaded with pickle disabled |
schema.json |
ordered feature contract, bounded transforms, source SHA-256 hashes, missing policy, and join diagnostics |
The candidate array must exactly match xScan metadata order when loaded.
Duplicate xScan rows may reuse one fit only when the duplicate stamps, splits,
and split groups are identical; duplicate xFit candidate IDs are rejected.
The default missing_policy: error requires every dataset row to have a fit.
indicator instead emits finite zero features with
fit_present=0 for unmatched rows. Invalid or non-converged fits set their
validity gates and invalid parameter-derived features to zero.
Every load recomputes the current dataset's difference-stamp hashes. Training
checkpoints retain the exact schema and feature artifact identity, and
inference records the independently validated identity of its target bundle.
The xFit input contains exact pixels and candidate identifiers. Fit and feature artifacts retain identifiers, hashes, residuals or derived fit values, and run summaries can retain resolved local paths. Publish these artifacts after their source data and metadata have been cleared for release.
Version 1 uses fixed, bounded transforms defined independently of the
training, validation, and test rows. A training config opts in with top-level
use_xfit_features: true plus xfit_feature_dir;
model.xfit_feature_names is then populated from and locked to the bundle
schema. The optional fusion controls are
xfit_hidden_dim, xfit_dropout, and xfit_modality_dropout. Inference and
evaluation of a fusion checkpoint require both --use-xfit-features and
--xfit-feature-dir. Omitting both preserves the image-only path; an
image-only checkpoint rejects xFit features. Indicator-policy evaluation also
rejects a validation/evaluated-split fit_present coverage difference greater
than 0.05 unless the caller explicitly supplies
--allow-xfit-coverage-mismatch.
Real/bogus training requires separately reviewed labels. Rubin
candidate_isDipole describes a dipole classification, and placeholder labels
support workflow checks. Partition reviewed labels into train/validation/test
groups by DiaObject or an equivalent stable source identity.
The xScan FITS inference API
accepts explicit FitsPlane descriptors and integer zero-based (y, x)
candidate centers. Channels are search, template, then optional difference.
Unlike an xFit split input, these are classifier image channels. The caller
supplies aligned images with the checkpoint's expected pixel preprocessing.
The API loads full selected HDUs, extracts odd-sized float32 stamps on one
GPU and returns host logits and probabilities in candidate order, with
reader receipts. This path does not produce a prepared training dataset or
interpret instrument mask bits.
CLI inputs are two-dimensional FITS images with a celestial WCS. Optional masks
must align with the source image. Python array workflows pair a 2D array with a
ReprojectionSpec containing the source-to-grid mapping, output bounding box,
interpolation, fill value, and mask policy.
Single-image runs write artifacts/reprojected.npy, an optional mask.npy, and
summary.json with backend, interpolation, grid reference, pixel scale,
bounding box, and timings. FITS output is optional. Stack runs use
reprojected_stack.npy and an optional mask stack on one shared grid.
xRay probes an HDF5 file before selecting a supported image cube, scan/delay
axis, entry counts, and intensity-normalization fields. On/off cubes must use
the same schema and image shape. Detector ROIs are supplied as (x, y) origins
and (width, height) dimensions; generated NumPy images remain row-major.
Trace NPZ files contain one-dimensional time/delay and signal arrays. Detector artifact runs persist a manifest plus detector-wide frequency, amplitude, and related NumPy arrays. Distributed runs add a plan, shard manifests, per-shard logs, and a merged manifest. Preserve the ROI, excluded rows, normalization, fit parameters, shard ranges, and package/hardware details with published results.
xRay detector artifacts from the optional iterative fitter use manifest version 3, including the fitting method and iterative controls in the configuration and resume identity. Existing linear-prediction artifacts keep their version 2 representation. The numerical diagnostic sidecar also identifies its schema: version 1 describes linear-prediction matrices; version 2 describes iterative convergence and residuals. A merge requires matching methods and controls as well as matching data and preprocessing.
Time-series inputs are aligned, uniformly sampled vectors from arrays or HDF5 datasets. Results contain cross-spectra, lag correlations, or both, with sample-grid and validity metadata. See the time-series contract for dataset layout, lag conventions and independent result shards.
Radio jobs describe station clocks, spectral bands, baselines and scan delay
models. They read two-bit, non-legacy VDIF files and write native xbeam-v1
or SWIN visibilities. Preserve timestamps, weights, baseline metadata and
integrity records when moving or merging outputs. See the
radio data contract and
sidecar guide.
Store generated run directories outside the checkout when practical and keep them out of source commits. A portable run includes structured configuration, numeric artifacts, and metadata alongside its HTML views so review files can be regenerated.
xImage reads numeric arrays from an NPY directory (images.npy, etc.), an
HDF5 file, or a Zarr store. Dataset names are the keys below. Images,
variances, and PSFs are required; all other arrays and metadata.json are
optional.
| Key | Shape | Meaning |
|---|---|---|
images |
(batch, y, x, frame) |
Registered image pixels |
variances |
(batch, y, x, frame) |
Input pixel variances |
psfs |
(psf_y, psf_x, channel, frame) |
Point-source images, including calibrated kernels |
masks |
(batch, y, x, frame) |
Optional boolean mask: True includes the pixel; absent means all pixels |
exp_times |
(frame,) |
Optional positive frame scales; absent means ones |
sky |
(batch, y, x, frame) |
Optional sky estimates retained with the bundle |
mcgauss |
(frame, kernel_y, kernel_x) |
Optional Gaussian kernels retained with the bundle |
PSF calibration and reconstruction require one PSF channel. The evaluation
workflow requires one image batch. All frame axes must agree. Reconstruction
divides images by exp_times and variances by its square. Supply images
with the intended background convention: sky is retained and is not
automatically subtracted. The experimental
background_model: scalar_per_exposure fits a constant background for each
frame.
The default mask.mode: provided uses the bundle's good-pixel mask;
mask.mode: none ignores it. Instrument-specific flags are converted by the
adapter before constructing a bundle. The legacy twelve-plane camera schema
can be converted with
cuphoton.ximage.adapters.legacy.legacy_masks_to_good_pixels, whose
conservative and non_conservative policies and dilation_factor apply
only during that conversion. The Rubin adapter directly converts its bad
pixel flags to boolean good-pixel masks.
ROI centers are (row, column) in the source image. VALID padding requires
real input pixels around the ROI: ceil((psf_size // 2) / srf) on each side.
A region without this context is rejected. Saved raw reconstructions include
convolution context; the evaluation workflow selects the requested ROI.
Exposure selection
applies to every array's exposure axis; saved calibrated kernels record their
exposure selection and are checked before use. A saved PSF file without
selection metadata is treated as being in caller-provided exposure order.
Runs contain the resolved config.yaml, a summary.json with paths and
execution metadata, numeric products under artifacts/, and traces under
trace/. Reconstruction produces a host NumPy image in exposure-time-normalized
units. It does not provide propagated output variance or an effective output
PSF. Interpolation and reconstruction can correlate pixels; input variances
are not an output uncertainty model. WCS and calibration remain caller-managed.
See xImage for examples and evaluation limitations.