Skip to content

Latest commit

 

History

History
384 lines (324 loc) · 21.3 KB

data-artifacts.md

File metadata and controls

384 lines (324 loc) · 21.3 KB

Data and artifact contracts

This page describes the stable concepts a workflow adapter should preserve. Individual commands may accept additional fields; use <command> help <subcommand> for the complete option contract.

General rules

  • Array dimensions are listed in storage order.
  • Image coordinates use (y, x) array order unless an option explicitly asks for (x, y) pixel coordinates.
  • Inputs must be finite unless a command documents a NaN policy.
  • Variance arrays use squared image units. Fit weights require positive variance on fitted pixels; xRep also accepts zero variance for propagation.
  • Boolean fit masks use True for selected pixels. Instrument bit masks must be translated with the command's mask policy.
  • Keep units and coordinate frames in FITS headers or adjacent metadata, including metadata accompanying NumPy arrays.

xFit dipole batches

xFit reads pickle-free .npz archives or JSON manifests describing FITS planes and candidate positions. The archive requires a unique one-dimensional candidate_id array and floating-point images. Difference images have shape (batch, y, x); split images have shape (batch, 3, y, x) with channels ordered as difference, positive, and negative. Optional initial parameters use (batch, parameter). Optional mask and variance arrays must align with the fitted image pixels; variances must be positive wherever the mask selects a pixel. The sampled-stamp model additionally consumes a numeric stamp_basis array. For split inputs, (batch, y, x) auxiliaries are per-candidate and broadcast over all three planes. This interpretation also wins when batch == 3; use (1, 3, y, x) to express per-plane auxiliaries explicitly.

The command rejects standalone .npy inputs, object arrays, and archives that require pickle. Callers supply the observational data.

FITS manifests use schema cuphoton.xfit.fits-input/v1. They specify one difference plane or three split planes in difference, positive, negative order; all planes share an already aligned pixel grid. Each candidate has a unique integer or string ID and integer zero-based (x, y) center, with an odd [height, width] stamp fully inside the image. Paths resolve relative to the manifest, and HDU indices are zero-based. Optional variance and integer mask HDUs come from the same file as their image plane. bad_mask_bits selects excluded bits; without it, any nonzero mask pixel is excluded. Split mode requires variance for every plane or none. See the complete FITS manifest example.

A successful fit-dipoles run contains:

File Meaning
summary.json requested and resolved model/backend, input and compute dtypes, device, input-archive or manifest hash, input counts, artifact paths, and artifact hashes; FITS inputs also record source hashes and read receipts
effective-config.yaml validated options used for the fit
fits.parquet one row per candidate with an exact input-stamp hash, parameters, status, convergence, valid-pixel coverage, fitted and zero-signal chi-square statistics, and uncertainties
fit-arrays.npz candidate indices and IDs, covariance matrices, and residual arrays without pickled objects

Covariance uses supplied variances when present and residual scaling otherwise. Non-converged or rank-deficient fits retain status information and mark their uncertainties invalid.

For distributed xFit and xScan inference, summary.json is the terminal execution summary. Merged scientific artifacts live in scientific/ for ordinary runs or rounds/<round-id>/scientific/ for benchmark runs. Round summaries describe individual passes; only the root summary includes the executor's final lifecycle status. Warmup artifacts are retained.

xScan HSC NPY inputs

xScan accepts an HSC NPY directory directly or as HSC_npy beneath a supplied base directory. The directory contains metadata.json and these arrays:

File Axis order Meaning
images.npy (batch, y, x, exposure) observed images
variances.npy (batch, y, x, exposure) per-pixel variances
psfs.npy (psf_y, psf_x, channel, exposure) exposure PSFs
exp_times.npy (exposure,) exposure times
masks.npy optional (batch, y, x, mask_class, exposure) mask planes
sky.npy optional (batch, y, x, exposure) sky estimates

Spatial dimensions and exposure counts must agree across corresponding arrays. Treat the directory as caller-supplied data and keep it outside the repository.

xPois image pairs

reference and target are two-dimensional arrays with the same shape. An optional target variance image, used for fit weights and chi-square, and input masks must also match that shape. Kernels have odd height and width. An explicit NPY fit mask is boolean or binary and uses True/1 for pixels included in the weighted solve.

The spatial ALS Python API and image-pair manifest can instead receive an explicit (sample, 2) array of zero-based (y, x) positions. In a v2 manifest, fit_positions names a strict integer NPY file whose coordinates refer to the post-crop image. Position files require solver=spatial-als and cannot be combined with a fit mask or automatic stamp selection. Duplicate positions are retained and therefore increase that pixel's weight. Every explicit position must reference a finite target, variance, and source footprint; an invalid explicit position raises an error for the selection. Mask and default selection use valid pixels. For spatial ALS, Chebyshev coordinates are normalized over the fitted image axes recorded in the summary's image_shape, which is the crop when a workflow crop is applied, so saved coefficient fields and kernel_at use crop coordinates. These positions and all image, variance, mask, registration, PSF, and instrument-calibration inputs come from the calling pipeline.

Explicit-position summaries distinguish row count, unique pixel count, and duplicate row count.

A successful fit writes summary.json and these arrays under artifacts/:

File Meaning
kernel.npy fitted matching kernel for the constant solver
matched.npy convolved reference plus fitted background
residual.npy target minus matched image
fit_mask.npy pixels used by the solve
fit_positions.npy ordered fit rows, including duplicates, when supplied
background.npy fitted differential background

Spatial ALS runs save their position-dependent kernel as horizontal and vertical reference profiles and bases, both coefficient matrices, and a center-evaluated preview in kernel_center.npy. They also save the background coefficients and objective history. The summary records the solver, term order, coordinate normalization, convergence, regularization, reference-profile scale, center kernel sum, and fit counts needed to interpret them, plus design_chunk_size, the row-batch cap used for normal equations and reconstruction. That cap is fixed on CPU and derived from free device memory on CuPy; different caps can produce floating-point rounding differences. The vertical_reference_scale field is always saved because it multiplies the vertical reference when the factors are evaluated. With flux conservation enabled, the summary also records it as flux_scale, the position-independent signed kernel sum. With flux conservation disabled, the summary stores the multiplier as vertical_reference_scale, and the local photometric scale comes from the sum of kernel_at(y, x). The center kernel sum is recorded in either case. Spatial ALS dof is the nominal fit-row count minus parameter count. Estimating effective degrees of freedom for this regularized nonlinear fit requires a separate statistical analysis.

Spatial Gaussian-polynomial runs save basis_kernels.npy, shape_coefficients.npy, photometric_coefficients.npy, and background_coefficients.npy. The spatial_gaussian_polynomial section of summary.json records the ordered (x_degree, y_degree) Chebyshev terms for each field and its spatial_domain: the array's parent origin and the half-open parent bounding box used for normalization. With a CLI crop, this box retains the original full-image bounds; the coefficients use parent coordinates. kernel_center.npy previews the kernel at the crop's center, and photometric_scale.npy contains the local kernel sum at each image pixel. When target variance is supplied, marginal_residual_variance.npy and marginal_standardized_residual.npy contain the target-only marginal noise diagnostics. See the model description for reconstruction and uncertainty conventions.

Auto-stamp selection also writes metadata describing the selected regions.

Benchmark runs write timings.json, comparisons.json, and each backend's result arrays as <backend>_<name>.npy. Constant-kernel benchmarks save the kernel, matched, residual, fit-mask, and background images. Spatial ALS benchmarks save the matched, residual, fit-mask, and background images, plus the line references and bases, both coefficient fields, the background coefficients, flux_scale, the objective history, the spatial and background term tables, kernel_sample_positions_yx (the four image corners and the center), and realized_kernels evaluated there; the kernels are reproducible from the saved factors. With flux conservation enabled, flux_scale is the signed kernel sum; otherwise it is the vertical reference multiplier, and local kernel sums must be evaluated from the saved factors. The summary separates first-solve from warm timings, states each timing boundary, and reports median speedups against the reference backend, per-backend solver facts including design_chunk_size, and the CPU thread environment. CuPy timing rows add CUDA-event intervals and memory-pool observations. parity.ok gates arrays, scalars, exact fields, and fit masks; the objective history, condition number, iteration count, and convergence flag are reported as non-gating diagnostics.

xScan datasets

A prepared dataset directory contains:

File Shape or format
search.npy (sample, y, x) floating-point stamps
template.npy (sample, y, x) floating-point stamps
difference.npy optional (sample, y, x) triplet channel
labels.npy (sample,) binary labels
split.npy (sample,); train 0, validation 1, test 2
metadata.jsonl one JSON object per sample when metadata is available
summary.json dataset kind, shapes, split counts, and saved files

All image arrays have the same shape and all first dimensions equal the label count. Pair models consume search and template channels; triplet models also require difference.npy. Train with reviewed labels whose provenance has been validated. Reserve smoke and unlabeled placeholder rows for workflow checks.

Training runs contain an effective config, checkpoint and metric artifacts, and a summary that records the selected device and split. Review annotations are separate append-only artifacts and should retain reviewer and source-run provenance.

xFit feature bundles for xScan

cuphoton xscan data-export-xfit-input writes a pickle-free .npz containing numeric or Unicode arrays: unique candidate_id and exact float images rows from difference.npy. It is the default bridge into cuphoton xfit fit-dipoles; optional --variance and --mask NPY files add xFit's existing variance and mask members. Each input must match the original difference.npy shape and row order. The exporter retains each plane's dtype and applies the same first-occurrence candidate deduplication to every member; duplicate auxiliary planes must contain equal values, including matching NaNs at excluded pixels.

Nonzero mask values include pixels. The mask must be finite everywhere; images must be finite and variance must be finite and positive at included pixels. Variance uses squared image units. The optional --image-unit label and source-file SHA-256 hashes appear in the emitted JSON summary, outside the NPZ's numeric-array contract. Redirect that summary to a file to retain it. Construct an archive manually when initial parameters or a stamp basis are required.

cuphoton xscan data-build-xfit-features joins a completed difference-mode xFit run to a prepared xScan dataset by candidate_id and writes a new bundle directory. It verifies the xFit artifact hashes and binds every matched row to the exact xScan difference.npy stamp by a dtype-, shape-, and content-aware hash:

File Meaning
candidate-id.npy numeric or Unicode candidate IDs in exact xScan metadata order, loaded with pickle disabled
features.npy row-aligned float32 feature matrix, loaded read-only through a memory map so split datasets and data-loader workers share the same file-backed values
input-image-sha256.npy fixed-width ASCII SHA-256 values for the exact difference stamps, loaded with pickle disabled
schema.json ordered feature contract, bounded transforms, source SHA-256 hashes, missing policy, and join diagnostics

The candidate array must exactly match xScan metadata order when loaded. Duplicate xScan rows may reuse one fit only when the duplicate stamps, splits, and split groups are identical; duplicate xFit candidate IDs are rejected. The default missing_policy: error requires every dataset row to have a fit. indicator instead emits finite zero features with fit_present=0 for unmatched rows. Invalid or non-converged fits set their validity gates and invalid parameter-derived features to zero. Every load recomputes the current dataset's difference-stamp hashes. Training checkpoints retain the exact schema and feature artifact identity, and inference records the independently validated identity of its target bundle.

The xFit input contains exact pixels and candidate identifiers. Fit and feature artifacts retain identifiers, hashes, residuals or derived fit values, and run summaries can retain resolved local paths. Publish these artifacts after their source data and metadata have been cleared for release.

Version 1 uses fixed, bounded transforms defined independently of the training, validation, and test rows. A training config opts in with top-level use_xfit_features: true plus xfit_feature_dir; model.xfit_feature_names is then populated from and locked to the bundle schema. The optional fusion controls are xfit_hidden_dim, xfit_dropout, and xfit_modality_dropout. Inference and evaluation of a fusion checkpoint require both --use-xfit-features and --xfit-feature-dir. Omitting both preserves the image-only path; an image-only checkpoint rejects xFit features. Indicator-policy evaluation also rejects a validation/evaluated-split fit_present coverage difference greater than 0.05 unless the caller explicitly supplies --allow-xfit-coverage-mismatch.

Real/bogus training requires separately reviewed labels. Rubin candidate_isDipole describes a dipole classification, and placeholder labels support workflow checks. Partition reviewed labels into train/validation/test groups by DiaObject or an equivalent stable source identity.

Direct FITS inference

The xScan FITS inference API accepts explicit FitsPlane descriptors and integer zero-based (y, x) candidate centers. Channels are search, template, then optional difference. Unlike an xFit split input, these are classifier image channels. The caller supplies aligned images with the checkpoint's expected pixel preprocessing. The API loads full selected HDUs, extracts odd-sized float32 stamps on one GPU and returns host logits and probabilities in candidate order, with reader receipts. This path does not produce a prepared training dataset or interpret instrument mask bits.

xRep FITS and arrays

CLI inputs are two-dimensional FITS images with a celestial WCS. Optional masks must align with the source image. Python array workflows pair a 2D array with a ReprojectionSpec containing the source-to-grid mapping, output bounding box, interpolation, fill value, and mask policy.

Single-image runs write artifacts/reprojected.npy, an optional mask.npy, and summary.json with backend, interpolation, grid reference, pixel scale, bounding box, and timings. FITS output is optional. Stack runs use reprojected_stack.npy and an optional mask stack on one shared grid.

xRay HDF5 and trace products

xRay probes an HDF5 file before selecting a supported image cube, scan/delay axis, entry counts, and intensity-normalization fields. On/off cubes must use the same schema and image shape. Detector ROIs are supplied as (x, y) origins and (width, height) dimensions; generated NumPy images remain row-major.

Trace NPZ files contain one-dimensional time/delay and signal arrays. Detector artifact runs persist a manifest plus detector-wide frequency, amplitude, and related NumPy arrays. Distributed runs add a plan, shard manifests, per-shard logs, and a merged manifest. Preserve the ROI, excluded rows, normalization, fit parameters, shard ranges, and package/hardware details with published results.

xRay detector artifacts from the optional iterative fitter use manifest version 3, including the fitting method and iterative controls in the configuration and resume identity. Existing linear-prediction artifacts keep their version 2 representation. The numerical diagnostic sidecar also identifies its schema: version 1 describes linear-prediction matrices; version 2 describes iterative convergence and residuals. A merge requires matching methods and controls as well as matching data and preprocessing.

xBeam signals and radio visibilities

Time-series inputs are aligned, uniformly sampled vectors from arrays or HDF5 datasets. Results contain cross-spectra, lag correlations, or both, with sample-grid and validity metadata. See the time-series contract for dataset layout, lag conventions and independent result shards.

Radio jobs describe station clocks, spectral bands, baselines and scan delay models. They read two-bit, non-legacy VDIF files and write native xbeam-v1 or SWIN visibilities. Preserve timestamps, weights, baseline metadata and integrity records when moving or merging outputs. See the radio data contract and sidecar guide.

Run-directory hygiene

Store generated run directories outside the checkout when practical and keep them out of source commits. A portable run includes structured configuration, numeric artifacts, and metadata alongside its HTML views so review files can be regenerated.

xImage reconstruction bundles

xImage reads numeric arrays from an NPY directory (images.npy, etc.), an HDF5 file, or a Zarr store. Dataset names are the keys below. Images, variances, and PSFs are required; all other arrays and metadata.json are optional.

Key Shape Meaning
images (batch, y, x, frame) Registered image pixels
variances (batch, y, x, frame) Input pixel variances
psfs (psf_y, psf_x, channel, frame) Point-source images, including calibrated kernels
masks (batch, y, x, frame) Optional boolean mask: True includes the pixel; absent means all pixels
exp_times (frame,) Optional positive frame scales; absent means ones
sky (batch, y, x, frame) Optional sky estimates retained with the bundle
mcgauss (frame, kernel_y, kernel_x) Optional Gaussian kernels retained with the bundle

PSF calibration and reconstruction require one PSF channel. The evaluation workflow requires one image batch. All frame axes must agree. Reconstruction divides images by exp_times and variances by its square. Supply images with the intended background convention: sky is retained and is not automatically subtracted. The experimental background_model: scalar_per_exposure fits a constant background for each frame.

The default mask.mode: provided uses the bundle's good-pixel mask; mask.mode: none ignores it. Instrument-specific flags are converted by the adapter before constructing a bundle. The legacy twelve-plane camera schema can be converted with cuphoton.ximage.adapters.legacy.legacy_masks_to_good_pixels, whose conservative and non_conservative policies and dilation_factor apply only during that conversion. The Rubin adapter directly converts its bad pixel flags to boolean good-pixel masks.

ROI centers are (row, column) in the source image. VALID padding requires real input pixels around the ROI: ceil((psf_size // 2) / srf) on each side. A region without this context is rejected. Saved raw reconstructions include convolution context; the evaluation workflow selects the requested ROI. Exposure selection applies to every array's exposure axis; saved calibrated kernels record their exposure selection and are checked before use. A saved PSF file without selection metadata is treated as being in caller-provided exposure order.

Runs contain the resolved config.yaml, a summary.json with paths and execution metadata, numeric products under artifacts/, and traces under trace/. Reconstruction produces a host NumPy image in exposure-time-normalized units. It does not provide propagated output variance or an effective output PSF. Interpolation and reconstruction can correlate pixels; input variances are not an output uncertainty model. WCS and calibration remain caller-managed. See xImage for examples and evaluation limitations.