Repository navigation
Conversation
|
No actionable comments were generated in the recent review. Walkthrough
Severity of issue fixed: Medium Merge Risk:
|
melo-gonzo
left a comment
There was a problem hiding this comment.
Thanks for the fix and the clear write-up. The change is right. float32 has a 24-bit significand, so it holds every 8- and 16-bit integer but not every 32-bit one. The new rule matches Astropy and cuphoton.core.fits_io._logical_dtype, which returns float64 for scaled data above 16 bits. All three promotion sites now agree: GpuImageReader.read, _probe_output_specs and _consume_image. No other itemsize <= 4 promotion is left in src/.
The new tests check the right values. 2**31 + 2**24 + 1 lies between float32 values, which are 256 apart at that size, so the old float32 path cannot pass the new _consume_image case. The new i4 case in test_probe_output_specs_promotes_scaled_integer_images expects f8 where the old code gives f4.
Considerations, none blocking:
- I checked this by reading the code and tests. I did not run them in this review, and CI has not run yet, because workflows from outside contributors wait for a maintainer to approve them.
- Scaled 32-bit images now come back as float64 instead of float32. That doubles their device memory and changes the result dtype for callers. A CHANGELOG line would help, as you offered.
- As you note, CI does not run
test_image_reader_scales_int32_images_without_rounding, because the GPU job'stests/xdrselection skipstest_reader.py. GpuCompImageReadernot applying BZERO/BSCALE is worth its own issue.
|
Thanks for the review. I added the CHANGELOG entry in 080ddef, as a new I left the CI selection as it is. Adding |
|
Thanks. The entry at 080ddef is accurate, and a separate change for the CI selection is fine. Please open the |
GpuImageReader, batch_to_device and batch_to_device_stream promote a scaled integer image (BSCALE != 1 or BZERO != 0) to float32 whenever the on-disk item size is at most 4 bytes. float32 has a 24-bit significand, so BITPIX = 32 values above 2**24 are rounded. With the FITS unsigned 32-bit convention (BZERO = 2**31), 4294967295 is returned as 4294967296.0, which is outside the uint32 range. Astropy, CFITSIO and cuphoton.core.fits_io._logical_dtype all use float64 for scaled BITPIX = 32. Use float32 only for 8- and 16-bit integers, and add BITPIX = 32 cases to the promotion tests plus a GPU test that compares GpuImageReader with Astropy. Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
Scaled BITPIX = 32 images from GpuImageReader, batch_to_device and batch_to_device_stream now come back as float64 instead of float32, which changes the result dtype and doubles their device memory. Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Include the scaled-int32 reader regression in the existing GPU test selection so its compatibility-mode cases run and cannot silently skip. Signed-off-by: Trent Nelson <trentn@nvidia.com>
374c3ad to
5e4b62b
Compare
Summary
Fixes #189 (
cuphoton.xdr).GpuImageReader.read(), and the output specs ofbatch_to_device/batch_to_device_stream, promote a scaled integer image (BSCALE != 1 or BZERO != 0) to float32 whenever the on-disk item size is at most 4 bytes. float32 has a 24-bit significand, so BITPIX = 32 values above 224 are rounded. With the FITS unsigned 32-bit convention (BZERO = 231), 4294967295 is returned as 4294967296.0, outside the uint32 range. Astropy, CFITSIO andcuphoton.core.fits_io._logical_dtypeall keep these values exact.The three sites (
xdr/reader.py, and_probe_output_specs/_consume_imageinxdr/prefetch.py) now use float32 only for 8- and 16-bit integers and float64 otherwise. 8- and 16-bit images are unchanged. xDR still returns float64 where Astropy returns uint32 for the unsigned convention, but the values are exact; matching uint32 would be a larger change and is not part of this PR.Tests: the existing promotion tests in
tests/xdr/test_prefetch.pyget ani4case each (CPU, fake CuPy, like the existing cases).tests/xdr/test_reader.py::test_image_reader_scales_int32_images_without_roundingwrites uint32 (BZERO = 2**31) and int32 + BZERO = 0.5 files with Astropy, reads them withGpuImageReaderunder KvikIO compatibility mode, and compares with Astropy. The routine GPU job explicitly selects both parametrizations and rejects skipped tests. The regression forces KvikIO compatibility mode, so it requires a CUDA device but does not require GPUDirect Storage.Residual risk: tile-compressed images (
GpuCompImageReader) do not apply BZERO/BSCALE at all; that is a separate question and is not changed here.Validation
x86_64 Linux, Python 3.12.3,
uv sync --locked --extra dev --extra ioatcc16e05. GPU steps on a Blackwell (sm_103) GPU withKVIKIO_COMPAT_MODE=ON, without the native extensions.pre-commit run --all-fileswas not run. The code change touches only Python, whichmake lintcovers. The CHANGELOG commit passes the pre-commit whitespace and end-of-file hooks.Repository hygiene
### xDRin the Unreleased section ofCHANGELOG.md.)This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.