Write ISTP BIN_LOCATION attributes as CDF_REAL4#330
Merged
Conversation
SPDF ISTP validation expects BIN_LOCATION variable attributes to be
stored as REAL*4/CDF_REAL4. xarray_to_cdf previously relied on generic
attribute type inference, which wrote integer values such as 0 as
CDF_INT8 and floating-point values such as 0.5 as CDF_DOUBLE.
This caused SPDF validation errors like:
Bin_location has a data type of CDF_INT8. It should be CDF_REAL4.
Bin_location has a data type of CDF_DOUBLE. It should be CDF_REAL4.
When writing with istp=True, explicitly serialize BIN_LOCATION as
[np.float32(value), "CDF_REAL4"]. Preserve the existing inferred
attribute typing behavior for non-ISTP writes.
Add regression tests covering ISTP BIN_LOCATION output for integer and
fractional values, and non-ISTP behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Write ISTP
BIN_LOCATIONvariable attributes asCDF_REAL4when usingxarray_to_cdf(..., istp=True).SPDF ISTP validation expects
BIN_LOCATIONto be stored as REAL*4 /CDF_REAL4. Previously, cdflib relied on generic attribute type inference, so values like0were written asCDF_INT8and values like0.5were written asCDF_DOUBLE.That produced SPDF validation errors such as:
Details
BIN_LOCATIONis an ISTP variable attribute describing where a timestamp falls within a measurement bin. The SPDF ISTP guide defines it as a REAL*4 attribute:https://spdf.gsfc.nasa.gov/istp_guide/vattributes.html#BIN_LOCATION
This PR adds a narrow ISTP-mode special case in
xarray_to_cdfsoBIN_LOCATIONis serialized as:The behavior only applies when
istp=True. Non-ISTP writes continue to use the existing inferred attribute type behavior.Validation
Added regression tests for:
BIN_LOCATIONvalues written asCDF_REAL4in ISTP mode.BIN_LOCATIONvalues written asCDF_REAL4in ISTP mode.Targeted test run:
pytest tests/test_xarray_reader_writer.py -k 'bin_location or compression_skip_vars or skips_epoch_compression'Result:
Validated against a real IMAP CDF that previously triggered the SPDF checker error. Before rewriting:
After rewriting through this branch:
The SPDF Java checker no longer reports the
Bin_location has a data type ... should be CDF_REAL4errors on the rewritten file.Related IMAP validation context:
IMAP-Science-Operations-Center/imap_processing#2577