Skip to content

chore: hypothesis property tests for fill-value handling#990

Open
maxrjones wants to merge 2 commits into
zarr-developers:mainfrom
maxrjones:test/fill-value-hypotheses
Open

chore: hypothesis property tests for fill-value handling#990
maxrjones wants to merge 2 commits into
zarr-developers:mainfrom
maxrjones:test/fill-value-hypotheses

Conversation

@maxrjones
Copy link
Copy Markdown
Member

@maxrjones maxrjones commented May 11, 2026

What this PR adds

Property-based test coverage for the fill-sentinel-values issue cluster, plus a zarr-spec compliance test that asserts virtualizarr's correctness independently of xarray's HDF5-style _FillValue encoding.

Three test modules under virtualizarr/tests/test_parsers/:

  • test_zarr_spec_compliance.py — primary correctness check. Asserts ZarrParser produces metadata matching zarr-python's view of the source, no xarray in the loop. 7/7 passing.
  • test_hdf/test_hdf_fill_value_equivalence.py — HDFParser interop check against xarray+h5netcdf. Hypothesis-driven dtype × fill matrix + curated @examples.
  • test_zarr_fill_value_equivalence.py — ZarrParser interop check against xr.open_zarr.

Plus _fill_value_common.py — shared infrastructure (sentinels, hypothesis profiles, two-layer assertion helpers with attribution).

Hypothesis is added to the dev dependency group with >=6.100. Module-level pytestmark = pytest.mark.hypothesis_tests allows skipping via pytest -m "not hypothesis_tests". VIRTUALIZARR_HYPOTHESIS_PROFILE=ci shrinks the random-draw count for fast CI runs.

What this PR doesn't change

No production code. No parser fixes. This is test infrastructure only. Failing tests are the to-fix list, tracked in https://github.com/NASA-IMPACT/veda-odd#371.

Why it matters

Test failures are attributed by category, so reviewers and maintainers can tell what to fix and where:

Attribution Meaning
BothEnginesFailedIdenticallyError Both engines fail identically — upstream issue (e.g. xarray's FillValueCoder, tracked at pydata/xarray#11332), not virtualizarr
observed (virtualizarr) failed; reference succeeded Virtualizarr-specific bug — fix lives in this repo
Plain assert_identical diff Real correctness gap — investigate

This separates virtualizarr-side bugs from upstream xarray gaps cleanly. For example, #989's _FillValue=0.0-as-JSON-number case shows up as BothEnginesFailedIdenticallyError in the equivalence module — and the spec-compliance module passes on the same fixture, proving the gap is upstream (pydata/xarray#11332).

Current results

31 passed, 10 failed in ~6s (ci profile)

The 10 failures decompose as:

  • 2 TestCompoundDtype — parser-side TypeError at hdf.py:364 (new finding; tracked in roadmap as Cluster A).
  • 5 TestBasicEquivalence::test_equivalence_curated[*] (HDF) — string-dtype _FillValue. Improved attribution surfaced this is a parser-side encode error, not the downstream decode error previously documented.
  • 1 TestBasicEquivalence::test_equivalence_random (HDF) — random draws hitting the same string-dtype cluster.
  • 1 TestBasicEquivalence::test_equivalence_curated[float64-nan-fill] (Zarr) — upstream xarray decode gap, pydata/xarray#11332 (also user-reported as FillValue decoding unexpected fill_value: null (on version 2.6.1) #989).
  • 1 TestBasicEquivalence::test_equivalence_random (Zarr) — random draws hitting the same xarray decode gap.

Each failure points at one of the clusters in the roadmap.

Notable findings surfaced while building this

  • ZarrParser get_metadata fails for LocalStore Zarr with dtype uint8 #811 doesn't reproduce in current main. The curated example for the original bug report (uint8 with no explicit fill_value) opens cleanly through both engines. Worth a separate verification but suggests it was silently fixed.
  • vlen-string _FillValue is virtualizarr-specific, not upstream. The improved attribution showed the failure is an encode error at parser manifest construction, not a decode error in xarray.

How to run

pixi run -e test pytest virtualizarr/tests/test_parsers/test_zarr_spec_compliance.py
pixi run -e test pytest virtualizarr/tests/test_parsers/test_hdf/test_hdf_fill_value_equivalence.py
pixi run -e test pytest virtualizarr/tests/test_parsers/test_zarr_fill_value_equivalence.py

# Skip hypothesis-driven tests (faster everyday CI):
pixi run -e test pytest -m "not hypothesis_tests"

# Smaller hypothesis matrix:
VIRTUALIZARR_HYPOTHESIS_PROFILE=ci pixi run -e test pytest ...

Refs

Acceptance criteria:

  • Closes #xxxx
  • Tests added
  • Tests passing
  • No test coverage regression
  • Full type hint coverage
  • Changes are documented in docs/releases.md
  • New functions/methods are listed in an appropriate *.md file under docs/api
  • New functionality has documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant