[Roadmap] Fill / sentinel-value handling in VirtualiZarr

[maxrjones]

Tracking the work to resolve the 15 open issues labelled fill-sentinel-values plus the related upstream gaps in xarray's FillValueCoder.

Framing

VirtualiZarr correctness is measured against the Zarr spec, not against xarray equivalence. Several recent reports (zarr-developers/VirtualiZarr#989, zarr-developers/VirtualiZarr#485, zarr-developers/VirtualiZarr#628) hit xarray's FillValueCoder.decode failing on JSON-native scalars in zarr metadata — the parser is producing spec-compliant output that xarray's HDF5-style coder can't consume. Tracked upstream at pydata/xarray#11332. Those are upstream xarray issues, not virtualizarr bugs.

The property-test infrastructure added in zarr-developers/VirtualiZarr#990 distinguishes failure categories:

Failure shape	Attribution	Action
Both engines fail identically	Upstream xarray / zarr-python	Track upstream — no virtualizarr PR
Observed (virtualizarr) fails; reference ok	Virtualizarr-specific bug	Fix in VirtualiZarr
Both succeed but differ	Real correctness gap	Fix in VirtualiZarr

Root-cause clusters

The 15 open issues plus two new findings collapse into 8 underlying problems. Each issue is listed under its primary cluster; cross-cluster cascades are noted inline.

A. Parser crashes during fill extraction — local parser fixes, ~5-20 lines each.

• Failure to read HDF5 data with data arrays of dtype='<U1' zarr-developers/VirtualiZarr#878 — PR Correctly handle HDF5 fillvalue for string dtype arrays. zarr-developers/VirtualiZarr#988 fixes the parser crash; post-fix, the same dataset hits cluster C.
• ZarrParser get_metadata fails for LocalStore Zarr with dtype uint8 zarr-developers/VirtualiZarr#811 — does not currently reproduce on main; confirm before closing.
• (new finding, no GH issue yet) structured-dtype _FillValue raises TypeError at _extract_attrs in parsers/hdf/hdf.py:364 due to a v == "DIMENSION_SCALE" comparison against a void scalar.

B. HDF parser _FillValue encoding gaps — local parser fix; emit base64 for kind S per docs/custom_parsers.md.

• _FillValue / NetCDF parser encoding error zarr-developers/VirtualiZarr#785
• Issue opening some virtual icechunk datasets due to _FillValue zarr-developers/VirtualiZarr#628 — same root cause as #785; symptom is xarray's base64 assertion downstream.

C. xarray FillValueCoder lacking branches — upstream, tracked at pydata/xarray#11332. Out of virtualizarr scope.

• Broken example notebook, error decoding FillValue zarr-developers/VirtualiZarr#485
• FillValue decoding unexpected fill_value: null (on version 2.6.1) zarr-developers/VirtualiZarr#989 — also a cluster-D root-cause.
• Failure to read HDF5 data with data arrays of dtype='<U1' zarr-developers/VirtualiZarr#878 (post-PR-#988) — once the parser stops crashing, the same dataset hits this cluster.

D. h5py default fillvalue propagated indiscriminately — parser fix: use dataset.id.get_create_plist().fill_value_defined() to skip propagating defaults. Fixing D removes the cascade into C for vlen-string-without-_FillValue cases.

E. Cross-parser inconsistency — different parsers produce different fill defaults / metadata for the same source. Architectural fix.

• Inconsistent fill_val between ZArray constructor and from_kerchunk_refs() zarr-developers/VirtualiZarr#287
• Datetime coordinate missing single timestep (NaT) when data is loaded zarr-developers/VirtualiZarr#352
• netcdf3 parser fails to preserve _FillValue attribute zarr-developers/VirtualiZarr#982
• DMRParser fails to parse time values correctly for netcdf4 file zarr-developers/VirtualiZarr#904

F. Writer-side fill semantics — writer-API design questions, distinct from parser fixes.

• Nan values when saving parq files with virtualize.to_kerchunk() zarr-developers/VirtualiZarr#339
• Using ArrayV3Metadata, kerchunk fill_value will always be coerced to a value, even though null is a valid fill_value for Zarr V2 zarr-developers/VirtualiZarr#478
• Padding arrays with NaNs zarr-developers/VirtualiZarr#22 — feature request, not a bug fix.

G. Attribute serialization fidelity — zarr v3 metadata is JSON; lossy for some attribute shapes.

• How to handle non-JSON serializable attributes? zarr-developers/VirtualiZarr#715
• (new finding, no GH issue yet) scalar attribute dtype widening across the JSON round-trip (float32 → float64).

H. Cross-cutting encoding model — meta-discussion; closes via the totality of the other clusters.

• How to handle encoding zarr-developers/VirtualiZarr#68

Phases

1. Local parser fixes (low risk): Correctly handle HDF5 fillvalue for string dtype arrays. zarr-developers/VirtualiZarr#988, structured-dtype guard at _extract_attrs, ZarrParser default lookup, S-dtype base64 encoding. ~50 lines total across several small PRs.
2. Upstream advocacy (parallel track, no virtualizarr PRs): pydata/xarray#11332 tracks the FillValueCoder JSON-native-scalar gap; engage with zarr-specs#351, zarr-extensions#33.
3. fill_value_defined() distinction: stop propagating h5py-default fills to zarr storage.
4. Cross-parser consistency: extend the property-test suite to Kerchunk, TIFF; port HDFParser conventions; document the contract in docs/custom_parsers.md.
5. Writer-side round-trips: Icechunk / Kerchunk writers preserve fill semantics.
6. Attribute fidelity: policy for non-JSON-serializable attrs (How to handle non-JSON serializable attributes? zarr-developers/VirtualiZarr#715), scalar dtype preservation across JSON metadata.

Phase 2 runs in parallel with all others. BothEnginesFailedIdenticallyError cases auto-resolve when xarray ships the fix; no virtualizarr code change required.

Status

• ✅ Property-test infrastructure: chore: hypothesis property tests for fill-value handling zarr-developers/VirtualiZarr#990
• ✅ ZarrParser get_metadata fails for LocalStore Zarr with dtype uint8 zarr-developers/VirtualiZarr#811 confirmed not reproducing in current main
• ⏳ PR Correctly handle HDF5 fillvalue for string dtype arrays. zarr-developers/VirtualiZarr#988 in review (Phase 1 partial: HDF parser _get_fill_value + StringDType + kind-skip)
• ⏳ Upstream tracker: pydata/xarray#11332 (Phase 2)
• Open: Phases 3, 4, 5, 6

References

• Property-test PR: chore: hypothesis property tests for fill-value handling zarr-developers/VirtualiZarr#990
• Zarr spec discussions: zarr-specs#351, zarr-extensions#33 (Optional[T] dtype)
• Related: PR Correctly handle HDF5 fillvalue for string dtype arrays. zarr-developers/VirtualiZarr#988 (fix/problem_fillvalues)
• Issue zarr-developers/VirtualiZarr#347 (meta-request for hypothesis-driven HDF tests — addressed by the property-test PR)