Repository navigation
Open
Description
As discussed in the TAB meeting and requested by Oliver, this issue covers the zarr-python 3 support and tracks the work across the ecosystem. hdmf-zarr 0.14 moved to zarr-python 3, and the rest of the stack has to follow so that NWB files stored as Zarr are written, inspected and uploaded with it.
- hdmf-zarr: 0.14.0 requires zarr-python 3.4 and Python 3.12, see [Feature]: Support zarr-python v3 hdmf-dev/hdmf-zarr#202.
- spikeinterface: 0.106.0 requires zarr-python 3 on every Python version, after Support zarr v3 SpikeInterface/spikeinterface#4260.
- roiextractors: 0.10.0 works with zarr-python 3, after Keep roiextractors working when zarr-python v3 is installed catalystneuro/roiextractors#608, Remove the zarr 2 pins catalystneuro/roiextractors#616 and Read Zarr-backed NWB files in the NWB extractors catalystneuro/roiextractors#618.
- pynwb and hdmf: their
zarrextras still pinzarr<3. The pull requests and the order to review them are in Drop the Zarr 2 dependency from thezarrextra #2279.- Select the legacy Zarr reader in read_nwb #2280 and Update the Zarr extra to use Zarr 3 #2281, which need each other for CI to pass
- Validate against the cached namespaces in
pynwb.validate(io=...)#2268, so thatpynwb.validateworks for Zarr files - Write zarr v3 arrays of strings with HDF5IO and wrap zarr v3 arrays with H5DataIO hdmf-dev/hdmf#1618 and Move the
zarrextra to Zarr 3 hdmf-dev/hdmf#1619 - a release of each
- nwbinspector: Validate through the open io so Zarr files can be inspected by path nwbinspector#770 after the pynwb release, then a release. The compression checks for zarr-python 3 are merged in Support hdmf-zarr 0.14 and zarr-python 3 in the compression checks nwbinspector#767.
- neuroconv: done on main with Stop installing
hdmf_zarron Python 3.11 catalystneuro/neuroconv#2231, Move to SpikeInterface 0.106.0 and lift theneoandzarrpins catalystneuro/neuroconv#2239 and the sharding in Zarr v3 backend configuration catalystneuro/neuroconv#1749. It needs a release. - dandi-cli: Validate Zarr v3 with zarr-python and make zarr an optional extra dandi/dandi-cli#1971 validates Zarr format 2 and 3 with zarr-python 3, and Support NWB files stored as Zarr (
.nwb.zarr) dandi/dandi-cli#1994 adds NWB validation for.nwb.zarrand fixes the depth limit in Zarr depth limit rejects NWB chunk paths dandi/dandi-cli#1919. - dandi-archive: publishing dandisets with Zarr assets, Track zarr upload URLS dandi/dandi-archive#3018 and then Allow publishing dandisets which contain zarrs dandi/dandi-archive#3006.
The pynwb part comes first, as hdmf and nwbinspector wait on its release.
Metadata
Metadata
Assignees
Labels
No labels
Activity