@tomwhite's old prototype for traversing HDF5 files (via NetCDF) using Spark is still in the singlecell module
It would be good to add parallelization options to the convert CLI; most likely a Spark-specific code-path will be necessary for that, though in general it would be great to plumb parallelization through cats.Traverse / using Cats' Parallel. Futures and cloud-functions might be workable there.
@tomwhite's old prototype for traversing HDF5 files (via NetCDF) using Spark is still in the
singlecellmoduleIt would be good to add parallelization options to the
convertCLI; most likely a Spark-specific code-path will be necessary for that, though in general it would be great to plumb parallelization throughcats.Traverse/ using Cats' Parallel.Futures and cloud-functions might be workable there.