Proposed HDF5 input format and draft implementation

[The9Cat]

Summary of changes

• Proposed and implement and HDF5 schema for body input data
• Provide a converter to read ascii and emit compressed HDF5 and vice versa
• Implement a body reader for initializing EXP
• This is documented in the Doxygen header for Component. Also, the PR HDF5 documentation EXP-docs#42 include a full write-up and discussion of the HDF5 initialization and the new HDF5 phase-space file format that we currently recommend instead of the older native PSP format.

History and intent

This started as a initial body file compressor. Using HDF5 to exploit the intrinsic stuctural similarities in a body file seemed like a good way to go. After completing the standalone routine, I realized that I was more than half way to an implementation of a reader for the Component class.

This is low priority and does not fix any bugs. But the ugly and inefficient ASCII tables should probably be replaced by HDF5 now that most of EXP is reading/writing using HDF5.

Implementation details

• The HDF5 schema takes the following form:
  /
  ├── Attributes
  │ ├── num_particles (int)
  │ ├── num_aux_ints (int)
  │ └── num_aux_floats (int)
  └── particles/ (Group)
  ├── index (Dataset, unsigned long)
  ├── m (Dataset, float/double)
  ├── x, y, z (Datasets, float/double)
  ├── u, v, w (Datasets, float/double)
  ├── aux_int_0, aux_int_1, ... (Datasets, int)
  └── aux_float_0, aux_float_1, ... (Datasets, float/double)
• The design goal is keeping like data together and using chunking, bit shuffling, and compression at the HDF5 level to achieve up to a factor of 5 savings in file size for float32 and a factor of 2.5 savings for float64.
• The index and aux_int_*, aux_float_* fields are optional.
• This schema is a precise mapping of the current body file data fields, except for the inverse ordering. One could make this more Gadget-like, but that would muddy the clarity here.
• I separated pos into x, y, z (and vel into u, v, w) to simply the std::variant coding to allow both float and double arrays. It is possible that consolidating into 2d arrays could be more efficient. But this is easier to code.
• The precision of the float data type is deduced for each dataset independently. One could mix precision between m, x, y, z, u, v, w if desired.
• The converter routine hdf5bods was initially designed to compress body files and it achieves that goal. It is implemented with OpenMP which gives some modest performance gains (but not MPI). A $10^6$ particle body files requires 10 seconds for conversion on a laptop.
• The Component class checks whether the specified body file is HDF5 and reverts to the original ascii method if not; so all of the prior behavior is retained for compatibility.
• If the optional index field (unsigned long) exists, it will populate the EXP Particle.indx field. Similarly, the stand-along conversion routine has can read and write the index field for consistency with the original ASCII body table format.
• The stand-alone hdf5bods converter has support for multiple HDF5 compression filters for testing. The default GZip has the highest compression ratio but it also takes the most CPU time.

Tests so far

• Code compiles
• The converter code, hdf5bods, has built in round-trip verification which passes
• Checked that exp can read the new HDF5 files and produce the same run for a simple case
• Review HDF5 schema
• Test initial condition body file with int and float auxiliary attributes
• Test initial condition body file with index values

[Copilot]

Pull request overview

This PR introduces an HDF5-based format for EXP particle “body” input data, along with a converter utility and initial integration into Component so EXP can initialize from either ASCII or HDF5 while preserving backward compatibility.

Changes:

• Added hdf5bods utility to convert EXP ASCII body files to compressed HDF5 (and back), with optional round-trip verification.
• Updated Component initialization to detect HDF5 body files and read/distribute particles from HDF5 when applicable (fallback to existing ASCII reader otherwise).
• Bumped project/documentation version to 7.11.0.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
utils/PhaseSpace/hdf5bods.cc	New ASCII↔HDF5 body-file converter using HighFive + OpenMP; writes proposed schema attributes/datasets.
utils/PhaseSpace/CMakeLists.txt	Builds/installs the new hdf5bods executable.
src/Component.H	Adds declarations for HDF5 initialization/read helpers.
src/Component.cc	Adds HDF5 detection and HDF5-based body ingest + MPI distribution.
include/ScopeTimer.H	New simple scope-based timing helper used by the converter.
doc/exp.cfg.breathe	Version bump for generated docs.
doc/exp.cfg	Version bump for generated docs.
CMakeLists.txt	Project version bump to 7.11.0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

[michael-petersen]

Thanks for developing this, @The9Cat ! I've tested the compile.

I wonder if we can get Copilot to help write the minimal test to read files with attributes / indices?

[michael-petersen]

I've rebased this to devel as well.

[The9Cat]

It would be interesting to see what copilot would do with that request!

On indexes and attributes, I realize that we can use psp2ascii to generate an ascii file with indexes and attributes from a PSP checkpoint file from an old DSMC run, which I did and ran the roundtrip test successfully and checked the generated HDF5 with h5dump. I also generated an input file with fake attributes and confirmed that EXP correctly read the input file and generated the same run output, even though those attributes were not being used directly. The roundtrip check in hdf5bods passed and I spot checked that that the auxiliary field values in the HDF5 using h5dump So that's enough of a check, perhaps? I'm going to tentatively mark those tests "done".

I fixed a minor bug in hdf5bods in the process: traling spaces in the header line in the ASCII input was being interpreted rather than ignored. I'll let copilot have an another go at the PR.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

[michael-petersen]

I haven't as extensive testing as you have, but I have checked that I can do a bods -> hdf5 -> bods, and that I can start a simulation from hdf5. As far as I can tell, this is self-contained and good to go -- and a very welcome addition!

[The9Cat]

Okay, as soon as the CI run from the last merge fixing the version (induced by the previous merge to devel), I'm going to merge this one as well. Yes, I can't see reason to hold off.