Stream precomputed velocity fields for memory-efficient scalar transport

[vulpes922]

Problem

Scalar transport with precomputed velocity is currently impractical for many cardiovascular simulations with high temporal resolution.

A typical Fontan CFD case uses approximately 1,000 time steps per cardiac cycle. To run scalar transport from a precomputed velocity field, we currently need to:

1. Save the velocity solution for every time step as a separate .vtu file.
2. Combine the 1,000 files into a single input for the precomputed-velocity scalar transport simulation.
3. Load the full velocity time history before or during the scalar transport solve.

For a representative case with approximately 2 million mesh cells and 1,000 time steps per cycle, this requires roughly 350 GB of memory. This makes the workflow inaccessible on many clusters and prevents running scalar transport separately from the flow simulation at the desired temporal resolution.

The memory cost scales with both mesh size and the number of saved time steps, even though the scalar transport solver only needs the velocity field at the current time (and possibly adjacent time points for temporal interpolation).

Solution

Implement an out-of-core / streamed precomputed-velocity input path for scalar transport, so that the solver does not need to keep the complete velocity time series in memory.

The preferred behavior would be to load only the velocity data required for the current scalar-transport time step, retain a small cache for temporal interpolation, and release data that are no longer needed.

Possible implementation options include:

1. Lazy loading of individual VTU files

   • Accept a time-indexed list or .pvd collection of velocity .vtu files.
   • Load one time step at a time, or the two neighboring time steps required for linear temporal interpolation.
   • Keep only a configurable small number of snapshots in memory.

2. Chunked VTU input

   • Store the velocity history in smaller groups of time steps rather than one fully combined file.
   • Load and discard each group as the scalar transport simulation progresses.
   • This could reduce repeated file-open overhead while keeping memory bounded.

3. HDF5-based time-series format

   • Add support for a format such as HDF5 (potentially with XDMF metadata) that supports efficient partial reads/hyperslabs.
   • Store the static mesh once and the velocity field as a time-dependent dataset.
   • This may be the more scalable long-term solution, particularly for MPI runs, because each rank can read only its local portion of the required time step.

Additional context

Desired capabilities

• Support approximately 1,000 velocity snapshots per cardiac cycle without loading the full cycle into memory.
• Read only the current velocity snapshot(s) needed by scalar transport.
• Support temporal interpolation between velocity snapshots.
• Support repeating periodic velocity data over multiple cardiac cycles.
• Maintain compatibility with distributed-memory/MPI simulations.
• Allow the number of cached snapshots or chunk size to be configurable.
• Retain the current full-load workflow as a backward-compatible option, if appropriate.

Code of Conduct

• I agree to follow this project's Code of Conduct and Contributing Guidelines

[ktbolt]

We don't want to save this velocity data as VTK VTU files because

1. Mesh data is duplicated for each VTU file and is not really needed
2. All the data from the VTU is read

We will need to figure out if HDF5 is suitable for the simple block structure of the velocity data and how useful its parallel I/O and compression filters might be.

We might could evaluate VTKHDF file format for this.

[vulpes922]

I also found two immediate improvements in the existing path. The full velocity history is stored in mesh.Ys as a 3D node by time array, so memory grows linearly with the number of snapshots. In addition, the interpolation loop currently copies the mesh object at every time step. Because the mesh owns Ys, this can duplicate the full velocity history temporarily. Changing that local mesh variable to a reference will remove the unnecessary copy.

I’ll apply these small changes in my local branch and measure their effect on memory usage as a temporary fix.