Motivation
A long GPEC run that crashes in the ODE integration currently loses everything, including the equilibrium that finished minutes earlier. Writing gpec.h5 incrementally per module — Equilibrium, then ForceFreeStates, then the downstream stages — means a crash leaves behind whatever completed. This is much more natural in HDF5 than it was in the Fortran netCDF interface, which is worth taking advantage of.
Current state — already partly done
Staged writing already exists at module granularity:
| Stage |
Open mode |
When it hits disk |
| Equilibrium + ForceFreeStates + Galerkin |
"w" (truncates) |
after the entire FFS stage |
| PerturbedEquilibrium |
"cw" |
after PE |
Coil snapshot (_write_coil_snapshot!) |
"r+" |
after PE |
| KineticForces |
"cw" |
after KF |
| Tearing |
"r+" ("w" if absent) |
last |
The Tearing stage is already the model for what we want: _run_slayer_stage wraps the whole thing in a try/catch whose comment states that a failure "must not discard the equilibrium / stability / PE results already computed."
The actual gap
Everything in row 1 is computed before anything is written. The equilibrium solve, the ODE integration, free_run!, the Δ′ BVP and the Galerkin solve all complete, and only then does write_outputs_to_HDF5 open the file with "w". So the longest, most crash-prone part of the run is also the part with nothing on disk behind it.
Proposed change
Split the stage-1 writer so Info/, Input/ and Equilibrium/ are written immediately after the equilibrium solve, and ForceFreeStates appends with "r+". The isfile(h5_path) ? "r+" : "w" idiom at GeneralizedPerturbedEquilibrium.jl:543 is already the pattern to follow.
What makes this more than a mechanical split
1. Partial files become readable-but-incomplete, and Rerun.jl will start from them.
Today a gpec.h5 either exists and is complete or doesn't exist. Once writes are incremental, a crashed run leaves a file containing a valid Input/gpec_toml_raw — which is exactly what the rerun path reads. A rerun would silently start from a truncated file.
This needs a completion marker: a root attribute (run_complete, or better a stages_written list so a reader knows which modules are trustworthy), plus readers taught to check it. #364 introduces Utilities.HDF5Annotations.write_root_attrs!, which is the natural home for this — so this work should be based on #364, not on #363.
2. There are no flush calls anywhere in src/.
Incremental writing without an explicit HDF5.flush at each stage boundary risks leaving a corrupt file on crash, which is strictly worse than no file — a corrupt HDF5 can't be opened at all, whereas today's all-or-nothing write at least fails cleanly.
3. The bit-for-bit rerun round-trip test needs re-verifying.
HDF5 orders groups by name rather than creation order by default, so changing write order should be invisible to test/runtests_rerun_from_h5.jl — but that should be confirmed, not assumed.
4. Idempotency. Any stage that can run twice needs the delete_object-then-recreate guard write_slayer_hdf5! already uses.
Sequencing
Should land after the #226 stack (#363 → #364 → #368), based on #364 for the root-attributes machinery. Deliberately not folded into #363: that PR is a pure rename whose safety evidence is "zero regression-harness movement," and a write-ordering change is a behaviour change needing different evidence (crash injection, partial-file reads). Bolting it on would void the three approvals already on it.
Origin
Raised by @jhalpern30 in review of #363 ("I'm wondering if it would simplify some code elsewhere to write other parts of the outputs incrementally too?"), refined by @d-burg into the per-module eq/FFS/tearing form described above.
Motivation
A long GPEC run that crashes in the ODE integration currently loses everything, including the equilibrium that finished minutes earlier. Writing
gpec.h5incrementally per module — Equilibrium, then ForceFreeStates, then the downstream stages — means a crash leaves behind whatever completed. This is much more natural in HDF5 than it was in the Fortran netCDF interface, which is worth taking advantage of.Current state — already partly done
Staged writing already exists at module granularity:
"w"(truncates)"cw"_write_coil_snapshot!)"r+""cw""r+"("w"if absent)The Tearing stage is already the model for what we want:
_run_slayer_stagewraps the whole thing in a try/catch whose comment states that a failure "must not discard the equilibrium / stability / PE results already computed."The actual gap
Everything in row 1 is computed before anything is written. The equilibrium solve, the ODE integration,
free_run!, the Δ′ BVP and the Galerkin solve all complete, and only then doeswrite_outputs_to_HDF5open the file with"w". So the longest, most crash-prone part of the run is also the part with nothing on disk behind it.Proposed change
Split the stage-1 writer so
Info/,Input/andEquilibrium/are written immediately after the equilibrium solve, and ForceFreeStates appends with"r+". Theisfile(h5_path) ? "r+" : "w"idiom atGeneralizedPerturbedEquilibrium.jl:543is already the pattern to follow.What makes this more than a mechanical split
1. Partial files become readable-but-incomplete, and
Rerun.jlwill start from them.Today a
gpec.h5either exists and is complete or doesn't exist. Once writes are incremental, a crashed run leaves a file containing a validInput/gpec_toml_raw— which is exactly what the rerun path reads. A rerun would silently start from a truncated file.This needs a completion marker: a root attribute (
run_complete, or better astages_writtenlist so a reader knows which modules are trustworthy), plus readers taught to check it. #364 introducesUtilities.HDF5Annotations.write_root_attrs!, which is the natural home for this — so this work should be based on #364, not on #363.2. There are no
flushcalls anywhere insrc/.Incremental writing without an explicit
HDF5.flushat each stage boundary risks leaving a corrupt file on crash, which is strictly worse than no file — a corrupt HDF5 can't be opened at all, whereas today's all-or-nothing write at least fails cleanly.3. The bit-for-bit rerun round-trip test needs re-verifying.
HDF5 orders groups by name rather than creation order by default, so changing write order should be invisible to
test/runtests_rerun_from_h5.jl— but that should be confirmed, not assumed.4. Idempotency. Any stage that can run twice needs the
delete_object-then-recreate guardwrite_slayer_hdf5!already uses.Sequencing
Should land after the #226 stack (#363 → #364 → #368), based on #364 for the root-attributes machinery. Deliberately not folded into #363: that PR is a pure rename whose safety evidence is "zero regression-harness movement," and a write-ordering change is a behaviour change needing different evidence (crash injection, partial-file reads). Bolting it on would void the three approvals already on it.
Origin
Raised by @jhalpern30 in review of #363 ("I'm wondering if it would simplify some code elsewhere to write other parts of the outputs incrementally too?"), refined by @d-burg into the per-module eq/FFS/tearing form described above.