Skip to content

Atomic checkpoint writes (fixes #13) - #74

Merged
CanReader merged 2 commits into
CanReader:masterfrom
sharmis-tp:fix/atomic-checkpoint-writes
Nov 12, 2026
Merged

CanReader merged 2 commits into
CanReader:masterfrom
sharmis-tp:fix/atomic-checkpoint-writes

Conversation

@sharmis-tp

Copy link
Copy Markdown
Contributor

Summary

  • Extracted a shared write_atomic(path, write_fn) helper in src/serialize/atomic.rs
  • save_tensors (fdl) and save_safetensors now write to {path}.partial.{pid}, sync_all(), then rename — a crash mid-write never destroys the existing checkpoint
  • Temp file is cleaned up on write error so it never accumulates
  • Added 4 robustness tests: truncated fdl/safetensors detection, temp cleanup, and round-trip through the atomic path

Closes #13

Test plan

  • cargo test — all 186 tests pass (including 4 new robustness tests)
  • Truncated fdl checkpoint does not destroy the good copy
  • Truncated safetensors does not destroy the good copy
  • Atomic write cleans up temp on fdl save error
  • save_tensors round-trips through atomic write

@CanReader
CanReader force-pushed the fix/atomic-checkpoint-writes branch from 3738c22 to 4703f92 Compare November 12, 2026 08:47
@CanReader
CanReader merged commit 8a13d2f into CanReader:master Nov 12, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Checkpoint writes are not atomic, a crash mid save destroys the previous one

2 participants