Skip to content

fix: keep local file publication atomic - #590

Draft
wchwawa wants to merge 2 commits into
mainfrom
fix/local-transfer-atomic-publication-20261026
Draft

wchwawa wants to merge 2 commits into
mainfrom
fix/local-transfer-atomic-publication-20261026

Conversation

@wchwawa

@wchwawa wchwawa commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

TLDR

Publish local materialized files only after their staged bytes are complete and synced.

Problem

Follow-up to PR #584. A fallback for filesystems that reject both no-replace rename and hard links created the final name before writing its bytes. No public issue was opened because SECURITY.md directs security reports privately.

Consequences

Another process could read an incomplete output while the write was in progress. A failed write could also leave a partial result or race with another change to that name.

Example

Previously, when both atomic publication methods were unavailable, materialization wrote directly to the output name. It now reports an error without publishing that name.

Solution

Keep the existing no-replace rename and hard-link paths. Remove the direct-create fallback from the CLI and Python adapter so both paths require a complete staged file before publication.

Gain

The successful paths do no extra work. The unsupported-filesystem path cannot expose an incomplete materialized file.

Tradeoffs

A filesystem that supports neither atomic method now rejects materialization. That combination has not been tested on a live NFS or FUSE deployment; there is no measured throughput claim.

Validation

  • At NoKV main 10e142ac (including merged DCP fix fix: reject executable DCP checkpoint formats #585), cargo test --workspace --locked -- --test-threads=2 passed on macOS, including 189 CLI unit tests, 1 CLI integration test, and 48 Python adapter Rust tests.
  • NOKV_TEST_REQUIRE_TORCH=1 ... python -m pytest -q crates/nokv-python/tests/test_torch_unit.py crates/nokv-python/tests/test_torch_dcp_integration.py passed 39 tests with real PyTorch 2.10 on the combined head.
  • cargo clippy -p nokv -p nokv-python --all-targets --locked -- -D warnings, cargo fmt --all -- --check, and git diff --check: passed.
  • Remote exact-head CI and live NFS/FUSE fault testing remain to be completed.

Signed-off-by: wchwawa <wch19961116@gmail.com>
Signed-off-by: wchwawa <wch19961116@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant