Skip to content

Align LocalFileSystem directory creation and multipart cleanup with the upstream drafts - #6

Open
reidkaufmann wants to merge 5 commits into
base-object-store-0132from
align-local-with-upstream-drafts
Open

reidkaufmann wants to merge 5 commits into
base-object-store-0132from
align-local-with-upstream-drafts

Conversation

@reidkaufmann

@reidkaufmann reidkaufmann commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Note

The directory-durability fixes are already in this fork (#3, #5). This PR changes them to the form proposed upstream in apache#890 and apache#891, so the fork can be dropped without a behaviour change once those merge. It also adds apache#891's staging-file cleanup.

Which issue does this PR close?

Mirrors apache#890 (issue apache#888) and apache#891.

Rationale for this change

Upstream review replaced the directory-creation registry used here with a simpler process-wide set. Keeping the fork in the upstream form makes the later switch to an upstream release a no-op.

What changes are included in this PR?

  • The registry and its waiting become one process-wide set of directories whose entries changed but are not fsynced yet. A change records its directory before it runs; a durable write fsyncs the recorded directories on its own path before returning; an entry is removed only after a sync that started after its last change succeeds.
  • Missing directories are created one at a time, so each is recorded.
  • The staging file is removed when publishing fails, but not after a rename that succeeded and whose fsync failed.
  • The two upstream PRs return different error types from rename; this branch uses DurableError::{Change, Sync} with the cleanup rule above. Upstream will hit the same conflict when the second PR merges.

Only src/local.rs changes, and it equals the two upstream PRs merged this way plus v0.13.2's differences from upstream main (checked byte-for-byte). Public API unchanged. Squash on merge.

Are there any user-facing changes?

With fsync enabled: writes no longer wait for another write's directory creation; once a directory's fsync has failed, durable writes beneath it fail until a sync of it succeeds; directory fsync failures after a put, copy or rename are UnableToSyncFile instead of UnableToRenameFile/UnableToCopyFile; a failed multipart complete no longer leaves <dest>#N behind. Without fsync, nothing changes.

local::tests pass locally ten times in a row, and CI passes, including win64 and the emulator tests. A power-cut run of influxdb_pro pinned to this head is queued; influxdb_pro re-pins after merge.

…directory chain

Mirrors upstream draft local-durable-dir-creation (8de2a2d, on apache main
279572e) and replaces the fork's implementation from
#3 and
#5.

`create_parent_dirs` delegates to `create_dir_all_durable(dir, create, sync)`,
which registers the missing directories in `CREATING_DIRS`, creates them one
component at a time while registering each one it tries, and syncs from `dir`
up to the parent of the highest registered directory. A durable write into a
directory being created waits on `DIRS_CHANGED` (parking_lot `Condvar`).

Differences from the fork's code this replaces: the walk starts at `dir` and
stops at the parent of the highest registered directory, not the highest one
this call created, so it also covers a directory an untracked creator made
after the snapshot and runs when this call created nothing; tests inject
`create` and `sync` closures instead of the `#[cfg(test)]` hooks
`BEFORE_CREATE_DIR` and `SYNCED_DIRS`, and assert the exact sync sequence; the
`with_fsync` doc paragraph is the draft's shorter one. The fork's unsynced
records and their tests are removed here and return in the next commit in the
upstream draft's form; the in-between-creation test is dropped, as upstream
did, because the new walk covers that case by construction.
…as unsynced

Mirrors upstream draft local-unsynced-after-failed-creation (fbb5ef0, stacked
on 8de2a2d).

`CREATING_DIRS` becomes `DIRS: Mutex<DirRegistry>` with the `creating` counts
and an `unsynced` set. `create_dir_all_registering` records every registered
directory that exists after its `create_dir` in `CreatingDirs::reached`; unless
the creation synced them, `Drop` moves those records to `unsynced`. A durable
write under an unsynced directory syncs from its parent up to the parent of the
highest unsynced ancestor (`fsync_parent_dir_with`, which tests call with a
recording closure) and then clears the records on that chain.

Differences from the fork's unsynced set
(#3, bounded in
#5): `Drop` does no
I/O; a registered directory whose `create_dir` was never tried is not
recorded, and one deleted after it was reached still is; the 0o555 `put` loop
test is replaced by a direct test of the recorded set;
`wait_for_ancestor_creation` returns early when nothing is registered and no
record exists.
…mplete fails

Mirrors upstream draft local-multipart-staging-cleanup (4577e8a, on apache
main 279572e).

`finish_staged_rename` and `finish_staged_hard_link` return the staging file's
metadata and remove the staging file themselves when the metadata read, sync,
close, hard link or rename fails, so `LocalUpload::complete` no longer leaks the
`#N` file on failure and `put_opts` only cleans up after a failed write.
`rename` returns `RenameError::{Rename, Fsync}`; after a successful rename whose
directory fsync failed the staging path may already belong to a new upload, so
`finish_staged_rename` and `copy_opts` (Overwrite) leave it alone in that case.
`rename_opts` converts the error back to `io::Error`.

`Error::Metadata` from `put_opts` now names the staging path, as `complete`
already did.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Crash-durability recovery needs correction, and concurrency-sensitive failure paths require final human review.

1 open issue
What changed in this PR

Aligns LocalFileSystem with three upstream drafts to ease eventual removal of the fork while preserving the public API.

Changes:

  • Reworks durable directory creation, synchronization, and failed-creation tracking.
  • Centralizes staging-file cleanup and distinguishes rename failures from subsequent sync failures.
  • Replaces global test hooks with injected closures and adds regression coverage.
File Description
src/​local.rs Updates directory durability coordination, staged publishing, documentation, and regression tests.

Review effort: Balanced

Comment thread src/local.rs Resolved
…creation reached

Mirrors the update to upstream draft local-unsynced-after-failed-creation
(reidkaufmann/arrow-rs-object-store@6b1e655): a failed durable creation
recorded only the directories its own create_dir found existing. A
registered directory an untracked creator made in between (a before this
call's create_dir(a/b)) was not recorded, so after a failed sync the next
write under a/b synced a/b and a but never the root holding a's entry.

The record is now the deepest registered directory the walk found existing
and the registered directories above it, which existed then too. Nothing
below the deepest one is recorded (it did not exist as a directory), and a
creation that reached none records nothing, so the set stays bounded by
directories that existed during a creation.

src/local.rs equals the merged upstream drafts (6b1e655 + 4577e8a) modulo
the v0.13.2-vs-main differences, checked byte-for-byte.

Fork PR: #6
Review thread: #6 (comment)
…cess-wide set

Mirrors the current head of upstream draft local-durable-dir-creation
(reidkaufmann/arrow-rs-object-store@8f8c2cb, apache#890),
reworked after maintainer review with apache#892 folded in. The directory-creation
registry, its condvar and the waiting are replaced by one set of directories
whose entries changed without an fsync since, shared by every store in the
process. Each change of a directory's entries (create_dir, rename, hard_link)
records the directory before it runs; each durable operation fsyncs the
recorded directories among the ancestors of what it wrote and removes an
entry only when no change to it is in flight and none finished after the
fsync started. A failed fsync leaves the entries to the next write under
them, which covers the directories a failed creation left unsynced (the case
6933196 recorded through `reached`). A failed directory fsync is reported as
UnableToSyncFile naming that directory; the ancestor walk is iterative.

Reconciles the draft with local-multipart-staging-cleanup
(reidkaufmann/arrow-rs-object-store@4577e8a, apache#891),
which the fork already carries: both change `rename`'s error type. The fork
keeps apache#890's `DurableError { Change, Sync }`, drops apache#891's `RenameError
{ Rename, Fsync }`, and applies apache#891's cleanup rule to it: `finish_staged_rename`
and `copy_opts` (Overwrite) remove the staging file only on `Change`, never
after `Sync`, when the rename has happened and the path may belong to a new
upload. `finish_staged_hard_link` reports a `Change` of kind AlreadyExists as
`Error::AlreadyExists`, everything else through `DurableError::reported`, and
removes the staging file either way, as apache#891 does. `rename_opts` keeps apache#890's
arms. Upstream will meet the same conflict when either PR merges.

src/local.rs equals apache#890 and apache#891 merged this way, modulo the v0.13.2-vs-main
differences (close_fd/close_handle, merge_ranges in get_ranges, extensions,
quoted ETag, get_ranges_coalesces), checked byte-for-byte.

Fork PR: #6
Upstream: apache#890
          apache#891
@reidkaufmann
reidkaufmann marked this pull request as ready for review November 6, 2026 22:12
@reidkaufmann
reidkaufmann requested a balanced review from Copilot November 6, 2026 23:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The rewritten durability concurrency protocol requires human validation of filesystem-specific fsync ordering and pending power-cut tests before its acknowledged-write guarantees can be trusted.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

reidkaufmann added a commit that referenced this pull request Nov 13, 2026
…creation reached

Mirrors the update to upstream draft local-unsynced-after-failed-creation
(reidkaufmann/arrow-rs-object-store@6b1e655): a failed durable creation
recorded only the directories its own create_dir found existing. A
registered directory an untracked creator made in between (a before this
call's create_dir(a/b)) was not recorded, so after a failed sync the next
write under a/b synced a/b and a but never the root holding a's entry.

The record is now the deepest registered directory the walk found existing
and the registered directories above it, which existed then too. Nothing
below the deepest one is recorded (it did not exist as a directory), and a
creation that reached none records nothing, so the set stays bounded by
directories that existed during a creation.

src/local.rs equals the merged upstream drafts (6b1e655 + 4577e8a) modulo
the v0.13.2-vs-main differences, checked byte-for-byte.

Fork PR: #6
Review thread: #6 (comment)

(cherry picked from commit 6933196)
reidkaufmann added a commit that referenced this pull request Nov 13, 2026
…cess-wide set

Mirrors the current head of upstream draft local-durable-dir-creation
(reidkaufmann/arrow-rs-object-store@8f8c2cb, apache#890),
reworked after maintainer review with apache#892 folded in. The directory-creation
registry, its condvar and the waiting are replaced by one set of directories
whose entries changed without an fsync since, shared by every store in the
process. Each change of a directory's entries (create_dir, rename, hard_link)
records the directory before it runs; each durable operation fsyncs the
recorded directories among the ancestors of what it wrote and removes an
entry only when no change to it is in flight and none finished after the
fsync started. A failed fsync leaves the entries to the next write under
them, which covers the directories a failed creation left unsynced (the case
6933196 recorded through `reached`). A failed directory fsync is reported as
UnableToSyncFile naming that directory; the ancestor walk is iterative.

Reconciles the draft with local-multipart-staging-cleanup
(reidkaufmann/arrow-rs-object-store@4577e8a, apache#891),
which the fork already carries: both change `rename`'s error type. The fork
keeps apache#890's `DurableError { Change, Sync }`, drops apache#891's `RenameError
{ Rename, Fsync }`, and applies apache#891's cleanup rule to it: `finish_staged_rename`
and `copy_opts` (Overwrite) remove the staging file only on `Change`, never
after `Sync`, when the rename has happened and the path may belong to a new
upload. `finish_staged_hard_link` reports a `Change` of kind AlreadyExists as
`Error::AlreadyExists`, everything else through `DurableError::reported`, and
removes the staging file either way, as apache#891 does. `rename_opts` keeps apache#890's
arms. Upstream will meet the same conflict when either PR merges.

src/local.rs equals apache#890 and apache#891 merged this way, modulo the v0.13.2-vs-main
differences (close_fd/close_handle, merge_ranges in get_ranges, extensions,
quoted ETag, get_ranges_coalesces), checked byte-for-byte.

Fork PR: #6
Upstream: apache#890
          apache#891

(cherry picked from commit 5802ce1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants