Skip to content

fix(local): sync the directories a durable write actually creates; bound the unsynced set - #5

Merged
reidkaufmann merged 3 commits into
base-object-store-0132from
fsync-dir-sync-fixes-0132
Oct 26, 2026
Merged

reidkaufmann merged 3 commits into
base-object-store-0132from
fsync-dir-sync-fixes-0132

Conversation

@reidkaufmann

Copy link
Copy Markdown
Collaborator

Follow-up to #4 for two of Copilot's findings on create_parent_dirs.

Snapshot race (#4 r4379279400). create_parent_dirs recorded the deepest existing ancestor, called create_dir_all, and synced up to that ancestor. With with_automatic_cleanup(true), a concurrent delete can remove the ancestor in between; create_dir_all recreated it, and its new entry in the directory above was never synced.

Fix: create the missing components one create_dir at a time, registering each before creating it (so concurrent durable writes into it keep waiting, as before), and sync every directory from the deepest one this call created up to the one holding the highest one's entry. That range also covers a directory another creation made in between, which holds an entry made here.

Tests, both failing without the fix: fsync_creation_syncs_an_ancestor_removed_by_concurrent_cleanup (a #[cfg(test)] hook before create_dir removes the ancestor; the root must be synced) and fsync_creation_syncs_a_directory_another_creation_made_in_between (the hook creates x/y between this call's creation of x and of x/y/z; x/y must be synced).

Unbounded unsynced set (#4 r4379279468). A creation that failed before creating anything (EACCES, ENOSPC) still recorded its would-be directories in the process-global unsynced set, and nothing cleared them.

Fix: on drop without a completed sync, record only directories that exist or whose state cannot be determined.

Test: fsync_failed_creation_records_only_directories_that_exist (50 failed puts under an unwritable directory leave the set unchanged); fails without the fix. fsync_write_syncs_directories_a_failed_creation_left still covers the existing-directory case.

Docs (#4 r4379279335): with_fsync notes that stores with different fsync settings must not share a root within one process. No behaviour change.

For influxdata/influxdb_pro#6332; influxdb_pro will pin this branch's head.

`create_parent_dirs` recorded the deepest existing ancestor, called
`create_dir_all`, and synced from the new parent up to that ancestor.
A concurrent delete with automatic cleanup can remove the ancestor in
between; `create_dir_all` recreated it, and its new entry in the
directory above was never synced.

Create the missing components one `create_dir` at a time, registering
each before creating it, and sync every directory from the deepest one
created up to the one holding the highest one's entry. That also covers
a directory another creation made in between, which holds an entry made
here.
…ed creation

A creation that failed before creating anything (EACCES, ENOSPC) still
recorded its would-be directories in the process-global unsynced set,
and nothing cleared them. Record only directories that exist, or whose
state cannot be determined.
Copilot AI balanced review requested due to automatic review settings October 26, 2026 16:55

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Concurrent directory creation and crash-durability guarantees require final human validation of synchronization interleavings.

No open issues 🎉

What changed in this PR

This follow-up strengthens LocalFileSystem durable writes by tracking directories actually created during concurrent cleanup and reducing stale entries in the process-wide unsynced set.

Changes:

  • Creates directories individually and syncs the range covering newly created directories and their parent entries.
  • Excludes confirmed nonexistent or non-directory paths from failed-creation tracking, with regression tests for cleanup races and failed writes.
  • Documents that stores with different fsync settings must not share a root within one process.
File Description
src/​local.rs Updates directory creation, durability tracking, documentation, and regression tests.

Review effort: Balanced

@reidkaufmann
reidkaufmann merged commit 41b600a into base-object-store-0132 Oct 26, 2026
10 checks passed
@reidkaufmann
reidkaufmann deleted the fsync-dir-sync-fixes-0132 branch October 26, 2026 17:39
reidkaufmann added a commit that referenced this pull request Nov 13, 2026
…directory chain

Mirrors upstream draft local-durable-dir-creation (8de2a2d, on apache main
279572e) and replaces the fork's implementation from
#3 and
#5.

`create_parent_dirs` delegates to `create_dir_all_durable(dir, create, sync)`,
which registers the missing directories in `CREATING_DIRS`, creates them one
component at a time while registering each one it tries, and syncs from `dir`
up to the parent of the highest registered directory. A durable write into a
directory being created waits on `DIRS_CHANGED` (parking_lot `Condvar`).

Differences from the fork's code this replaces: the walk starts at `dir` and
stops at the parent of the highest registered directory, not the highest one
this call created, so it also covers a directory an untracked creator made
after the snapshot and runs when this call created nothing; tests inject
`create` and `sync` closures instead of the `#[cfg(test)]` hooks
`BEFORE_CREATE_DIR` and `SYNCED_DIRS`, and assert the exact sync sequence; the
`with_fsync` doc paragraph is the draft's shorter one. The fork's unsynced
records and their tests are removed here and return in the next commit in the
upstream draft's form; the in-between-creation test is dropped, as upstream
did, because the new walk covers that case by construction.

(cherry picked from commit f21a592)
reidkaufmann added a commit that referenced this pull request Nov 13, 2026
…as unsynced

Mirrors upstream draft local-unsynced-after-failed-creation (fbb5ef0, stacked
on 8de2a2d).

`CREATING_DIRS` becomes `DIRS: Mutex<DirRegistry>` with the `creating` counts
and an `unsynced` set. `create_dir_all_registering` records every registered
directory that exists after its `create_dir` in `CreatingDirs::reached`; unless
the creation synced them, `Drop` moves those records to `unsynced`. A durable
write under an unsynced directory syncs from its parent up to the parent of the
highest unsynced ancestor (`fsync_parent_dir_with`, which tests call with a
recording closure) and then clears the records on that chain.

Differences from the fork's unsynced set
(#3, bounded in
#5): `Drop` does no
I/O; a registered directory whose `create_dir` was never tried is not
recorded, and one deleted after it was reached still is; the 0o555 `put` loop
test is replaced by a direct test of the recorded set;
`wait_for_ancestor_creation` returns early when nothing is registered and no
record exists.

(cherry picked from commit 22ad666)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants