Skip to content

LocalFileSystem: per-store directory-handle cache for durable writes - #7

Open
reidkaufmann wants to merge 14 commits into
base-object-store-0132from
local-dirfd-cache-0132
Open

reidkaufmann wants to merge 14 commits into
base-object-store-0132from
local-dirfd-cache-0132

Conversation

@reidkaufmann

@reidkaufmann reidkaufmann commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Alternative to #6. Instead of a process-wide set of unsynced directories (the form of apache#890), each LocalFileSystem keeps open directory handles and does every entry change through them with openat/mkdirat/linkat/renameat/unlinkat, as suggested in review of apache#890. Keeps apache#891 staging-file cleanup. Draft for a power-cut round of influxdb_pro pinned to this head.

Which issue does this PR close?

Same races as apache#888 (fixed here in #3, #5 and reworked in #6).

What changes are included in this PR?

With fsync enabled, on Linux and Android:

  • A per-store cache maps directories under the root to open O_RDONLY|O_DIRECTORY handles, each carrying its unsynced-change state. A change is recorded on the handle before it runs; a durable write fsyncs the dirty handles on its own chain, deepest first, before returning. No process-wide state, no registry, no waiting on other writers.
  • Creates, publishes and removals go through the parent handle, so a directory removed and recreated by another call (automatic cleanup) cannot receive a write meant for the old one. A removed directory is detected (st_nlink == 0 plus a path check) and the operation retries from the root, bounded by chain depth + 16.
  • put, create, copy, rename, delete (with cleanup) and multipart all use the cache. Deletes without sync walk only cached handles and do not open new ones.
  • Capacity 4096 handles, clamped to a quarter of the soft RLIMIT_NOFILE (floor 16), per store. Clean handles not in use are evicted (second chance); dirty or in-use handles are never evicted.
  • Staging files are removed when publishing fails (fix(local): remove the staging file when publishing it fails apache/arrow-rs-object-store#891), best effort.
  • The root's own entry is fsynced in its parent whenever the root is opened fresh. If that parent cannot be opened or fsynced (including EACCES), writes fail with UnableToSyncRootParent. New option with_best_effort_root_parent_sync(true) skips it on permission errors only; off by default.

Targets Linux. Other targets keep the path-based code of upstream main, without this fork's directory registry (see follow-ups). Without fsync, nothing changes.

Benchmark (public API, xfs and ext4, 5 rounds, paired ratios of ops/s, this design / #6, 8 / 64 writers)
Operation xfs ext4
put, existing directory 1.03 / 0.99 1.02 / 0.98
put, shared directory 1.01 / 1.00 1.04 / 1.01
put spread over 12,288 directories (more than the cache) 1.00 / 0.98 0.97 / 1.04
put, each creating 3 directories 1.36 / 1.23 2.00 / 1.39
create (PutMode::Create), each creating 3 directories 1.39 / 1.26 2.07 / 1.48
delete with automatic cleanup 0.98 / 1.00 1.02 / 1.02
copy 1.00 / 1.14 1.02 / 0.95
rename 1.01 / 0.93 1.18 / 1.16
multipart, 8 writers 1.04 0.98

Same fsync count as #6 in every cell. Delete measured on the final delete path; other rows on an earlier commit of this branch (2d0366f). The xfs rename loss and ext4 rename gain at 64 writers were consistent across rounds; cause not identified. Host had external I/O load; absolute numbers are not comparable across rounds.

Behaviour differences and limits
  • Every directory on a write's chain, including existing ancestors, must be openable O_RDONLY; an ancestor with --x only fails with UnableToOpenFile.
  • The root's parent must be readable unless with_best_effort_root_parent_sync(true) is set. With the option, a root created while its parent was unreadable may have an unsynced entry (the path code never fsyncs an existing root's entry either).
  • Directory fsync failures are UnableToSyncFile naming the directory on Linux; the path code keeps UnableToRenameFile/UnableToCopyFile. Deleting through a path with a file component is UnableToDeleteFile, as before. A failed root creation is UnableToCreateDir with a nested error.
  • The cache and the fd clamp are per store: two LocalFileSystems on one root do not coordinate, and N stores take N quarters of the limit.
  • Directories moved by another process are not detected; a write in flight during such a move may leave an unsynced entry in the moved directory.
  • Without a tokio runtime, dropping an unfinished multipart upload removes the staging file and fsyncs its chain on the caller's thread.
  • Staging-file removal is best effort; a crash can leave <name>#<n> files, hidden from list.
  • Rename's retry bound uses the deeper of its two chains.
Covered elsewhere / follow-ups
  • Upstream version of this change: prepared on the upstream main base, not opened; depends on the outcome of the power-cut round.
  • macOS / FreeBSD and other non-Linux targets: this PR removes the fork's directory registry there, so they lose the LocalFileSystem with fsync can return before a new directory's entry is durable apache/arrow-rs-object-store#888 race fixes base-object-store-0132 has (Copilot findings 1 and 2). Out of scope here; the cache could be widened after a CI test confirms removed-directory detection (APFS behaviour is not documented).
  • influxdb_pro: raise the soft RLIMIT_NOFILE at startup when configured for local files: https://github.com/influxdata/influxdb_pro/issues/7289.
  • influxdb_pro does not expose with_best_effort_root_parent_sync.
  • No orphan staging-file sweep in object_store.
  • xfs rename at 64 writers (0.93×) not investigated.
  • Not addressed in this PR (review nits):
    • The cap on re-creating a directory removed before it can be opened (16 rounds, then UnableToCreateDir) has no test.
    • A put into a directory being renamed through the store can return before the moved directory's entries are fsynced (fsyncs run outside the cache lock, then the old handles are evicted). Directory moves in flight are not covered; file renames are unaffected.
    • If the handle of a finished multipart staging task is dropped without being awaited, its cleanup (unlink plus directory fsyncs) runs on the async worker instead of the blocking pool.

Are there any user-facing changes?

With fsync on Linux: new option with_best_effort_root_parent_sync, new error UnableToSyncRootParent, the error-variant and permission differences above, and up to 4096 open directory descriptors per store. Without fsync, none.

cargo test --all-features and cargo test pass at the head and cargo test at every commit; clippy -D warnings and fmt pass. cargo +1.85.0 check reports only the src/client/token.rs errors already on base-object-store-0132. A power-cut run of influxdb_pro pinned to this head is queued.

@reidkaufmann
reidkaufmann marked this pull request as ready for review November 13, 2026 16:23
@reidkaufmann
reidkaufmann requested review from crepererum and a balanced review from Copilot November 13, 2026 16:23

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical directory-sync and cache-coordination bugs can acknowledge writes before ancestor entries are durable, risking data loss after power failure.

6 open issues
What changed in this PR

This PR replaces process-wide directory coordination with a per-store directory-handle cache for LocalFileSystem durable writes on Linux and Android.

Changes:

  • Routes durable mutations through cached directory handles, with dirty-state tracking, stale-handle retries, and eviction.
  • Adds configurable best-effort root-parent syncing and staging-file cleanup on publication failures.
  • Adds cache and durability tests while retaining path-based operations on other targets.
File Description
src/​local/​durable.rs Implements handle-based writes, copies, renames, deletes, and multipart uploads.
src/​local/​dir_cache.rs Adds directory caching, dirty-state tracking, syncing, and eviction.
src/​local.rs Integrates durable operations, adds the root-sync option, revises fallback helpers, and expands tests.
Cargo.toml Enables nix filesystem and resource-limit APIs.

Review effort: Balanced

Comment thread src/local.rs
None => break,
// Record the deepest already-existing ancestor *before* creating any directories, so that
// afterwards we know exactly which directories are new and need to be fsynced.
let first_existing = fsync.then(|| {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of scope for this PR, which targets Linux; non-Linux targets keep upstream's path code. Listed under follow-ups in the description.

Comment thread src/local.rs
Comment on lines 1158 to +1161
fn fsync_parent_dir(path: &std::path::Path) -> io::Result<()> {
let unsynced = wait_for_ancestor_creation(path);
let Some(parent) = path.parent() else {
return Ok(());
};
let highest = unsynced
.as_deref()
.and_then(|dir| dir.parent())
.unwrap_or(parent);
for dir in parent.ancestors() {
fsync_dir(dir)?;
if dir == highest {
break;
}
match path.parent() {
Some(parent) => fsync_dir(parent),
None => Ok(()),

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of scope for this PR, which targets Linux; the registry stays out of it. Listed under follow-ups in the description.

Comment thread src/local/dir_cache.rs Outdated
Comment on lines +414 to +415
if let Some(root) = self.inner.lock().dirs.get(&key) {
return Ok(Arc::clone(root));

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 44a1087.

Comment thread src/local/durable.rs Outdated
});
// A renamed directory's handle still answers to its old path, whatever the fsyncs do.
if renamed.is_ok() {
cache.evict_moved(&from_rel.join(from_name));

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6a36c57.

Comment thread src/local/dir_cache.rs Outdated
match openat(&parent.fd, name, DIR_FLAGS, Mode::empty()) {
Ok(fd) => break fd,
// Removed again since this call made it: make it again.
Err(Errno::ENOENT) if made => continue,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0359b07.

Comment thread src/local/durable.rs
Comment on lines +484 to +487
pub(super) struct Upload {
chain: Vec<Arc<DirHandle>>,
staging: Staging,
name: OsString,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6a89cde.

…mplete fails

Mirrors upstream draft local-multipart-staging-cleanup (4577e8a, on apache
main 279572e).

`finish_staged_rename` and `finish_staged_hard_link` return the staging file's
metadata and remove the staging file themselves when the metadata read, sync,
close, hard link or rename fails, so `LocalUpload::complete` no longer leaks the
`#N` file on failure and `put_opts` only cleans up after a failed write.
`rename` returns `RenameError::{Rename, Fsync}`; after a successful rename whose
directory fsync failed the staging path may already belong to a new upload, so
`finish_staged_rename` and `copy_opts` (Overwrite) leave it alone in that case.
`rename_opts` converts the error back to `io::Error`.

`Error::Metadata` from `put_opts` now names the staging path, as `complete`
already did.
…y-handle cache

Port of the upstream draft local-dirfd-cache-e (0367694..20f8bb0 on
apache main 279572e + 4577e8a) onto object_store 0.13.2.

With fsync enabled on Linux and Android, put, copy, rename, delete and
multipart uploads change directory entries through directory descriptors
the store keeps open (openat, mkdirat, renameat, linkat, unlinkat), one
cache per store. Each handle records the entry changes made through it
and the changes an fsync has covered; a write fsyncs every handle on the
chains it changed that still owes one, whoever changed it, before
returning, also when it fails. Handles found stale (the directory removed
or replaced, confirmed by inode) are evicted and the operation retried
from the root. The cache holds up to 4096 clean handles, clamped to a
quarter of the soft RLIMIT_NOFILE; dirty handles are never evicted.

This removes the process-wide DirRegistry (creating/unsynced sets and the
condvar wait in fsync_parent_dir), its creation-recording helpers and the
test hooks; nothing is shared between stores or processes any more. Every
other target keeps the path-based fsync code as it was before the
registry. A directory that cannot be opened is an open error, not a sync
error; UnableToSyncFile is only an fsync failure.

Staging entries are removed when publishing fails and left alone after a
successful rename whose fsync failed (the name may belong to a new
upload); the removal is best effort, a crash leaves a `name#N` entry that
`list` hides.
…, evict renamed directories

Port of upstream local-dirfd-cache-e 10c6107..4147531 (review findings
S1-S6, N1-N4, N6, N7, N9 on b3402b8):

- Capacity eviction keeps handles an operation still holds; eviction is
  by handle identity, so a late eviction cannot drop a live replacement;
  a cached child must have been opened under the parent in the chain.
- A handle is stale when its inode has no links left (fstat, no path
  lookup); a stale root is reopened or recreated with its entry fsynced;
  a directory removed between mkdirat and openat is made again; retries
  are bounded by handle identity.
- The eviction order holds weak references and is compacted; descriptors
  are closed after the cache lock is released.
- After a rename whose source is a directory, the handles at and under
  the old path are evicted.
- with_fsync states again what is not coordinated (separate stores on one
  root, other processes, symlink/case aliases, stores with fsync off,
  external moves); the chmod rename test is kept for non-Linux Unix.
- Shared staging-file and AlreadyExists helpers.
…nc, fsync a reopened root's entry

Port of upstream local-dirfd-cache-e 2f74a4a (re-review R1-R5 and the
N7 leftover on 2d0366f). A directory counts as removed only when fstat
reports no links left and its path no longer names it; an operation
retries at most 16 times. The handle of a renamed directory is evicted
whenever the rename itself succeeded, checking only the renamed key.
Whoever opens an uncached root fsyncs the directory holding its entry,
and a recreated root has every created ancestor fsynced. A cached child
opened under another parent is replaced only while it is still that entry
and its directory is removed or moved. EEXIST is AlreadyExists in create
mode only. The losing root handle of a concurrent first open is closed
outside the cache lock.
A cascade of removed cached directories evicts one per retry, deepest
first, so a chain deeper than the fixed bound of 16 failed with NotFound
where the path-based code succeeds. The bound is now the chain depth
plus 16.
Every uncached open of the root fsynced the root's parent, so a store
whose root sits under a traverse-only directory (0311, a 0711 home) failed
every durable write with a sync error, which neither the base nor the
cache before the root recreation did. A PermissionDenied from opening
that parent is now skipped unless this call created the root, whose entry
then still has to be durable.
A delete without a sync opened and cached a handle on the object's
directory (one openat and one fstat per delete), which automatic cleanup
then removed again. The object and the directories left empty below the
cached chain are now removed through the deepest cached ancestor with the
rest of the path; ENOENT through that handle is checked for staleness as
before. Deletes that sync still resolve the whole chain.
…he cached chain

Cleanup removed uncached levels through the deepest cached ancestor, but a
handle cached below such a level (its parent evicted for capacity) stayed
cached with its descriptor and removed inode until the clock reached it,
and for good if it was dirty. Each removed level's handle is evicted.
A failure to open or fsync the directory holding the store root, permission
errors included, is an error again whether or not this call created the
root; the error names that directory and the opt-out.
with_best_effort_root_parent_sync(true) skips the fsync on permission
errors only, at the cost that the root's directory entry may not survive
a power loss. Disabled by default; set per store, as with_fsync; no effect on other
targets.
@reidkaufmann
reidkaufmann force-pushed the local-dirfd-cache-0132 branch from c5e6ec4 to 2547870 Compare November 13, 2026 16:34
…kipped

The root handle is shared by clones while the best-effort option is per
store, so once a clone with the option had opened and cached the root
without fsyncing its parent, a store without the option found the root
cached and wrote without that fsync. The cache records whether the
parent was fsynced since the root was opened; a caller without the
option fsyncs it on a cache hit when it was not, and the record is
cleared when the root is evicted. One relaxed load per operation.
… them

A directory rename evicted the moved directory's handle while it, or a
handle under it, still owed an fsync for a change (a staged multipart
upload, a write in flight). The next write into the moved directory opens
new handles that know nothing of those changes and returns without
fsyncing them. The dirty handles at and under the moved path are fsynced
before the eviction; a change still in flight during the rename stays
uncovered. File renames are unchanged; directory renames scan the cache
once and fsync each dirty handle under the moved path.
… opened

A directory this call made and then found removed before it could open
it was made again without limit, so a remover winning every round kept
the operation looping, as the path code's directory creation does. After
16 rounds the creation fails with the ENOENT.
A durable multipart upload whose future was dropped after it had staged
its file (a cancelled put_multipart) removed the staging file through
the Staging guard but left every handle on its chain dirty, and dirty
handles are never evicted: each cancelled upload into a new directory
pinned a descriptor for good. Upload now removes its staging file and
fsyncs its chain when dropped unpublished; complete and discard take the
staging first, so nothing is cleaned up twice.
Only the directory-handle cache raises UnableToSyncRootParent; on the
other targets the variant was never constructed, which fails a
-D warnings build there.
@reidkaufmann

reidkaufmann commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Retest at 257b341 (Copilot fixes 3-6) against 2547870 (the head before those fixes).

Checks pass at 257b341: tests, clippy, fmt, simulated non-Linux.
  • cargo test --all-features: 219 passed, 4 ignored. cargo test: 118 passed, 1 ignored.
  • clippy -D warnings and cargo fmt --check pass.
  • cargo +1.85.0 check --all-features: only the two src/client/token.rs errors already on base-object-store-0132.
  • Simulated non-Linux build (cfg gate swapped to redox|hermit): cargo test --lib local:: 26 passed, clippy clean.
Benchmark: no op differs from 2547870 beyond run-to-run variation (~5%), xfs and ext4.

5 rounds per file system, same cells as the round-3 table in the description (minus multipart at 1 writer). Paired 257b341 / 2547870 ops/s ratios have medians of 0.95-1.04 on every cell except xfs delete 64 (0.90) and ext4 rename 8 (0.94). Those two, both multipart cells, and ext4 put steady 64 (one round at 0.53) were rerun. The targeted rerun (sync + 3 s before each run, each variant run first in turn) put them back within variation:

Cell Matrix ratio, median (min-max) Targeted rerun, median ops/s: #6 / 2547870 / 257b341 Targeted ratio
xfs delete, 64 writers 0.90 (0.77-1.03) 107k / 143k / 138k 1.02
ext4 rename, 8 writers 0.94 (0.82-1.06) 3156 / 3507 / 3577 1.00
ext4 put steady, 64 writers 0.97 (0.53-1.03) 13.7k / 14.0k / 13.3k 0.97 (0.92-1.02)
xfs multipart, 8 writers 0.96 (0.81-1.13) 212 / 219 / 214 — (single runs 133-451)
ext4 multipart, 8 writers 0.95 (0.69-1.05) 235 / 239 / 237 —

Multipart is order-confounded in the matrix: the variant that runs first in the cell gets ~2.5x the throughput (seen for #6, the only one of the three that ran first). Compared with #6, the results match round 3 (cold put/create faster, everything else ~1.0).

Syscalls per op match 2547870 (strace, 14 op/fs cells).

13 of 14 cells differ by at most 0.02 syscalls per op. The exception, xfs multipart, showed +6 fsyncs per 160 ops in one traced run. A second traced run with per-path counts (strace -y) gave 354 fsyncs for both heads with identical per-path counts, so the difference did not reproduce.

The power-cut rerun of influxdb_pro pinned to 257b341 is running and is expected to pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants