Repository navigation
LocalFileSystem: per-store directory-handle cache for durable writes - #7
reidkaufmann wants to merge 14 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical directory-sync and cache-coordination bugs can acknowledge writes before ancestor entries are durable, risking data loss after power failure.
6 open issues
Recreated directory can leave its parent entry unsynced · New Non-DirCache writes lose ancestor sync tracking and coordination · New Strict clone may reuse root without syncing its parent · New Rename eviction loses dirty state for the moved subtree · New Child recreation loop bypasses the retry budget · New Canceled staging uploads bypass cleanup and leave dirty handles · New
What changed in this PR
This PR replaces process-wide directory coordination with a per-store directory-handle cache for LocalFileSystem durable writes on Linux and Android.
Changes:
- Routes durable mutations through cached directory handles, with dirty-state tracking, stale-handle retries, and eviction.
- Adds configurable best-effort root-parent syncing and staging-file cleanup on publication failures.
- Adds cache and durability tests while retaining path-based operations on other targets.
| File | Description |
|---|---|
src/local/durable.rs |
Implements handle-based writes, copies, renames, deletes, and multipart uploads. |
src/local/dir_cache.rs |
Adds directory caching, dirty-state tracking, syncing, and eviction. |
src/local.rs |
Integrates durable operations, adds the root-sync option, revises fallback helpers, and expands tests. |
Cargo.toml |
Enables nix filesystem and resource-limit APIs. |
Review effort: Balanced
| None => break, | ||
| // Record the deepest already-existing ancestor *before* creating any directories, so that | ||
| // afterwards we know exactly which directories are new and need to be fsynced. | ||
| let first_existing = fsync.then(|| { |
There was a problem hiding this comment.
Out of scope for this PR, which targets Linux; non-Linux targets keep upstream's path code. Listed under follow-ups in the description.
| fn fsync_parent_dir(path: &std::path::Path) -> io::Result<()> { | ||
| let unsynced = wait_for_ancestor_creation(path); | ||
| let Some(parent) = path.parent() else { | ||
| return Ok(()); | ||
| }; | ||
| let highest = unsynced | ||
| .as_deref() | ||
| .and_then(|dir| dir.parent()) | ||
| .unwrap_or(parent); | ||
| for dir in parent.ancestors() { | ||
| fsync_dir(dir)?; | ||
| if dir == highest { | ||
| break; | ||
| } | ||
| match path.parent() { | ||
| Some(parent) => fsync_dir(parent), | ||
| None => Ok(()), |
There was a problem hiding this comment.
Out of scope for this PR, which targets Linux; the registry stays out of it. Listed under follow-ups in the description.
| if let Some(root) = self.inner.lock().dirs.get(&key) { | ||
| return Ok(Arc::clone(root)); |
| }); | ||
| // A renamed directory's handle still answers to its old path, whatever the fsyncs do. | ||
| if renamed.is_ok() { | ||
| cache.evict_moved(&from_rel.join(from_name)); |
| match openat(&parent.fd, name, DIR_FLAGS, Mode::empty()) { | ||
| Ok(fd) => break fd, | ||
| // Removed again since this call made it: make it again. | ||
| Err(Errno::ENOENT) if made => continue, |
| pub(super) struct Upload { | ||
| chain: Vec<Arc<DirHandle>>, | ||
| staging: Staging, | ||
| name: OsString, |
…mplete fails Mirrors upstream draft local-multipart-staging-cleanup (4577e8a, on apache main 279572e). `finish_staged_rename` and `finish_staged_hard_link` return the staging file's metadata and remove the staging file themselves when the metadata read, sync, close, hard link or rename fails, so `LocalUpload::complete` no longer leaks the `#N` file on failure and `put_opts` only cleans up after a failed write. `rename` returns `RenameError::{Rename, Fsync}`; after a successful rename whose directory fsync failed the staging path may already belong to a new upload, so `finish_staged_rename` and `copy_opts` (Overwrite) leave it alone in that case. `rename_opts` converts the error back to `io::Error`. `Error::Metadata` from `put_opts` now names the staging path, as `complete` already did.
…y-handle cache Port of the upstream draft local-dirfd-cache-e (0367694..20f8bb0 on apache main 279572e + 4577e8a) onto object_store 0.13.2. With fsync enabled on Linux and Android, put, copy, rename, delete and multipart uploads change directory entries through directory descriptors the store keeps open (openat, mkdirat, renameat, linkat, unlinkat), one cache per store. Each handle records the entry changes made through it and the changes an fsync has covered; a write fsyncs every handle on the chains it changed that still owes one, whoever changed it, before returning, also when it fails. Handles found stale (the directory removed or replaced, confirmed by inode) are evicted and the operation retried from the root. The cache holds up to 4096 clean handles, clamped to a quarter of the soft RLIMIT_NOFILE; dirty handles are never evicted. This removes the process-wide DirRegistry (creating/unsynced sets and the condvar wait in fsync_parent_dir), its creation-recording helpers and the test hooks; nothing is shared between stores or processes any more. Every other target keeps the path-based fsync code as it was before the registry. A directory that cannot be opened is an open error, not a sync error; UnableToSyncFile is only an fsync failure. Staging entries are removed when publishing fails and left alone after a successful rename whose fsync failed (the name may belong to a new upload); the removal is best effort, a crash leaves a `name#N` entry that `list` hides.
…, evict renamed directories Port of upstream local-dirfd-cache-e 10c6107..4147531 (review findings S1-S6, N1-N4, N6, N7, N9 on b3402b8): - Capacity eviction keeps handles an operation still holds; eviction is by handle identity, so a late eviction cannot drop a live replacement; a cached child must have been opened under the parent in the chain. - A handle is stale when its inode has no links left (fstat, no path lookup); a stale root is reopened or recreated with its entry fsynced; a directory removed between mkdirat and openat is made again; retries are bounded by handle identity. - The eviction order holds weak references and is compacted; descriptors are closed after the cache lock is released. - After a rename whose source is a directory, the handles at and under the old path are evicted. - with_fsync states again what is not coordinated (separate stores on one root, other processes, symlink/case aliases, stores with fsync off, external moves); the chmod rename test is kept for non-Linux Unix. - Shared staging-file and AlreadyExists helpers.
…nc, fsync a reopened root's entry Port of upstream local-dirfd-cache-e 2f74a4a (re-review R1-R5 and the N7 leftover on 2d0366f). A directory counts as removed only when fstat reports no links left and its path no longer names it; an operation retries at most 16 times. The handle of a renamed directory is evicted whenever the rename itself succeeded, checking only the renamed key. Whoever opens an uncached root fsyncs the directory holding its entry, and a recreated root has every created ancestor fsynced. A cached child opened under another parent is replaced only while it is still that entry and its directory is removed or moved. EEXIST is AlreadyExists in create mode only. The losing root handle of a concurrent first open is closed outside the cache lock.
A cascade of removed cached directories evicts one per retry, deepest first, so a chain deeper than the fixed bound of 16 failed with NotFound where the path-based code succeeds. The bound is now the chain depth plus 16.
Every uncached open of the root fsynced the root's parent, so a store whose root sits under a traverse-only directory (0311, a 0711 home) failed every durable write with a sync error, which neither the base nor the cache before the root recreation did. A PermissionDenied from opening that parent is now skipped unless this call created the root, whose entry then still has to be durable.
A delete without a sync opened and cached a handle on the object's directory (one openat and one fstat per delete), which automatic cleanup then removed again. The object and the directories left empty below the cached chain are now removed through the deepest cached ancestor with the rest of the path; ENOENT through that handle is checked for staleness as before. Deletes that sync still resolve the whole chain.
…he cached chain Cleanup removed uncached levels through the deepest cached ancestor, but a handle cached below such a level (its parent evicted for capacity) stayed cached with its descriptor and removed inode until the clock reached it, and for good if it was dirty. Each removed level's handle is evicted.
A failure to open or fsync the directory holding the store root, permission errors included, is an error again whether or not this call created the root; the error names that directory and the opt-out. with_best_effort_root_parent_sync(true) skips the fsync on permission errors only, at the cost that the root's directory entry may not survive a power loss. Disabled by default; set per store, as with_fsync; no effect on other targets.
c5e6ec4 to
2547870
Compare
…kipped The root handle is shared by clones while the best-effort option is per store, so once a clone with the option had opened and cached the root without fsyncing its parent, a store without the option found the root cached and wrote without that fsync. The cache records whether the parent was fsynced since the root was opened; a caller without the option fsyncs it on a cache hit when it was not, and the record is cleared when the root is evicted. One relaxed load per operation.
… them A directory rename evicted the moved directory's handle while it, or a handle under it, still owed an fsync for a change (a staged multipart upload, a write in flight). The next write into the moved directory opens new handles that know nothing of those changes and returns without fsyncing them. The dirty handles at and under the moved path are fsynced before the eviction; a change still in flight during the rename stays uncovered. File renames are unchanged; directory renames scan the cache once and fsync each dirty handle under the moved path.
… opened A directory this call made and then found removed before it could open it was made again without limit, so a remover winning every round kept the operation looping, as the path code's directory creation does. After 16 rounds the creation fails with the ENOENT.
A durable multipart upload whose future was dropped after it had staged its file (a cancelled put_multipart) removed the staging file through the Staging guard but left every handle on its chain dirty, and dirty handles are never evicted: each cancelled upload into a new directory pinned a descriptor for good. Upload now removes its staging file and fsyncs its chain when dropped unpublished; complete and discard take the staging first, so nothing is cleaned up twice.
Only the directory-handle cache raises UnableToSyncRootParent; on the other targets the variant was never constructed, which fails a -D warnings build there.
|
Retest at 257b341 (Copilot fixes 3-6) against 2547870 (the head before those fixes). Checks pass at
|
| Cell | Matrix ratio, median (min-max) | Targeted rerun, median ops/s: #6 / 2547870 / 257b341 |
Targeted ratio |
|---|---|---|---|
| xfs delete, 64 writers | 0.90 (0.77-1.03) | 107k / 143k / 138k | 1.02 |
| ext4 rename, 8 writers | 0.94 (0.82-1.06) | 3156 / 3507 / 3577 | 1.00 |
| ext4 put steady, 64 writers | 0.97 (0.53-1.03) | 13.7k / 14.0k / 13.3k | 0.97 (0.92-1.02) |
| xfs multipart, 8 writers | 0.96 (0.81-1.13) | 212 / 219 / 214 | — (single runs 133-451) |
| ext4 multipart, 8 writers | 0.95 (0.69-1.05) | 235 / 239 / 237 | — |
Multipart is order-confounded in the matrix: the variant that runs first in the cell gets ~2.5x the throughput (seen for #6, the only one of the three that ran first). Compared with #6, the results match round 3 (cold put/create faster, everything else ~1.0).
Syscalls per op match 2547870 (strace, 14 op/fs cells).
13 of 14 cells differ by at most 0.02 syscalls per op. The exception, xfs multipart, showed +6 fsyncs per 160 ops in one traced run. A second traced run with per-path counts (strace -y) gave 354 fsyncs for both heads with identical per-path counts, so the difference did not reproduce.
The power-cut rerun of influxdb_pro pinned to 257b341 is running and is expected to pass.


Note
Alternative to #6. Instead of a process-wide set of unsynced directories (the form of apache#890), each
LocalFileSystemkeeps open directory handles and does every entry change through them withopenat/mkdirat/linkat/renameat/unlinkat, as suggested in review of apache#890. Keeps apache#891 staging-file cleanup. Draft for a power-cut round of influxdb_pro pinned to this head.Which issue does this PR close?
Same races as apache#888 (fixed here in #3, #5 and reworked in #6).
What changes are included in this PR?
With
fsyncenabled, on Linux and Android:O_RDONLY|O_DIRECTORYhandles, each carrying its unsynced-change state. A change is recorded on the handle before it runs; a durable write fsyncs the dirty handles on its own chain, deepest first, before returning. No process-wide state, no registry, no waiting on other writers.st_nlink == 0plus a path check) and the operation retries from the root, bounded by chain depth + 16.RLIMIT_NOFILE(floor 16), per store. Clean handles not in use are evicted (second chance); dirty or in-use handles are never evicted.UnableToSyncRootParent. New optionwith_best_effort_root_parent_sync(true)skips it on permission errors only; off by default.Targets Linux. Other targets keep the path-based code of upstream
main, without this fork's directory registry (see follow-ups). Withoutfsync, nothing changes.Benchmark (public API, xfs and ext4, 5 rounds, paired ratios of ops/s, this design / #6, 8 / 64 writers)
PutMode::Create), each creating 3 directoriesSame fsync count as #6 in every cell. Delete measured on the final delete path; other rows on an earlier commit of this branch (
2d0366f). The xfs rename loss and ext4 rename gain at 64 writers were consistent across rounds; cause not identified. Host had external I/O load; absolute numbers are not comparable across rounds.Behaviour differences and limits
O_RDONLY; an ancestor with--xonly fails withUnableToOpenFile.with_best_effort_root_parent_sync(true)is set. With the option, a root created while its parent was unreadable may have an unsynced entry (the path code never fsyncs an existing root's entry either).UnableToSyncFilenaming the directory on Linux; the path code keepsUnableToRenameFile/UnableToCopyFile. Deleting through a path with a file component isUnableToDeleteFile, as before. A failed root creation isUnableToCreateDirwith a nested error.LocalFileSystems on one root do not coordinate, and N stores take N quarters of the limit.<name>#<n>files, hidden fromlist.Covered elsewhere / follow-ups
mainbase, not opened; depends on the outcome of the power-cut round.LocalFileSystemwith fsync can return before a new directory's entry is durable apache/arrow-rs-object-store#888 race fixesbase-object-store-0132has (Copilot findings 1 and 2). Out of scope here; the cache could be widened after a CI test confirms removed-directory detection (APFS behaviour is not documented).RLIMIT_NOFILEat startup when configured for local files: https://github.com/influxdata/influxdb_pro/issues/7289.with_best_effort_root_parent_sync.UnableToCreateDir) has no test.Are there any user-facing changes?
With
fsyncon Linux: new optionwith_best_effort_root_parent_sync, new errorUnableToSyncRootParent, the error-variant and permission differences above, and up to 4096 open directory descriptors per store. Withoutfsync, none.cargo test --all-featuresandcargo testpass at the head andcargo testat every commit; clippy-D warningsand fmt pass.cargo +1.85.0 checkreports only thesrc/client/token.rserrors already onbase-object-store-0132. A power-cut run of influxdb_pro pinned to this head is queued.