Skip to content

Align LocalFileSystem directory creation and multipart cleanup with the upstream drafts (0.12.5) - #11

Closed
reidkaufmann wants to merge 8 commits into
base-object-store-0125from
align-local-with-upstream-drafts-0125
Closed

reidkaufmann wants to merge 8 commits into
base-object-store-0125from
align-local-with-upstream-drafts-0125

Conversation

@reidkaufmann

Copy link
Copy Markdown
Collaborator

Mirror of #6 onto base-object-store-0125 (object_store 0.12.5) for the influxdb_pro 3.12 backport; stacked on #9 until it merges. One commit per original commit (5), cherry-picked with -x.

Conflicts resolved (0.12.5 vs 0.13.2): futures/edition-2021 import lines; the changes to copy_opts/rename_opts applied to 0.12.5's copy, rename and copy_if_not_exists (same RenameError/DurableError handling); the test helper returning impl FnMut needs + '_ under edition 2021 (use<'_> needs Rust 1.82 > MSRV 1.79).

Checks: cargo test --all-features, cargo test, cargo clippy --all-targets --all-features -- -D warnings, cargo fmt --check, cargo msrv verify -- cargo check --all-features (1.79.0) pass locally.

`create_parent_dirs` recorded the deepest existing ancestor, called
`create_dir_all`, and synced from the new parent up to that ancestor.
A concurrent delete with automatic cleanup can remove the ancestor in
between; `create_dir_all` recreated it, and its new entry in the
directory above was never synced.

Create the missing components one `create_dir` at a time, registering
each before creating it, and sync every directory from the deepest one
created up to the one holding the highest one's entry. That also covers
a directory another creation made in between, which holds an entry made
here.

(cherry picked from commit 4d3a27f)
…ed creation

A creation that failed before creating anything (EACCES, ENOSPC) still
recorded its would-be directories in the process-global unsynced set,
and nothing cleared them. Record only directories that exist, or whose
state cannot be determined.

(cherry picked from commit 1d068f4)
…directory chain

Mirrors upstream draft local-durable-dir-creation (8de2a2d, on apache main
279572e) and replaces the fork's implementation from
#3 and
#5.

`create_parent_dirs` delegates to `create_dir_all_durable(dir, create, sync)`,
which registers the missing directories in `CREATING_DIRS`, creates them one
component at a time while registering each one it tries, and syncs from `dir`
up to the parent of the highest registered directory. A durable write into a
directory being created waits on `DIRS_CHANGED` (parking_lot `Condvar`).

Differences from the fork's code this replaces: the walk starts at `dir` and
stops at the parent of the highest registered directory, not the highest one
this call created, so it also covers a directory an untracked creator made
after the snapshot and runs when this call created nothing; tests inject
`create` and `sync` closures instead of the `#[cfg(test)]` hooks
`BEFORE_CREATE_DIR` and `SYNCED_DIRS`, and assert the exact sync sequence; the
`with_fsync` doc paragraph is the draft's shorter one. The fork's unsynced
records and their tests are removed here and return in the next commit in the
upstream draft's form; the in-between-creation test is dropped, as upstream
did, because the new walk covers that case by construction.

(cherry picked from commit f21a592)
…as unsynced

Mirrors upstream draft local-unsynced-after-failed-creation (fbb5ef0, stacked
on 8de2a2d).

`CREATING_DIRS` becomes `DIRS: Mutex<DirRegistry>` with the `creating` counts
and an `unsynced` set. `create_dir_all_registering` records every registered
directory that exists after its `create_dir` in `CreatingDirs::reached`; unless
the creation synced them, `Drop` moves those records to `unsynced`. A durable
write under an unsynced directory syncs from its parent up to the parent of the
highest unsynced ancestor (`fsync_parent_dir_with`, which tests call with a
recording closure) and then clears the records on that chain.

Differences from the fork's unsynced set
(#3, bounded in
#5): `Drop` does no
I/O; a registered directory whose `create_dir` was never tried is not
recorded, and one deleted after it was reached still is; the 0o555 `put` loop
test is replaced by a direct test of the recorded set;
`wait_for_ancestor_creation` returns early when nothing is registered and no
record exists.

(cherry picked from commit 22ad666)
…mplete fails

Mirrors upstream draft local-multipart-staging-cleanup (4577e8a, on apache
main 279572e).

`finish_staged_rename` and `finish_staged_hard_link` return the staging file's
metadata and remove the staging file themselves when the metadata read, sync,
close, hard link or rename fails, so `LocalUpload::complete` no longer leaks the
`#N` file on failure and `put_opts` only cleans up after a failed write.
`rename` returns `RenameError::{Rename, Fsync}`; after a successful rename whose
directory fsync failed the staging path may already belong to a new upload, so
`finish_staged_rename` and `copy_opts` (Overwrite) leave it alone in that case.
`rename_opts` converts the error back to `io::Error`.

`Error::Metadata` from `put_opts` now names the staging path, as `complete`
already did.

(cherry picked from commit 64d125a)
…creation reached

Mirrors the update to upstream draft local-unsynced-after-failed-creation
(reidkaufmann/arrow-rs-object-store@6b1e655): a failed durable creation
recorded only the directories its own create_dir found existing. A
registered directory an untracked creator made in between (a before this
call's create_dir(a/b)) was not recorded, so after a failed sync the next
write under a/b synced a/b and a but never the root holding a's entry.

The record is now the deepest registered directory the walk found existing
and the registered directories above it, which existed then too. Nothing
below the deepest one is recorded (it did not exist as a directory), and a
creation that reached none records nothing, so the set stays bounded by
directories that existed during a creation.

src/local.rs equals the merged upstream drafts (6b1e655 + 4577e8a) modulo
the v0.13.2-vs-main differences, checked byte-for-byte.

Fork PR: #6
Review thread: #6 (comment)

(cherry picked from commit 6933196)
…cess-wide set

Mirrors the current head of upstream draft local-durable-dir-creation
(reidkaufmann/arrow-rs-object-store@8f8c2cb, apache#890),
reworked after maintainer review with apache#892 folded in. The directory-creation
registry, its condvar and the waiting are replaced by one set of directories
whose entries changed without an fsync since, shared by every store in the
process. Each change of a directory's entries (create_dir, rename, hard_link)
records the directory before it runs; each durable operation fsyncs the
recorded directories among the ancestors of what it wrote and removes an
entry only when no change to it is in flight and none finished after the
fsync started. A failed fsync leaves the entries to the next write under
them, which covers the directories a failed creation left unsynced (the case
6933196 recorded through `reached`). A failed directory fsync is reported as
UnableToSyncFile naming that directory; the ancestor walk is iterative.

Reconciles the draft with local-multipart-staging-cleanup
(reidkaufmann/arrow-rs-object-store@4577e8a, apache#891),
which the fork already carries: both change `rename`'s error type. The fork
keeps apache#890's `DurableError { Change, Sync }`, drops apache#891's `RenameError
{ Rename, Fsync }`, and applies apache#891's cleanup rule to it: `finish_staged_rename`
and `copy_opts` (Overwrite) remove the staging file only on `Change`, never
after `Sync`, when the rename has happened and the path may belong to a new
upload. `finish_staged_hard_link` reports a `Change` of kind AlreadyExists as
`Error::AlreadyExists`, everything else through `DurableError::reported`, and
removes the staging file either way, as apache#891 does. `rename_opts` keeps apache#890's
arms. Upstream will meet the same conflict when either PR merges.

src/local.rs equals apache#890 and apache#891 merged this way, modulo the v0.13.2-vs-main
differences (close_fd/close_handle, merge_ranges in get_ranges, extensions,
quoted ETag, get_ranges_coalesces), checked byte-for-byte.

Fork PR: #6
Upstream: apache#890
          apache#891

(cherry picked from commit 5802ce1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant