Repository navigation
Patched object_store 0.12.5: LocalFileSystem fsync backport - #2
Conversation
…pache#676) (cherry picked from commit 9b8cf52) Backport to 0.12.5: Cargo.toml conflict only. 0.12.5 has no `tokio` feature, so `fs = ["walkdir", "nix", "windows-sys"]`, and `gcp` keeps 0.12.5's dependency list. The new optional `nix` dependency uses 0.30, matching 0.12.5's existing `nix` dev-dependency (upstream uses 0.31).
(cherry picked from commit dd0afcbc7535c2b8e78a1aad4396e7ce43c7a42a) Backport to 0.12.5: put_opts, multipart and the helper functions applied unchanged. object_store 0.13 merged copy/copy_if_not_exists/rename into copy_opts/rename_opts; here the fsync changes are ported by hand onto 0.12.5's separate copy, copy_if_not_exists and rename methods, and the with_fsync docs name those methods. file_test_fsync drops the call to list_with_offset_exclusivity, an integration helper 0.12.5 does not have.
(cherry picked from commit 596e1f6) Backport to 0.12.5: 0.12.5 has no rename_opts, and rename_if_not_exists is the ObjectStore trait's default copy_if_not_exists + delete. LocalFileSystem now overrides rename_if_not_exists so the source delete goes through delete_location with fsync. delete_location takes the filesystem path already validated by LocalFileSystem::path_to_filesystem, so delete keeps 0.12.5's path validation (which rejects #<digits> staging names).
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
New tests exceed the declared MSRV and construct files from invalid closed descriptors or handles.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Backports durable local filesystem writes and explicit close-error handling from upstream.
Changes:
- Adds opt-in fsync for writes, copies, renames, and multipart uploads.
- Adds checked file closing and durability tests.
- Adds platform-specific close dependencies.
| File | Description |
|---|---|
src/local.rs |
Implements fsync, checked closes, and tests. |
Cargo.toml |
Adds Unix and Windows dependencies. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| use std::os::fd::FromRawFd; | ||
| use std::os::unix::io::AsRawFd; |
There was a problem hiding this comment.
Addressed differently in 2d44269: the backport's own dependencies already need Rust 1.71 (nix 0.30 declares 1.69, windows-sys 0.61 declares 1.71), so rust-version now says 1.71.0 instead of 1.64.0. std::os::fd is stable there. The test that used it was also rewritten in 6b7c9a0 (see the other thread).
| let file = { | ||
| let fd = file.as_raw_fd(); | ||
| super::close_file(file).unwrap(); | ||
| unsafe { std::fs::File::from_raw_fd(fd) } |
There was a problem hiding this comment.
Fixed in 6b7c9a0. close_file now delegates to close_raw_fd (Unix) and close_raw_handle (Windows). The tests call those with values that can never be open (a descriptor at the RLIMIT_NOFILE soft limit, and a null handle), so no File is rebuilt from a freed descriptor and no other test's descriptor can be closed. Upstream main still has the original test; we'll propose this fix there too.
The close-error tests from apache#676 closed a file and then rebuilt a File from the freed descriptor or handle to close it again. That breaks the from_raw_fd/from_raw_handle contract, and in a parallel test run another thread can be handed the same descriptor in between, so the test could close an unrelated file. Split close_file into close_raw_fd (Unix) and close_raw_handle (Windows) and test those with a value that can never be open: a descriptor at the open-file limit, and a null handle. close_file's behavior is unchanged. Raised in review (Copilot) on #2.
apache#676 added nix 0.30 (rust-version 1.69) and windows-sys 0.61 (rust-version 1.71), so the crate no longer builds on the declared 1.64. Upstream is at 1.85.
(cherry picked from commit 0c380a1)
…he#602) Updates the requirements on [quick-xml](https://github.com/tafia/quick-xml) to permit the latest version. - [Release notes](https://github.com/tafia/quick-xml/releases) - [Changelog](https://github.com/tafia/quick-xml/blob/master/Changelog.md) - [Commits](tafia/quick-xml@v0.38.0...v0.39.0) --- updated-dependencies: - dependency-name: quick-xml dependency-version: 0.39.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit b3027f8)
…he#727) Updates the requirements on [quick-xml](https://github.com/tafia/quick-xml) to permit the latest version. - [Release notes](https://github.com/tafia/quick-xml/releases) - [Changelog](https://github.com/tafia/quick-xml/blob/master/Changelog.md) - [Commits](tafia/quick-xml@v0.39.0...v0.40.1) --- updated-dependencies: - dependency-name: quick-xml dependency-version: 0.40.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit b058903)
…he#785) Updates the requirements on [quick-xml](https://github.com/tafia/quick-xml) to permit the latest version. - [Release notes](https://github.com/tafia/quick-xml/releases) - [Changelog](https://github.com/tafia/quick-xml/blob/master/Changelog.md) - [Commits](tafia/quick-xml@v0.40.1...v0.41.0) --- updated-dependencies: - dependency-name: quick-xml dependency-version: 0.41.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit fd44ebb)
…ting (apache#799) Adds a rust-toolchain.toml (mirroring apache/arrow-rs) pinning the toolchain to 1.97.0 so CI uses a stable, reproducible Rust version and is not surprised by new lints in future releases. Rust 1.97.0's clippy added the useless_borrows_in_formatting lint, which flags redundant references in format! arguments. Fixes the resulting failures in the azure module. Closes apache#798 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit e64e8e3)
quick-xml 0.41 (cherry-picked from apache#785 for RUSTSEC-2026-0194/0195) declares rust-version 1.79; it is used by the aws, azure and gcp features.
The job pinned a hand-maintained list of dependency versions. The list went stale: cargo update -p quote --precise 1.0.41 now fails because zerovec-derive 0.11.6 requires quote >= 1.0.44, so the job failed before checking anything. Generate the lockfile with CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=fallback instead, which picks the newest versions compatible with the crate's rust-version.
cargo msrv verify ran cargo check with default features only, so the rust-version requirement of feature-only dependencies (quick-xml 0.41 for aws/azure/gcp needs 1.79) went unchecked.
|
Pushed review and CI fixes; all checks now pass, including Verify MSRV. The PR description has been rewritten to cover how clean the backport is (which parts were hand-ported and need the closest review), the commits that are ours rather than upstream's, the upstream CI fixes cherry-picked with |

Tracking issue: influxdata/influxdb_pro#6262
Patched
object_store0.12.5 for influxdb_pro (and the OSS influxdb code undeross/), which usesLocalFileSystemfor--object-store file. In 0.12.5LocalFileSystemnever fsyncs, so a power loss can leave files that exist under their final names with no data (influxdata/EAR#7132). This branch adds upstream's opt-inLocalFileSystem::with_fsync, which influxdb_pro turns on.base-object-store-0125is upstream tagv0.12.5(8f16588). Every upstream change listed here first shipped in object_store 0.14.0 or 0.14.1.How clean the backport is
b5b651fclose with error checkingCargo.tomlconflict only: 0.12.5 has notokiofeature, and the new optionalnixdependency uses 0.30 to match 0.12.5's dev-dependency. About 6 changed lines against upstream.8b19aedwith_fsyncput_opts, multipart and the helper functions applied unchanged. 0.13 mergedcopy/copy_if_not_exists/renameintocopy_opts/rename_opts, so the fsync changes to those three methods were ported by hand onto 0.12.5's separate methods. About 68 of roughly 340 changed lines differ from upstream.8f73b1dfsync the source delete in create-mode renamerename_opts, so the same behaviour is a newrename_if_not_existsoverride, plus adelete_locationhelper that keeps 0.12.5's path validation. About 89 changed lines differ from a 71-line upstream change.The hand-ported parts are the ones to review closely.
cargo testcovers them with fsync on (file_test_fsyncruns the integration suite, including copy, rename andcopy_if_not_exists, withwith_fsync(true)), and influxdb_pro's crash-simulation tests exercise them end to end.Changes of our own, not upstream:
6b7c9a0: the close-error tests from feat(local): explicit close with error checking for LocalFileSystem apache/arrow-rs-object-store#676 rebuilt aFilefrom a closed descriptor or handle. Fixed after review. Upstreammainstill has the original; we plan to propose the fix there.2d44269,421b019:rust-versionraised from 1.64 to 1.71, then 1.79, to match what the new dependencies declare (nix0.30,windows-sys0.61,quick-xml0.41).fde97c6,4d67bb3: the MSRV job's hand-pinned dependency list had gone stale and failed before checking anything. It now uses Cargo's Rust-version-aware resolver and checks all features.98cf6e8: formatting.Cherry-picked from upstream to fix CI failures that were already present on
v0.12.5, all with-x:useless_conversion.quick-xml0.38 → 0.41, for RUSTSEC-2026-0194 and RUSTSEC-2026-0195. Only thequick-xmlline was taken; upstream's neighbouringmd-5/randbumps were left out.useless_borrows_in_formatting.Getting rid of the fork
The fork exists only because influxdb_pro cannot move to object_store 0.14 yet: its DataFusion fork (52.5) and
parquet57 require 0.12. DataFusionmainmoved to object_store 0.14.2 and arrow/parquet 60 on 2026-09-18 (apache/datafusion#25335), but the latest release, 55.1.0, still uses 0.13.When influxdb_pro rebases its DataFusion fork onto a release that uses 0.14:
[patch.crates-io] object_storeentry inent/Cargo.tomlandoss/Cargo.tomland depend onobject_store0.14.x. Everything picked here is in 0.14.1, andwith_fsync(bool)has the same name and meaning there, so influxdb_pro's.with_fsync(true)call stays as it is.6b7c9a0). It affects this crate's own tests, not users of it, and should go upstream regardless.ObjectStoretrait (copy_opts/rename_opts, convenience methods moved out of the trait), which touches influxdb_pro's roughly 72ObjectStoreimplementations. That cost comes with any upgrade to 0.13 or later.Known gap, same as upstream
with_fsyncfsyncs only the directories a call itself creates. A write into a directory that a concurrent write has just created can return before that directory's own entry is durable. On ext4 one fsync commits the whole journal, so this probably does not occur there; POSIX does not guarantee it. We can address that gap if proves to be more than theoretical.Testing
cargo check --all-features --all-targets;cargo clippy -- -D warningswith CI's feature sets (default, no-default, fs, aws, gcp, azure, http, all-features/all-targets);cargo fmt --check;cargo test: 80 passed and 1 ignored (lib), 19 passed (doc), 2 passed (other).cargo checkwith default features and with--all-featureson Rust 1.79.0, using a lockfile generated withCARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=fallback.with_fsync(true)enabled, crash-simulation tests (an strace trace of the server, replayed to the state a power cut would leave) find no file left non-durable. Without it, every file written during the run is left empty.