Skip to content

Patched object_store 0.12.5: LocalFileSystem fsync backport - #2

Merged
reidkaufmann merged 14 commits into
base-object-store-0125from
upgrade-object-store-ver0125-a
Oct 5, 2026
Merged

reidkaufmann merged 14 commits into
base-object-store-0125from
upgrade-object-store-ver0125-a

Conversation

@reidkaufmann

@reidkaufmann reidkaufmann commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Tracking issue: influxdata/influxdb_pro#6262

Patched object_store 0.12.5 for influxdb_pro (and the OSS influxdb code under oss/), which uses LocalFileSystem for --object-store file. In 0.12.5 LocalFileSystem never fsyncs, so a power loss can leave files that exist under their final names with no data (influxdata/EAR#7132). This branch adds upstream's opt-in LocalFileSystem::with_fsync, which influxdb_pro turns on.

base-object-store-0125 is upstream tag v0.12.5 (8f16588). Every upstream change listed here first shipped in object_store 0.14.0 or 0.14.1.

How clean the backport is

Commit Upstream How it applied
b5b651f close with error checking apache#676 Code unchanged. Cargo.toml conflict only: 0.12.5 has no tokio feature, and the new optional nix dependency uses 0.30 to match 0.12.5's dev-dependency. About 6 changed lines against upstream.
8b19aed with_fsync apache#643 put_opts, multipart and the helper functions applied unchanged. 0.13 merged copy/copy_if_not_exists/rename into copy_opts/rename_opts, so the fsync changes to those three methods were ported by hand onto 0.12.5's separate methods. About 68 of roughly 340 changed lines differ from upstream.
8f73b1d fsync the source delete in create-mode rename apache#758 Effectively re-implemented. 0.12.5 has no rename_opts, so the same behaviour is a new rename_if_not_exists override, plus a delete_location helper that keeps 0.12.5's path validation. About 89 changed lines differ from a 71-line upstream change.

The hand-ported parts are the ones to review closely. cargo test covers them with fsync on (file_test_fsync runs the integration suite, including copy, rename and copy_if_not_exists, with with_fsync(true)), and influxdb_pro's crash-simulation tests exercise them end to end.

Changes of our own, not upstream:

  • 6b7c9a0: the close-error tests from feat(local): explicit close with error checking for LocalFileSystem apache/arrow-rs-object-store#676 rebuilt a File from a closed descriptor or handle. Fixed after review. Upstream main still has the original; we plan to propose the fix there.
  • 2d44269, 421b019: rust-version raised from 1.64 to 1.71, then 1.79, to match what the new dependencies declare (nix 0.30, windows-sys 0.61, quick-xml 0.41).
  • fde97c6, 4d67bb3: the MSRV job's hand-pinned dependency list had gone stale and failed before checking anything. It now uses Cargo's Rust-version-aware resolver and checks all features.
  • 98cf6e8: formatting.

Cherry-picked from upstream to fix CI failures that were already present on v0.12.5, all with -x:

Getting rid of the fork

The fork exists only because influxdb_pro cannot move to object_store 0.14 yet: its DataFusion fork (52.5) and parquet 57 require 0.12. DataFusion main moved to object_store 0.14.2 and arrow/parquet 60 on 2026-09-18 (apache/datafusion#25335), but the latest release, 55.1.0, still uses 0.13.

When influxdb_pro rebases its DataFusion fork onto a release that uses 0.14:

  • Removing the fork itself is small. Delete the [patch.crates-io] object_store entry in ent/Cargo.toml and oss/Cargo.toml and depend on object_store 0.14.x. Everything picked here is in 0.14.1, and with_fsync(bool) has the same name and meaning there, so influxdb_pro's .with_fsync(true) call stays as it is.
  • What is lost: only the close-test fix (6b7c9a0). It affects this crate's own tests, not users of it, and should go upstream regardless.
  • The real cost is elsewhere and is not caused by this fork. object_store 0.13 changed the ObjectStore trait (copy_opts/rename_opts, convenience methods moved out of the trait), which touches influxdb_pro's roughly 72 ObjectStore implementations. That cost comes with any upgrade to 0.13 or later.

Known gap, same as upstream

with_fsync fsyncs only the directories a call itself creates. A write into a directory that a concurrent write has just created can return before that directory's own entry is durable. On ext4 one fsync commits the whole journal, so this probably does not occur there; POSIX does not guarantee it. We can address that gap if proves to be more than theoretical.

Testing

  • Locally on 1.97.0:
    • cargo check --all-features --all-targets;
    • cargo clippy -- -D warnings with CI's feature sets (default, no-default, fs, aws, gcp, azure, http, all-features/all-targets);
    • cargo fmt --check;
    • cargo test: 80 passed and 1 ignored (lib), 19 passed (doc), 2 passed (other).
  • cargo check with default features and with --all-features on Rust 1.79.0, using a lockfile generated with CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=fallback.
  • In influxdb_pro, with this branch pinned and with_fsync(true) enabled, crash-simulation tests (an strace trace of the server, replayed to the state a power cut would leave) find no file left non-durable. Without it, every file written during the run is left empty.

jgiannuzzi and others added 3 commits October 2, 2026 15:46
…pache#676)

(cherry picked from commit 9b8cf52)

Backport to 0.12.5: Cargo.toml conflict only. 0.12.5 has no `tokio` feature,
so `fs = ["walkdir", "nix", "windows-sys"]`, and `gcp` keeps 0.12.5's
dependency list. The new optional `nix` dependency uses 0.30, matching
0.12.5's existing `nix` dev-dependency (upstream uses 0.31).
(cherry picked from commit dd0afcbc7535c2b8e78a1aad4396e7ce43c7a42a)

Backport to 0.12.5: put_opts, multipart and the helper functions applied
unchanged. object_store 0.13 merged copy/copy_if_not_exists/rename into
copy_opts/rename_opts; here the fsync changes are ported by hand onto
0.12.5's separate copy, copy_if_not_exists and rename methods, and the
with_fsync docs name those methods. file_test_fsync drops the call to
list_with_offset_exclusivity, an integration helper 0.12.5 does not have.
(cherry picked from commit 596e1f6)

Backport to 0.12.5: 0.12.5 has no rename_opts, and rename_if_not_exists is
the ObjectStore trait's default copy_if_not_exists + delete. LocalFileSystem
now overrides rename_if_not_exists so the source delete goes through
delete_location with fsync. delete_location takes the filesystem path
already validated by LocalFileSystem::path_to_filesystem, so delete keeps
0.12.5's path validation (which rejects #<digits> staging names).
Copilot AI balanced review requested due to automatic review settings October 2, 2026 15:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

New tests exceed the declared MSRV and construct files from invalid closed descriptors or handles.

Review effort: Balanced
Findings: 2 High severity

Open (2)
What changed in this PR

Backports durable local filesystem writes and explicit close-error handling from upstream.

Changes:

  • Adds opt-in fsync for writes, copies, renames, and multipart uploads.
  • Adds checked file closing and durability tests.
  • Adds platform-specific close dependencies.
File Description
src/​local.rs Implements fsync, checked closes, and tests.
Cargo.toml Adds Unix and Windows dependencies.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/local.rs Outdated
Comment on lines +2022 to +2023
use std::os::fd::FromRawFd;
use std::os::unix::io::AsRawFd;

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed differently in 2d44269: the backport's own dependencies already need Rust 1.71 (nix 0.30 declares 1.69, windows-sys 0.61 declares 1.71), so rust-version now says 1.71.0 instead of 1.64.0. std::os::fd is stable there. The test that used it was also rewritten in 6b7c9a0 (see the other thread).

Comment thread src/local.rs Outdated
let file = {
let fd = file.as_raw_fd();
super::close_file(file).unwrap();
unsafe { std::fs::File::from_raw_fd(fd) }

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6b7c9a0. close_file now delegates to close_raw_fd (Unix) and close_raw_handle (Windows). The tests call those with values that can never be open (a descriptor at the RLIMIT_NOFILE soft limit, and a null handle), so no File is rebuilt from a freed descriptor and no other test's descriptor can be closed. Upstream main still has the original test; we'll propose this fix there too.

reidkaufmann and others added 11 commits October 2, 2026 17:50
The close-error tests from apache#676 closed a file and then rebuilt a File from
the freed descriptor or handle to close it again. That breaks the
from_raw_fd/from_raw_handle contract, and in a parallel test run another
thread can be handed the same descriptor in between, so the test could
close an unrelated file.

Split close_file into close_raw_fd (Unix) and close_raw_handle (Windows)
and test those with a value that can never be open: a descriptor at the
open-file limit, and a null handle. close_file's behavior is unchanged.

Raised in review (Copilot) on #2.
apache#676 added nix 0.30 (rust-version 1.69) and windows-sys 0.61 (rust-version
1.71), so the crate no longer builds on the declared 1.64. Upstream is at
1.85.
(cherry picked from commit 0c380a1)
…he#602)

Updates the requirements on [quick-xml](https://github.com/tafia/quick-xml) to permit the latest version.
- [Release notes](https://github.com/tafia/quick-xml/releases)
- [Changelog](https://github.com/tafia/quick-xml/blob/master/Changelog.md)
- [Commits](tafia/quick-xml@v0.38.0...v0.39.0)

---
updated-dependencies:
- dependency-name: quick-xml
  dependency-version: 0.39.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit b3027f8)
…he#727)

Updates the requirements on [quick-xml](https://github.com/tafia/quick-xml) to permit the latest version.
- [Release notes](https://github.com/tafia/quick-xml/releases)
- [Changelog](https://github.com/tafia/quick-xml/blob/master/Changelog.md)
- [Commits](tafia/quick-xml@v0.39.0...v0.40.1)

---
updated-dependencies:
- dependency-name: quick-xml
  dependency-version: 0.40.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit b058903)
…he#785)

Updates the requirements on [quick-xml](https://github.com/tafia/quick-xml) to permit the latest version.
- [Release notes](https://github.com/tafia/quick-xml/releases)
- [Changelog](https://github.com/tafia/quick-xml/blob/master/Changelog.md)
- [Commits](tafia/quick-xml@v0.40.1...v0.41.0)

---
updated-dependencies:
- dependency-name: quick-xml
  dependency-version: 0.41.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit fd44ebb)
…ting (apache#799)

Adds a rust-toolchain.toml (mirroring apache/arrow-rs) pinning the
toolchain to 1.97.0 so CI uses a stable, reproducible Rust version and
is not surprised by new lints in future releases.

Rust 1.97.0's clippy added the useless_borrows_in_formatting lint, which
flags redundant references in format! arguments. Fixes the resulting
failures in the azure module.

Closes apache#798

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit e64e8e3)
quick-xml 0.41 (cherry-picked from apache#785 for RUSTSEC-2026-0194/0195)
declares rust-version 1.79; it is used by the aws, azure and gcp features.
The job pinned a hand-maintained list of dependency versions. The list
went stale: cargo update -p quote --precise 1.0.41 now fails because
zerovec-derive 0.11.6 requires quote >= 1.0.44, so the job failed before
checking anything. Generate the lockfile with
CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=fallback instead, which picks the
newest versions compatible with the crate's rust-version.
cargo msrv verify ran cargo check with default features only, so the
rust-version requirement of feature-only dependencies (quick-xml 0.41 for
aws/azure/gcp needs 1.79) went unchecked.
@reidkaufmann

Copy link
Copy Markdown
Collaborator Author

Pushed review and CI fixes; all checks now pass, including Verify MSRV. The PR description has been rewritten to cover how clean the backport is (which parts were hand-ported and need the closest review), the commits that are ours rather than upstream's, the upstream CI fixes cherry-picked with -x, and what it takes to drop this fork once influxdb_pro can move to object_store 0.14.

@reidkaufmann
reidkaufmann requested review from a team, alamb and crepererum October 5, 2026 19:38
@reidkaufmann
reidkaufmann merged commit c789dc7 into base-object-store-0125 Oct 5, 2026
9 checks passed
@reidkaufmann
reidkaufmann deleted the upgrade-object-store-ver0125-a branch October 5, 2026 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants