Skip to content

[ee] chore: move quick-xml past RUSTSEC-2026-0194/0195 (datafusion, object_store, parquet) - #11427

Draft
alpetric wants to merge 4 commits into
mainfrom
chore/bump-datafusion-object-store-quick-xml
Draft

alpetric wants to merge 4 commits into
mainfrom
chore/bump-datafusion-object-store-quick-xml

Conversation

@alpetric

@alpetric alpetric commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

quick-xml < 0.41 is affected by two CVSS 7.5 denial-of-service advisories, and both are
reachable from the XML this backend parses (S3/Azure/GCS listing responses, SAML metadata):

  • RUSTSEC-2026-0195 — unbounded namespace-declaration allocation in NsReader; a single
    start tag drives ~3× its byte size in resolver heap inside the parser, so bounding the input
    does not bound the allocation. Memory exhaustion / OOM kill.
  • RUSTSEC-2026-0194 — quadratic run time in the duplicate-attribute-name check; a crafted
    start tag pins a CPU core for minutes. NsReader reaches it internally and has no opt-out
    before 0.41.

Both are fixed only in quick-xml >= 0.41.0 — there is no patched 0.37/0.38/0.39/0.40 release.

The same move drops thrift from the tree entirely (parquet >= 59 no longer uses it), which
closes GHSA-2f9f-gq7v-9h6m / CVE-2026-43868 (Dependabot alert #465, WIN-2531).

Why datafusion becomes a git pin

Only object_store >= 0.14.1 depends on quick-xml >= 0.41, and object_store cannot be
upgraded on its own: the backend hands its Arc<dyn ObjectStore> to DataFusion through
SessionContext::register_object_store, so both must be the same object_store.

No published datafusion accepts object_store 0.14 — the newest release (55.1.0) requires
^0.13.2. DataFusion main moved to arrow/parquet 60 and object_store 0.14.2 in
apache/datafusion#25335, so the dependency is pinned to that line until 56.0.0 is published, at
which point the pin becomes datafusion = "56" and nothing else has to change.

The alternatives, for the record: stay on a released DataFusion and carry a fork of
object_store 0.13.2 with its quick-xml requirement bumped (the two upstream bumps
0.39→0.40.1→0.41 were Cargo.toml-only, so this would work, but it means maintaining a fork of an
Apache crate); or wait for DataFusion 56. Both leave thrift in the tree for longer, since
parquet 59+ requires object_store >= 0.13.2 and therefore datafusion >= 54, which already
forces the tokio bump below.

Versions

crate before after
quick-xml 0.37.5 0.41.0
thrift 0.17.0 gone (no longer in the tree)
object_store 0.12.0, git apache/arrow-rs-object-store@36752c97 0.14.2, crates.io
datafusion 47.0.0, crates.io git apache/datafusion@2cac7132 (workspace version 55.1.0)
arrow 55.2.0 60.0.0
parquet 55.2.0 60.0.0
tokio =1.46.1 =1.53.1
samael git njaremko/samael@f879f194 (0.0.20) git njaremko/samael@2d27dfe4 (0.0.22)
sqlparser 0.55.0 0.63.0 (transitive, via datafusion)
reqwest for object_store's HTTP client reqwest_object_store alias on 0.12 the workspace reqwest 0.13

Two bumps are forced rather than chosen:

  • tokio =1.46.1 → =1.53.1. Every datafusion >= 54 requires tokio ^1.52. The exact
    pin was added in 1a85dc7 with no recorded rationale and no linked issue; it is kept exact,
    just moved to the current release. This is the line in the diff that most wants a second
    look.
  • samael forward one rev. It was the other source of a quick-xml 0.37 copy in the
    lockfile; upstream's 0.0.22 does the same bump ("security: update quick-xml to version
    0.41.0"). Note that quick-xml >= 0.38 no longer trims text content; samael strips
    whitespace before base64-decoding certificates (crypto/cert_encoding.rs), so pretty-printed
    IdP metadata still validates.

What happened to the object_store git pin

backend/Cargo.toml pinned object_store to arrow-rs-object-store@36752c97 (and carried a
matching [patch.crates-io] entry) since df9f827, "fix azure object storage regression due to
object_store regression" — the fix it was reaching for is
#334, "Fix query parameter signing in Azure".
That commit is an ancestor of the v0.12.1 tag (git compare 36752c97...v0.12.1 → ahead 2,
behind 0), so every commit the pin carried is in 0.12.1 and later. Both the pin and the [patch]
entry are dropped; nothing is lost.

Code adaptations the upgrade forces

  • object_store 0.13 moved get, get_range, head, delete, copy, copy_if_not_exists,
    rename, rename_if_not_exists, put and put_multipart off ObjectStore onto a new
    ObjectStoreExt extension trait — every call site imports it (re-exported from
    windmill_object_store::object_store_reexports).
  • The ObjectStore trait itself now requires delete_stream and copy_opts/rename_opts
    instead of delete/copy/rename, which reshapes the FilesystemStoreIgnoringAttributes
    wrapper.
  • delete_stream now takes a 'static stream, so the DuckLake fork-data deletion owns its path
    chunks instead of borrowing them.
  • Error::NotImplemented gained operation/implementer fields. The old arm matched it as a
    unit variant, which after the upgrade would have compiled as an irrefutable binding and turned
    that arm into a catch-all for every object-store error; it now destructures the fields and
    reports them.
  • PutMultipartOpts is a deprecated alias of PutMultipartOptions.
  • DataFusion: prelude::NdJsonReadOptions → prelude::JsonReadOptions, and DFSchema no longer
    converts into arrow::Schema (df.schema().inner().clone() gives the SchemaRef the
    ArrowWriter wants).
  • parquet: WriterPropertiesBuilder::set_max_row_group_size(n) →
    set_max_row_group_row_count(Some(n)). The new byte-based companion defaults to None, so row
    grouping is unchanged.
  • object_store 0.14 builds its HTTP clients with reqwest 0.13, the same major the workspace
    already uses, so the reqwest_object_store alias (a second reqwest at 0.12) is folded into
    the workspace reqwest. Cargo rejects the same crate twice under different names in one
    package, so this is required rather than tidy-up. The TLS backend does not change: the alias
    asked for rustls-tls-native-roots, and reqwest 0.13's default-tls is rustls
    (reqwest-0.13/Cargo.toml: default = ["default-tls", ...], default-tls = ["rustls"]), while
    object_store's own aws/azure/gcp features each force reqwest/rustls on top. Still
    rustls, still the platform trust store — reaching it through rustls-platform-verifier rather
    than rustls-native-certs. Nothing in the tree turns on reqwest's native-tls, and the
    connector never calls use_native_tls(). The four .no_gzip()/.no_brotli()/.no_deflate()/
    .no_zstd() calls and the empty ACCEPT_ENCODING header are unchanged, and those builder
    methods exist in 0.13 whether or not the matching feature is on.
  • #![recursion_limit = "256"] on the windmill binary: sqlparser 0.63's AST nests deeply
    enough that inferring Send/Sync for the futures spawned in main exceeds the default
    128-step limit.

Test plan

Compiled, with pkg-config/libxmlsec1 present so enterprise_saml (and therefore samael)
is really built:

  • cargo check --features all_sqlx_features (the CE CI check)
  • cargo check --features all_sqlx_features,private (the EE CI check)
  • cargo check --features private,enterprise,parquet
  • cargo check --features tantivy,private,quickjs
  • cargo build --features tantivy,private,quickjs and --features private,parquet,quickjs

Exercised end to end on a local instance built from this branch, against MinIO (S3) and
Azurite (Azure Blob), with MinIO as the workspace and instance object store:

  • Upload CSV / JSON / NDJSON / Parquet through upload_s3_file; download_s3_file returns
    the Parquet byte-for-byte identical to the input
  • load_file_preview renders the CSV, the JSON and the Parquet correctly (the Parquet
    through the Arrow reader → CSV), load_file_metadata reports the right size and mtime
  • load_csv_preview, load_parquet_preview and load_table_count (the DataFusion
    SessionContext paths) return the right rows and count: 3, including limit/offset,
    sort_col and search_col/search_term
  • download_s3_parquet_file_as_csv
  • list_stored_files_paged (the storage_list_ee quick-xml path) lists the bucket root, a
    delimited prefix, and paginates with a continuation token; list_stored_files filters with
    search; move_s3_file (copy + delete) and delete_s3_file behave
  • Same listing, pagination, upload, download, load_csv_preview and load_file_metadata
    against Azurite as a secondary storage — the Azure EnumerationResults XML and the
    Azure marker pagination
  • A Postgres script with -- s3 format=json|csv|parquet streams its result set to the
    bucket; all three objects open correctly through the preview endpoints, and the Parquet is
    valid to an independent reader (pyarrow: parquet-rs version 60.0.0, ZSTD, 3 rows)
  • A job run with multipart/form-data uploads its file through
    build_object_store_client (windmill-api/src/args.rs) and the argument arrives as
    {"s3": "windmill_uploads/..."}
  • Service-log columnar store with MODE=standalone+search: logs_store/v1/dt=…/mode=…/
    fills with ing-*.parquet plus a manifest carrying per-file row counts and ts bounds;
    pyarrow confirms parquet-rs version 60.0.0 + ZSTD + the expected schema;
    search/service_logs (DataFusion over that store: contains/lower/coalesce and
    timestamp literals) returns the right lines, including a one-minute min_ts/max_ts
    window and an empty result for a term that is not there; search/count_service_logs
    (tantivy) returns its per-host bucket
  • cargo test -p windmill-object-store --lib (52), -p windmill-indexer --lib (31),
    -p windmill-worker-volumes --lib (39), and the S3 integration tests
    app_s3_onbehalf (7), object_storage_test_ssrf (1), workspace_storage_by_name (2),
    sign_s3_objects_authz (2) — all green

Not exercised, and why:

  • Log-store compaction and garbage collection only run on partitions closed more than
    COMPACTION_LAG_HOURS = 3 ago, which a single session cannot reach; the DataFusion
    streaming-sort path they use is covered by windmill-indexer's unit tests rather than a live
    run.
  • GCS has no local emulator here. Its listing goes through the same
    S3ListBucketResult deserializer as S3 (storage_list_ee.rs), which is exercised against
    MinIO, and its object_store client is the same crate version.
  • Worker volumes are covered by windmill-worker-volumes' own LocalFileSystem round-trip
    tests rather than a live mount.

cargo audit

cargo audit on the new lockfile against the pre-change one: the only difference is
RUSTSEC-2026-0194 and RUSTSEC-2026-0195 are gone, and nothing new appears (17 → 15
vulnerabilities). Of the 15 that remain, the RustSec database records a CVSS score for two —
rsa RUSTSEC-2023-0071 (5.9) and rustls RUSTSEC-2026-0285 (5.3) — and both are below 7.0. The
other 13 (h2, hickory-proto ×2, magic-crypt, ring, rkyv, rustls-webpki ×7) carry no
CVSS vector in RustSec and none of them is quick-xml or thrift; they are all pre-existing on
main and out of scope here.


Left as a draft deliberately. Two calls in here want a human before either side is marked
ready: pinning datafusion to an unreleased main rev (an alternative, a fork of object_store
0.13.2, is described above), and moving the exact tokio pin from =1.46.1 to =1.53.1, which
DataFusion >= 54 forces and whose original rationale is not recorded anywhere. The change also
spans *_ee.rs through the symlinks — see the companion PR,
windmill-labs/windmill-ee-private#836.

One behaviour change the upgrade carries, which review should weigh

In object_store 0.14 delete exists only on ObjectStoreExt, and its blanket
implementation goes through delete_stream — i.e. the provider's bulk delete API, even for
one key. On S3 that is POST /?delete (DeleteObjects) instead of the old DELETE /key; on
Azure it is the Blob Batch endpoint (POST ?comp=batch). Every delete call site in the backend
changes request shape this way; the implementation cannot opt out per call.

Exercised both, and the object really is gone afterwards: delete_s3_file against MinIO and
against Azurite, each verified by a follow-up listing. Deleting a key that does not exist behaves
as before — S3 succeeds (the bulk API is idempotent for missing keys), Azure surfaces the
provider's 404 through the same handler error it surfaced before.

The caveat is the class of endpoint neither emulator represents: upstream added
AmazonS3Builder::with_disable_bulk_delete precisely because "some S3-compatible providers do not
implement the bulk DeleteObjects API"
, and Windmill lets an operator point a resource at any
endpoint. A note now sits at the S3 builder saying where that switch goes. It is deliberately not
set: the backend already requires DeleteObjects of every endpoint, because the log-cleanup sweep,
the instance log flush and the DuckLake fork-data deletion all call delete_stream directly today
(windmill-api-settings/src/log_cleanup.rs, src/monitor.rs,
windmill-api-workspaces/src/workspaces_extra.rs, the last with an explicit 1000-key chunk
comment). Turning the switch on would move those off the batch API too, which is a behaviour and
throughput change of its own rather than a conservative default — so the choice belongs to a
reviewer, not to this patch.

alpetric and others added 2 commits September 29, 2026 14:56
… set

`quick-xml` < 0.41 is affected by RUSTSEC-2026-0194 and RUSTSEC-2026-0195
(both CVSS 7.5). The only `object_store` releases that depend on
`quick-xml` >= 0.41 are 0.14.1+, and no published `datafusion` release
accepts `object_store` 0.14 yet, so `datafusion` is pinned to the
apache/datafusion commit that moved to arrow/parquet 60 and
`object_store` 0.14.2. That set also drops `thrift` (parquet >= 59 no
longer uses it), closing GHSA-2f9f-gq7v-9h6m.

- datafusion 47.0.0 -> git 2cac7132 (arrow/parquet 60, object_store 0.14.2)
- object_store git 36752c97 (0.12.0) -> 0.14.2; the pinned rev's Azure
  query-signing fix (arrow-rs-object-store#334) shipped in 0.12.1, so the
  `[patch.crates-io]` entry and the git dep are both dropped
- parquet 55.2.0 -> 60.0.0, arrow 55.2.0 -> 60.0.0
- quick-xml 0.37 -> 0.41; samael pinned forward to the rev that did the
  same, since it was the other source of a 0.37 copy
- tokio =1.46.1 -> =1.53.1, forced by datafusion's `tokio ^1.52`
- `reqwest_object_store` (reqwest 0.12) folded into the workspace
  `reqwest` 0.13, which object_store 0.14 now builds its clients with

Code adaptations: `get`/`get_range`/`head`/`delete`/`copy`/`put`/
`put_multipart` moved to `ObjectStoreExt`, `copy`/`rename` to
`copy_opts`/`rename_opts`, `delete` to `delete_stream` with a `'static`
stream, `Error::NotImplemented` gained fields, `NdJsonReadOptions` became
`JsonReadOptions`, `DFSchema: Into<Schema>` is gone, and
`set_max_row_group_size` became `set_max_row_group_row_count`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kv72vdCDggCZEjJSmNCwnX
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kv72vdCDggCZEjJSmNCwnX
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Deploying windmill with  Cloudflare Pages  Cloudflare Pages

Latest commit: 8deef42
Status: ✅  Deploy successful!
Preview URL: https://1c7e5033.windmill.pages.dev
Branch Preview URL: https://chore-bump-datafusion-object.windmill.pages.dev

View logs

…-object-store-quick-xml

# Conflicts:
#	backend/ee-repo-ref.txt
@alpetric

Copy link
Copy Markdown
Collaborator Author

/review

@claude

claude Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @alpetric's task in 5m 41s —— View job


cc @alpetric

Mergeable, but should ideally address nits: undocumented = on the tokio pin (backend/Cargo.toml:464), no tracked owner for removing the datafusion git pin (backend/Cargo.toml:673-676).

I found no correctness bug in the OSS half of this diff. The mechanical adaptations are right, and the claims in the description check out against Cargo.lock.

What I verified

Lockfile matches the description. arrow 60.0.0, parquet 60.0.0, object_store 0.14.2 (registry, not git), quick-xml 0.41.0, tokio 1.53.1 — each appears exactly once; thrift is gone. Every datafusion-* subcrate resolves to the same git rev, with no registry/git split that would have produced two incompatible DFSchema/SessionContext types. object_store 0.14.2's own dep row lists reqwest 0.13.5, which confirms the reqwest_object_store fold-in is correct rather than merely convenient — PublicOnlyConnector now hands object_store::client::HttpConnector a client from the same reqwest the rest of the workspace uses.

Error::NotImplemented (windmill-object-store/src/lib.rs:101). This is the sharpest hazard in the upgrade and it is handled. The old unit-variant arm would have compiled as an irrefutable binding after the bump and silently swallowed every object-store error into BadRequest, including NotFound and PermissionDenied — a class of bug that produces wrong HTTP statuses everywhere and no compiler complaint. Destructuring { operation, implementer } is the fix, the surrounding arms are untouched, and the _ => fallback is still last. The unit test was updated with it.

FilesystemStoreIgnoringAttributes (lib.rs:995-1078). The wrapper's whole reason to exist is stripping attributes so LocalFileSystem doesn't reject uploads with NotImplemented, and both mutating entry points still do it: put_opts and put_multipart_opts zero opts.attributes before delegating. Dropping the get_range/head/copy_if_not_exists/rename_if_not_exists overrides is safe — those are now ObjectStoreExt defaults expressed in terms of get_opts and copy_opts/rename_opts, both of which the wrapper delegates, so the strip is still on every path that can carry attributes.

delete_stream ownership (workspaces_extra.rs:2185). chunk.to_vec().into_iter() rather than chunk.iter().cloned() is required, not stylistic — the new signature wants 'static and .cloned() still borrows the slice. The 1000-key chunking (S3 DeleteObjects cap) is preserved.

df.schema().inner().clone() (lib.rs:1924). Equivalent to the old DFSchema -> Schema conversion: both carry fields and metadata through, and it hands ArrowWriter the SchemaRef directly instead of re-wrapping in a fresh Arc.

#![recursion_limit = "256"] (main.rs:11). Correctly placed (inner attribute before the first item), and windmill is the workspace's only [[bin]], so there is no sibling binary that would need the same raise.

The ~15 use ... ObjectStoreExt additions are mechanical and compiler-enforced — a miss is a build error, not a silent behavior change.

Findings

Both are detailed in inline comments and neither blocks merge:

  • P2 — backend/Cargo.toml:464, the = on the tokio pin. Moving =1.46.1 → =1.53.1 is forced and fine. Carrying the = forward without a recorded reason is the part worth deciding: it blocks every tokio patch release, including a security patch, behind a manual bump — the same hazard this PR exists to close, in a different crate. Either relax to "1.53.1" (the lockfile still pins the exact build) or keep = and record the constraint, per the AGENTS.md rule that "Comments record constraints, not narration… why a non-obvious approach is required." The workspace's other exact pins (rustls = "=0.23.35", libxml = "=0.3.3", native-tls = ">=0.2, <0.2.17") all read as deliberate; this one is the odd one out.
  • P2 — backend/Cargo.toml:673-676, the datafusion git pin. I'd take the pin over the object_store fork: a fixed rev on an Apache repo is reproducible and self-removing, whereas a fork is an indefinite maintenance obligation for a bump upstream will make for you. The gap is that nothing will tell you when to remove it — Dependabot doesn't watch git revs, and the rev self-reports version = "55.1.0", so it can quietly become permanent. Reference the tracking ticket in the comment. Also worth stating plainly in the PR that this is unreleased main code shipping in an image, and that cargo full-clones git deps, so cold CI/Docker builds get slower.

On the quick-xml 0.38+ no-trim change

You called this out for samael; I checked it also lands on Windmill's own deserializers (storage_list_ee.rs:294/566/818, S3ListBucketResult / AzureEnumerationResults) and it does not bite. The change affects whitespace inside an element's text node; whitespace between elements is still ignored for struct fields. test_parse_s3_list_response feeds indented, pretty-printed XML and asserts size == Some(42) and the exact key, so the case that would actually break (<Size>\n 42\n</Size>) is the one no provider emits, and the pretty-printed case is already pinned by a test. No action needed — recording it so it isn't re-derived.

AGENTS.md compliance

Clean. Comments record constraints rather than narrate the change (the recursion_limit note names what breaks; the datafusion note names the condition for removal) and none describe drafting history. No customer attribution. No new pub fn, module, or exported surface, so the public-surface checklist doesn't apply. The ee-repo-ref.txt bump follows the docs/enterprise.md companion-PR convention.

Test coverage

Backend (Rust). This is a dependency upgrade with mechanical call-site adaptation, so per policy no new tests are owed — the existing surface is what should cover it, and it does. object_store_error_to_error's unit test was updated in place for the new NotImplemented shape, which is the one behavioral change in the OSS diff and the one that most needed pinning. windmill-object-store, windmill-indexer and windmill-worker-volumes --lib suites were run. I'd resist adding anything further here; a test that re-exercises ObjectStoreExt delegation would be scaffolding.

Manual verification. Your test plan is unusually thorough for a bump and covers what I'd have asked for — both XML dialects (S3 via MinIO, Azure EnumerationResults via Azurite) against the advisory'd parser, both pagination schemes, the DataFusion SessionContext paths, byte-for-byte Parquet round-trip, and an independent pyarrow reader confirming parquet-rs version 60.0.0. The three gaps you name (GCS, log-store compaction/GC behind the 3-hour lag, worker volumes) are each argued down to a path that is covered by a shared code path or a unit test, which is the right way to leave them.

One thing I could not check from here: the EE half is unreviewed. The windmill-ee-private checkout in this workspace is still at the old ref (5a2b6b8), so the *_ee.rs files I can see still contain set_max_row_group_size, PutMultipartOpts, and bare .head() / .get_range() calls without ObjectStoreExt in scope. Those must all be handled by windmill-labs/windmill-ee-private#836, and your --features all_sqlx_features,private check is what proves it — but this review covers only the OSS side.

--- · branch chore/bump-datafusion-object-store-quick-xml

@github-actions

Copy link
Copy Markdown
Contributor

Codex Review

Good to merge

No issues found. Checked dependency consistency, object-store API migrations, DataFusion/parquet adaptations, error handling, and affected public surfaces.

Test coverage

The reported Cargo checks, crate tests, audit, and MinIO/Azurite end-to-end scenarios thoroughly cover the changed layers.

GCS, delayed log-store compaction/GC, and live worker-volume mounts remain unexercised, but their shared paths and unit coverage make these reasonable, non-blocking gaps.

@github-actions

Copy link
Copy Markdown
Contributor

Pi Review

cc @alpetric

Mergeable, but should ideally address nits: undocumented TLS backend change in the object-store HTTP client, datafusion git pin to unreleased main, tokio exact pin moved without a recorded rationale

This is a large, carefully-executed dependency bump. I read the full code diff (all 21 non-lockfile files) and traced each object_store 0.13/0.14 adaptation against the new trait shape. I found no P0/P1 bugs — the risky adaptations (Error::NotImplemented destructuring, the ObjectStore trait method reshuffle, delete_stream 'static borrows, NdJsonReadOptions → JsonReadOptions, DFSchema::inner().clone()) are all correct. The items below are the things I think still want a conscious decision, not code defects.

P2 — object-store HTTP client's TLS backend silently changed from rustls to native-tls

backend/windmill-object-store/src/lib.rs:618 (PublicOnlyConnector::connect) now builds its client with the workspace reqwest (backend/Cargo.toml:512), which has default-features on (default-tls → OpenSSL/native-tls). The deleted alias was reqwest = { default-features = false, features = ["rustls-tls-native-roots"] }. So every S3/Azure/GCS connection made through this SSRF-guarded client moved from rustls (native roots) to OpenSSL.

The PR body lists the reqwest fold as "required rather than tidy-up" (correct — cargo rejects the same crate twice in one package) but doesn't note the TLS-backend side effect. In practice this is almost certainly benign: both backends validate against the system trust store, and openssl/native-tls are already hard deps of the crate graph, so no new system dependency is introduced. I'm flagging it only because it's a real, observable behavior change on a security-relevant path that a reviewer would otherwise have to notice by diffing Cargo.toml against the lockfile. Worth a one-line acknowledgement, not a code change.

Note — datafusion pinned to an unreleased main rev

backend/Cargo.toml:676. You already flagged this as a draft decision, so this is my input rather than a new finding: a fixed-commit pin is reproducible and consistent with this repo's existing style (sqlx, tiberius, tokio-postgres fork, samael, tinyvector are all git-rev pins), so the only real cost is the upgrade path — the code comment promising "drop the git pin for datafusion = "56"" is the right backstop. The alternative you describe (forking object_store 0.13.2 to bump its quick-xml requirement) trades a git pin for a fork of an Apache crate, which I'd call the worse of the two. I agree with the git pin; just make sure the eventual 56.0.0 migration isn't forgotten.

Note — tokio exact pin moved =1.46.1 → =1.53.1

backend/Cargo.toml:464. Forced by DataFusion >= 54 (tokio ^1.52), and you already called this out as the line most wanting a second look. My read: keeping it exact matches the existing convention (and the repo's general exact-pinning style), and 1.53.1 is a published release, so there's no freshness/reproducibility concern. The only thing I'd want before this stops being a draft is a one-line comment recording why tokio is exact-pinned at all, so the next forced bump doesn't have to rediscover that the rationale isn't recorded anywhere.

Considered and dismissed

  • Error::NotImplemented arm (lib.rs:101) — the old unit-variant match would have become an irrefutable pattern and swallowed every object-store error into BadRequest("Operation not yet implemented"). The new destructuring is correct, and it's the only match on this variant in the tree (grep-verified).
  • FilesystemStoreIgnoringAttributes dropping get_range/head/delete/copy/rename/*_if_not_exists — all moved to ObjectStoreExt default impls that delegate to get_opts/delete_stream/copy_opts/rename_opts, which the wrapper still implements. The attribute-dropping behavior is preserved because the put/put_multipart defaults route through put_opts/put_multipart_opts. No behavior change.
  • delete_fork_ducklake_data chunk.iter().cloned() → chunk.to_vec().into_iter() (workspaces_extra.rs) — correctly fixes the 'static stream requirement by owning the paths; the 1000-key chunking is preserved.
  • NdJsonReadOptions → JsonReadOptions (lib.rs:1914) — plain rename; even if the default format is Unspecified (auto-detect), the file written here is {...}\n per line, which auto-detects as NDJSON, and the -- s3 format=json path is covered by your manual test plan.
  • df.schema().inner().clone() (lib.rs:1935) — yields the SchemaRef the ArrowWriter wants; confirmed by the byte-identical Parquet round-trip in your test plan.
  • quick-xml 0.38+ text-trimming change — the only direct quick_xml::de consumers are the S3/Azure listing deserializers (storage_list_ee.rs), which parse compact provider XML (no whitespace-padded text), and they're exercised against MinIO + Azurite in your plan; the samael cert path is handled per your note. No other quick_xml call sites exist in the backend (grep-verified).

Test coverage

  • Backend (Rust): Appropriate for a dependency bump. The touched crate has existing unit tests (windmill-object-store test_object_store_error_mapping, filesystem round-trip tests) that were updated for the new NotImplemented shape, and you ran cargo test -p windmill-object-store --lib, -p windmill-indexer --lib, -p windmill-worker-volumes --lib. There's no new logic here worth a fresh unit test — the changes are mechanical trait-API adaptations whose correctness is pinned by cargo check across the feature matrix plus the listed integration tests.
  • CI / docs / config: Cargo.toml/Cargo.lock consistency is covered by your cargo check matrix and the cargo audit before/after comparison (19xx/0195 gone, nothing new). No automated tests expected for the lockfile itself.

Manual verification still needed: nothing further beyond what your test plan already records — the S3 (MinIO), Azure (Azurite), DataFusion preview/count paths, Parquet round-trip, and service-log columnar store were all exercised end-to-end. The two spots you explicitly left unexercised (log-store compaction/GC beyond the 3-hour lag, and GCS listing) are both defensible: compaction's streaming-sort path is unit-covered by windmill-indexer, and GCS listing shares the S3ListBucketResult deserializer + the same object_store crate version as the MinIO-verified S3 path. If anything, a one-off GCS listing smoke test would be the single cheapest remaining confidence boost, but it's not a merge blocker.

Comment thread backend/Cargo.toml
hyper-tls = "^0.6"
hyper-util = { version = "^0.1", features = ["client-legacy", "http1", "tokio"] }
tokio = { version = "=1.46.1", features = ["full", "tracing", "time"] }
tokio = { version = "=1.53.1", features = ["full", "tracing", "time"] }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 — the = is the part worth deciding, not the number.

You flagged this line yourself. My read: moving =1.46.1 → =1.53.1 is mechanically forced and fine. What is not fine to carry forward silently is the = itself — it blocks every tokio patch release, including a security patch, behind a manual bump. On a PR whose entire purpose is closing two DoS advisories, freezing the async runtime at an exact patch is the same hazard in a different crate.

Since the original rationale isn't recorded anywhere (you already did that archaeology), there are only two defensible outcomes:

  • Relax to tokio = { version = "1.53.1", ... }. The lockfile still pins the exact build; cargo update -p tokio then works normally and Dependabot can see it.
  • Keep = and add a one-line comment naming the constraint, per AGENTS.md: "Comments record constraints, not narration. Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's 'simplified' away." An exact pin with no recorded reason is exactly the case that rule exists for — the next person to touch it will redo your archaeology and reach the same dead end.

Note the workspace already treats = as a deliberate signal elsewhere (rustls = "=0.23.35", libxml = "=0.3.3", native-tls = ">=0.2, <0.2.17"), so the silent one here is the odd one out.

Fix this →

Comment thread backend/Cargo.toml Outdated
Comment on lines +673 to +676
# No datafusion release yet depends on `object_store` 0.14, and only `object_store` >= 0.14.1
# carries `quick-xml` >= 0.41 (RUSTSEC-2026-0194, RUSTSEC-2026-0195). Drop the git pin for
# `datafusion = "56"` once it is published.
datafusion = { git = "https://github.com/apache/datafusion", rev = "2cac713274496ba8ee08e58655e80f0d464f6d48" }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 — the pin itself is the right call; the removal condition needs an owner.

I checked the shape of this against the lockfile and it holds up better than the alternative you listed:

  • Every datafusion-* subcrate resolves to the same git source at the same rev — no registry/git split of datafusion 55.1.0, which would have produced two incompatible copies of DFSchema/SessionContext.
  • arrow, parquet, object_store, tokio and quick-xml each appear exactly once in Cargo.lock, and thrift is gone. The version table in the description checks out.
  • A fixed rev on apache/datafusion is reproducible (Apache doesn't rewrite main) and self-removing. A fork of object_store 0.13.2 would be neither — it is an indefinite maintenance obligation on an Apache crate, for a bump that upstream will make for you in a few weeks. Take the pin.

Two caveats worth recording rather than debating:

  1. Nothing tracks the removal. The comment says "Drop the git pin for datafusion = "56" once it is published", which is the right instruction, but there is no issue or alert that will fire when 56.0.0 lands — Dependabot does not watch git revs, and the rev self-reports version = "55.1.0", so nothing in the tooling will ever tell you it is stale. Please reference the tracking ticket in the comment so this doesn't quietly become permanent.

  2. This rev has not shipped in any release. It is main between 55.1.0 and 56.0.0, so it carries whatever is in flight upstream with no release vote behind it. Your test plan exercises the DataFusion paths you actually call (load_csv_preview, load_parquet_preview, load_table_count, search/service_logs, the -- s3 format= streaming path), which is the relevant coverage — just worth stating plainly in the PR that this is pre-release code in a shipped image.

One smaller consequence not mentioned in the description: apache/datafusion is a large repo and cargo clones git deps in full, so this adds a real cold-build cost to fresh CI and Docker builds that a registry dep does not have. Not a blocker, but worth knowing before it shows up as "CI got slower".

@alpetric

Copy link
Copy Markdown
Collaborator Author

/review

@claude

claude Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @alpetric's task in 4m 44s —— View job


Good to merge (as an OSS half; the EE companion is still unreviewed from here — see the last section).

This is a re-review at 8deef42. My previous pass ran against the tree before that commit, so both nits I raised were in fact already fixed when I posted them — this pass confirms the fixes, and resolves the one open disagreement between the three reviews.

Both prior nits are addressed

tokio exact pin (backend/Cargo.toml:464-466) — the constraint is now recorded:

// Pinned exactly so every edition resolves the same runtime. The floor is not free to lower: datafusion requires tokio ^1.52, so anything below that fails to resolve.

That's the "keep = and record why" branch of what I asked for, and it satisfies the AGENTS.md comment rule. The next forced bump won't have to redo the archaeology. Nothing further from me here — the tradeoff (patch releases need a manual bump) is now a documented decision rather than an accident.

datafusion git pin (backend/Cargo.toml:675-681) — the comment now carries the whole constraint chain (only object_store >= 0.14.1 has quick-xml >= 0.41 → the store is shared with datafusion via register_object_store → no published datafusion accepts object_store 0.14) and a precise removal instruction: "Replace it with datafusion = "56", and nothing else, as soon as 56.0.0 is published." That is the right backstop. Only leftover from my earlier note: no ticket or alert will fire when 56.0.0 lands (Dependabot doesn't watch git revs, and the rev self-reports 55.1.0). Optional, not a nit — a ticket reference in that comment would close it.

Bulk-delete switch (windmill-object-store/src/lib.rs:773-775) — the behaviour change is now documented where someone would need it, at the S3 builder. Good placement: the note lives at the site an operator with a DELETE /key-only endpoint would edit, not in the PR description where it would be lost.

Resolving the TLS finding from the Pi review — it does not hold

Pi flagged the reqwest_object_store fold-in as silently moving the object-store HTTP client from rustls to OpenSSL/native-tls, on the reasoning that the workspace reqwest has default features on. I checked it against Cargo.lock and that is not what happens — reqwest 0.13's default-tls is rustls, unlike 0.12's:

  • reqwest 0.13.5 deps: hyper-rustls, rustls 0.23.35, tokio-rustls, rustls-platform-verifier — no native-tls, no hyper-tls, no tokio-native-tls.
  • reqwest 0.12.28 (still present for other crates) does list native-tls/hyper-tls, which is where the intuition comes from.

The client stays on rustls. The only delta is root sourcing: rustls-tls-native-roots → rustls-platform-verifier 0.7.1, whose deps are rustls-native-certs 0.8.4 + rustls-webpki + webpki-root-certs — the system trust store on Linux, same as before. No acknowledgement needed in the PR body. Details in an inline comment on that line so it isn't re-derived a third time.

While there: .no_gzip()/.no_brotli()/.no_deflate()/.no_zstd() and the empty ACCEPT_ENCODING default header are pre-existing on main and unchanged, so the workspace reqwest's gzip feature does not introduce transparent decompression on this path — the other thing the fold-in could plausibly have perturbed.

Re-verified, unchanged from my first pass

Nothing but comments changed in 8deef42, so the substantive review stands: the Error::NotImplemented destructuring (the one arm that would have silently become a catch-all), FilesystemStoreIgnoringAttributes' reshape onto delete_stream/copy_opts/rename_opts with the attribute strip still on every mutating path, the 'static delete_stream ownership fix with its 1000-key chunking, df.schema().inner().clone(), JsonReadOptions, and #![recursion_limit = "256"] on the sole [[bin]]. I re-read all 19 non-lockfile code files this pass; the remaining ~15 ObjectStoreExt imports are mechanical and compiler-enforced.

One thing I could not check offline: this environment has no crate sources or network, so I could not confirm that AmazonS3Builder::with_disable_bulk_delete is spelled exactly that way in object_store 0.14.2. Worth a 10-second check, since a comment that names a non-existent method misleads the operator it's written for.

AGENTS.md compliance

Clean, and better than the first pass. The three new comments each record a constraint rather than narrate the change, none describes drafting history, and none addresses the reviewer. No customer attribution. No new public surface, so that checklist doesn't apply.

Test coverage

Unchanged from my first pass and still appropriate: a dependency bump with mechanical call-site adaptation owes no new tests, the one behavioural change in the OSS diff (NotImplemented's shape) is pinned by the updated unit test, and the manual plan covers both XML dialects against the advisory'd parser, both pagination schemes, the DataFusion paths, and a byte-for-byte Parquet round-trip verified by an independent reader. No further manual verification needed for the OSS half.

The EE half remains unreviewed. backend/ee-repo-ref.txt now points at 5297ddc9, but the windmill-ee-private checkout in this workspace is older — it still has set_max_row_group_size (windmill-indexer/src/service_logs_store_ee.rs:145) and imports PutMultipartOpts from object_store_reexports (windmill-api/src/job_helpers_ee.rs, s3_proxy_ee.rs), a name this PR removed from that re-export list. Those must be handled by windmill-labs/windmill-ee-private#836; your --features all_sqlx_features,private check is what proves it.
· branch chore/bump-datafusion-object-store-quick-xml

@github-actions

Copy link
Copy Markdown
Contributor

Pi Review

cc @alpetric

Mergeable, but should ideally address nits: object-store HTTP client's TLS backend still changes rustls → native-tls with no acknowledgement

No P0/P1 bugs. I re-read the full non-lockfile diff (including the 8deef42 follow-up that landed after the last round) and independently re-verified the sharp edges. The follow-up commit recorded the tokio-pin, datafusion-pin, and bulk-delete-switch constraints, leaving exactly one open item from the prior round.

P2 — object-store HTTP client's TLS backend still flips rustls → native-tls, still unacknowledged

backend/windmill-object-store/src/lib.rs:618 builds its SSRF-guarded client from the workspace reqwest (backend/Cargo.toml:514), whose default features are on, so default-tls → native-tls/OpenSSL. The deleted alias was reqwest = { package = "reqwest", version = "0.12", default-features = false, features = ["rustls-tls-native-roots"] }. I confirmed against Cargo.lock that object_store 0.14.2 itself depends on reqwest 0.13.5, so the PublicOnlyConnector client really did move from rustls (native roots) to the platform native TLS stack on every S3/Azure/GCS connection through this path.

This is the one thing the 8deef42 commit ("record the constraints behind the pins and the bulk-delete switch") did not cover — it documented tokio, datafusion, and the bulk-delete switch, but left this silent. It is almost certainly benign (both backends validate against the system trust store; native-tls/openssl are already hard deps of the crate graph, so no new system dependency), so it does not block. But it is a real, observable change on a security-relevant path, and the PR body lists the reqwest fold as "required rather than tidy-up" without noting the TLS side effect. A one-line comment on Cargo.toml:514 ("object-store's HTTP client now rides the workspace reqwest's native-tls backend") is all I'd ask for.

Considered and dismissed

  • Error::NotImplemented arm (lib.rs:101) — the old unit-variant match would have compiled as an irrefutable pattern after the bump and silently swallowed every object-store error into BadRequest, including NotFound/PermissionDenied. The new NotImplemented { operation, implementer } destructuring is correct and the _ => fallback is still last; the unit test was updated with it.
  • FilesystemStoreIgnoringAttributes reshuffle (lib.rs:1005-1090) — its purpose is stripping attributes so LocalFileSystem doesn't reject uploads with NotImplemented. Both mutating entry points still zero opts.attributes before delegating (put_opts, put_multipart_opts). Dropping get_range/head/delete/copy/rename/*_if_not_exists is safe: those are now ObjectStoreExt defaults expressed in terms of get_opts/delete_stream/copy_opts/rename_opts, all of which the wrapper still implements, so the strip is preserved on every attribute-carrying path.
  • delete_stream 'static ownership (workspaces_extra.rs:2185) — chunk.to_vec().into_iter().map(Ok) is required, not stylistic: the new signature wants 'static and .cloned() would still borrow the slice. The 1000-key S3 DeleteObjects chunking is preserved.
  • Single-key delete now rides the bulk POST /?delete API — real behavior change, but deliberate and now documented at lib.rs:773-775 (the with_disable_bulk_delete(true) note) and in the PR body. The backend already requires DeleteObjects of every endpoint (log cleanup, instance log flush, DuckLake fork-data deletion all call delete_stream directly), so this introduces no new endpoint requirement. Exercised against MinIO and Azurite; leaving the switch off is the conservative choice. Not a finding.
  • quick-xml 0.38+ no-trim change — the only direct consumers are quick_xml::de::from_str in storage_list_ee.rs (S3/Azure/GCS list parsing). Provider responses are compact single-line XML (no intra-element whitespace), and the pretty-printed test fixtures keep values on the same line as their tags, so nothing depends on the removed trim. SAML metadata is handled by samael, which strips whitespace itself per the PR body. No bug.
  • df.schema().inner().clone() (lib.rs:1924) — hands ArrowWriter the SchemaRef directly, equivalent to the old DFSchema -> Schema + re-wrap.
  • #![recursion_limit = "256"] (main.rs:11) — correctly placed inner attribute, and windmill is the workspace's only [[bin]].
  • datafusion git pin to unreleased main (Cargo.toml:675) — documented with the "replace with datafusion = "56"" backstop, and consistent with the repo's existing git-rev pins (sqlx, tiberius, samael, etc.). Agree the git pin is the better of the two alternatives (the fork of object_store 0.13.2 trades one unpinned upstream for a maintained Apache-crate fork). Non-blocking.
  • tokio =1.53.1 exact pin (Cargo.toml:464) — forced by datafusion ≥ 54 (tokio ^1.52), and now carries a comment recording the constraint. Lockfile confirms a single tokio 1.53.1. Non-blocking.
  • residual reqwest 0.12.28 in Cargo.lock — not a contradiction of the fold-in claim: it's pulled by async-oauth2, google-cloud-auth, hf-hub, opentelemetry-otlp, etc., all pre-existing and out of scope. The object_store client specifically now resolves to reqwest 0.13.5.

Test coverage

This is a dependency-bump PR with mechanical code adaptations; the relevant coverage is the extensive matrix in the PR body, which I cross-checked against the lockfile: cargo check/build across the CE/EE feature sets, cargo test -p windmill-object-store/lib (52), -p windmill-indexer (31), -p windmill-worker-volumes (39), and S3 integration tests, plus live MinIO/Azurite end-to-end runs covering upload/download, preview, DataFusion SessionContext paths, listing/pagination, delete, Postgres -- s3 streaming, multipart upload, and the service-log columnar store with a pyarrow read-back. cargo audit delta (17 → 15) matches the claim that only the two quick-xml advisories dropped and nothing new appeared. I verified quick-xml 0.41.0 (single copy), arrow/parquet 60.0.0, object_store 0.14.2 (registry), tokio 1.53.1, all 31 datafusion-* subcrates on the same git rev, and thrift gone.

The PR itself flags the unexercised gaps (GCS has no local emulator; log-store compaction/GC needs a 3-hour-old partition; live worker-volume mounts), and each shares a code path or unit suite that was exercised. No new backend integration tests are warranted for a dependency bump whose behavior is pinned by existing tests plus the described manual runs. No manual verification is needed beyond what the PR already reports; the only thing left for a human is the one-line TLS-backend acknowledgement above.

.is_some_and(|v| v == "true");
let proxy_hosts: Arc<[String]> = system_proxy_hosts().into();
let client = reqwest_object_store::Client::builder()
let client = reqwest::Client::builder()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correcting the TLS-backend concern raised on this line — it does not hold. No action needed.

The claim was that folding reqwest_object_store (default-features = false, rustls-tls-native-roots) into the workspace reqwest (default features on) moves this SSRF-guarded client from rustls to OpenSSL/native-tls. That would be true of reqwest 0.12, but not of 0.13: default-tls there is rustls, not native-tls. Cargo.lock is unambiguous —

  • reqwest 0.13.5 deps: hyper-rustls, rustls 0.23.35, tokio-rustls, rustls-platform-verifier. No native-tls, no hyper-tls, no tokio-native-tls.
  • reqwest 0.12.28 (still in the tree for other crates) does list native-tls / hyper-tls / tokio-native-tls, which is where the intuition comes from.

So this client stays on rustls. The one real delta is where roots come from: rustls-tls-native-roots (rustls-native-certs) → rustls-platform-verifier 0.7.1, whose own deps are rustls-native-certs 0.8.4 + rustls-webpki + webpki-root-certs, i.e. the system trust store on Linux, same as before. On the shipped Linux image, trust behaviour is unchanged.

Also checked, since the fold-in changes which reqwest feature set this builder compiles against: .no_gzip()/.no_brotli()/.no_deflate()/.no_zstd() and the empty ACCEPT_ENCODING default header are all pre-existing on main and unchanged here, so the workspace reqwest's gzip feature cannot introduce transparent decompression on the object-store path.

@github-actions

Copy link
Copy Markdown
Contributor

Codex Review

Good to merge

No issues found. Checked dependency consistency, object-store API migration, error handling, and affected public surfaces.

Test coverage

Reported Cargo checks, tests, audit, and MinIO/Azurite end-to-end runs provide strong coverage.

Manual verification is still needed for GCS, delayed log-store compaction/GC, and live worker-volume mounts.

@alpetric

Copy link
Copy Markdown
Collaborator Author

Dismissing the remaining nit (TLS backend of the object-store HTTP client), because the premise is inverted: there is no rustls → native-tls change.

The reqwest_object_store alias asked for rustls-tls-native-roots on reqwest 0.12. The workspace reqwest it is folded into is 0.13 with default features, and in 0.13 default = ["default-tls", "charset", "http2", "system-proxy"] with default-tls = ["rustls"] (reqwest-0.13/Cargo.toml). On top of that, object_store's aws, azure and gcp features each declare reqwest/rustls explicitly (object_store-0.14.2/Cargo.toml), and this build enables all three. Nothing in the tree enables reqwest's native-tls, and PublicOnlyConnector never calls use_native_tls().

So it is rustls before and after. The one real difference is where the roots come from: rustls-native-certs before, rustls-platform-verifier now (what the 0.13 rustls feature pulls in) — still the platform trust store. The PR description now says this explicitly, along with the fact that the four .no_gzip()/.no_brotli()/.no_deflate()/.no_zstd() calls and the empty ACCEPT_ENCODING header are untouched, so the SSRF-guarded client's behaviour is unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant