Repository navigation
Conversation
… set `quick-xml` < 0.41 is affected by RUSTSEC-2026-0194 and RUSTSEC-2026-0195 (both CVSS 7.5). The only `object_store` releases that depend on `quick-xml` >= 0.41 are 0.14.1+, and no published `datafusion` release accepts `object_store` 0.14 yet, so `datafusion` is pinned to the apache/datafusion commit that moved to arrow/parquet 60 and `object_store` 0.14.2. That set also drops `thrift` (parquet >= 59 no longer uses it), closing GHSA-2f9f-gq7v-9h6m. - datafusion 47.0.0 -> git 2cac7132 (arrow/parquet 60, object_store 0.14.2) - object_store git 36752c97 (0.12.0) -> 0.14.2; the pinned rev's Azure query-signing fix (arrow-rs-object-store#334) shipped in 0.12.1, so the `[patch.crates-io]` entry and the git dep are both dropped - parquet 55.2.0 -> 60.0.0, arrow 55.2.0 -> 60.0.0 - quick-xml 0.37 -> 0.41; samael pinned forward to the rev that did the same, since it was the other source of a 0.37 copy - tokio =1.46.1 -> =1.53.1, forced by datafusion's `tokio ^1.52` - `reqwest_object_store` (reqwest 0.12) folded into the workspace `reqwest` 0.13, which object_store 0.14 now builds its clients with Code adaptations: `get`/`get_range`/`head`/`delete`/`copy`/`put`/ `put_multipart` moved to `ObjectStoreExt`, `copy`/`rename` to `copy_opts`/`rename_opts`, `delete` to `delete_stream` with a `'static` stream, `Error::NotImplemented` gained fields, `NdJsonReadOptions` became `JsonReadOptions`, `DFSchema: Into<Schema>` is gone, and `set_max_row_group_size` became `set_max_row_group_row_count`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kv72vdCDggCZEjJSmNCwnX
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kv72vdCDggCZEjJSmNCwnX
Deploying windmill with
|
| Latest commit: |
8deef42
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://1c7e5033.windmill.pages.dev |
| Branch Preview URL: | https://chore-bump-datafusion-object.windmill.pages.dev |
…-object-store-quick-xml # Conflicts: # backend/ee-repo-ref.txt
|
/review |
|
Claude finished @alpetric's task in 5m 41s —— View job cc @alpetric Mergeable, but should ideally address nits: undocumented I found no correctness bug in the OSS half of this diff. The mechanical adaptations are right, and the claims in the description check out against What I verifiedLockfile matches the description.
The ~15 FindingsBoth are detailed in inline comments and neither blocks merge:
On the
|
Codex ReviewGood to merge No issues found. Checked dependency consistency, object-store API migrations, DataFusion/parquet adaptations, error handling, and affected public surfaces. Test coverageThe reported Cargo checks, crate tests, audit, and MinIO/Azurite end-to-end scenarios thoroughly cover the changed layers. GCS, delayed log-store compaction/GC, and live worker-volume mounts remain unexercised, but their shared paths and unit coverage make these reasonable, non-blocking gaps. |
Pi Reviewcc @alpetric Mergeable, but should ideally address nits: undocumented TLS backend change in the object-store HTTP client, datafusion git pin to unreleased This is a large, carefully-executed dependency bump. I read the full code diff (all 21 non-lockfile files) and traced each P2 — object-store HTTP client's TLS backend silently changed from rustls to native-tls
The PR body lists the reqwest fold as "required rather than tidy-up" (correct — cargo rejects the same crate twice in one package) but doesn't note the TLS-backend side effect. In practice this is almost certainly benign: both backends validate against the system trust store, and Note —
|
| hyper-tls = "^0.6" | ||
| hyper-util = { version = "^0.1", features = ["client-legacy", "http1", "tokio"] } | ||
| tokio = { version = "=1.46.1", features = ["full", "tracing", "time"] } | ||
| tokio = { version = "=1.53.1", features = ["full", "tracing", "time"] } |
There was a problem hiding this comment.
P2 — the = is the part worth deciding, not the number.
You flagged this line yourself. My read: moving =1.46.1 → =1.53.1 is mechanically forced and fine. What is not fine to carry forward silently is the = itself — it blocks every tokio patch release, including a security patch, behind a manual bump. On a PR whose entire purpose is closing two DoS advisories, freezing the async runtime at an exact patch is the same hazard in a different crate.
Since the original rationale isn't recorded anywhere (you already did that archaeology), there are only two defensible outcomes:
- Relax to
tokio = { version = "1.53.1", ... }. The lockfile still pins the exact build;cargo update -p tokiothen works normally and Dependabot can see it. - Keep
=and add a one-line comment naming the constraint, perAGENTS.md: "Comments record constraints, not narration. Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's 'simplified' away." An exact pin with no recorded reason is exactly the case that rule exists for — the next person to touch it will redo your archaeology and reach the same dead end.
Note the workspace already treats = as a deliberate signal elsewhere (rustls = "=0.23.35", libxml = "=0.3.3", native-tls = ">=0.2, <0.2.17"), so the silent one here is the odd one out.
| # No datafusion release yet depends on `object_store` 0.14, and only `object_store` >= 0.14.1 | ||
| # carries `quick-xml` >= 0.41 (RUSTSEC-2026-0194, RUSTSEC-2026-0195). Drop the git pin for | ||
| # `datafusion = "56"` once it is published. | ||
| datafusion = { git = "https://github.com/apache/datafusion", rev = "2cac713274496ba8ee08e58655e80f0d464f6d48" } |
There was a problem hiding this comment.
P2 — the pin itself is the right call; the removal condition needs an owner.
I checked the shape of this against the lockfile and it holds up better than the alternative you listed:
- Every
datafusion-*subcrate resolves to the same git source at the same rev — no registry/git split ofdatafusion55.1.0, which would have produced two incompatible copies ofDFSchema/SessionContext. arrow,parquet,object_store,tokioandquick-xmleach appear exactly once inCargo.lock, andthriftis gone. The version table in the description checks out.- A fixed rev on
apache/datafusionis reproducible (Apache doesn't rewritemain) and self-removing. A fork ofobject_store0.13.2 would be neither — it is an indefinite maintenance obligation on an Apache crate, for a bump that upstream will make for you in a few weeks. Take the pin.
Two caveats worth recording rather than debating:
-
Nothing tracks the removal. The comment says "Drop the git pin for
datafusion = "56"once it is published", which is the right instruction, but there is no issue or alert that will fire when 56.0.0 lands — Dependabot does not watch git revs, and the rev self-reportsversion = "55.1.0", so nothing in the tooling will ever tell you it is stale. Please reference the tracking ticket in the comment so this doesn't quietly become permanent. -
This rev has not shipped in any release. It is
mainbetween 55.1.0 and 56.0.0, so it carries whatever is in flight upstream with no release vote behind it. Your test plan exercises the DataFusion paths you actually call (load_csv_preview,load_parquet_preview,load_table_count,search/service_logs, the-- s3 format=streaming path), which is the relevant coverage — just worth stating plainly in the PR that this is pre-release code in a shipped image.
One smaller consequence not mentioned in the description: apache/datafusion is a large repo and cargo clones git deps in full, so this adds a real cold-build cost to fresh CI and Docker builds that a registry dep does not have. Not a blocker, but worth knowing before it shows up as "CI got slower".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kv72vdCDggCZEjJSmNCwnX
|
/review |
|
Claude finished @alpetric's task in 4m 44s —— View job Good to merge (as an OSS half; the EE companion is still unreviewed from here — see the last section). This is a re-review at Both prior nits are addressed
That's the "keep
Bulk-delete switch ( Resolving the TLS finding from the Pi review — it does not holdPi flagged the
The client stays on rustls. The only delta is root sourcing: While there: Re-verified, unchanged from my first passNothing but comments changed in One thing I could not check offline: this environment has no crate sources or network, so I could not confirm that AGENTS.md complianceClean, and better than the first pass. The three new comments each record a constraint rather than narrate the change, none describes drafting history, and none addresses the reviewer. No customer attribution. No new public surface, so that checklist doesn't apply. Test coverageUnchanged from my first pass and still appropriate: a dependency bump with mechanical call-site adaptation owes no new tests, the one behavioural change in the OSS diff ( The EE half remains unreviewed. |
Pi Reviewcc @alpetric Mergeable, but should ideally address nits: object-store HTTP client's TLS backend still changes rustls → native-tls with no acknowledgement No P0/P1 bugs. I re-read the full non-lockfile diff (including the P2 — object-store HTTP client's TLS backend still flips rustls → native-tls, still unacknowledged
This is the one thing the Considered and dismissed
Test coverageThis is a dependency-bump PR with mechanical code adaptations; the relevant coverage is the extensive matrix in the PR body, which I cross-checked against the lockfile: The PR itself flags the unexercised gaps (GCS has no local emulator; log-store compaction/GC needs a 3-hour-old partition; live worker-volume mounts), and each shares a code path or unit suite that was exercised. No new backend integration tests are warranted for a dependency bump whose behavior is pinned by existing tests plus the described manual runs. No manual verification is needed beyond what the PR already reports; the only thing left for a human is the one-line TLS-backend acknowledgement above. |
| .is_some_and(|v| v == "true"); | ||
| let proxy_hosts: Arc<[String]> = system_proxy_hosts().into(); | ||
| let client = reqwest_object_store::Client::builder() | ||
| let client = reqwest::Client::builder() |
There was a problem hiding this comment.
Correcting the TLS-backend concern raised on this line — it does not hold. No action needed.
The claim was that folding reqwest_object_store (default-features = false, rustls-tls-native-roots) into the workspace reqwest (default features on) moves this SSRF-guarded client from rustls to OpenSSL/native-tls. That would be true of reqwest 0.12, but not of 0.13: default-tls there is rustls, not native-tls. Cargo.lock is unambiguous —
reqwest 0.13.5deps:hyper-rustls,rustls 0.23.35,tokio-rustls,rustls-platform-verifier. Nonative-tls, nohyper-tls, notokio-native-tls.reqwest 0.12.28(still in the tree for other crates) does listnative-tls/hyper-tls/tokio-native-tls, which is where the intuition comes from.
So this client stays on rustls. The one real delta is where roots come from: rustls-tls-native-roots (rustls-native-certs) → rustls-platform-verifier 0.7.1, whose own deps are rustls-native-certs 0.8.4 + rustls-webpki + webpki-root-certs, i.e. the system trust store on Linux, same as before. On the shipped Linux image, trust behaviour is unchanged.
Also checked, since the fold-in changes which reqwest feature set this builder compiles against: .no_gzip()/.no_brotli()/.no_deflate()/.no_zstd() and the empty ACCEPT_ENCODING default header are all pre-existing on main and unchanged here, so the workspace reqwest's gzip feature cannot introduce transparent decompression on the object-store path.
Codex ReviewGood to merge No issues found. Checked dependency consistency, object-store API migration, error handling, and affected public surfaces. Test coverageReported Cargo checks, tests, audit, and MinIO/Azurite end-to-end runs provide strong coverage. Manual verification is still needed for GCS, delayed log-store compaction/GC, and live worker-volume mounts. |
|
Dismissing the remaining nit (TLS backend of the object-store HTTP client), because the premise is inverted: there is no rustls → native-tls change. The So it is rustls before and after. The one real difference is where the roots come from: |
Summary
quick-xml< 0.41 is affected by two CVSS 7.5 denial-of-service advisories, and both arereachable from the XML this backend parses (S3/Azure/GCS listing responses, SAML metadata):
NsReader; a singlestart tag drives ~3× its byte size in resolver heap inside the parser, so bounding the input
does not bound the allocation. Memory exhaustion / OOM kill.
start tag pins a CPU core for minutes.
NsReaderreaches it internally and has no opt-outbefore 0.41.
Both are fixed only in
quick-xml>= 0.41.0 — there is no patched 0.37/0.38/0.39/0.40 release.The same move drops
thriftfrom the tree entirely (parquet>= 59 no longer uses it), whichcloses GHSA-2f9f-gq7v-9h6m / CVE-2026-43868 (Dependabot alert #465, WIN-2531).
Why
datafusionbecomes a git pinOnly
object_store>= 0.14.1 depends onquick-xml>= 0.41, andobject_storecannot beupgraded on its own: the backend hands its
Arc<dyn ObjectStore>to DataFusion throughSessionContext::register_object_store, so both must be the sameobject_store.No published
datafusionacceptsobject_store0.14 — the newest release (55.1.0) requires^0.13.2. DataFusionmainmoved toarrow/parquet60 andobject_store0.14.2 inapache/datafusion#25335, so the dependency is pinned to that line until 56.0.0 is published, at
which point the pin becomes
datafusion = "56"and nothing else has to change.The alternatives, for the record: stay on a released DataFusion and carry a fork of
object_store0.13.2 with itsquick-xmlrequirement bumped (the two upstream bumps0.39→0.40.1→0.41 were Cargo.toml-only, so this would work, but it means maintaining a fork of an
Apache crate); or wait for DataFusion 56. Both leave
thriftin the tree for longer, sinceparquet59+ requiresobject_store>= 0.13.2 and thereforedatafusion>= 54, which alreadyforces the
tokiobump below.Versions
quick-xmlthriftobject_storeapache/arrow-rs-object-store@36752c97datafusionapache/datafusion@2cac7132(workspace version 55.1.0)arrowparquettokio=1.46.1=1.53.1samaelnjaremko/samael@f879f194(0.0.20)njaremko/samael@2d27dfe4(0.0.22)sqlparserreqwestfor object_store's HTTP clientreqwest_object_storealias on 0.12reqwest0.13Two bumps are forced rather than chosen:
tokio=1.46.1→=1.53.1. Everydatafusion>= 54 requirestokio ^1.52. The exactpin was added in 1a85dc7 with no recorded rationale and no linked issue; it is kept exact,
just moved to the current release. This is the line in the diff that most wants a second
look.
samaelforward one rev. It was the other source of aquick-xml0.37 copy in thelockfile; upstream's 0.0.22 does the same bump ("security: update
quick-xmlto version0.41.0"). Note that
quick-xml>= 0.38 no longer trims text content;samaelstripswhitespace before base64-decoding certificates (
crypto/cert_encoding.rs), so pretty-printedIdP metadata still validates.
What happened to the
object_storegit pinbackend/Cargo.tomlpinnedobject_storetoarrow-rs-object-store@36752c97(and carried amatching
[patch.crates-io]entry) since df9f827, "fix azure object storage regression due toobject_store regression" — the fix it was reaching for is
#334, "Fix query parameter signing in Azure".
That commit is an ancestor of the
v0.12.1tag (git compare 36752c97...v0.12.1→ ahead 2,behind 0), so every commit the pin carried is in 0.12.1 and later. Both the pin and the
[patch]entry are dropped; nothing is lost.
Code adaptations the upgrade forces
object_store0.13 movedget,get_range,head,delete,copy,copy_if_not_exists,rename,rename_if_not_exists,putandput_multipartoffObjectStoreonto a newObjectStoreExtextension trait — every call site imports it (re-exported fromwindmill_object_store::object_store_reexports).ObjectStoretrait itself now requiresdelete_streamandcopy_opts/rename_optsinstead of
delete/copy/rename, which reshapes theFilesystemStoreIgnoringAttributeswrapper.
delete_streamnow takes a'staticstream, so the DuckLake fork-data deletion owns its pathchunks instead of borrowing them.
Error::NotImplementedgainedoperation/implementerfields. The old arm matched it as aunit variant, which after the upgrade would have compiled as an irrefutable binding and turned
that arm into a catch-all for every object-store error; it now destructures the fields and
reports them.
PutMultipartOptsis a deprecated alias ofPutMultipartOptions.prelude::NdJsonReadOptions→prelude::JsonReadOptions, andDFSchemano longerconverts into
arrow::Schema(df.schema().inner().clone()gives theSchemaReftheArrowWriterwants).parquet:WriterPropertiesBuilder::set_max_row_group_size(n)→set_max_row_group_row_count(Some(n)). The new byte-based companion defaults toNone, so rowgrouping is unchanged.
object_store0.14 builds its HTTP clients withreqwest0.13, the same major the workspacealready uses, so the
reqwest_object_storealias (a secondreqwestat 0.12) is folded intothe workspace
reqwest. Cargo rejects the same crate twice under different names in onepackage, so this is required rather than tidy-up. The TLS backend does not change: the alias
asked for
rustls-tls-native-roots, and reqwest 0.13'sdefault-tlsisrustls(
reqwest-0.13/Cargo.toml:default = ["default-tls", ...],default-tls = ["rustls"]), whileobject_store's ownaws/azure/gcpfeatures each forcereqwest/rustlson top. Stillrustls, still the platform trust store — reaching it through
rustls-platform-verifierratherthan
rustls-native-certs. Nothing in the tree turns on reqwest'snative-tls, and theconnector never calls
use_native_tls(). The four.no_gzip()/.no_brotli()/.no_deflate()/.no_zstd()calls and the emptyACCEPT_ENCODINGheader are unchanged, and those buildermethods exist in 0.13 whether or not the matching feature is on.
#![recursion_limit = "256"]on thewindmillbinary:sqlparser0.63's AST nests deeplyenough that inferring
Send/Syncfor the futures spawned inmainexceeds the default128-step limit.
Test plan
Compiled, with
pkg-config/libxmlsec1present soenterprise_saml(and thereforesamael)is really built:
cargo check --features all_sqlx_features(the CE CI check)cargo check --features all_sqlx_features,private(the EE CI check)cargo check --features private,enterprise,parquetcargo check --features tantivy,private,quickjscargo build --features tantivy,private,quickjsand--features private,parquet,quickjsExercised end to end on a local instance built from this branch, against MinIO (S3) and
Azurite (Azure Blob), with MinIO as the workspace and instance object store:
upload_s3_file;download_s3_filereturnsthe Parquet byte-for-byte identical to the input
load_file_previewrenders the CSV, the JSON and the Parquet correctly (the Parquetthrough the Arrow reader → CSV),
load_file_metadatareports the right size and mtimeload_csv_preview,load_parquet_previewandload_table_count(the DataFusionSessionContextpaths) return the right rows andcount: 3, includinglimit/offset,sort_colandsearch_col/search_termdownload_s3_parquet_file_as_csvlist_stored_files_paged(thestorage_list_eequick-xml path) lists the bucket root, adelimited prefix, and paginates with a continuation token;
list_stored_filesfilters withsearch;move_s3_file(copy + delete) anddelete_s3_filebehaveload_csv_previewandload_file_metadataagainst Azurite as a secondary storage — the Azure
EnumerationResultsXML and theAzure marker pagination
-- s3 format=json|csv|parquetstreams its result set to thebucket; all three objects open correctly through the preview endpoints, and the Parquet is
valid to an independent reader (pyarrow:
parquet-rs version 60.0.0, ZSTD, 3 rows)multipart/form-datauploads its file throughbuild_object_store_client(windmill-api/src/args.rs) and the argument arrives as{"s3": "windmill_uploads/..."}MODE=standalone+search:logs_store/v1/dt=…/mode=…/fills with
ing-*.parquetplus a manifest carrying per-file row counts and ts bounds;pyarrow confirms
parquet-rs version 60.0.0+ ZSTD + the expected schema;search/service_logs(DataFusion over that store:contains/lower/coalesceandtimestamp literals) returns the right lines, including a one-minute
min_ts/max_tswindow and an empty result for a term that is not there;
search/count_service_logs(tantivy) returns its per-host bucket
cargo test -p windmill-object-store --lib(52),-p windmill-indexer --lib(31),-p windmill-worker-volumes --lib(39), and the S3 integration testsapp_s3_onbehalf(7),object_storage_test_ssrf(1),workspace_storage_by_name(2),sign_s3_objects_authz(2) — all greenNot exercised, and why:
COMPACTION_LAG_HOURS = 3ago, which a single session cannot reach; the DataFusionstreaming-sort path they use is covered by
windmill-indexer's unit tests rather than a liverun.
S3ListBucketResultdeserializer as S3 (storage_list_ee.rs), which is exercised againstMinIO, and its
object_storeclient is the same crate version.windmill-worker-volumes' ownLocalFileSystemround-triptests rather than a live mount.
cargo auditcargo auditon the new lockfile against the pre-change one: the only difference isRUSTSEC-2026-0194 and RUSTSEC-2026-0195 are gone, and nothing new appears (17 → 15
vulnerabilities). Of the 15 that remain, the RustSec database records a CVSS score for two —
rsaRUSTSEC-2023-0071 (5.9) andrustlsRUSTSEC-2026-0285 (5.3) — and both are below 7.0. Theother 13 (
h2,hickory-proto×2,magic-crypt,ring,rkyv,rustls-webpki×7) carry noCVSS vector in RustSec and none of them is
quick-xmlorthrift; they are all pre-existing onmainand out of scope here.Left as a draft deliberately. Two calls in here want a human before either side is marked
ready: pinning
datafusionto an unreleasedmainrev (an alternative, a fork ofobject_store0.13.2, is described above), and moving the exact
tokiopin from=1.46.1to=1.53.1, whichDataFusion >= 54 forces and whose original rationale is not recorded anywhere. The change also
spans
*_ee.rsthrough the symlinks — see the companion PR,windmill-labs/windmill-ee-private#836.
One behaviour change the upgrade carries, which review should weigh
In
object_store0.14deleteexists only onObjectStoreExt, and its blanketimplementation goes through
delete_stream— i.e. the provider's bulk delete API, even forone key. On S3 that is
POST /?delete(DeleteObjects) instead of the oldDELETE /key; onAzure it is the Blob Batch endpoint (
POST ?comp=batch). Everydeletecall site in the backendchanges request shape this way; the implementation cannot opt out per call.
Exercised both, and the object really is gone afterwards:
delete_s3_fileagainst MinIO andagainst Azurite, each verified by a follow-up listing. Deleting a key that does not exist behaves
as before — S3 succeeds (the bulk API is idempotent for missing keys), Azure surfaces the
provider's 404 through the same handler error it surfaced before.
The caveat is the class of endpoint neither emulator represents: upstream added
AmazonS3Builder::with_disable_bulk_deleteprecisely because "some S3-compatible providers do notimplement the bulk
DeleteObjectsAPI", and Windmill lets an operator point a resource at anyendpoint. A note now sits at the S3 builder saying where that switch goes. It is deliberately not
set: the backend already requires
DeleteObjectsof every endpoint, because the log-cleanup sweep,the instance log flush and the DuckLake fork-data deletion all call
delete_streamdirectly today(
windmill-api-settings/src/log_cleanup.rs,src/monitor.rs,windmill-api-workspaces/src/workspaces_extra.rs, the last with an explicit 1000-key chunkcomment). Turning the switch on would move those off the batch API too, which is a behaviour and
throughput change of its own rather than a conservative default — so the choice belongs to a
reviewer, not to this patch.